| *** jamesmcarthur has quit IRC | 00:06 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 00:08 | |
| *** jamesmcarthur has quit IRC | 00:09 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 00:09 | |
| *** jamesmcarthur has quit IRC | 00:33 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 00:33 | |
| *** jamesmcarthur has quit IRC | 00:39 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 01:09 | |
| *** jamesmcarthur has quit IRC | 01:15 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 01:40 | |
| *** jamesmcarthur has quit IRC | 02:54 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 02:55 | |
| *** jamesmcarthur has quit IRC | 02:59 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 03:35 | |
| *** jamesmcarthur has quit IRC | 05:10 | |
| *** logan- has quit IRC | 06:00 | |
| *** logan_ has joined #openstack-publiccloud | 06:01 | |
| *** logan_ is now known as logan- | 06:01 | |
| *** gtema has quit IRC | 06:03 | |
| *** gtema has joined #openstack-publiccloud | 06:14 | |
| *** gtema has quit IRC | 06:29 | |
| *** gtema has joined #openstack-publiccloud | 06:35 | |
| *** gtema has quit IRC | 06:50 | |
| *** gtema has joined #openstack-publiccloud | 06:50 | |
| *** gtema has quit IRC | 06:55 | |
| *** gtema has joined #openstack-publiccloud | 07:05 | |
| *** hberaud|gone is now known as hberaud | 07:46 | |
| *** witek has joined #openstack-publiccloud | 08:43 | |
| *** gtema_ has joined #openstack-publiccloud | 09:28 | |
| *** gtema has quit IRC | 09:31 | |
| *** hberaud is now known as hberaud|lunch | 09:56 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 10:15 | |
| *** jamesmcarthur has quit IRC | 10:19 | |
| *** hberaud|lunch is now known as hberaud | 10:50 | |
| tobberydberg | o/ | 14:00 |
|---|---|---|
| peschk_l | o/ | 14:00 |
| jferrieu | o/ | 14:00 |
| peschk_l | jferrieu and I are form the CloudKitty team | 14:01 |
| tobberydberg | Nice to have you here folks! | 14:02 |
| peschk_l | tobberydberg: sorry we missed the previous meetings.. | 14:02 |
| tobberydberg | mnaser ,gtema_, witek (and maybe more) ... are you around for this meeting as well? | 14:02 |
| tobberydberg | No worries peschk_l | 14:03 |
| mnaser | :)\ | 14:03 |
| tobberydberg | Summer have slowed it all up a bit is easy to say | 14:03 |
| gtema_ | I'm around, a bit over-busy unfortunately | 14:03 |
| tobberydberg | I'll start the meeting and you contribute with the time you can spare :-) | 14:03 |
| tobberydberg | #startmeeting publiccloud_wg | 14:04 |
| openstack | Meeting started Thu Jul 18 14:04:09 2019 UTC and is due to finish in 60 minutes. The chair is tobberydberg. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:04 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:04 |
| *** openstack changes topic to " (Meeting topic: publiccloud_wg)" | 14:04 | |
| openstack | The meeting name has been set to 'publiccloud_wg' | 14:04 |
| tobberydberg | Agenda found at https://etherpad.openstack.org/p/publiccloud-wg | 14:04 |
| tobberydberg | Simple agenda today, one topic :-) | 14:04 |
| witek | hi | 14:06 |
| *** ncastele has joined #openstack-publiccloud | 14:06 | |
| ncastele | Hi | 14:06 |
| tobberydberg | welcome folks | 14:06 |
| tobberydberg | meeting notes for this topic is located here: https://etherpad.openstack.org/p/publiccloud-sig-billing-implementation-proposal | 14:07 |
| tobberydberg | Short recap first maybe ... | 14:07 |
| ncastele | yep, needed, long time no see :D | 14:07 |
| peschk_l | would be great :) | 14:07 |
| tobberydberg | What we said last time was to do a metric / collection method mapping | 14:07 |
| tobberydberg | We have identified some metrics that we need to collect, as well as a few different suggestions of collection methods | 14:08 |
| tobberydberg | Just created a google spreadsheet for that purpose | 14:09 |
| tobberydberg | https://docs.google.com/spreadsheets/d/15HtA15Lrf8UhkPqSTzM4Nan08aEUDQmHUl8XS7T2KO0/edit#gid=0 | 14:09 |
| tobberydberg | haven't had the time to fill in any data unfortunate | 14:10 |
| tobberydberg | The thought was to complete this mapping so we get a clear overview which metric can be collected with which method ... this to be able to make some good decisions for implementation | 14:11 |
| tobberydberg | (you correct me if I'm out flying here:-) ) | 14:12 |
| tobberydberg | So, we took one step back to analyze the situation a little bit | 14:13 |
| ncastele | so the spreadsheet is here to name all the metrics we want, and to check if we can have them regarding each collector ? | 14:13 |
| tobberydberg | yes, that was my thought | 14:14 |
| tobberydberg | To fins a solution that will cover them all, I'm pretty sure we will end up with more than one collector tool/collection method | 14:15 |
| peschk_l | definitely | 14:15 |
| tobberydberg | I might be totally off in my thinking though, so you correct me if I'm wrong :-) | 14:15 |
| peschk_l | I think an "SQL collection" column may also be needed :) several cloudkitty users did that | 14:16 |
| tobberydberg | Feel free to add | 14:17 |
| peschk_l | for example for octavia loadbalancers, they directly scrapped the SQL database and pushed metrics into gnocchi | 14:17 |
| tobberydberg | Personally, IF that is possible to avoid and be collected in any other way that is preferable | 14:18 |
| peschk_l | of course, adding these metrics to ceilometer or monasca would be way better, but it was the fastest solution for them | 14:18 |
| tobberydberg | I'm happy if we can avoid direct db queries ... but maybe that isn't possible | 14:18 |
| tobberydberg | understood | 14:19 |
| ncastele | +1, scrapping DB is a way to go but not the best one, even if it can be handled by querying slaves to avoid impacting the production | 14:19 |
| *** jamesmcarthur has joined #openstack-publiccloud | 14:19 | |
| peschk_l | ncastele: agreed. For pure-openstack, I like how ceilometer listens to rabbitMQ notifications | 14:21 |
| tobberydberg | Do you all feel this is the way to go initially? | 14:21 |
| tobberydberg | the mapping stuff that is | 14:21 |
| peschk_l | tobberydberg: yes | 14:22 |
| ncastele | yes | 14:22 |
| jferrieu | yes | 14:22 |
| tobberydberg | Ok, cool | 14:23 |
| ncastele | scrapping DB is the simplest/quickest way to go, but if we can rely on other components (by relying I mean 100% sure it does the job and we do not risk to lose data), then it's a better way to go | 14:23 |
| tobberydberg | If we all help out with this mapping we can probably have a good initial view until next meeting | 14:23 |
| *** jamesmcarthur has quit IRC | 14:23 | |
| *** jamesmcarthur has joined #openstack-publiccloud | 14:24 | |
| peschk_l | tobberydberg: I'll add stuff on monday, some customers sent us very detailed lists of the metrics they wanted to rate | 14:24 |
| tobberydberg | reliability is important I would say :-) | 14:24 |
| tobberydberg | cool peschk_l | 14:24 |
| tobberydberg | I mean, neutron deletes everything from db after existence ... har to rely on that :-) | 14:25 |
| peschk_l | I believe pretty much every "core" projet does this, except for nova and maybe keystone | 14:26 |
| tobberydberg | You are probably right about that | 14:26 |
| peschk_l | that's a big plus for the notification system, even resources that are only up for a few seconds are taken into account | 14:27 |
| tobberydberg | +1 | 14:27 |
| tobberydberg | What are the next steps after that mapping? Don't want to rush but maybe good to be able to start thinking about that as well... | 14:28 |
| tobberydberg | collection method/s will be a big key, but also backed storage for this and how to be able to query this in a real time fashion | 14:29 |
| peschk_l | For storage: cloudkitty uses influxDB, and I'm working on an ElasticSearch driver for ck | 14:30 |
| peschk_l | from my experience, a flexible backend with support for complex aggregations is important | 14:31 |
| tobberydberg | I'll add a section for that in the etherpad | 14:31 |
| ncastele | +1 | 14:31 |
| ncastele | we are running thousands of VMs here, we need the backend to be extensible | 14:32 |
| peschk_l | we used SQL or gnocchi in the past, and both caused problems | 14:32 |
| peschk_l | and I believe ceilometer used MongoDB as a backend several releases ago, but it had terrible performance | 14:33 |
| *** ricolin_ is now known as ricolin | 14:34 | |
| peschk_l | ncastele: agreed, extensibilty is also a key point | 14:34 |
| peschk_l | in some european countries, companies may be required to store unaggregated billing data for up to 10 years | 14:35 |
| tobberydberg | yea, I think the raw data will be important for some companies as well | 14:36 |
| tobberydberg | yea, mongo was not super :-) | 14:36 |
| ncastele | raw data in the past can be stored in a cold storage, I don't think we need it to be quickly queryable after a couple of months | 14:37 |
| peschk_l | +1 | 14:38 |
| tobberydberg | totally agree | 14:38 |
| peschk_l | but even a few months can be a huge amount of data, especially if you want real-time precision | 14:38 |
| ncastele | yup | 14:38 |
| peschk_l | speaking of that, does anyone have an idea of a datamodel for that ? | 14:40 |
| peschk_l | ck has a collect period which can be configured (1h hour by default), but when this period becomes too small, there is just too much data | 14:41 |
| peschk_l | (we're storing one point per resource per collect period) | 14:41 |
| tobberydberg | Haven't thought about the details around that | 14:42 |
| tobberydberg | BUT .... per second (for resources measured in time) precision will be a requrement | 14:42 |
| peschk_l | gnocchi's model is great for this kind of issue because metadata is stored only once (+ updates), but aggregation queries are not very flexible | 14:43 |
| *** jamesmcarthur has quit IRC | 14:43 | |
| peschk_l | tobberydberg: yes, that's the impression we had at the summit | 14:43 |
| tobberydberg | that doesn't say that data must be stored every second though | 14:44 |
| peschk_l | everybody wants do to FaaS, so at least second-precision is required | 14:44 |
| tobberydberg | +1 | 14:45 |
| ncastele | This is why we need to scope the metrics we need, for which purpose, and challenge the collect period and how we store them | 14:45 |
| peschk_l | maybe a model similar to gnocchi's, but stored in a document-oriented DB ? it would avoid metadata duplication, and we could store exact creation, update and deletion timestamps | 14:47 |
| peschk_l | (or maybe it is too soon to think about this) | 14:47 |
| tobberydberg | would be good to avoid duplication of that, agreed, and that sounds resonable to me | 14:48 |
| tobberydberg | hehe, maybe ... but good to have something to think about a little in the back of the head ;-) | 14:48 |
| tobberydberg | mnaser might have had some thoughts around this earlier ... you correct me if I'm wrong | 14:49 |
| peschk_l | if mnaser thoughts are still available on eavesdrop or somewhere, I'd love to read them :) | 14:52 |
| peschk_l | oh, something else that caused us a few headaches: users WILL want/need to modify existing rating rules | 14:52 |
| tobberydberg | All meetings have been recorded ... | 14:52 |
| tobberydberg | I might be wrong there though, been some weeks since first meetings :-) | 14:53 |
| peschk_l | the problem is that you can't just compute the final price on-the-fly when an API request is made | 14:53 |
| tobberydberg | right ... I'm pretty sure you are right about that | 14:54 |
| peschk_l | (all right, I'll try to find them then :) ) | 14:54 |
| tobberydberg | what do you mean by that peschk_l ? | 14:54 |
| tobberydberg | In what way? | 14:54 |
| peschk_l | fox example, if a company decides that for internal billing, metric X should be taken into account for the current month | 14:56 |
| peschk_l | or that the price for metric Y was not right | 14:56 |
| peschk_l | the data collected since the beginning of the month will have to be modified | 14:57 |
| tobberydberg | aha, ok ok | 14:57 |
| *** hberaud has quit IRC | 14:57 | |
| tobberydberg | Time flies...almost up for today | 14:58 |
| ncastele | yes | 14:58 |
| peschk_l | so there are two possibilities: either the price is calculated on collection, and stored along with the qty (that's what ck does). In that case, you can have exact invoices in a few seconds through the API, but you need to plan for data updates | 14:59 |
| ncastele | do we agree we have a look on the spreadsheet to fulfill metrics | 14:59 |
| peschk_l | ncastele: +1 | 14:59 |
| jferrieu | +1 | 14:59 |
| tobberydberg | Short summary ... if we all can help out identifying the metrics and collections, as well as suggestions for backend storage until next meeting, that would be super | 14:59 |
| ncastele | price should be computed through another brick/layer | 14:59 |
| *** hberaud has joined #openstack-publiccloud | 14:59 | |
| jferrieu | tobberydberg: agreed | 14:59 |
| peschk_l | agreed | 15:00 |
| tobberydberg | sounds good! | 15:00 |
| tobberydberg | Thanks a lot for today folks! Happy we are moving forward in these discussions :-) | 15:01 |
| peschk_l | tobberydberg: thanks for organizing :-) | 15:01 |
| tobberydberg | Leaving for vacation here tomorrow, but I hope that I will be able to make every other thursdays for this meeting to keep this going | 15:01 |
| tobberydberg | See you all in 2 weeks :-) | 15:02 |
| jferrieu | see you, thanks again for the meeting | 15:02 |
| tobberydberg | #endmeeting | 15:02 |
| *** openstack changes topic to "New meeting time!! Thursday odd weeks at 1400 UTC in this channel!!" | 15:02 | |
| openstack | Meeting ended Thu Jul 18 15:02:41 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:02 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-07-18-14.04.html | 15:02 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-07-18-14.04.txt | 15:02 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-07-18-14.04.log.html | 15:02 |
| *** ncastele has quit IRC | 15:09 | |
| *** witek has quit IRC | 16:16 | |
| *** gtema_ has quit IRC | 16:59 | |
| *** gtema has joined #openstack-publiccloud | 16:59 | |
| *** gtema has quit IRC | 17:04 | |
| *** gtema has joined #openstack-publiccloud | 17:14 | |
| *** aprice has quit IRC | 17:45 | |
| *** hogepodge has quit IRC | 17:45 | |
| *** hogepodge has joined #openstack-publiccloud | 17:47 | |
| *** aprice has joined #openstack-publiccloud | 17:47 | |
| *** hberaud is now known as hberaud|gone | 18:03 | |
| *** irclogbot_3 has quit IRC | 18:07 | |
| *** altlogbot_3 has quit IRC | 18:07 | |
| *** irclogbot_2 has joined #openstack-publiccloud | 18:08 | |
| *** altlogbot_0 has joined #openstack-publiccloud | 18:09 | |
| *** ricolin_ has joined #openstack-publiccloud | 19:03 | |
| *** ricolin has quit IRC | 19:05 | |
| *** gtema has quit IRC | 19:29 | |
| *** gtema has joined #openstack-publiccloud | 19:30 | |
| *** gtema has quit IRC | 19:35 | |
| *** blake has joined #openstack-publiccloud | 20:38 | |
| *** blake has quit IRC | 20:58 | |
| *** blake has joined #openstack-publiccloud | 20:59 | |
| *** gtema has joined #openstack-publiccloud | 21:31 | |
| *** gtema has quit IRC | 21:36 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!