| *** bobh has joined #openstack-monasca | 01:53 | |
| *** mbindlish has joined #openstack-monasca | 02:24 | |
| *** bobh has quit IRC | 02:47 | |
| *** toabctl has quit IRC | 04:31 | |
| *** pcaruana has joined #openstack-monasca | 05:06 | |
| *** joadavis has quit IRC | 05:24 | |
| *** joadavis has joined #openstack-monasca | 05:25 | |
| *** joadavis has quit IRC | 05:27 | |
| *** witek has joined #openstack-monasca | 06:36 | |
| *** chaconpiza has joined #openstack-monasca | 07:09 | |
| *** toabctl has joined #openstack-monasca | 07:30 | |
| *** dougsz has joined #openstack-monasca | 08:08 | |
| *** trident has quit IRC | 08:08 | |
| *** trident has joined #openstack-monasca | 08:09 | |
| *** ivve has joined #openstack-monasca | 08:34 | |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-tempest-plugin master: Use unicode literals in test_metrics https://review.opendev.org/663927 | 08:43 |
|---|---|---|
| openstackgerrit | Witold Bedyk proposed openstack/monasca-tempest-plugin master: Use unicode literals in test_metrics https://review.opendev.org/663927 | 08:57 |
| *** sum12 has joined #openstack-monasca | 09:10 | |
| *** witek has left #openstack-monasca | 09:39 | |
| *** witek has joined #openstack-monasca | 09:39 | |
| openstackgerrit | Adrian Czarnecki proposed openstack/monasca-api master: [WIP] Merge log-api and api https://review.opendev.org/651249 | 09:44 |
| *** Wasaac has joined #openstack-monasca | 09:57 | |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-api master: Upgrade Storm to 1.2.2 https://review.opendev.org/658371 | 11:26 |
| *** Wasaac has quit IRC | 11:32 | |
| *** Wasaac has joined #openstack-monasca | 11:33 | |
| *** Wasaac has quit IRC | 11:38 | |
| *** elod has quit IRC | 12:11 | |
| *** Wasaac has joined #openstack-monasca | 12:11 | |
| witek | chaconpiza: https://review.opendev.org/663927 | 12:12 |
| chaconpiza | it pass all checks | 12:16 |
| witek | yes, finally | 12:16 |
| chaconpiza | Do you have an idea of which thing is creating the random behavior? | 12:17 |
| *** Wasaac has quit IRC | 12:18 | |
| witek | you mean randomly failing tempest tests? | 12:18 |
| chaconpiza | yes | 12:20 |
| witek | I think it's resources issue | 12:21 |
| witek | the node where the tests are run has 8GB RAM | 12:22 |
| witek | http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007493.html | 12:23 |
| *** Wasaac has joined #openstack-monasca | 12:23 | |
| openstackgerrit | Witold Bedyk proposed openstack/monasca-api master: Set topology.max.spout.pending for Apache Storm https://review.opendev.org/668887 | 12:25 |
| *** elod has joined #openstack-monasca | 12:48 | |
| *** elod_ has joined #openstack-monasca | 12:58 | |
| *** elod has quit IRC | 12:58 | |
| *** elod_ is now known as elod | 12:59 | |
| *** elod has quit IRC | 13:06 | |
| *** elod has joined #openstack-monasca | 13:07 | |
| *** dougsz has quit IRC | 13:18 | |
| *** dougsz has joined #openstack-monasca | 13:18 | |
| *** dougsz has quit IRC | 13:22 | |
| *** Wasaac has quit IRC | 13:43 | |
| *** joadavis has joined #openstack-monasca | 13:43 | |
| *** Wasaac_ has joined #openstack-monasca | 13:43 | |
| *** mbindlish has quit IRC | 13:44 | |
| *** Wasaac_ has quit IRC | 13:58 | |
| *** ivve has quit IRC | 14:14 | |
| *** sgrasley has joined #openstack-monasca | 14:17 | |
| *** Wasaac has joined #openstack-monasca | 14:46 | |
| *** koji_n has joined #openstack-monasca | 14:53 | |
| *** hosanai has joined #openstack-monasca | 14:56 | |
| witek | Courtesy Monasca meeting reminder in #openstack-monasca: witek, jayahn,iurygregory,ezpz,igorn,haad,sc,joadavis, akiraY,tobiajo,dougsz,fouadben, amofakhar, haru5ny (haruki),kaiokmo,pandiyan,charana,guilhermesp,chaconpiza,toabctl,hosanai,Wasaac | 14:58 |
| witek | #startmeeting monasca | 15:00 |
| openstack | Meeting started Wed Jul 3 15:00:15 2019 UTC and is due to finish in 60 minutes. The chair is witek. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: monasca)" | 15:00 | |
| openstack | The meeting name has been set to 'monasca' | 15:00 |
| witek | hello everyone | 15:00 |
| Wasaac | o/ | 15:00 |
| joadavis | o/ | 15:00 |
| kaiokmo | \o | 15:00 |
| koji_n | hi | 15:00 |
| hosanai | hi | 15:01 |
| witek | the agenda as usual: | 15:01 |
| witek | https://etherpad.openstack.org/p/monasca-team-meeting-agenda | 15:01 |
| Dobroslaw | hi | 15:01 |
| witek | #topic Batching messages for Kafka | 15:01 |
| *** openstack changes topic to "Batching messages for Kafka (Meeting topic: monasca)" | 15:01 | |
| witek | I don't see Doug | 15:02 |
| witek | he proposed this change: | 15:02 |
| witek | https://review.opendev.org/667931 | 15:02 |
| *** haru5ny has joined #openstack-monasca | 15:02 | |
| *** dougsz has joined #openstack-monasca | 15:02 | |
| witek | the purpose is split the list of messages if the batch is too big | 15:02 |
| witek | hi dougsz | 15:03 |
| dougsz | sorry i'm late, hi | 15:03 |
| witek | I've just introduced your change https://review.opendev.org/667931 | 15:03 |
| dougsz | ah, thanks | 15:03 |
| witek | please take over | 15:03 |
| dougsz | So we saw this problem with Monasca Agent with large buffers | 15:04 |
| dougsz | When they were disconnected from the Monasca API due to a network reconfigure | 15:04 |
| dougsz | They buffered up a lot of data | 15:04 |
| dougsz | Then when the API was available, they made posts of large amounts of data | 15:05 |
| dougsz | Monasca API tried to write this to Kafka, which failed due to the chunk being to large | 15:05 |
| dougsz | There are a series of example errors in the Story | 15:05 |
| *** bobh has joined #openstack-monasca | 15:05 | |
| dougsz | The fix simply writes a post in chunks | 15:05 |
| dougsz | The chunks consist of a fixed number of metrics | 15:06 |
| dougsz | It's not trivial to calculate the size of the metrics, as would be written to Kafka | 15:06 |
| dougsz | We've tested in production - we could handle 20GB metric posts with this change. | 15:06 |
| dougsz | That's it really | 15:07 |
| witek | have you checked performance impact? | 15:07 |
| joadavis | seems like an effective approach | 15:07 |
| dougsz | I havent measured it, but I think the bottle neck is still in the monasca persister | 15:08 |
| dougsz | we wrote about 300GB of metrics in a couple of hours to Kafka | 15:08 |
| joadavis | good question - what is the 'normal' batch size | 15:08 |
| dougsz | and then the persister spent the next few days writing them to InfluxDB | 15:08 |
| witek | :) | 15:08 |
| joadavis | lol | 15:09 |
| dougsz | joadavis: yeah - it depends on your max write size to kafka | 15:09 |
| kaiokmo | :) | 15:09 |
| witek | the default value of message.max.bytes is ca. 1 MB | 15:10 |
| dougsz | yeah, that's correct, I think roughly the default for this change should be about 0.5MB writes | 15:10 |
| dougsz | Anyway, it was good proof that the architecture works. Kafka absorbed the massive spike in metrics very easily. | 15:11 |
| witek | I just noticed that the initialization of `batch_size` variable could probably be moved to a better place | 15:11 |
| dougsz | ah yes, I can do that | 15:12 |
| witek | ok, thanks for the update | 15:12 |
| dougsz | ok, thanks, will update asap | 15:12 |
| witek | folks, please leave your comments in review if you have any thoughts | 15:12 |
| witek | #topic Python 3.7 | 15:13 |
| *** openstack changes topic to "Python 3.7 (Meeting topic: monasca)" | 15:13 | |
| witek | there has been a series of changes recently updating our CI jobs for Python 3.7 | 15:13 |
| witek | most of merged well, but there are some exceptions | 15:13 |
| witek | which I wanted to bring to attention | 15:14 |
| witek | https://review.opendev.org/667060 | 15:14 |
| witek | https://review.opendev.org/667058 | 15:14 |
| witek | the last one: | 15:14 |
| witek | https://review.opendev.org/#/c/657829/ | 15:14 |
| witek | is general Python 3 support for monasca-agent | 15:14 |
| joadavis | and I noticed in reviewing the other changes that the monasca-agent change could be using the -train job | 15:15 |
| witek | yes, I haven't checked the other repos, some might need update as well | 15:16 |
| witek | so if anyone looks for a task to pick up, help very welcome | 15:16 |
| joadavis | I don't think all the monasca repos are on the list that the goal team is working on | 15:16 |
| joadavis | so checking through them ourselves is a good idea | 15:16 |
| witek | hm, do you know which list they're using? | 15:17 |
| joadavis | I don't, I just looked at the list of changes with the python3-train branch | 15:18 |
| joadavis | s/branch/topic/ | 15:18 |
| joadavis | we could ask in openstack-discuss | 15:19 |
| witek | ok, I'll try to find out if they have some etherpad for tracking this | 15:19 |
| witek | #topic monasca-notification config update | 15:19 |
| *** openstack changes topic to "monasca-notification config update (Meeting topic: monasca)" | 15:19 | |
| witek | I've picked oslo.config for monasca-notification in DevStack again | 15:20 |
| witek | and it finally seems to work | 15:20 |
| joadavis | :thumbs-up: | 15:20 |
| witek | we needed a bugfix from Isaac: | 15:20 |
| witek | https://review.opendev.org/663960 | 15:20 |
| witek | and set the default enabled plugins: | 15:21 |
| witek | https://review.opendev.org/668508 | 15:21 |
| witek | with them DevStack can be updated: | 15:21 |
| witek | https://review.opendev.org/633194 | 15:21 |
| witek | these changes were long overdue | 15:22 |
| witek | any comments? | 15:23 |
| witek | #topic mid-cycle meeting | 15:23 |
| *** openstack changes topic to "mid-cycle meeting (Meeting topic: monasca)" | 15:23 | |
| witek | I though it might be a good idea to have an audio meeting in the middle of the cycle | 15:24 |
| witek | to sync up on progress | 15:24 |
| witek | what do you think? | 15:25 |
| witek | we could do it around milestone-2 | 15:25 |
| witek | I don't expect it to be a long meeting | 15:25 |
| Wasaac | No objections | 15:26 |
| joadavis | I like the idea | 15:26 |
| Dobroslaw | fine | 15:27 |
| witek | cool, milestone-2 is in the week Jul 22 - Jul 26 | 15:27 |
| witek | I'd suggest to set up a one hour slot, we could extend it to two if needed | 15:28 |
| witek | do we want to use the same time as this meeting, or better make a doodle list? | 15:28 |
| dougsz | +1 for audio meeting | 15:28 |
| witek | I'll send the email to the mailing list and set up a doodle with time proposals | 15:30 |
| witek | I think that'll be fair for everyone | 15:30 |
| witek | #topic other business | 15:31 |
| *** openstack changes topic to "other business (Meeting topic: monasca)" | 15:31 | |
| witek | any other topics? | 15:31 |
| witek | if not, we can finish earlier | 15:32 |
| witek | oh, the deadline for Shanghai has been extended if anyone is interested | 15:33 |
| witek | thanks for joining | 15:33 |
| witek | see you next week | 15:33 |
| joadavis | Thanks all | 15:33 |
| koji_n | thanks | 15:34 |
| hosanai | thx | 15:34 |
| witek | #endmeeting | 15:34 |
| *** openstack changes topic to "OpenStack Monitoring as a Service | https://wiki.openstack.org/wiki/Monasca" | 15:34 | |
| openstack | Meeting ended Wed Jul 3 15:34:31 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:34 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-07-03-15.00.html | 15:34 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-07-03-15.00.txt | 15:34 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/monasca/2019/monasca.2019-07-03-15.00.log.html | 15:34 |
| *** hosanai has quit IRC | 15:34 | |
| *** koji_n has quit IRC | 15:34 | |
| *** haru5ny has quit IRC | 15:53 | |
| *** bobh has quit IRC | 15:58 | |
| *** Wasaac has quit IRC | 16:17 | |
| *** Wasaac has joined #openstack-monasca | 16:21 | |
| *** Wasaac has quit IRC | 16:25 | |
| *** bobh has joined #openstack-monasca | 16:33 | |
| *** bobh has quit IRC | 16:38 | |
| openstackgerrit | Merged openstack/monasca-notification master: Fix notification type plugin install loop https://review.opendev.org/663960 | 16:50 |
| *** dougsz has quit IRC | 16:58 | |
| *** openstackgerrit has quit IRC | 17:04 | |
| *** witek has quit IRC | 17:04 | |
| *** elod has quit IRC | 17:05 | |
| *** elod has joined #openstack-monasca | 17:07 | |
| *** trident has quit IRC | 20:43 | |
| *** trident has joined #openstack-monasca | 20:45 | |
| *** pcaruana has quit IRC | 21:16 | |
| *** bobh has joined #openstack-monasca | 22:31 | |
| *** bobh has quit IRC | 23:32 | |
| *** bobh has joined #openstack-monasca | 23:33 | |
| *** bobh has quit IRC | 23:37 | |
| *** bobh has joined #openstack-monasca | 23:48 | |
| *** bobh has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!