| *** slaweq has joined #openstack-meeting-4 | 00:11 | |
| *** dviroel has quit IRC | 00:11 | |
| *** slaweq has quit IRC | 00:16 | |
| *** vishalmanchanda has joined #openstack-meeting-4 | 00:50 | |
| *** enriquetaso has quit IRC | 00:54 | |
| *** slaweq has joined #openstack-meeting-4 | 01:11 | |
| *** dwalt has quit IRC | 01:16 | |
| *** slaweq has quit IRC | 01:17 | |
| *** dave-mccowan has quit IRC | 01:32 | |
| *** dave-mccowan has joined #openstack-meeting-4 | 01:35 | |
| *** dave-mccowan has quit IRC | 01:40 | |
| *** hongbin has joined #openstack-meeting-4 | 02:09 | |
| *** slaweq has joined #openstack-meeting-4 | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** igordc has quit IRC | 02:17 | |
| *** enriquetaso has joined #openstack-meeting-4 | 02:49 | |
| *** slaweq has joined #openstack-meeting-4 | 03:11 | |
| *** slaweq has quit IRC | 03:16 | |
| *** enriquetaso has quit IRC | 03:17 | |
| *** yamamoto has joined #openstack-meeting-4 | 03:23 | |
| *** psachin has joined #openstack-meeting-4 | 03:41 | |
| *** links has joined #openstack-meeting-4 | 04:11 | |
| *** slaweq has joined #openstack-meeting-4 | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| *** yamamoto has quit IRC | 04:29 | |
| *** yamamoto has joined #openstack-meeting-4 | 04:35 | |
| *** hongbin has quit IRC | 05:10 | |
| *** slaweq has joined #openstack-meeting-4 | 05:11 | |
| *** slaweq has quit IRC | 05:16 | |
| *** dave-mccowan has joined #openstack-meeting-4 | 05:18 | |
| *** yamamoto has quit IRC | 05:20 | |
| *** yamamoto has joined #openstack-meeting-4 | 05:22 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-meeting-4 | 05:34 | |
| *** slaweq has joined #openstack-meeting-4 | 06:11 | |
| *** slaweq has quit IRC | 06:16 | |
| *** e0ne has joined #openstack-meeting-4 | 06:43 | |
| *** e0ne has quit IRC | 06:49 | |
| *** yamamoto has quit IRC | 06:51 | |
| *** slaweq has joined #openstack-meeting-4 | 07:11 | |
| *** slaweq has quit IRC | 07:16 | |
| *** yamamoto has joined #openstack-meeting-4 | 07:16 | |
| *** lpetrut has joined #openstack-meeting-4 | 07:20 | |
| *** yamamoto has quit IRC | 07:34 | |
| *** yamamoto has joined #openstack-meeting-4 | 07:34 | |
| *** gcheresh has joined #openstack-meeting-4 | 07:48 | |
| *** gcheresh has quit IRC | 07:57 | |
| *** gcheresh has joined #openstack-meeting-4 | 07:58 | |
| *** slaweq has joined #openstack-meeting-4 | 07:59 | |
| *** roman_g has quit IRC | 08:18 | |
| *** ralonsoh has joined #openstack-meeting-4 | 08:35 | |
| *** jokke_ has joined #openstack-meeting-4 | 09:10 | |
| *** k_mouza has joined #openstack-meeting-4 | 09:15 | |
| *** e0ne has joined #openstack-meeting-4 | 09:17 | |
| *** ktibi has joined #openstack-meeting-4 | 10:12 | |
| *** dviroel has joined #openstack-meeting-4 | 10:27 | |
| *** fdegir has quit IRC | 10:34 | |
| *** georgk has quit IRC | 10:34 | |
| *** rihabb2527 has quit IRC | 10:34 | |
| *** georgk has joined #openstack-meeting-4 | 10:35 | |
| *** fdegir has joined #openstack-meeting-4 | 10:35 | |
| *** rihabb2527 has joined #openstack-meeting-4 | 10:35 | |
| *** ktibi has quit IRC | 11:04 | |
| *** k_mouza has quit IRC | 11:24 | |
| *** k_mouza has joined #openstack-meeting-4 | 11:25 | |
| *** Luzi has joined #openstack-meeting-4 | 12:14 | |
| *** yamamoto has quit IRC | 12:28 | |
| *** fdegir has quit IRC | 12:29 | |
| *** rihabb2527 has quit IRC | 12:29 | |
| *** georgk has quit IRC | 12:29 | |
| *** yamamoto has joined #openstack-meeting-4 | 12:31 | |
| *** fdegir has joined #openstack-meeting-4 | 12:31 | |
| *** rihabb2527 has joined #openstack-meeting-4 | 12:31 | |
| *** georgk has joined #openstack-meeting-4 | 12:31 | |
| *** ktibi has joined #openstack-meeting-4 | 12:56 | |
| *** enriquetaso has joined #openstack-meeting-4 | 13:16 | |
| *** gcheresh_ has joined #openstack-meeting-4 | 13:42 | |
| *** gcheresh has quit IRC | 13:43 | |
| *** yamamoto has quit IRC | 13:52 | |
| *** enriquetaso has quit IRC | 14:02 | |
| *** igordc has joined #openstack-meeting-4 | 14:04 | |
| *** enriquetaso has joined #openstack-meeting-4 | 14:05 | |
| *** dwalt has joined #openstack-meeting-4 | 14:06 | |
| *** yamamoto has joined #openstack-meeting-4 | 14:10 | |
| *** Luzi has quit IRC | 14:14 | |
| *** enriquetaso has quit IRC | 14:14 | |
| *** enriquetaso has joined #openstack-meeting-4 | 14:16 | |
| *** zigo has quit IRC | 14:42 | |
| *** michael-beaver has joined #openstack-meeting-4 | 14:59 | |
| *** igordc has quit IRC | 15:00 | |
| *** roman_g has joined #openstack-meeting-4 | 15:02 | |
| *** lpetrut has quit IRC | 15:11 | |
| zhipeng[m] | The meeting is over? | 15:12 |
|---|---|---|
| *** links has quit IRC | 15:21 | |
| *** macz_ has joined #openstack-meeting-4 | 15:42 | |
| *** gagehugo has joined #openstack-meeting-4 | 15:54 | |
| *** gcheresh_ has quit IRC | 15:56 | |
| portdirect | o/ | 16:01 |
| portdirect | #startmeeting openstack-helm | 16:01 |
| openstack | Meeting started Tue Feb 18 16:01:56 2020 UTC and is due to finish in 60 minutes. The chair is portdirect. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:01 |
| *** openstack changes topic to " (Meeting topic: openstack-helm)" | 16:01 | |
| openstack | The meeting name has been set to 'openstack_helm' | 16:02 |
| gagehugo | o/ | 16:02 |
| dwalt | o/ | 16:02 |
| megheisler | \o | 16:02 |
| portdirect | lets give it a fews misn for people to roll up | 16:02 |
| *** stevthedev has joined #openstack-meeting-4 | 16:02 | |
| *** lamt has joined #openstack-meeting-4 | 16:02 | |
| portdirect | lamt and gagehugo prepared the agenda for today, as they both rock: https://etherpad.openstack.org/p/openstack-helm-meeting-2020-02-18 | 16:02 |
| stevthedev | O/ | 16:02 |
| portdirect | also its great to be back | 16:02 |
| lamt | o/ | 16:03 |
| lamt | wb | 16:03 |
| gagehugo | wb | 16:03 |
| portdirect | ok lets go | 16:06 |
| portdirect | #topic virtual ptg | 16:06 |
| *** openstack changes topic to "virtual ptg (Meeting topic: openstack-helm)" | 16:06 | |
| mattmceuen | o/ | 16:06 |
| portdirect | before i got roped into one million and one things we were talking about holding a virtual midcycle ptg | 16:06 |
| portdirect | primarily focused upon documentation and gating | 16:06 |
| portdirect | we still like this idea? if so I'll send out a few dates in march for us to choose from | 16:07 |
| gagehugo | I think it's a good idea | 16:07 |
| mattmceuen | +1 | 16:07 |
| lamt | ++ | 16:07 |
| gagehugo | Did we find a good video conferencing tool? | 16:07 |
| portdirect | i think webex should work, unless anyone objects? | 16:08 |
| *** zigo has joined #openstack-meeting-4 | 16:09 | |
| portdirect | ok, i'll take silence as approval ;) | 16:09 |
| portdirect | #action, portdirect to pull finger out and get email sent for virtualptg | 16:09 |
| gagehugo | webex should be fine | 16:10 |
| portdirect | so - that actually is it for the agenda today | 16:10 |
| portdirect | however i can offer a tale of osh in use that may be of interest? | 16:11 |
| megheisler | certainly sounds interesting | 16:11 |
| lamt | I am eared. | 16:11 |
| stevthedev | consider my interests piqued | 16:11 |
| portdirect | so - a couple of weeks ago an osh deployment was happily humming along... | 16:12 |
| portdirect | when several things hit it at once | 16:12 |
| portdirect | 1) network fabric instability | 16:12 |
| portdirect | 2) signifigant resource contention on control plane and compute nodes nodes | 16:13 |
| portdirect | these two factors combined resulted in us not living up to the mantra of our humble mascot, le grande honeybadger | 16:13 |
| portdirect | ultimatly resulting in the oom-killer taking out (i believe) qemu and kvm processes | 16:14 |
| portdirect | which is not a good thing | 16:14 |
| gagehugo | oh | 16:15 |
| portdirect | so i think theres some really valible learnings we can take away from this: | 16:15 |
| lamt | is that the issue with the 1000 probes? | 16:16 |
| portdirect | >2k probes, but yes ;) also you ruin stories lamt | 16:16 |
| portdirect | 1) we need to ensure that the blast radius of any component in osh is limited | 16:16 |
| portdirect | 2) we need to do work that ensures that even when a node is unhealty - we take out 'non-essential' services and infra prior to tenants being impacted | 16:17 |
| portdirect | as lamt has pointed out, it was observed that we had a massive buildup of exec probes on some compute nodes | 16:18 |
| portdirect | which were caused by an unhealty rabbit cluster* | 16:18 |
| portdirect | * the rabbit cluster was infact working, but due to load on the control plane, exec probes there were timing out, resulting in k8s flapping endpoints | 16:19 |
| portdirect | some of the community memebers involved in this have started to make some great patchsets to improve the 'honey badger' ideals we strive for | 16:20 |
| portdirect | eg: https://review.opendev.org/#/c/706590/ | 16:20 |
| portdirect | but i'd also like to ask that we all look at the areas of osh that we know best | 16:20 |
| portdirect | and see if there is any areas there where we can make similar hardening improvements | 16:21 |
| portdirect | eg phil recently made this chaneg to mariadb: https://review.opendev.org/#/c/708071/ | 16:21 |
| portdirect | which improves how the deployment copes with instability of the k8s api | 16:21 |
| portdirect | but im sure there are many other areas where we can make improvement | 16:22 |
| portdirect | anyone have any thoughts or insight into this? | 16:22 |
| mattmceuen | Thanks for sharing the background and the work to date - those changes both look great | 16:23 |
| mattmceuen | all hail the honey badger, may he live forever! | 16:24 |
| lamt | would probably look closer at mariadb and rabbitmq - those two components are always the trouble maker | 16:24 |
| gagehugo | yeah | 16:24 |
| gagehugo | those two and their clustering | 16:24 |
| portdirect | amen to that - but please also look at their consumers | 16:24 |
| portdirect | the probes issue above is testement to that | 16:25 |
| portdirect | i think there is a ps being worked on to ensure the oom-score is adjusted appropriately for libvirt etc | 16:25 |
| portdirect | in summary - we will never be able to protect against all control plane issues | 16:25 |
| portdirect | s/protect/prevent | 16:26 |
| lamt | part of that may also be tuning those probes (timeout, delay, etc.) | 16:26 |
| portdirect | so we need to do all we can to mitigate them | 16:26 |
| portdirect | lamt: +100 | 16:26 |
| portdirect | i think oveverall we are probably much more agressive than we need to be with livelyness probes | 16:26 |
| portdirect | it would also be great to see resource limits turned on by default | 16:27 |
| portdirect | eg - if say a neutron agen is consuming more than 8gb of ram | 16:27 |
| portdirect | its pretty safe to assume somthing is wrong | 16:27 |
| *** psachin has quit IRC | 16:27 | |
| portdirect | the other side of this - is things like lma | 16:27 |
| portdirect | should we be reading from the head, or tail of files? what are the tradeoffs etc | 16:28 |
| portdirect | anyway - food for thought | 16:29 |
| stevthedev | I don't think it's been done upstream yet, but dsmith was able to implement the pos file for fluentd, which will allow it to 'remember' where it was reading from in case of a restart | 16:29 |
| portdirect | what does that do? | 16:30 |
| *** vishalmanchanda has quit IRC | 16:30 | |
| stevthedev | It records the position in the log file which fluentd is reading | 16:31 |
| stevthedev | So after a restart, fluentd won't start at the top of the file and send out a bunch of duplicated messaged | 16:31 |
| portdirect | oh nice - that sounds like a great improvement | 16:32 |
| stevthedev | Yeah, I'll ask him to make the changes to the chart upstream as well, as an example to other OSH users | 16:33 |
| portdirect | ok should we move onto the weekly plea for reviews? | 16:33 |
| portdirect | stevthedev: +++ | 16:33 |
| stevthedev | Sure | 16:33 |
| portdirect | #topic reviews | 16:33 |
| *** openstack changes topic to "reviews (Meeting topic: openstack-helm)" | 16:33 | |
| portdirect | we have a few this week that could do with some eyes: | 16:33 |
| portdirect | https://review.opendev.org/#/c/706590/ (sangeet) | 16:33 |
| portdirect | https://review.opendev.org/#/c/708046/ | 16:33 |
| portdirect | https://review.opendev.org/#/c/702983/ | 16:33 |
| portdirect | https://review.opendev.org/#/c/697554/ | 16:33 |
| portdirect | https://review.opendev.org/#/c/701839/ | 16:33 |
| portdirect | and with that - if we have nothing left to discuss this week, we can take 20 mins back? | 16:34 |
| gagehugo | wfm | 16:37 |
| portdirect | ok - thanks for coming everyone! | 16:37 |
| portdirect | #endmeeting | 16:37 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:37 | |
| openstack | Meeting ended Tue Feb 18 16:37:57 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:38 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_helm/2020/openstack_helm.2020-02-18-16.01.html | 16:38 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_helm/2020/openstack_helm.2020-02-18-16.01.txt | 16:38 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_helm/2020/openstack_helm.2020-02-18-16.01.log.html | 16:38 |
| *** ktibi has quit IRC | 16:50 | |
| *** gagehugo has left #openstack-meeting-4 | 16:52 | |
| *** enriquetaso has quit IRC | 17:00 | |
| *** e0ne has quit IRC | 17:05 | |
| *** igordc has joined #openstack-meeting-4 | 17:09 | |
| *** enriquetaso has joined #openstack-meeting-4 | 17:26 | |
| *** k_mouza has quit IRC | 17:32 | |
| *** yamamoto has quit IRC | 17:33 | |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-meeting-4 | 17:34 | |
| *** gcheresh_ has joined #openstack-meeting-4 | 17:55 | |
| *** e0ne has joined #openstack-meeting-4 | 18:26 | |
| *** ralonsoh has quit IRC | 18:36 | |
| *** k_mouza has joined #openstack-meeting-4 | 18:56 | |
| *** k_mouza has quit IRC | 18:57 | |
| *** k_mouza has joined #openstack-meeting-4 | 18:57 | |
| *** k_mouza has quit IRC | 19:02 | |
| *** gcheresh_ has quit IRC | 19:09 | |
| *** gcheresh_ has joined #openstack-meeting-4 | 19:17 | |
| *** e0ne has quit IRC | 19:32 | |
| *** enriquetaso has quit IRC | 19:35 | |
| *** gcheresh_ has quit IRC | 20:00 | |
| *** e0ne has joined #openstack-meeting-4 | 20:08 | |
| *** e0ne has quit IRC | 20:08 | |
| *** enriquetaso has joined #openstack-meeting-4 | 20:33 | |
| *** michael-beaver has quit IRC | 20:49 | |
| *** k_mouza has joined #openstack-meeting-4 | 20:58 | |
| *** k_mouza has quit IRC | 21:03 | |
| *** e0ne has joined #openstack-meeting-4 | 21:52 | |
| *** enriquetaso has quit IRC | 22:02 | |
| *** k_mouza has joined #openstack-meeting-4 | 22:16 | |
| *** k_mouza has quit IRC | 22:19 | |
| *** klindgren_ has quit IRC | 22:35 | |
| *** klindgren has joined #openstack-meeting-4 | 22:36 | |
| *** slaweq has quit IRC | 22:38 | |
| *** dave-mccowan has quit IRC | 22:55 | |
| *** slaweq has joined #openstack-meeting-4 | 23:11 | |
| *** slaweq has quit IRC | 23:16 | |
| *** e0ne has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!