Wednesday, 2022-04-13

johnsomWell that was fun. A git update is the root cause.00:11
johnsomdevstack gate jobs are not going to run until a workaround is merged00:12
opendevreviewGregory Thiemonge proposed openstack/octavia-tempest-plugin master: WIP Testing c9s job with new ssh key type  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/83759705:48
opendevreviewGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Update c9s job with new ssh key type  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/83759707:55
opendevreviewCedric Jeanneret proposed openstack/octavia master: Save the HAProxy state outside of its systemd unit  https://review.opendev.org/c/openstack/octavia/+/83772111:56
opendevreviewCedric Jeanneret proposed openstack/octavia master: Save the HAProxy state outside of its systemd unit  https://review.opendev.org/c/openstack/octavia/+/83772112:33
opendevreviewCedric Jeanneret proposed openstack/octavia master: Save the HAProxy state outside of its systemd unit  https://review.opendev.org/c/openstack/octavia/+/83772113:21
opendevreviewCedric Jeanneret proposed openstack/octavia master: Save the HAProxy state outside of its systemd unit  https://review.opendev.org/c/openstack/octavia/+/83772113:49
opendevreviewGregory Thiemonge proposed openstack/octavia master: Save the HAProxy state outside of its systemd unit  https://review.opendev.org/c/openstack/octavia/+/83772115:40
gthiemonge#startmeeting Octavia16:00
opendevmeetMeeting started Wed Apr 13 16:00:29 2022 UTC and is due to finish in 60 minutes.  The chair is gthiemonge. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'octavia'16:00
gthiemongeHi!16:00
tweininghi16:00
johnsomo/16:00
gthiemonge#topic Announcements16:02
gthiemonge* New CI status etherpad16:02
rm_workHi16:02
gthiemongeas discussed during the PTG, I started building an etherpad that collects all the (not so) "random" CI failures16:02
gthiemonge#link https://etherpad.opendev.org/p/octavia-ci-failures16:02
gthiemongeif you notice new issues (Octavia issues) please fill the etherpad or ping me on IRC16:03
johnsomThat second one makes sense. The first one I don't fully understand where the SG update is getting dropped16:03
gthiemongejohnsom: an update on the 2nd one: it happens after a listener update (PUT), nearly 10 second after its creation, so this is still weird16:04
gthiemongeI have DNM patch that tries to add debug for both cases16:04
gthiemongeI reproduced twice the "0 listener reported"16:05
johnsomAt least in the wild it shouldn't impact anything as the next heartbeat should clear the situation16:06
johnsomJust some wonky log messages I suspect16:06
johnsomOh, nevermind, it shows up in the operating status16:06
gthiemongeyeah, one workaround would be to ignore the ERROR while wiating for the ONLINE status16:06
gthiemongeI will update the test to ignore the ERROR status, at least it will show if the operating status switches back to ONLINE after a few seconds16:08
gthiemongein the current test, if the status is ERROR the listener is deleted immediatly16:08
gthiemongeAbout CI, a gentle reminder about the recheck policy:16:09
gthiemongedon't recheck without adding a reason to the message16:09
gthiemongefor instance "recheck octavia-v2-dsvm-scenario Amphora reports 0 listeners when 1 expected"16:09
gthiemongeor "recheck octavia-v2-dsvm-noop-api timeout", etc...16:09
johnsomWell, don't recheck until the devstack/git issue is resolved either. Every devstack based job on Ubuntu is broken right now.16:10
gthiemongeIf you need help to understand why a job failed, ping me ;-)16:10
* johnsom raises his hand for help16:10
johnsomgrin16:10
gthiemongeyou broke devstack, you have to fix it yourself /s16:11
johnsomlol16:11
johnsomOctavia first saw it, so it must be an Octavia bug16:11
gthiemonge:D16:11
gthiemonge* Zed Release Schedule16:12
tweiningjohnsom: is that why you mainly work on Designate? ;)16:12
gthiemongeWe had a great PTG last week, thank you all!16:12
gthiemongeJust a reminder, you can follow the Zed release cycle on this page:16:13
* johnsom thinks tweining picks things up quickly....16:13
gthiemonge#link https://releases.openstack.org/zed/schedule.html16:13
gthiemongelol16:13
rm_workSo what I forgot to bring up in the PTG was this spec we talked about: https://review.opendev.org/c/openstack/octavia/+/83161916:14
rm_workHoping to get this done Zed16:14
gthiemongerm_work: ok, thanks, I'll take a look16:15
gthiemongeany other announcements?16:16
gthiemongenop16:19
gthiemonge#topic Octavia CI status16:19
gthiemongeWell, I added this new weekly topic about the CI16:19
johnsomI noticed the scoped tokens job isn't running outside of the tempest tests. Is that right?16:20
gthiemongeto talk about patches that would fix/improve our CI (we have a lot of open review in o-t-p)16:20
gthiemongedo you mean it's not in the projects.yaml file in Octavia?16:20
johnsomRight16:20
gthiemongeYeah we need to add it to the long list :D16:21
johnsomOr pick a job to merge it with16:21
gthiemongeI'm fine if we add a new job in octavia16:21
gthiemongewe have the same gap with the c9s job16:22
gthiemongeit's only in octavia-tempest-plugin16:22
gthiemongeand about that, there was a recent update in c9s16:22
gthiemongethe ssh client in c9s now excludes rsa ssh keys (like the key we generate in the devstack plugin and we use in octavia-tempest-plugin)16:23
gthiemongeI have an update for the job:16:23
gthiemonge#link https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/83759716:23
johnsomShould we prune the ussuri jobs?16:24
gthiemongeyeah we should16:24
gthiemongeand we still have the spare pool jobs for master16:24
gthiemonge(I have an open patch to remove it)16:24
gthiemongebut I think we cannot move forward until devstack is fixed16:25
johnsomRight16:25
gthiemongethere are also some patches to (re)enable act-stdby jobs as voting jobs in octavia and tempest16:27
gthiemongeI will create a list of patches for the next meeting16:27
gthiemonge#topic Brief progress reports / bugs needing review16:30
gthiemongeI wanted to talk about a new patch about a SELinux issue (something I introduced a few months ago), but I saw that johnsom and tweining already commented in the review, so this is fine16:31
gthiemongebut FYI16:31
gthiemonge#link https://review.opendev.org/c/openstack/octavia/+/83772116:31
gthiemongeand FYI we have a lot of backports in review:16:32
johnsomI have two patches looking for a second +2:16:32
johnsomhttps://review.opendev.org/c/openstack/octavia/+/83106216:32
johnsomhttps://review.opendev.org/c/openstack/octavia/+/83108716:32
johnsomQuick and easy reviews....16:32
gthiemonge#link https://review.opendev.org/q/(project:openstack/octavia+OR+project:openstack/octavia-lib+OR+project:openstack/octavia-dashboard+OR+project:openstack/octavia-tempest-plugin+OR+project:openstack/python-octaviaclient)+status:open+branch:%255Estable/.*16:32
johnsomDang, forgot the link...16:32
tweiningwe discussed a few changes that need review during PTG. See https://etherpad.opendev.org/p/zed-ptg-octavia16:33
tweiningbtw. it seems the bot picks up links without #link too16:33
johnsomOh, does it now?16:33
tweiningyes, I saw it in the last notes that it picked up one of my links and I never use the tag16:34
gthiemongeprobably some smart regex ;-)16:35
johnsomNeat16:35
gthiemongetweining: are you talking about the failover stop threshold patch?16:35
tweiningyes, for instance.16:36
tweiningfirst of all we need to agree about the new state16:36
tweiningsee https://review.opendev.org/c/openstack/octavia-lib/+/83504916:37
gthiemongeso I'm fine with the new state16:37
tweiningbtw, I am currently working on tempest tests for that change16:39
johnsomI think I commented on the main patch. The topics don't align, so I would have to track it down16:39
tweiningfor reference: main patch is https://review.opendev.org/c/openstack/octavia/+/65681116:40
johnsomOh, looks like I didn't finish, I have pending comments16:41
gthiemongeadded a comment16:41
tweiningthanks16:42
gthiemonge#topic Open Discussion16:45
gthiemongeanything else folks?16:45
tweiningnot from my side16:45
johnsomNothing else here16:46
gthiemongeok! thank you all!16:46
gthiemonge#endmeeting16:46
opendevmeetMeeting ended Wed Apr 13 16:46:43 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:46
opendevmeetMinutes:        https://meetings.opendev.org/meetings/octavia/2022/octavia.2022-04-13-16.00.html16:46
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/octavia/2022/octavia.2022-04-13-16.00.txt16:46
opendevmeetLog:            https://meetings.opendev.org/meetings/octavia/2022/octavia.2022-04-13-16.00.log.html16:46
tweiningok, I think the bot only picks it up if the line starts with http. because here it did it without the tag https://meetings.opendev.org/meetings/octavia/2022/octavia.2022-03-30-16.00.html16:49
johnsomgthiemonge FYI, even victoria goes EM next week per the schedule, so maybe disable both the ussuri and victoria jobs.17:02
gthiemongejohnsom: ack17:04
johnsomA workaround to the git/devstack issue has merged so the gates should be functional again.22:15
johnsomAnnnnd..... DIB is broken too23:11
opendevreviewMichael Johnson proposed openstack/octavia-tempest-plugin master: DNM: Testing DIB fix for git issue  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/83777823:24

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!