Wednesday, 2021-09-08

*** amoralej|off is now known as amoralej06:04
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Move to ansible-core and collections  https://review.opendev.org/c/openstack/kolla/+/80727906:29
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Remove haproxy,keepalived groups  https://review.opendev.org/c/openstack/kolla-ansible/+/80734406:52
mnasiadkamorning07:01
opendevreviewMichal Nasiadka proposed openstack/kayobe stable/victoria: CI: Add CentOS Stream 8  https://review.opendev.org/c/openstack/kayobe/+/80782907:14
mgoddardmorning08:28
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Remove haproxy,keepalived groups  https://review.opendev.org/c/openstack/kolla-ansible/+/80734408:33
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Bump up Ansible to 4.x / ansible-core 2.11  https://review.opendev.org/c/openstack/kolla-ansible/+/79675808:37
mnasiadkamgoddard, yoctozepto: should I pursue getting rid of ansible in favor of ansible-core in K-A, or are we sticking to Ansible 5 as maximum (which includes ansible-core 2.11 plus a lot of collections)?08:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Bump up Ansible max supported ver to 5.x  https://review.opendev.org/c/openstack/kolla-ansible/+/79675808:44
opendevreviewMichal Nasiadka proposed openstack/kolla master: toolbox: Move to ansible-core 2.11 and collections  https://review.opendev.org/c/openstack/kolla/+/80727908:47
mnasiadkawell, we could support both, just deliver a requirements.yml for installing collections...08:48
mgoddardmnasiadka: 5.x is not released yet08:56
mgoddardmnasiadka: at this stage in the cycle I would suggest staying with ansible08:57
mgoddardwhat do we benefit from ansible-core?08:57
mnasiadkaNothing really, apart being able to pin to specific versions of collections, if there are new features/fixed bugs08:58
yoctozeptoslimmer installs for sure08:58
mnasiadkaand I mixed up Ansible versions, we need to support max 4 - because 5 will come with ansible-core 2.1208:58
mnasiadkalet me update the change08:58
opendevreviewMark Goddard proposed openstack/kolla-ansible master: CI: monasca: stop checking Elasticsearch index  https://review.opendev.org/c/openstack/kolla-ansible/+/80768909:04
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Bump up Ansible max supported ver to 4.x  https://review.opendev.org/c/openstack/kolla-ansible/+/79675809:25
mnasiadkaok, sorted out the numbering mess09:26
opendevreviewMerged openstack/kolla-ansible stable/victoria: Fix kolla-toolbox with IPv6 and disabled RabbitMQ  https://review.opendev.org/c/openstack/kolla-ansible/+/80764010:09
opendevreviewMerged openstack/kolla-ansible stable/ussuri: Fix kolla-toolbox with IPv6 and disabled RabbitMQ  https://review.opendev.org/c/openstack/kolla-ansible/+/80764110:10
*** amoralej is now known as amoralej|lunch12:09
opendevreviewMark Goddard proposed openstack/kayobe master: Allow passing arguments to dev/environment-setup.sh  https://review.opendev.org/c/openstack/kayobe/+/80786412:26
blmtis anyone here using a clustered OVN deployment (i.e. one with 3 controllers nodes that leverages the clustered ovsdb) ? ref. https://docs.openvswitch.org/en/latest/ref/ovsdb.7/12:27
blmtof course deployed via kolla :)12:27
opendevreviewMark Goddard proposed openstack/kolla-ansible master: CI: monasca: stop checking Elasticsearch index  https://review.opendev.org/c/openstack/kolla-ansible/+/80768912:32
opendevreviewalecorps proposed openstack/kolla-ansible master: Add support for VMware NSXP  https://review.opendev.org/c/openstack/kolla-ansible/+/80740413:18
*** amoralej|lunch is now known as amoralej13:20
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Fix kolla-toolbox with IPv6 and disabled RabbitMQ  https://review.opendev.org/c/openstack/kolla-ansible/+/80763913:20
opendevreviewVerification of a change to openstack/kolla-ansible stable/victoria failed: Zun: Temporarily skip capsule test for ubuntu  https://review.opendev.org/c/openstack/kolla-ansible/+/80775613:21
mnasiadkayoctozepto: https://review.opendev.org/c/openstack/kolla/+/786719 - do we need that still?13:48
yoctozeptomnasiadka: it's a cleanup, approve13:58
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Zun: Temporarily skip capsule test for ubuntu  https://review.opendev.org/c/openstack/kolla-ansible/+/80775514:20
opendevreviewMerged openstack/kolla-ansible stable/ussuri: Zun: Temporarily skip capsule test for ubuntu  https://review.opendev.org/c/openstack/kolla-ansible/+/80775714:20
opendevreviewMark Goddard proposed openstack/kayobe master: Allow passing arguments to dev/environment-setup.sh  https://review.opendev.org/c/openstack/kayobe/+/80786414:43
opendevreviewMerged openstack/kolla master: ceph: Update CentOS packages to Pacific  https://review.opendev.org/c/openstack/kolla/+/80704314:50
opendevreviewMerged openstack/kolla stable/victoria: [doc] Fix Ceph sources  https://review.opendev.org/c/openstack/kolla/+/80708814:50
opendevreviewMerged openstack/kolla stable/ussuri: [doc] Fix Ceph sources  https://review.opendev.org/c/openstack/kolla/+/80708714:50
mnasiadkamgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt - meeting in 714:53
opendevreviewMerged openstack/kolla master: Move clustercheck script to mariadb-base  https://review.opendev.org/c/openstack/kolla/+/80561314:53
opendevreviewRadosÅ‚aw Piliszek proposed openstack/kolla master: [CI] Drop support for publishing versioned image tags  https://review.opendev.org/c/openstack/kolla/+/78671914:54
* hrw out14:54
mnasiadka#startmeeting kolla15:00
opendevmeetMeeting started Wed Sep  8 15:00:00 2021 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'kolla'15:00
mnasiadka#topic rollcall15:00
mgoddard\o15:01
mnasiadkao/15:02
parallax.15:03
yoctozeptoo/15:04
*** imtiazc is now known as Guest669815:04
mnasiadkaok, I think it's time to start :)15:05
mnasiadka#topic agenda15:05
*** Guest6698 is now known as imtiazc_15:05
mnasiadka* Announcements15:05
mnasiadka* Review action items from the last meeting15:05
mnasiadka* CI status15:05
mnasiadka* Forward upgrade testing ( re: https://bugs.launchpad.net/kolla-ansible/+bug/1941706 )15:05
mnasiadka* DockerHub limits hurting users https://bugs.launchpad.net/kolla-ansible/+bug/194213415:05
mnasiadka* Future of Monasca - in context  of CI failures and Elasticsearch switch to AWS Opensearch15:05
mnasiadka* Release tasks15:05
mnasiadka* Xena cycle planning15:05
mnasiadka* Yoga PTG planning15:05
mnasiadka* Open discussion15:05
mnasiadka#topic Announcements15:05
mnasiadkaNo announcements from me - well, seems there was a mail with PTL announcements - so it's official now.15:06
mnasiadkaAnybody has anything else?15:06
mnasiadkaGuess not.15:07
mnasiadka#topic Review action items from the last meeting15:07
mnasiadkaNo action items.15:08
mnasiadka#topic CI status15:08
mnasiadkaSeems all green based on the whiteboard.15:08
mnasiadka#topic Forward upgrade testing ( re: https://bugs.launchpad.net/kolla-ansible/+bug/1941706 )15:10
mnasiadkayoctozepto: I guess it's yours.15:10
*** imtiazc_ is now known as imtiazc15:12
yoctozeptooh, sorry15:12
yoctozeptoI wanted to speak up on CI status but missed it15:13
mnasiadkaYou still can :)15:13
yoctozeptoyeah :-)15:13
yoctozeptoso, relating to CI status; the gates are green, obviously :-)15:13
yoctozeptoI asked hongbin for help with the zun scenario (because it was red on ubuntu and prevented us to see other issues)15:13
*** imtiazc is now known as Guest670015:13
yoctozeptoand so he proposed to drop the capsule testing as upstream did as well15:14
*** Guest6700 is now known as imtiazc_15:14
yoctozeptoI merged the change on all supported branches15:14
yoctozeptoyou can see it in the workarounds15:14
yoctozeptonow the zun scenario is green15:14
mnasiadkaGreat to hear.15:14
yoctozeptoremember zun scenario tests multinode cinder with iscsi15:14
yoctozeptoso it's useful for this purpose15:14
yoctozeptook, another15:14
yoctozeptocinder failure in xena15:14
yoctozeptoit got promoted to critical before release15:15
yoctozeptothe cinder folks are debating on the best approach to it15:15
yoctozeptobut it's in progress15:15
mnasiadkaGood, let's track it - I'll subscribe to the bug.15:15
yoctozeptoour cephadm upgrade seems to work when any patch is applied15:15
yoctozeptothey just need to merge something finally (-:15:15
mnasiadkaWhat I'm more concerned with is the Monasca scenario, but maybe let's discsus that during the Monasca/Elasticsearch topic parallax raised.15:16
yoctozeptoremember it's our only multinode upgrade job15:16
yoctozeptoyeah, Monasca is to be discussed later so I'm not starting this now15:16
yoctozeptoDebian I will fix another time15:16
yoctozeptothat's all for CI status; now onto the "current topic"15:16
yoctozeptoso "Forward upgrade testing ( re: https://bugs.launchpad.net/kolla-ansible/+bug/1941706 )"15:16
yoctozeptowhat I mean in here is that it's obviously very recommended to merge additional testing15:17
yoctozeptobut I think we need one more job variant - forward upgrades15:17
yoctozeptoso that we actually exercise the proposed code's ability to allow to be *upgraded-from*15:17
yoctozeptoi.e., even with the stricter testing we would not catch that libvirt issue immediately15:18
yoctozeptoonly after it merged on wallaby, would we see failures on master and scratch our heads15:18
mgoddarddo we have a feel for how many bugs this would have caught15:18
yoctozeptolikely not that many tbh; thus have not worked on that upfront and just put a discussion point15:19
yoctozeptothat we have this obvious hole in testing15:19
yoctozeptowe can always just "ack" it and go ahead; other projects don't seem to do forward testing either15:20
yoctozepto(unless perhaps tripleo does but their CI scares me)15:20
mnasiadkatripleo's CI is not the scariest element of tripleo ;-)15:20
yoctozeptohard to tell really15:20
mgoddardwould it prevent us from merging some changes? we already have some cases where we have to push a fix back to N-3 before it can be merged in later branches15:21
mnasiadkaThat's my concern, we are doing reverse-backports from time to time, so this could complicate such scenarios.15:22
yoctozeptoyeah, I was wondering the same thing but I would recommend the wisdom of the elders - when you have an emergency and know the root cause, disable the blocking jobs and merge the necessary fixes ;-)15:22
yoctozeptomy biggest concern is general stability and gate time hit would not be compensated by the extra testing scope15:22
yoctozeptoif we are all sceptic, then it's perhaps best to just ack this point and move on15:23
yoctozeptoget back to it when we hit another similar issue15:23
yoctozepto(could be like in 3 years time and we don't even remember lol)15:24
mgoddardwhile we may merge some breaking changes currently, we would at least catch them when running upgrade jobs in later branches15:24
mgoddardassuming our CI catches the issue15:24
yoctozeptoyeah, the issue is: only after and you have to guess which change broke15:24
yoctozeptootherwise definitely15:24
mgoddardwhich it didn't with the libvirt issue, since you added testing to catch it15:25
yoctozeptoyes15:25
yoctozeptoi.e.15:25
yoctozeptowith new testing in place for now, we would have this situation15:25
yoctozepto1) we actually break master15:25
yoctozepto2) we backport the breakage to wallaby15:25
yoctozepto3) we merge the breakage in wallaby because everything is nice and dandy15:26
yoctozepto4) suddenly we see master jobs break15:26
yoctozepto5) we investigate and figure out (after some time) that it was a change merged on wallaby that broke it15:26
yoctozeptoso15:26
mnasiadkaSo from my perspective it's already better ;-)15:26
yoctozeptoat least we are already reducing the time window for affecting users15:26
yoctozeptoit is15:26
yoctozeptothe point was if we want to make it even tighter15:27
yoctozeptobut it comes with all the already mentioned drawbacks ;-)15:27
opendevreviewwu.chunyang proposed openstack/kolla-ansible master: Remove chrony role from kolla  https://review.opendev.org/c/openstack/kolla-ansible/+/79174315:27
mnasiadkaSo, let's agree to revisit it if we think about it again/happen to be in a similar situation?15:28
yoctozeptoworks for me; glad now you also understand the matter :-)15:29
mnasiadka#agreed to not pursue forward-upgrade-testing currently and get back to it when we hit another similar issue.15:29
mnasiadka#topic DockerHub limits hurting users https://bugs.launchpad.net/kolla-ansible/+bug/194213415:29
mnasiadkayoctozepto: you again!15:29
yoctozeptomgoddard: also agrees?15:29
yoctozeptomnasiadka: me again, yes (-:15:29
mgoddardagree15:30
yoctozeptook, so users are legitimately giving up on kolla because of dockerhub15:30
mnasiadkagood that we all agree to agreeing.15:30
yoctozeptowhat could we do? could we just point to quay.io?15:30
yoctozeptoand forget dockerhub?15:30
mnasiadkaI thought that will sooner or later come to that point.15:30
*** amoralej is now known as amoralej|off15:30
yoctozeptomnasiadka: me kinda too15:31
mnasiadkaWill it be better with quay.io for a regular user?15:31
mnasiadkaWell, CI is better - so I guess it kind of will.15:31
yoctozeptosame thinking here ;-)15:31
yoctozeptoit also gets those daily updates :D15:31
mnasiadkaI think we should also encourage in the docs, with bold blinking letters on orange background, to build your own images and setup your own registry.15:31
yoctozeptowild mgoddard needs to appear in the discussion too15:32
yoctozeptomnasiadka: ++15:32
mgoddardI'm reading15:32
yoctozepto+2 for blinking, like it's 90s still (-:15:32
mnasiadkaStart writing :)15:32
yoctozeptomnasiadka: mgoddard in read-only mode15:32
opendevreviewalecorps proposed openstack/kolla-ansible master: Add support for VMware NSXP  https://review.opendev.org/c/openstack/kolla-ansible/+/80740415:32
mgoddardI'm just biding my time before saying quay.io++15:32
yoctozeptook, action me on that15:33
yoctozeptoand now - what about the docs? mnasiadka to propose better wording?15:33
yoctozeptoor mgoddard (as English native)?15:33
mgoddard-1 blinking15:33
parallaxhow about this: registry deployed by default as a caching proxy to the outer docker hub / quay.io15:33
mnasiadka#action yoctozepto to point all deployments to quay.io15:33
yoctozeptoparallax: dockerhub still breaks behind proxy15:33
yoctozepto:-(((15:34
mnasiadka#action mnasiadka to update docs encouraging to build your own containers and run your own registry15:34
mnasiadkashould we also write a contrib/ playbook to fetch quay.io containers and push it to your own registry?15:34
parallaxtbh - never had these issues when deploying all in one dev envs 15:34
yoctozeptomnasiadka: yeah, that could help for sure15:34
mgoddardmnasiadka: build/retag/push was a kolla feature request15:34
mgoddards/build/pull/15:35
yoctozeptoparallax: yeah, aio does not break, but all-in-two does afair lol (-:15:35
yoctozeptoespecially with retries15:35
mnasiadkaDo we have a volunteer to work on the playbook? Or even a kolla-ansible command?15:35
yoctozeptodockerhub is a joke really15:35
mgoddardkolla command15:35
yoctozeptomgoddard: now we talk pull-push15:36
mnasiadkaI don't think we have a blueprint around that feature.15:36
yoctozeptohmm15:37
mnasiadkaOk, so - any volunteer to create a blueprint, write down what it should do - and then we can find a volunteer to write the code?15:38
mgoddardhere is the original kayobe RFE: https://storyboard.openstack.org/#!/story/200773115:38
mnasiadkaok then, doesn't sound very detailed, but it's a start.15:40
mnasiadkafor the sake of meeting time, I'll create the blueprint and reference it in Kayobe's RFE - if there will be anyone interested in writing that, he can pick it up.15:41
mnasiadka#action mnasiadka to create pull-retag-push blueprint based on kayobe RFE: https://storyboard.openstack.org/#!/story/200773115:41
mnasiadkaIs there anything else left in that topic to discuss, or we can move on?15:41
mnasiadka#topic Future of Monasca - in context  of CI failures and Elasticsearch switch to AWS Opensearch15:42
mnasiadkaparallax: stage is yours15:42
parallaxWe don't know if there are actual users of it, do we?15:43
parallaxThinking about recent releases, eg. Wallaby15:44
mnasiadkamgoddard: any past user inquiries about Monasca in Kolla? Do you remember?15:44
yoctozeptohave you checked bug reports?15:45
mnasiadkaThere's a Wallaby bug report15:45
mgoddardwell, we are jumping the gun a little here - StackHPC is still a monasca user15:45
yoctozeptowe have quite a few open it seems15:46
mnasiadkaYes, so we need to assume it needs to live.15:46
mgoddardand other people do use it15:46
yoctozeptomgoddard: I see at least two StackHPC folks wishing to kill Monasca15:46
mnasiadkaparallax: what's the main problem (apart of CI liveliness)15:46
yoctozeptoanyhow, any idea what broke in CI?15:47
parallaxElasticsearch in Kolla is getting slightly outdated; however, Monasca is depending on it15:47
mgoddardI proposed https://review.opendev.org/c/openstack/kolla-ansible/+/807689 to try to fix the CI15:48
parallaxThis blocks e.g. migration to OpenSearch15:48
mgoddardparallax: what about centralised logging?15:48
yoctozeptoparallax: we can have both streams15:48
parallaxWhich is not impossible, but likely requires development effort in Monasca15:48
yoctozeptomgoddard: that is WIP and looks like stopping to test Monasca in the Monasca scenario ;p15:49
mgoddardnot really15:49
yoctozeptoso, uhm, the index in es means nothing?15:50
mgoddardit does, but that is just one part of the test15:50
parallaxOK, maybe we could leave ES as it is and proceed with OpenSearch for centralised logging15:51
yoctozeptook, I now see it's still read there15:51
yoctozeptored*15:51
mnasiadkaAnyway, so from what I understand Monasca is persisting on using Elasticsearch - and we'd like to move our central logging to use OpenSearch15:51
mnasiadkaAnything that blocks us from keeping Elasticsearch for Monasca (and waiting for them to move in some direction) and add OpenSearch as new container images and use them in central logging feature?15:51
mgoddardin the absence of someone willing to properly investigate, I'd rather have a green job15:51
yoctozeptomnasiadka: ++15:51
yoctozeptomgoddard: ++15:51
parallaxNot removing / deprecating anything, just adding15:52
mgoddardseems reasonable15:52
mnasiadka#agreed Keep Elasticsearch for Monasca and work on separate container images for OpenSearch and include it in central_logging feature15:53
mnasiadkaGreat, another one solved.15:53
parallaxNice15:53
mnasiadkaLet's go forward...15:53
mnasiadka#topic Release tasks15:53
mnasiadkaThe winter is coming, err - the feature freeze is coming.15:54
mnasiadkaSep 27 - Oct 01 is R-1 - Feature Freeze date15:55
mnasiadka3 weeks from now15:55
mnasiadka#topic Xena cycle planning15:56
mnasiadkaShould we go quickly through priorities once again?15:56
mnasiadkaAnsible - it's on track, three changes awaiting reviews.15:56
mnasiadkaor even four.15:56
mnasiadkaCeph packages have been bumped to Pacific15:57
mnasiadkakevko is again not here, and there's a lot of changes in the proxysql series15:57
mgoddardI've been reviewing system scope patches recently15:58
yoctozeptoI will not be doing consul this cycle15:58
yoctozeptoas masakari delayed this15:58
mgoddardI think we won't be landing system scope this cycle15:58
mgoddardbut maybe we can start it if we break it up15:59
mnasiadkaI didn't have time to review modernisation of the Swift role - did you mgoddard ?15:59
mgoddardnope15:59
mnasiadkaceph radosgw and gather facts command are awaiting other core reviews 15:59
mnasiadkaok, keystone system scope feels like a big thing in just one patch, so it would be nice to break it up and come up with a plan16:00
mnasiadkaheadphoneJames: online?16:00
mnasiadkawell, it's already time to end the meeting16:00
mnasiadkalet's try to work on those reviews16:01
mnasiadka#endmeeting kolla16:01
opendevmeetMeeting ended Wed Sep  8 16:01:23 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:01
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-09-08-15.00.html16:01
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-09-08-15.00.txt16:01
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-09-08-15.00.log.html16:01
mnasiadkaThanks for the meeting!16:01
parallaxThanks16:01
yoctozeptothanks mnasiadka16:02
mgoddardthanks!16:02
opendevreviewMerged openstack/kolla-ansible master: Use Docker healthchecks for mistral services  https://review.opendev.org/c/openstack/kolla-ansible/+/77907516:07
opendevreviewMerged openstack/kolla-ansible stable/victoria: Zun: Temporarily skip capsule test for ubuntu  https://review.opendev.org/c/openstack/kolla-ansible/+/80775616:13
headphoneJamesmnasiadka: hi16:32
headphoneJamesmnasiadka: I am open to splitting the patch into different parts, but not sure what that would look like16:34
*** imtiazc_ is now known as imtiazc17:07
opendevreviewMerged openstack/kolla master: [CI] Drop support for publishing versioned image tags  https://review.opendev.org/c/openstack/kolla/+/78671920:00
-opendevstatus- NOTICE: The Gerrit service on review.opendev.org is going offline momentarily for a host migration and zuul upgrade, downtime should be only a few minutes.21:05

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!