Wednesday, 2022-04-20

*** amoralej|off is now known as amoralej06:48
yoctozeptomorning07:21
mgoddardmorning07:52
mnasiadkamorning!07:55
ironfootHello! When upgrading openstack to latest stable/xena I hit an issue because saslpasswd2 is not available in nova_libvirt container. As a result the handler "nova-cell : Create libvirt SASL user" would fail.08:28
ironfootMy guess is that previous nova_libvirt container doesn't have that tool installed, so I installed it manually just to be able to do the upgrade08:29
hrwmorning08:29
ironfootI just though I should mention as this could happen again08:29
hrwmnasiadka: yoga/cs8 should build now. started recheck08:30
hrwironfoot: which distro in containers?08:30
mnasiadkahrw: thanks08:31
ironfoothrw: Ubuntu 20.0408:31
hrwironfoot: and you installed sasl2-bin package?08:31
ironfootyes08:32
hrwironfoot: https://review.opendev.org/c/openstack/kolla/+/833414 added that in xena. please update images08:32
ironfootaha! fixed problem then, thank you!08:34
hrwironfoot: yw08:34
hrwironfoot: and thanks for coming here and telling about problem.08:34
yoctozeptook, the images are available on quay, time to switch k-a, hrw09:37
opendevreviewMichal Arbet proposed openstack/kolla master: Add support for checking more daemons  https://review.opendev.org/c/openstack/kolla/+/76578110:12
kevko\o/10:14
hrwyoctozepto: super!10:28
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: drop install_type from image names  https://review.opendev.org/c/openstack/kolla-ansible/+/83729310:30
hrwthis should use short names10:30
hrwyoctozepto: asked on #ubuntu-server about horizon10:35
hrwadded agenda entry to whiteboard - we need to handle Y->Z upgrades in some new scenarios 10:39
hrwhttps://zuul.openstack.org/status/change/837293,8 has some greens already. short names work ;D10:57
hrwmeeting is in 2 or 3 hours from now?11:03
mnasiadka2 and a half (16:00 polish time) ;)11:24
hrwgracias11:26
hrwheh. ussuri lacks network=host11:30
hrwanyway. added to my global kolla-build.conf ;D11:31
yoctozeptohrw: reds mostly expected11:43
yoctozeptoonly swift broke, needs fixing11:43
hrwyep11:43
opendevreviewMarcin Juszkiewicz proposed openstack/kolla stable/ussuri: fluentd: pin yajl-ruby 1.4.1 as we have old Ruby (!x86-64 only)  https://review.opendev.org/c/openstack/kolla/+/83871611:44
hrwussuri only11:45
hrwyoctozepto: is there any useful way to scroll through openstack.kolla images on quay.io?11:50
hrwah. swift wants dockerhub...11:53
hrwand there we lack image. it is on quay11:53
yoctozeptoyeah, it is probably because of how CI is set up11:53
yoctozepto(for swift)11:53
hrwyoctozepto: that ussuri patch is aarch64 only. I would normally ignore it but we want to EM ussuri to let it get it fixed ;D11:55
opendevreviewVerification of a change to openstack/kolla stable/yoga failed: [release] centos: switch to CentOS cloud SIG packages  https://review.opendev.org/c/openstack/kolla/+/83730011:57
hrwUCA?11:57
hrwhm. centos11:58
hrwyoga not on mirrors...11:59
yoctozeptohow did it pass in check though?12:00
yoctozeptoone mirror issue?12:00
hrwnot checked logs12:01
yoctozeptohrw: hmm, gnocchi on ubuntu aarch64 INFO:kolla.common.utils.gnocchi-base:      LookupError: setuptools-scm was unable to detect version for /gnocchi-base-source/gnocchi-base-archive-4.4.1.12:01
yoctozepto(on master)12:01
yoctozeptobut we fixed the perms everywhere, no?12:02
hrwgive me link to change please12:02
yoctozeptohttps://review.opendev.org/c/openstack/kolla/+/76578112:05
yoctozeptosorry, it was from Apr 1312:05
yoctozeptomeh12:05
*** amoralej is now known as amoralej|lunch12:06
opendevreviewLin PeiWen proposed openstack/kolla-ansible master: Add support for dedicated RabbitMQ clusters for cell controllers  https://review.opendev.org/c/openstack/kolla-ansible/+/82308412:06
hrwhttps://zuul.openstack.org/status/change/765781,21 is red12:06
hrwINFO:kolla.common.utils.cron:W: Failed to fetch http://mirror.regionone.osuosl.opendev.org/ubuntu-ports/dists/focal-security/InRelease  Unable to connect to mirror.regionone.osuosl.opendev.org:http:12:07
hrwmirror issue12:07
opendevreviewLin PeiWen proposed openstack/kolla-ansible master: Add support for dedicated RabbitMQ clusters for cell controllers  https://review.opendev.org/c/openstack/kolla-ansible/+/82308412:11
hrwyoctozepto: that swift thing needs own fixing. similar for magnum, monasca, ovn, masakari failures12:16
opendevreviewMerged openstack/kayobe-config-dev stable/yoga: Update .gitreview for stable/yoga  https://review.opendev.org/c/openstack/kayobe-config-dev/+/83783212:19
opendevreviewMerged openstack/kayobe-config-dev stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga  https://review.opendev.org/c/openstack/kayobe-config-dev/+/83783312:19
opendevreviewVladislav Belogrudov proposed openstack/kolla-ansible master: Fix ansible version in install command  https://review.opendev.org/c/openstack/kolla-ansible/+/83871912:24
hrwmnasiadka: openstack-yoga should be on CI mirrors on next mirror sync12:28
mnasiadkahrw: ok, thanks12:29
opendevreviewLin PeiWen proposed openstack/kolla-ansible master: Add support for dedicated RabbitMQ clusters for cell controllers  https://review.opendev.org/c/openstack/kolla-ansible/+/82308412:33
yoctozeptohrw: really? doh!12:33
hrwreminder: ubuntu 22.04 release next week. this time I want to review patch adding it. not to write it ;D12:35
yoctozeptoxD12:38
hrwI moved Kolla to new releases of Debian, CentOS, Ubuntu. So to minimalize bus factor I refuse to do ubuntu 22.04 one12:44
hrwespecially when it involves kolla, system-config and probably also project-config repos12:44
yoctozeptostackhpc cares about ubuntu now so I assume they will help12:47
opendevreviewVladislav Belogrudov proposed openstack/kolla-ansible master: Fix ansible version in install command  https://review.opendev.org/c/openstack/kolla-ansible/+/83871912:50
mnasiadkayoctozepto: that's the plan12:53
hrwI am out. will be at meeting. hope for being from start12:56
*** amoralej|lunch is now known as amoralej13:06
opendevreviewVerification of a change to openstack/kolla stable/yoga failed: [release] centos: switch to CentOS cloud SIG packages  https://review.opendev.org/c/openstack/kolla/+/83730013:06
yoctozeptoI might be late13:11
yoctozeptojust saying13:11
opendevreviewPierre Riteau proposed openstack/kolla-ansible master: Add support for deploying prometheus-msteams  https://review.opendev.org/c/openstack/kolla-ansible/+/81267913:21
opendevreviewJuan Pablo Suazo proposed openstack/kolla-ansible master: Adds ability to group stacktraces into single fluentd message  https://review.opendev.org/c/openstack/kolla-ansible/+/83856413:47
mnasiadkamgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt frickler adrian-a - meeting in 1013:50
mnasiadka#startmeeting kolla14:00
opendevmeetMeeting started Wed Apr 20 14:00:01 2022 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'kolla'14:00
mnasiadka#topic rollcall14:00
mnasiadkao/14:00
hrw /o\/o\14:08
mnasiadka#topic agenda14:09
mnasiadka* CI status14:09
mnasiadka* Release tasks14:09
mnasiadka* Current cycle planning14:09
mnasiadka* Additional agenda (from whiteboard)14:09
mnasiadka* Open discussion14:09
mnasiadka#topic CI Status14:09
mnasiadkaI think that's quite ok now, since we fixed Victoria and Ussuri?14:09
hrwYoga will be ready once CI mirror centOS 14:10
mnasiadkayup, that's release task :D14:11
hrwI would say that CI is fine14:11
mnasiadka#topic Release tasks14:12
mnasiadkaSo the only patch that needs to be merged is the CentOS one - and we're waiting for infra mirror sync14:12
mnasiadkaI haven't seen other disturbing issues14:13
mnasiadkahrw: what about the Ubuntu horizon thing?14:13
hrwUCA horizon is a problem14:13
hrwnot heard anything from UCA team on irc. will reask14:13
mnasiadkaok, do they have a bug filed for this?14:13
hrwnot by me and not checked14:14
mnasiadkaok14:14
mnasiadka#topic Current cycle planning14:14
mnasiadkaWe're on a good path to get rid of binary once and for all14:15
hrwwe got rid of it14:15
mnasiadkaSo I guess that's the ,,theme'' that is progressing now14:15
hrwwe are now on cleaning14:15
mnasiadkaYeah, I was thinking about cleaning and getting rid of -source in the image name14:15
hrwyep. short names as we call it14:16
hrwuhttps://review.opendev.org/c/openstack/kolla-ansible/+/837293 needs merge and then I do one patch to kolla and we are on source only with short names14:16
mnasiadkagreat14:16
hrwthen upgrade paths need work but that in 'additional agenda'14:16
mnasiadkaok14:16
mnasiadka#topic Additional agenda (from whiteboard)14:17
mnasiadkalet's start with yours14:17
mnasiadka(hrw) Yoga->Zed upgrades14:17
hrwpaste rest14:17
mnasiadkanew CI job: Y/binary -> Z/source14:17
mnasiadkawe need to handle removal of old images (Y/binary ones or Y/source ones as we will have shorter names in Z)14:17
hrwok14:17
hrwso for now we only test Y/source -> Z/source on CI14:18
hrwas there is no Y/binary -> Z/binary path we need to test Y/binary -> Z/source one14:18
hrwand then we need a way to get rid of old containers as names are changed in both scenarios14:18
hrwkolla/ubuntu-binary-nova-compute -> kolla/ubuntu-nova-compute etc14:19
mnasiadkaimage names have changed, so that should be handled semi-automatically in existing code14:19
hrwwould be bad to have both copies running14:19
mnasiadkaif the image name variable gets templated properly14:19
hrwNOTE: I never did upgrades14:19
mnasiadkaOk, I guess we (me, yoctozepto, mgoddard) can help with this one14:20
hrwyoctozepto: you here or not yet?14:20
mnasiadkawe don't have CI for binary-upgrade currently, but it makes sense to test the move from binary to source14:21
hrwwe need to have it as this will be popular path14:21
hrwand better get it running in first weeks of cycle before we grab Z changes from projects14:21
mnasiadkayup, agree14:22
mnasiadkaseems yoctozepto is very late ;)14:22
mnasiadkashould we move his topics to next week?14:22
hrwwe can discuss them now too14:23
hrwit is EM/EOL14:23
mnasiadka(yoctozepto) EM Victoria14:23
mnasiadkathat's obvious14:23
mnasiadkado we need to merge any Victoria patches before we do that last release and mark as EM?14:23
hrwiirc we did but moment14:23
hrwnothing in queue14:24
hrwso we can #action14:24
mnasiadkano open changes in Kolla, one open in K-A (make kolla-build quiet)14:24
hrwok, merge it and release14:24
mnasiadkaand two in Kayobe14:24
mnasiadkamgoddard: https://review.opendev.org/c/openstack/kayobe/+/782703 and https://review.opendev.org/c/openstack/kayobe/+/830069 - either let's merge them or abandon14:25
mnasiadka(yoctozepto) EOL Train14:25
mnasiadkathat's more complicated14:25
hrwI vote for EOL despite it being migration point14:26
mnasiadkawe have some (not a lot, just a couple) customers on Train, I'll check internally if there are any reasons for not doing EOL - but I doubt there are14:26
hrwok14:27
hrw#note EMHPC Train then14:27
hrw;D14:27
mnasiadka#action mnasiadka work on marking Train EOL14:27
mnasiadkaok then, seems like we're done14:28
mnasiadka#topic Open discussion14:28
mnasiadkaanything?14:28
hrwubuntu 22.04 is coming - another thing to your plate ;D14:28
mnasiadkayeah, we know, we'll manage it14:28
hrwexternal repos will be main pain probably14:28
yoctozeptoo/14:29
hrwinfra team will mirror main repos, you need to add UCA mirror probably (I did most of previous ones iirc)14:29
mnasiadkayoctozepto: you just made it to open discussion ;-)14:29
hrwone thing: do we checked core members lists?14:30
yoctozeptomnasiadka: to close the meeting :-)14:30
hrwthere are people who I have not seen for a while there14:30
mnasiadkawe did, last cycle, I did not remove some people that promised to continue working on Kolla14:30
hrwok14:30
mnasiadkaI can review if they did anything14:30
hrwyoctozepto: re-read meeting and add what you want14:31
* yoctozepto finished reading14:32
yoctozeptoI agree that we should revisit the cores' list14:32
yoctozeptoas other projects do14:32
yoctozeptore upgrades from binary to source (typeless) - will do14:33
opendevreviewMaksim Malchuk proposed openstack/kayobe master: Cleanup old and deprecated Swift configuration  https://review.opendev.org/c/openstack/kayobe/+/83873014:33
mnasiadkaI have a question - we're left with admin endpoint for keystone (removed all the others) - what's the ultimate plan with it?14:33
hrwbtw - ussuri is EM already?14:34
mnasiadkayes14:35
mnasiadkaIfa0b1b059ca8aafb76b2c25c1dd03209bc001f2d14:35
hrwok. have to remember 'ignore <W'14:36
yoctozeptomnasiadka: remove it14:36
yoctozeptomnasiadka: it even has a random bug (on the whiteboard)14:36
yoctozeptoas it uses a high port14:36
yoctozeptoanything can randomly pick it up and block the deploy14:36
mnasiadkayoctozepto: I stumbled upon Magnum having issues with lack of admin endpoint today, I think keystoneclient/keystoneauth1 defaults to that...14:37
yoctozeptomnasiadka: ugh? lack of admin endpoint for what?14:37
mnasiadkayoctozepto: lack for admin endpoint for keystone :)14:39
yoctozeptomnasiadka: but it's not removed (-:14:39
yoctozeptoor do you mean it is somewhere where you support it?14:39
mnasiadkaYeah, I know - but it's not created by default in devstack14:39
yoctozeptodevstack, devstack14:40
mnasiadkaUsed devstack to test out one thing in Magnum and stumbled across this14:40
yoctozeptothis is kolla meeting14:40
mnasiadkaSo just asked, if we are planning to remove admin endpoint in keystone14:40
yoctozeptojust use kolla to test out one thing...14:40
mnasiadkaif yes - I need to fix it there :D14:40
yoctozeptoare you sure it's not devstack's misconfig?14:40
mnasiadkayoctozepto: write tempest tests in Kolla? come on14:40
yoctozeptomnasiadka: we should run tempest at some point tbh14:40
yoctozeptono need to write, just run14:40
mnasiadkawrite NEW tests for Magnum, calm down :)14:41
hrwmnasiadka: https://bugs.launchpad.net/cloud-archive/+bug/1969630 is yoga/uca/horizon bug14:41
hrwdeprecate magnum in yoga, remove in zed14:41
mnasiadkaTempest in K-A CI? Well, sure, why not14:41
hrwtomorrow we will only worry of Magnum as guns ;D14:42
mnasiadkadeprecate and remove OpenStack, why not ;)14:42
yoctozeptomnasiadka: well, you write tempest tests and run them against k-a; I think I am missing the point you are trying to make by running devstack ;-)14:42
mnasiadkayoctozepto: replicate CI and find new problems in devstack :D14:42
yoctozeptomnasiadka: well, magnum is not necessarily the best faring project14:43
yoctozeptomnasiadka: ah, you are now a devstack fanboy, very well14:43
mnasiadkayoctozepto: there's a couple of them, trying to make it better14:43
yoctozeptogo get those bash scripts14:43
mnasiadkayoctozepto: don't even get me started :)14:43
mnasiadkaanyway, let's finish14:43
yoctozepto:-)14:43
mnasiadkathanks for the meeting!14:43
yoctozeptothus go kolla14:43
mnasiadka#endmeeting14:43
opendevmeetMeeting ended Wed Apr 20 14:43:42 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:43
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-04-20-14.00.html14:43
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-04-20-14.00.txt14:43
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-04-20-14.00.log.html14:43
yoctozeptoSean uses kolla for devel :-)14:43
yoctozeptothanks mnasiadka for chairing14:43
yoctozeptobut I'm serious regarding kolla14:44
wuchunyangwhy do we remove magnum ?14:44
yoctozeptowuchunyang: that was hrw's joke14:44
mnasiadkawuchunyang: we don't, hrw is teasing14:44
hrwtempest in kolla(-ansible) CI would be nice test14:44
wuchunyanghaha,, thanks for explanation.14:44
hrwwuchunyang: last days before fresh release is 'what do we want to remove in next cycle so we can do it in 2h now'14:45
hrwwuchunyang: like now: deprecate in Y, remove in Z == 2 patches done one by one, sent for review, CI, gate, merge, done14:46
wuchunyangOh, I see.14:47
yoctozeptoyes, we like removing cruft14:47
yoctozeptomagnum is not that bad14:47
yoctozepto(though it could probably be better, as most things)14:48
yoctozeptotbh, there is a queue of projects to remove BEFORE magnum so no rush14:48
hrwspeaking of evil...14:48
yoctozepto*cough* murano *cough*14:48
hrwwuchunyang: iirc you are kolla core still?14:48
hrw*cough* ^base *cough*14:49
wuchunyangyes14:49
hrwok, just was not sure14:50
wuchunyangI would like to add some tests for trove in kolla in this cycle.14:51
hrw15:00 meeting (on headphones), 16:00 kolla & rdo meetings. now coffee time as in a moment another meeting...14:51
mnasiadkano more meetings14:52
yoctozeptowuchunyang: lovely14:55
mnasiadkawuchunyang: how is trove? having fun? ;-)14:56
wuchunyangtrove is hard to use now, and no maintainer.14:57
wuchunyangI love kolla more, haha.14:58
yoctozepto<315:01
yoctozeptofwiw, magnum jobs in CI fail on trove in horizon15:06
yoctozeptohttps://storyboard.openstack.org/#!/story/200995515:11
yoctozeptoalready reported15:11
yoctozeptohttps://review.opendev.org/c/openstack/trove-dashboard/+/83870015:14
yoctozeptoI like quick solutions15:14
yoctozeptos/like/love/15:14
yoctozeptomnasiadka, mgoddard: merge https://review.opendev.org/c/openstack/kolla-ansible/+/837293 ?15:19
hrwand whole set of 'quiet kolla' waits for +2+W too15:24
hrwbut that's after 837293 ;D15:24
opendevreviewRadosÅ‚aw Piliszek proposed openstack/kolla-ansible master: [CI] Always use quay.io via infra's mirror  https://review.opendev.org/c/openstack/kolla-ansible/+/83874815:34
yoctozeptothat fixes swift's CI ^15:36
yoctozeptoplease also read my writeup on hrw's change regarding other jobs - if nobody cares about monasca, we can drop this job from CI15:37
opendevreviewMerged openstack/kolla stable/ussuri: fluentd: pin yajl-ruby 1.4.1 as we have old Ruby (!x86-64 only)  https://review.opendev.org/c/openstack/kolla/+/83871616:25
opendevreviewMerged openstack/kolla-ansible master: Fix ansible version in install command  https://review.opendev.org/c/openstack/kolla-ansible/+/83871916:25
hrwcs8/yoga recheck started - CI got mirror16:30
*** amoralej is now known as amoralej|off16:35
hrwhttps://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=1 - someone uses it?16:41
opendevreviewRadosÅ‚aw Piliszek proposed openstack/kolla-ansible stable/yoga: Fix ansible version in install command  https://review.opendev.org/c/openstack/kolla-ansible/+/83870116:49
yoctozeptohrw: unlikely16:50
opendevreviewMerged openstack/kolla-ansible stable/yoga: Fix ansible version in install command  https://review.opendev.org/c/openstack/kolla-ansible/+/83870117:11
hrwmnasiadka: you had your fingers in https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=1 - is someone using it? 18:11
*** hrww is now known as hrw19:10
*** hrww is now known as hrw19:21
opendevreviewJuan Pablo Suazo proposed openstack/kolla master: Adds fluent-plugin-detect-exceptions as a default fluentd plugin  https://review.opendev.org/c/openstack/kolla/+/83878720:43
opendevreviewJuan Pablo Suazo proposed openstack/kolla master: Adds fluent-plugin-detect-exceptions as a default fluentd plugin  https://review.opendev.org/c/openstack/kolla/+/83878720:44
opendevreviewJuan Pablo Suazo proposed openstack/kolla master: Adds fluent-plugin-detect-exceptions as a default fluentd plugin  https://review.opendev.org/c/openstack/kolla/+/83878720:45
opendevreviewJuan Pablo Suazo proposed openstack/kolla master: Adds fluent-plugin-detect-exceptions as a default fluentd plugin  https://review.opendev.org/c/openstack/kolla/+/83878720:45
opendevreviewVerification of a change to openstack/kolla stable/yoga failed: [release] centos: switch to CentOS cloud SIG packages  https://review.opendev.org/c/openstack/kolla/+/83730023:35

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!