Friday, 2022-09-09

mnasiadkamorning07:20
rockeymornings07:20
frickleryoctozepto: reading https://review.opendev.org/c/openstack/large-scale/+/855508 I wonder whether we should adopt that option in k-a by default, wdyt?07:44
mnasiadkayoctozepto: cleaned up quay.io a bit (those images without binary and source in the name that we pushed one time 13th of May)08:15
hrwyoctozepto: pip==22.* setuptools==65.* to both pip==latest patches08:27
hrwI prefer stable branches to behave stable rather to fail due to external changes08:27
yoctozeptofrickler: replied, it's the default so we are using it08:57
yoctozeptomnasiadka: awesome, thanks!08:57
mnasiadkayoctozepto: after we start publishing rocky linux - I'll clean up master centos8 and focal tags in the new image scheme08:58
yoctozeptohrw: hmm, makes sense, will amend08:58
yoctozeptomnasiadka: +++08:58
mnasiadkaThis is probably what I forgot to touch on the weekly meeting08:58
mnasiadkawe're not going to publish centos 9s images, right?08:58
yoctozeptowe need them for CI so we will08:58
yoctozeptowe can document them differently from others though08:59
hrwyoctozepto: I remember moments when we had to workaround setuptools change. IIRC we have it done twice even.08:59
yoctozeptowe also don't know whether rocky images will be feasible or if it is better to stay with stream in images and rocky on hosts (which sounds reasonable enough to me)08:59
yoctozeptohrw: yeah, I agree it is good to do it here as well09:00
yoctozeptoI will be backporting that so it makes double sense09:00
yoctozepto(or quadruple sense, aye?)09:00
hrwit is nice to have latest and greatest but who will remember to set them in stone during release cycle?09:01
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: veryWIP: Add Opensearch support  https://review.opendev.org/c/openstack/kolla-ansible/+/85661009:02
mnasiadkayoctozepto: what do you mean by feasible?09:08
yoctozeptomnasiadka: are not we missing SIGs' work in Rocky still?09:10
mnasiadkayoctozepto: Rocky (just like other clones) are using CentOS Stream SIG package repos, the centos-release-* packages are in Extras repo in RL8/909:10
mnasiadkaAs long as it works fine, they are not going to put additional work into just rebuilding this.09:11
mnasiadkaWhich is probably fine09:11
mnasiadkaAnd they would rather need to rebuild what is in SRPMs in RHEL, but that's another discussion.09:12
mnasiadkabut that's next step, first we need the host os c9s/rl9 patch merged09:12
hrwmnasiadka: el9 clones are rhel srpm rebuilds ;D09:13
mnasiadkahrw: just saying that Alma and others are using CentOS Stream SIG packages :D09:13
mnasiadkaanyway, let's not try to save the world in one day09:14
hrwmnasiadka: ;d09:14
mnasiadkaI think we should only expose the rl9 builds, and the c9s ones should be only to test what breakage will come soon to rl9 (just as we agreed on PTG)09:15
mnasiadkaso, do we want to publish images for a single kolla-ansible CI job?09:15
mnasiadka(which will be non voting)09:15
mnasiadkathat's the question for next weekly meeting :)09:15
mnasiadkaIn the meantime I need another pair of eyes on https://review.opendev.org/c/openstack/kolla-ansible/+/841916 - is that change of default so daunting, that we need to persist the not really sane default? yoctozepto can you have a look?09:18
yoctozeptowhat default09:19
* yoctozepto looking09:19
mnasiadkaprometheus-alertmanager web external url is pointing to internal API address :D09:20
mnasiadkaso if you send alerts via webhook to Slack or something else - the url that you can click on to get alert details - points to internal API url09:20
mnasiadkaSo either we change the default, which makes sense to me at least, and allow users to configure it, or we persist that default, but still allow users to configure it ;)09:21
yoctozeptobut this url works already, right?09:22
yoctozeptoit's just the advertisement that is broken?09:23
yoctozeptook, confirmed it works09:24
mnasiadkayes, the external url already works, just advertisement is broken (users should be directed to public api url, not internal one)09:24
yoctozepto+209:24
yoctozeptoyeah, it's a simple fix09:24
mnasiadkaWell, there's some disagreement in this review that it will hugely impact users.09:25
mnasiadkajust wanted another pair of eyes09:25
mnasiadkaAnd really surprised nobody has been complaining over this in the past09:25
frickleryoctozepto: ah, good that we have so many standa^Wdefaults to choose from ;)09:27
yoctozeptofrickler: yeah, I love it too09:30
yoctozeptomnasiadka: "hugely impact"?09:31
yoctozeptomaybe it has been misunderstood09:31
mnasiadkayoctozepto: check earlier comments in this patch ;)09:31
yoctozeptoif you want to improve, you can go explicit in the reno on how to restore the previous (wrong) behaviour09:31
yoctozeptoah, read them09:32
yoctozeptoI think dougsz misinterpreted the role of this option09:32
yoctozeptothough I hope you verified it and it works like you described, not like he describes it09:32
mnasiadkayeah, parallax verified that, but let me check once again09:34
mnasiadkaalertmanager docs and code skim tells me it's ONLY used in HTTP POST to webhooks09:37
yoctozeptothat would make sense09:38
opendevreviewRafal Lewandowski proposed openstack/kayobe master: added new dib upper constraints variables  https://review.opendev.org/c/openstack/kayobe/+/85590909:51
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Fix AlertManager's external web url  https://review.opendev.org/c/openstack/kolla-ansible/+/84191610:05
mnasiadkayoctozepto: https://review.opendev.org/c/openstack/kolla-ansible/+/841916 can you reapply +2 and let's end this drama?10:08
opendevreviewRafal Lewandowski proposed openstack/kayobe master: added new dib upper constraints variables  https://review.opendev.org/c/openstack/kayobe/+/85590910:16
kevkoINFO:kolla.common.utils.neutron-base: ---> Running in f07cb9417abb10:26
kevkoINFO:kolla.common.utils.neutron-base:Processing /neutron10:26
kevkoINFO:kolla.common.utils.neutron-base:  Preparing metadata (setup.py): started10:26
kevkoINFO:kolla.common.utils.neutron-base:  Preparing metadata (setup.py): finished with status 'done'10:26
kevkoINFO:kolla.common.utils.neutron-base:[91mERROR: Cannot install neutron 20.2.1.dev32 (from /neutron) because these package versions have conflicting dependencies.10:26
kevkoINFO:kolla.common.utils.neutron-base:[0m10:26
kevkoINFO:kolla.common.utils.neutron-base:The conflict is caused by:10:26
kevkoINFO:kolla.common.utils.neutron-base:    The user requested neutron 20.2.1.dev32 (from /neutron)10:26
kevkoINFO:kolla.common.utils.neutron-base:    The user requested (constraint) neutron===20.2.010:26
kevkoINFO:kolla.common.utils.neutron-base:To fix this you could try to:10:26
kevkoINFO:kolla.common.utils.neutron-base:1. loosen the range of package versions you've specified10:26
kevkoINFO:kolla.common.utils.neutron-base:2. remove package versions to allow pip attempt to solve the dependency conflict10:26
kevkoINFO:kolla.common.utils.neutron-base:[91mERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts10:27
kevkoINFO:kolla.common.utils.neutron-base:[0m10:27
kevkoINFO:kolla.common.utils.neutron-base:Removing intermediate container f07cb9417abb10:27
kevkoERROR:kolla.common.utils.neutron-base:Error'd with the following message10:27
kevkokolla yoga images build failing locally because of global requirements right ? 10:27
kevkoshould I propose patch for global requirements and update upper-constraints ? 10:27
kevkoneutron 20.2.0 -> 20.2.1 ... or what ? 10:27
mnasiadkaI think we've seen that before somewhere else, should we remove neutron from upper-constraints.txt?10:33
fricklerseems like a bug, yes. likely we don't see it in CI because we include latest tagged version there instead of latest stable? which would be a debatable option, too10:38
kevkohmm, upper-constraints includes neutron 20.2.0 as latest yoga release, but kolla is installing 20.2.1.dev32 ...last tag in git is 20.2.010:41
frickleractually we do install latest stable/yoga neutron here https://1af994232f33a248864f-7c57593ce71ac5e536068fa74069f258.ssl.cf1.rackcdn.com/855565/1/check/kolla-build-debian-source/4845005/kolla/build/neutron-base.log10:42
fricklerso something else must be different in your local build10:42
kevkohmmmm10:43
kevkofrickler: how old is log above ? 10:43
fricklertimestamp is at the top10:43
kevkohmm, that's weird10:47
kevkofrickler: can I start build manually somehow ? because when I download tarball for neutron ..there is 20.2.1dev310:48
kevkoi mean remotely ..10:49
kevkoand i think it has to fail also in upstream 10:49
fricklerkevko: https://opendev.org/openstack/kolla/src/branch/stable/yoga/docker/openstack-base/Dockerfile.j2#L32610:53
fricklerare you sure you are using kolla-build from stable/yoga and not from master?10:54
fricklerbut that also explains why mnasiadka has seen this https://review.opendev.org/c/openstack/kolla/+/84779310:55
kevkofrickler: hmm, it looks like this is needed to backport ..11:00
kevkolet me check 11:00
kevkobut i still don't understand why neutron tarball is 20.2.1dev32 but not released ! 11:01
kevkono tag in git ..nothing 11:01
yoctozeptomnasiadka, dougszu: what is the reasoning in that "internal discussion"? 11:01
fricklerkevko: because it is not released. it is the current head of the stable branch, which is tarred up as a convenience for consumers like kolla11:02
kevkofrickler: check below 11:05
kevkomichalarbet@pixla:~/ultimum/git/upstream/neutron$ git tag | grep '20\.2\.[0-9]'11:05
kevko20.2.011:05
kevkomichalarbet@pixla:~/ultimum/git/upstream/neutron$ grep -Ri '20\.2\.1'11:05
kevkomichalarbet@pixla:~/ultimum/git/upstream/neutron$ grep -Ri '20\.2\.1' .11:05
yoctozeptomnasiadka, dougszu: please reply on the patch, thanks11:05
kevkofrickler: hmm, ok .. now it's clear ..i just thought that it is adding "devXY" suffix to latest tag in git 11:09
hrwkevko: INFO:kolla.common.utils.neutron-base:Successfully tagged yoga/debian-source-neutron-base:debian11:12
kevkohrw: yeah,it looks like i should rebase :P 11:13
hrwkevko: INFO:kolla.common.utils.neutron-base:Successfully installed neutron-20.2.1.dev32 neutron-lib-2.20.0 os-ken-2.3.1 os-resource-classes-1.1.0 os-vif-2.7.1 ovsdbapp-1.15.211:13
hrwkevko: like frickler said - we fetch source tarball, not latest release11:14
hrwkevko: that's why 'dev' in version11:14
kevkohrw: yes i know, but i thought that python3 setup.py bdist_wheel will build version from LATEST_TAG.devXY version 11:14
kevkobut it looks like it is bumping last number in version and adding devXY suffix also 11:15
kevkoand i thought it is not bumbing 11:15
hrwkevko: and latest tag is 20.2.1?11:16
kevkono, it is 20.2.0 11:16
fricklerthat is because dev releases are sorted to come before the tagged release https://peps.python.org/pep-0440/#developmental-releases11:19
hrwI never understood how that versioning is done tbh11:19
kevkohrw: but now it makes sense  (python3 setup.py bdist_wheel will build package from TAG+1+.devXY) ...so for example for neutron if latest tag is 20.2.0 and top of the git is not tagged ..it will be 20.2.1.devXY ..so requirements can't work11:19
hrwkevko: ok, where you got that 20.2.0 u-c from?11:20
kevkohmm, and is it safe globally ? i mean ..shouldn't we install sources from TAGs ? (it sounds little bit as step backwards ..but just wondering)11:21
kevkohrw: upper-constraints classic kolla11:21
kevkonow when i am rebased ..it is working ..because we removed from upper-constraints11:22
fricklerhrw: https://review.opendev.org/c/openstack/requirements/+/84678511:22
fricklerI think that would really better be some <21.0 cap11:22
hrwah. did not got it with stable/yoga kolla11:26
hrwkevko: it was discussed in past. we moved to stable/* HEAD to get all fixes11:28
hrwkevko: we used TAG releases in past.11:28
kevkohrw: ok, i also agree that stable/* is better than tag 11:29
kevkofrickler: or take upper-constraints and replace last number in version for tarballs which are downloaded 11:31
kevkonow we rely on the fact that the tarball will always be correct and noone will ever make a mistake in tarball release 11:32
kevkonow we just removed neutron entry from upper constraints :D 11:33
fricklerwhich is the same situation as for every other service, no big deal11:34
opendevreviewBartosz Bezak proposed openstack/kayobe master: Move to CentOS Stream 9 / Rocky Linux 9  https://review.opendev.org/c/openstack/kayobe/+/85565612:00
dougszuhello, yoctozepto, frickler has kindly summarised12:02
opendevreviewBartosz Bezak proposed openstack/kayobe master: Move to CentOS Stream 9 / Rocky Linux 9  https://review.opendev.org/c/openstack/kayobe/+/85565612:06
mnasiadkayoctozepto: sorry, was busy, do I still need to do something? :)12:08
dougszuW+1 :D12:09
mnasiadkayoctozepto: we basically come to a conclusion to not change the default, because it might break existing deployments on upgrade, and it's only me and parallax complaining - nobody else have raised a bug :)12:09
mnasiadka+w it is12:09
opendevreviewPierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax  https://review.opendev.org/c/openstack/kayobe/+/85553813:12
yoctozeptomnasiadka: I would suggest to backport this but set a different default for zed13:12
mnasiadkayoctozepto: I would do that too, but dougszu seems to have different opinion - my knowledge about prometheus ends on deploying it and clicking in alertmanager :)13:13
yoctozeptoack13:14
yoctozeptoI am not pushing, I am not using this functionality13:14
mnasiadkayoctozepto: do you plan on updating the tenks change? we need to get it merged ASAP I think13:14
opendevreviewMerged openstack/kolla-ansible master: Fix AlertManager's external web url  https://review.opendev.org/c/openstack/kolla-ansible/+/84191613:17
yoctozeptoyoctozepto: yes, soon, do not treat me with "ASAP" ;-) 13:20
hrwmnasiadka: commented on your k-a opensearch patch.13:22
mnasiadkareplying to yourself? good one :)13:22
mnasiadkahrw: sure, working on it now, but thanks13:23
dougszuyoctozepto: what would be your reason for changing the default?13:31
yoctozeptodougszu: as the public interface is already on there, I think it makes more sense to advertise that one to the external systems13:33
yoctozeptomnasiadka: mobile client13:33
opendevreviewPierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax  https://review.opendev.org/c/openstack/kayobe/+/85553813:39
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: veryWIP: Add Opensearch support  https://review.opendev.org/c/openstack/kolla-ansible/+/85661013:42
rockeylulz; veryWIP:: might be a new status for sure13:43
dougszuyoctozepto: One downside is that you have to deal with the static password from HAProxy services for the public interface13:43
mnasiadkarockey: it just indicates it's a moving target for a couple of days ;)13:44
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Add Opensearch image(s)  https://review.opendev.org/c/openstack/kolla/+/83037313:46
rockeymnasiadka: "couple of days", planning on working on the weekend i see ;)13:46
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Add Opensearch image(s)  https://review.opendev.org/c/openstack/kolla/+/83037313:47
mnasiadkarockey: surely not over the weekend ;)13:47
opendevreviewBartosz Bezak proposed openstack/kayobe master: Move to CentOS Stream 9 / Rocky Linux 9  https://review.opendev.org/c/openstack/kayobe/+/85565614:08
BobZ_Annapolisusing the online kolla-ansible aio - works fine the 1st time thru - after the machine is rebooted and everything restarts, nova-compute never comes back up - crash-restarting w/various errors, eg nova.exception.HypervisorUnavailable: Connection to the hypervisor is broken on host - seen before / suggestions ? thx14:20
yoctozeptoBobZ_Annapolis: any chance your hostname has been reset?14:24
yoctozeptoor /etc/hosts ? some clouds may garble it between restarts14:29
yoctozeptoI *think* kolla-ansible tries to disable this behaviour of cloud-init14:29
priteauBobZ_Annapolis: Is nova-libvirt running and showing any error?14:31
BobZ_AnnapolisI didn't check /etc/hosts before shutting the machine down. upon restart, since it is an all-in-one, there are a couple entries for local host but they are all the same and 1 of the aliases matches 'hostname'14:34
BobZ_Annapolislibvirt logs say : irNetSASLSessionServerStep:603 : authentication failed: Failed to start SASL negotiation: -20 (SASL(-13): user not found: unable to canonify user and get auxprops)14:35
yoctozeptoBobZ_Annapolis: fwiw, you can check the timestamp on /etc/hosts if it does not correlate with the machine reboot ;-)14:35
BobZ_Annapolisdate -r /etc/hosts indicates it was modified today :-(14:39
BobZ_Annapolisi stood up the aio yesterday to show off how easy-quick it was, was able to create several vms, etc - wanted to show it off again today and this started happening before the demo :-(14:41
yoctozeptoso AWS setup mangled the hosts file14:42
yoctozeptoyou can try the following14:42
yoctozeptorerun the k-a bootstrap command14:42
yoctozeptoand then restart nova-compute and nova-libvirt manually14:42
opendevreviewRafal Lewandowski proposed openstack/kayobe master: Add new DIB upper constraints variables  https://review.opendev.org/c/openstack/kayobe/+/85590914:48
BobZ_Annapolistrying those steps now...since it is so quick to do, instead of bugging everyone all day, while debugging this machine, i'll go repeat the process on a brand new machine and when that is running successfully, start looking for discrepancies - thanks for the suggestions folks14:53
yoctozeptolet us know with all the details14:55
yoctozeptoit might be bugging others as well14:55
BobZ_Annapolisok, will do - fyi, ran those commands on the old system and no joy - nova-compute & libvirt still logging the same issues :-( ok, thanks again - i'm off to stand up another aio15:02
yoctozeptoack, maybe something else got messed as well15:28
spateldid you guys see high cpu utilization on neutron-ovn-metadata-agent in yoga release? (Logs are clean)15:48
opendevreviewPierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax  https://review.opendev.org/c/openstack/kayobe/+/85553815:54
opendevreviewPierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax  https://review.opendev.org/c/openstack/kayobe/+/85553815:54
opendevreviewPierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax  https://review.opendev.org/c/openstack/kayobe/+/85553815:55
opendevreviewPierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax  https://review.opendev.org/c/openstack/kayobe/+/85553815:57
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [CI] Run Kolla Ansible from its own venv  https://review.opendev.org/c/openstack/kolla-ansible/+/85644315:59
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [CI] Run Kolla Ansible from its own venv  https://review.opendev.org/c/openstack/kolla-ansible/+/85644316:01
opendevreviewMerged openstack/kayobe stable/wallaby: Ubuntu systemd-networkd: VLAN ifname heuristics  https://review.opendev.org/c/openstack/kayobe/+/85555416:06
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [CI] Run Kolla Ansible from its own venv  https://review.opendev.org/c/openstack/kolla-ansible/+/85644316:28
BobZ_Annapoliswhelp. . .as far as getting the kolla-ansible all-in-one yoga code up and running in AWS goes - it does appear that a machine reboot modified the /etc/hosts file in such a way so as to prevent the containers/services from communicating successfully again.  I built out a new one and looked at the working/good /etc/hosts file - when i modified my non-working one to mimic the format in the working one and did a k-a16:35
BobZ_AnnapolisSo thank you for addressing the all-in-one issue - i'm not sure if that applies to every AWS attempt or how to document that for anyone else though - is this known ? If not, is there a Troubleshooting FAQ that it could be added to ? 16:36
BobZ_AnnapolisNow that that is corrected - i'm going to keep that in mind and stand up a multinode - thanks a lot !!!16:37
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Make Cinder with iSCSI use fewer volumes  https://review.opendev.org/c/openstack/kolla-ansible/+/79721521:21
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Make Cinder with iSCSI use fewer volumes  https://review.opendev.org/c/openstack/kolla-ansible/+/79721521:22

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!