| mnasiadka | morning | 07:20 |
|---|---|---|
| rockey | mornings | 07:20 |
| frickler | yoctozepto: reading https://review.opendev.org/c/openstack/large-scale/+/855508 I wonder whether we should adopt that option in k-a by default, wdyt? | 07:44 |
| mnasiadka | yoctozepto: cleaned up quay.io a bit (those images without binary and source in the name that we pushed one time 13th of May) | 08:15 |
| hrw | yoctozepto: pip==22.* setuptools==65.* to both pip==latest patches | 08:27 |
| hrw | I prefer stable branches to behave stable rather to fail due to external changes | 08:27 |
| yoctozepto | frickler: replied, it's the default so we are using it | 08:57 |
| yoctozepto | mnasiadka: awesome, thanks! | 08:57 |
| mnasiadka | yoctozepto: after we start publishing rocky linux - I'll clean up master centos8 and focal tags in the new image scheme | 08:58 |
| yoctozepto | hrw: hmm, makes sense, will amend | 08:58 |
| yoctozepto | mnasiadka: +++ | 08:58 |
| mnasiadka | This is probably what I forgot to touch on the weekly meeting | 08:58 |
| mnasiadka | we're not going to publish centos 9s images, right? | 08:58 |
| yoctozepto | we need them for CI so we will | 08:58 |
| yoctozepto | we can document them differently from others though | 08:59 |
| hrw | yoctozepto: I remember moments when we had to workaround setuptools change. IIRC we have it done twice even. | 08:59 |
| yoctozepto | we also don't know whether rocky images will be feasible or if it is better to stay with stream in images and rocky on hosts (which sounds reasonable enough to me) | 08:59 |
| yoctozepto | hrw: yeah, I agree it is good to do it here as well | 09:00 |
| yoctozepto | I will be backporting that so it makes double sense | 09:00 |
| yoctozepto | (or quadruple sense, aye?) | 09:00 |
| hrw | it is nice to have latest and greatest but who will remember to set them in stone during release cycle? | 09:01 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: veryWIP: Add Opensearch support https://review.opendev.org/c/openstack/kolla-ansible/+/856610 | 09:02 |
| mnasiadka | yoctozepto: what do you mean by feasible? | 09:08 |
| yoctozepto | mnasiadka: are not we missing SIGs' work in Rocky still? | 09:10 |
| mnasiadka | yoctozepto: Rocky (just like other clones) are using CentOS Stream SIG package repos, the centos-release-* packages are in Extras repo in RL8/9 | 09:10 |
| mnasiadka | As long as it works fine, they are not going to put additional work into just rebuilding this. | 09:11 |
| mnasiadka | Which is probably fine | 09:11 |
| mnasiadka | And they would rather need to rebuild what is in SRPMs in RHEL, but that's another discussion. | 09:12 |
| mnasiadka | but that's next step, first we need the host os c9s/rl9 patch merged | 09:12 |
| hrw | mnasiadka: el9 clones are rhel srpm rebuilds ;D | 09:13 |
| mnasiadka | hrw: just saying that Alma and others are using CentOS Stream SIG packages :D | 09:13 |
| mnasiadka | anyway, let's not try to save the world in one day | 09:14 |
| hrw | mnasiadka: ;d | 09:14 |
| mnasiadka | I think we should only expose the rl9 builds, and the c9s ones should be only to test what breakage will come soon to rl9 (just as we agreed on PTG) | 09:15 |
| mnasiadka | so, do we want to publish images for a single kolla-ansible CI job? | 09:15 |
| mnasiadka | (which will be non voting) | 09:15 |
| mnasiadka | that's the question for next weekly meeting :) | 09:15 |
| mnasiadka | In the meantime I need another pair of eyes on https://review.opendev.org/c/openstack/kolla-ansible/+/841916 - is that change of default so daunting, that we need to persist the not really sane default? yoctozepto can you have a look? | 09:18 |
| yoctozepto | what default | 09:19 |
| * yoctozepto looking | 09:19 | |
| mnasiadka | prometheus-alertmanager web external url is pointing to internal API address :D | 09:20 |
| mnasiadka | so if you send alerts via webhook to Slack or something else - the url that you can click on to get alert details - points to internal API url | 09:20 |
| mnasiadka | So either we change the default, which makes sense to me at least, and allow users to configure it, or we persist that default, but still allow users to configure it ;) | 09:21 |
| yoctozepto | but this url works already, right? | 09:22 |
| yoctozepto | it's just the advertisement that is broken? | 09:23 |
| yoctozepto | ok, confirmed it works | 09:24 |
| mnasiadka | yes, the external url already works, just advertisement is broken (users should be directed to public api url, not internal one) | 09:24 |
| yoctozepto | +2 | 09:24 |
| yoctozepto | yeah, it's a simple fix | 09:24 |
| mnasiadka | Well, there's some disagreement in this review that it will hugely impact users. | 09:25 |
| mnasiadka | just wanted another pair of eyes | 09:25 |
| mnasiadka | And really surprised nobody has been complaining over this in the past | 09:25 |
| frickler | yoctozepto: ah, good that we have so many standa^Wdefaults to choose from ;) | 09:27 |
| yoctozepto | frickler: yeah, I love it too | 09:30 |
| yoctozepto | mnasiadka: "hugely impact"? | 09:31 |
| yoctozepto | maybe it has been misunderstood | 09:31 |
| mnasiadka | yoctozepto: check earlier comments in this patch ;) | 09:31 |
| yoctozepto | if you want to improve, you can go explicit in the reno on how to restore the previous (wrong) behaviour | 09:31 |
| yoctozepto | ah, read them | 09:32 |
| yoctozepto | I think dougsz misinterpreted the role of this option | 09:32 |
| yoctozepto | though I hope you verified it and it works like you described, not like he describes it | 09:32 |
| mnasiadka | yeah, parallax verified that, but let me check once again | 09:34 |
| mnasiadka | alertmanager docs and code skim tells me it's ONLY used in HTTP POST to webhooks | 09:37 |
| yoctozepto | that would make sense | 09:38 |
| opendevreview | Rafal Lewandowski proposed openstack/kayobe master: added new dib upper constraints variables https://review.opendev.org/c/openstack/kayobe/+/855909 | 09:51 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Fix AlertManager's external web url https://review.opendev.org/c/openstack/kolla-ansible/+/841916 | 10:05 |
| mnasiadka | yoctozepto: https://review.opendev.org/c/openstack/kolla-ansible/+/841916 can you reapply +2 and let's end this drama? | 10:08 |
| opendevreview | Rafal Lewandowski proposed openstack/kayobe master: added new dib upper constraints variables https://review.opendev.org/c/openstack/kayobe/+/855909 | 10:16 |
| kevko | INFO:kolla.common.utils.neutron-base: ---> Running in f07cb9417abb | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base:Processing /neutron | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base: Preparing metadata (setup.py): started | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base: Preparing metadata (setup.py): finished with status 'done' | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base:[91mERROR: Cannot install neutron 20.2.1.dev32 (from /neutron) because these package versions have conflicting dependencies. | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base:[0m | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base:The conflict is caused by: | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base: The user requested neutron 20.2.1.dev32 (from /neutron) | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base: The user requested (constraint) neutron===20.2.0 | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base:To fix this you could try to: | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base:1. loosen the range of package versions you've specified | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base:2. remove package versions to allow pip attempt to solve the dependency conflict | 10:26 |
| kevko | INFO:kolla.common.utils.neutron-base:[91mERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts | 10:27 |
| kevko | INFO:kolla.common.utils.neutron-base:[0m | 10:27 |
| kevko | INFO:kolla.common.utils.neutron-base:Removing intermediate container f07cb9417abb | 10:27 |
| kevko | ERROR:kolla.common.utils.neutron-base:Error'd with the following message | 10:27 |
| kevko | kolla yoga images build failing locally because of global requirements right ? | 10:27 |
| kevko | should I propose patch for global requirements and update upper-constraints ? | 10:27 |
| kevko | neutron 20.2.0 -> 20.2.1 ... or what ? | 10:27 |
| mnasiadka | I think we've seen that before somewhere else, should we remove neutron from upper-constraints.txt? | 10:33 |
| frickler | seems like a bug, yes. likely we don't see it in CI because we include latest tagged version there instead of latest stable? which would be a debatable option, too | 10:38 |
| kevko | hmm, upper-constraints includes neutron 20.2.0 as latest yoga release, but kolla is installing 20.2.1.dev32 ...last tag in git is 20.2.0 | 10:41 |
| frickler | actually we do install latest stable/yoga neutron here https://1af994232f33a248864f-7c57593ce71ac5e536068fa74069f258.ssl.cf1.rackcdn.com/855565/1/check/kolla-build-debian-source/4845005/kolla/build/neutron-base.log | 10:42 |
| frickler | so something else must be different in your local build | 10:42 |
| kevko | hmmmm | 10:43 |
| kevko | frickler: how old is log above ? | 10:43 |
| frickler | timestamp is at the top | 10:43 |
| kevko | hmm, that's weird | 10:47 |
| kevko | frickler: can I start build manually somehow ? because when I download tarball for neutron ..there is 20.2.1dev3 | 10:48 |
| kevko | i mean remotely .. | 10:49 |
| kevko | and i think it has to fail also in upstream | 10:49 |
| frickler | kevko: https://opendev.org/openstack/kolla/src/branch/stable/yoga/docker/openstack-base/Dockerfile.j2#L326 | 10:53 |
| frickler | are you sure you are using kolla-build from stable/yoga and not from master? | 10:54 |
| frickler | but that also explains why mnasiadka has seen this https://review.opendev.org/c/openstack/kolla/+/847793 | 10:55 |
| kevko | frickler: hmm, it looks like this is needed to backport .. | 11:00 |
| kevko | let me check | 11:00 |
| kevko | but i still don't understand why neutron tarball is 20.2.1dev32 but not released ! | 11:01 |
| kevko | no tag in git ..nothing | 11:01 |
| yoctozepto | mnasiadka, dougszu: what is the reasoning in that "internal discussion"? | 11:01 |
| frickler | kevko: because it is not released. it is the current head of the stable branch, which is tarred up as a convenience for consumers like kolla | 11:02 |
| kevko | frickler: check below | 11:05 |
| kevko | michalarbet@pixla:~/ultimum/git/upstream/neutron$ git tag | grep '20\.2\.[0-9]' | 11:05 |
| kevko | 20.2.0 | 11:05 |
| kevko | michalarbet@pixla:~/ultimum/git/upstream/neutron$ grep -Ri '20\.2\.1' | 11:05 |
| kevko | michalarbet@pixla:~/ultimum/git/upstream/neutron$ grep -Ri '20\.2\.1' . | 11:05 |
| yoctozepto | mnasiadka, dougszu: please reply on the patch, thanks | 11:05 |
| kevko | frickler: hmm, ok .. now it's clear ..i just thought that it is adding "devXY" suffix to latest tag in git | 11:09 |
| hrw | kevko: INFO:kolla.common.utils.neutron-base:Successfully tagged yoga/debian-source-neutron-base:debian | 11:12 |
| kevko | hrw: yeah,it looks like i should rebase :P | 11:13 |
| hrw | kevko: INFO:kolla.common.utils.neutron-base:Successfully installed neutron-20.2.1.dev32 neutron-lib-2.20.0 os-ken-2.3.1 os-resource-classes-1.1.0 os-vif-2.7.1 ovsdbapp-1.15.2 | 11:13 |
| hrw | kevko: like frickler said - we fetch source tarball, not latest release | 11:14 |
| hrw | kevko: that's why 'dev' in version | 11:14 |
| kevko | hrw: yes i know, but i thought that python3 setup.py bdist_wheel will build version from LATEST_TAG.devXY version | 11:14 |
| kevko | but it looks like it is bumping last number in version and adding devXY suffix also | 11:15 |
| kevko | and i thought it is not bumbing | 11:15 |
| hrw | kevko: and latest tag is 20.2.1? | 11:16 |
| kevko | no, it is 20.2.0 | 11:16 |
| frickler | that is because dev releases are sorted to come before the tagged release https://peps.python.org/pep-0440/#developmental-releases | 11:19 |
| hrw | I never understood how that versioning is done tbh | 11:19 |
| kevko | hrw: but now it makes sense (python3 setup.py bdist_wheel will build package from TAG+1+.devXY) ...so for example for neutron if latest tag is 20.2.0 and top of the git is not tagged ..it will be 20.2.1.devXY ..so requirements can't work | 11:19 |
| hrw | kevko: ok, where you got that 20.2.0 u-c from? | 11:20 |
| kevko | hmm, and is it safe globally ? i mean ..shouldn't we install sources from TAGs ? (it sounds little bit as step backwards ..but just wondering) | 11:21 |
| kevko | hrw: upper-constraints classic kolla | 11:21 |
| kevko | now when i am rebased ..it is working ..because we removed from upper-constraints | 11:22 |
| frickler | hrw: https://review.opendev.org/c/openstack/requirements/+/846785 | 11:22 |
| frickler | I think that would really better be some <21.0 cap | 11:22 |
| hrw | ah. did not got it with stable/yoga kolla | 11:26 |
| hrw | kevko: it was discussed in past. we moved to stable/* HEAD to get all fixes | 11:28 |
| hrw | kevko: we used TAG releases in past. | 11:28 |
| kevko | hrw: ok, i also agree that stable/* is better than tag | 11:29 |
| kevko | frickler: or take upper-constraints and replace last number in version for tarballs which are downloaded | 11:31 |
| kevko | now we rely on the fact that the tarball will always be correct and noone will ever make a mistake in tarball release | 11:32 |
| kevko | now we just removed neutron entry from upper constraints :D | 11:33 |
| frickler | which is the same situation as for every other service, no big deal | 11:34 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe master: Move to CentOS Stream 9 / Rocky Linux 9 https://review.opendev.org/c/openstack/kayobe/+/855656 | 12:00 |
| dougszu | hello, yoctozepto, frickler has kindly summarised | 12:02 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe master: Move to CentOS Stream 9 / Rocky Linux 9 https://review.opendev.org/c/openstack/kayobe/+/855656 | 12:06 |
| mnasiadka | yoctozepto: sorry, was busy, do I still need to do something? :) | 12:08 |
| dougszu | W+1 :D | 12:09 |
| mnasiadka | yoctozepto: we basically come to a conclusion to not change the default, because it might break existing deployments on upgrade, and it's only me and parallax complaining - nobody else have raised a bug :) | 12:09 |
| mnasiadka | +w it is | 12:09 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax https://review.opendev.org/c/openstack/kayobe/+/855538 | 13:12 |
| yoctozepto | mnasiadka: I would suggest to backport this but set a different default for zed | 13:12 |
| mnasiadka | yoctozepto: I would do that too, but dougszu seems to have different opinion - my knowledge about prometheus ends on deploying it and clicking in alertmanager :) | 13:13 |
| yoctozepto | ack | 13:14 |
| yoctozepto | I am not pushing, I am not using this functionality | 13:14 |
| mnasiadka | yoctozepto: do you plan on updating the tenks change? we need to get it merged ASAP I think | 13:14 |
| opendevreview | Merged openstack/kolla-ansible master: Fix AlertManager's external web url https://review.opendev.org/c/openstack/kolla-ansible/+/841916 | 13:17 |
| yoctozepto | yoctozepto: yes, soon, do not treat me with "ASAP" ;-) | 13:20 |
| hrw | mnasiadka: commented on your k-a opensearch patch. | 13:22 |
| mnasiadka | replying to yourself? good one :) | 13:22 |
| mnasiadka | hrw: sure, working on it now, but thanks | 13:23 |
| dougszu | yoctozepto: what would be your reason for changing the default? | 13:31 |
| yoctozepto | dougszu: as the public interface is already on there, I think it makes more sense to advertise that one to the external systems | 13:33 |
| yoctozepto | mnasiadka: mobile client | 13:33 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax https://review.opendev.org/c/openstack/kayobe/+/855538 | 13:39 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: veryWIP: Add Opensearch support https://review.opendev.org/c/openstack/kolla-ansible/+/856610 | 13:42 |
| rockey | lulz; veryWIP:: might be a new status for sure | 13:43 |
| dougszu | yoctozepto: One downside is that you have to deal with the static password from HAProxy services for the public interface | 13:43 |
| mnasiadka | rockey: it just indicates it's a moving target for a couple of days ;) | 13:44 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Add Opensearch image(s) https://review.opendev.org/c/openstack/kolla/+/830373 | 13:46 |
| rockey | mnasiadka: "couple of days", planning on working on the weekend i see ;) | 13:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Add Opensearch image(s) https://review.opendev.org/c/openstack/kolla/+/830373 | 13:47 |
| mnasiadka | rockey: surely not over the weekend ;) | 13:47 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe master: Move to CentOS Stream 9 / Rocky Linux 9 https://review.opendev.org/c/openstack/kayobe/+/855656 | 14:08 |
| BobZ_Annapolis | using the online kolla-ansible aio - works fine the 1st time thru - after the machine is rebooted and everything restarts, nova-compute never comes back up - crash-restarting w/various errors, eg nova.exception.HypervisorUnavailable: Connection to the hypervisor is broken on host - seen before / suggestions ? thx | 14:20 |
| yoctozepto | BobZ_Annapolis: any chance your hostname has been reset? | 14:24 |
| yoctozepto | or /etc/hosts ? some clouds may garble it between restarts | 14:29 |
| yoctozepto | I *think* kolla-ansible tries to disable this behaviour of cloud-init | 14:29 |
| priteau | BobZ_Annapolis: Is nova-libvirt running and showing any error? | 14:31 |
| BobZ_Annapolis | I didn't check /etc/hosts before shutting the machine down. upon restart, since it is an all-in-one, there are a couple entries for local host but they are all the same and 1 of the aliases matches 'hostname' | 14:34 |
| BobZ_Annapolis | libvirt logs say : irNetSASLSessionServerStep:603 : authentication failed: Failed to start SASL negotiation: -20 (SASL(-13): user not found: unable to canonify user and get auxprops) | 14:35 |
| yoctozepto | BobZ_Annapolis: fwiw, you can check the timestamp on /etc/hosts if it does not correlate with the machine reboot ;-) | 14:35 |
| BobZ_Annapolis | date -r /etc/hosts indicates it was modified today :-( | 14:39 |
| BobZ_Annapolis | i stood up the aio yesterday to show off how easy-quick it was, was able to create several vms, etc - wanted to show it off again today and this started happening before the demo :-( | 14:41 |
| yoctozepto | so AWS setup mangled the hosts file | 14:42 |
| yoctozepto | you can try the following | 14:42 |
| yoctozepto | rerun the k-a bootstrap command | 14:42 |
| yoctozepto | and then restart nova-compute and nova-libvirt manually | 14:42 |
| opendevreview | Rafal Lewandowski proposed openstack/kayobe master: Add new DIB upper constraints variables https://review.opendev.org/c/openstack/kayobe/+/855909 | 14:48 |
| BobZ_Annapolis | trying those steps now...since it is so quick to do, instead of bugging everyone all day, while debugging this machine, i'll go repeat the process on a brand new machine and when that is running successfully, start looking for discrepancies - thanks for the suggestions folks | 14:53 |
| yoctozepto | let us know with all the details | 14:55 |
| yoctozepto | it might be bugging others as well | 14:55 |
| BobZ_Annapolis | ok, will do - fyi, ran those commands on the old system and no joy - nova-compute & libvirt still logging the same issues :-( ok, thanks again - i'm off to stand up another aio | 15:02 |
| yoctozepto | ack, maybe something else got messed as well | 15:28 |
| spatel | did you guys see high cpu utilization on neutron-ovn-metadata-agent in yoga release? (Logs are clean) | 15:48 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax https://review.opendev.org/c/openstack/kayobe/+/855538 | 15:54 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax https://review.opendev.org/c/openstack/kayobe/+/855538 | 15:54 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax https://review.opendev.org/c/openstack/kayobe/+/855538 | 15:55 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Support configuring VLANs with systemd-networkd syntax https://review.opendev.org/c/openstack/kayobe/+/855538 | 15:57 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Run Kolla Ansible from its own venv https://review.opendev.org/c/openstack/kolla-ansible/+/856443 | 15:59 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Run Kolla Ansible from its own venv https://review.opendev.org/c/openstack/kolla-ansible/+/856443 | 16:01 |
| opendevreview | Merged openstack/kayobe stable/wallaby: Ubuntu systemd-networkd: VLAN ifname heuristics https://review.opendev.org/c/openstack/kayobe/+/855554 | 16:06 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Run Kolla Ansible from its own venv https://review.opendev.org/c/openstack/kolla-ansible/+/856443 | 16:28 |
| BobZ_Annapolis | whelp. . .as far as getting the kolla-ansible all-in-one yoga code up and running in AWS goes - it does appear that a machine reboot modified the /etc/hosts file in such a way so as to prevent the containers/services from communicating successfully again. I built out a new one and looked at the working/good /etc/hosts file - when i modified my non-working one to mimic the format in the working one and did a k-a | 16:35 |
| BobZ_Annapolis | So thank you for addressing the all-in-one issue - i'm not sure if that applies to every AWS attempt or how to document that for anyone else though - is this known ? If not, is there a Troubleshooting FAQ that it could be added to ? | 16:36 |
| BobZ_Annapolis | Now that that is corrected - i'm going to keep that in mind and stand up a multinode - thanks a lot !!! | 16:37 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Make Cinder with iSCSI use fewer volumes https://review.opendev.org/c/openstack/kolla-ansible/+/797215 | 21:21 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Make Cinder with iSCSI use fewer volumes https://review.opendev.org/c/openstack/kolla-ansible/+/797215 | 21:22 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!