Wednesday, 2023-09-06

mnasiadkaspatel: we're not doing the host part, only the definition of networking in Neutron, see https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/octavia/tasks/prepare.yml03:32
spatelmnasiadka hmm03:33
spatelmnasiadka In my case what I should do ? 03:33
spatelmnasiadka I have posed question in mailing list - https://paste.opendev.org/show/bUpwFvtZyPC4zZNONBRV/03:34
mnasiadkaspatel: configure the network interfaces on the controllers and let Kolla configure the networking in Neutron?03:35
spatelI have assigned dedicated VLAN 41 for LBaaS management traffic 03:35
mnasiadkaspatel: is bond0 part of external bridge in OVS?03:36
spatelYes bond0 is my external neutron interface 03:36
spatelall my VMs using bond0 03:36
mnasiadkaso you need an ip address on bond0.41 on the controllers to be able to reach amphora VMs (so in the same subnet)03:37
spatelI did assigned IP address on bond0.41 - 192.168.41.1 03:38
spatelI have seeing strange issue, when I remove bond0.41 interface then VM get IP from DHCP and VM can ping DHCP namespace IP (ip netns) 03:39
spatelif I create bond0.41 interface then VM stopped pinging DHCP name space 03:39
mnasiadkaah, right - then you need to have connectivity to vlan 41 on some other interface - or do what kayobe is doing (create a bridge on bond0, create veth pair there, and use that veth pair to plug into OVS, and move ip address to brbond0.41)03:40
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: Prevent libvirtd reload when only generating config  https://review.opendev.org/c/openstack/kolla-ansible/+/89376303:41
spatelhttps://paste.opendev.org/show/bdpDU937A3HHKqUzHA6H/ 03:41
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/zed: Prevent libvirtd reload when only generating config  https://review.opendev.org/c/openstack/kolla-ansible/+/89376403:41
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/yoga: Prevent libvirtd reload when only generating config  https://review.opendev.org/c/openstack/kolla-ansible/+/89376503:41
spatelmnasiadka yes.. I think you are correct, I can't just create bond0.41 on bond0 link. I need to create bridge and attach it to OVS or something03:43
spatelDo you have any short of doc or clue how to do that?03:43
spatelHow are we doing that magic with o-hm0 stuff?03:44
spatelI have tired this but it also doesn't work - ovs-vsctl add-port br-int br-lb0 -- set Interface br-lb0 type=internal03:46
spateljust attach port to OVS br-int and assign IP address to br-lb0 but it doesn't work and not able to ping dhcp namespace03:47
mnasiadkao-hm0 uses an OVS port in vxlan network AFAIR, I don't know how ,,production problematic'' is the systemd unit running dhclient03:48
spatelmnasiadka I can live with o-hm0 with tenant network to make it work but not sure its recommended way to use in production or not 03:49
spatelcurrently in my lab I am using o-hm0 with tenant network and life is good. 03:49
spatelNow I want to deploy in production so thought use best practice 03:50
spatelI found this doc - http://www.panticz.de/octavia/create-ovs-port 03:50
mnasiadkaas I said, we don't use it in production, if something wrong happens with that dhclient and ovs port tandem, you need to monitor it closely ;)03:50
spatelLook like every doc using o-hm0 for octavia deployment :(03:50
spatelagreed, I don't want to use o-hm0 tenant way to run octavia in production. 03:52
spatelJust trying to understand how do I attach octavia interface to OVS 03:52
spatelNot able to find any good doc or example 03:52
spatelAt present everything is working, vm getting IP from DHCP and my dhcp namespace can ping VM. (all I need interface on controller node to attach to OVS and that is where I am stuck)03:53
mnasiadkaspatel: well, your bridge is already attached there, it's the ip address on the host level that is making things difficult - so the only good solution here is to create a bridge (brbond0), create a veth pair and use the veth interface as neutron_external_interface03:54
mnasiadkaand then you can set the ip address for octavia health manager on brbond0.4103:54
mnasiadkaspatel: maybe our CI role for doing that can help you - https://opendev.org/openstack/kolla-ansible/commit/337771143ba7c392665a14e8aa2434817d9e947303:55
spatelI already have neutron_external_interface: br-vlan  in that case how do i add second interface to config?  are you suggesting neutron_external_interface: br-vlan, brbond0  ?03:56
mnasiadkaso, br-vlan has bond0 as a member?04:06
spatelyes04:07
spatelbond0 is my interface and br-vlan is bridge on it 04:07
spatelI think I figured out.. let me try and get back to you04:14
spatelmnasiadka found the solution :) and it works 04:25
spatelmnasiadka This is what I did and it works - https://paste.opendev.org/show/bRDwUHcHImnMUn9glOCt/04:29
spatelI wish we can add some kind of reference doc to kolla-ansible document to help our new folks to get some idea how internal wiring works with octavia 04:31
mnasiadkaspatel: feel free to contribute :)04:31
spatelSure but its ok to add this snippet in kolla-ansible octavia page?04:32
spatelHow would you like to handle it?04:32
spatelWish we can add this in ansible playbook and set variable in global.yml to execute it 04:34
mnasiadkaadd it somewhere in https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/reference/networking/octavia.rst for starters04:34
mnasiadkakolla-ansible never configured host os networking, so I think a guide is enough - that's for kayobe to handle04:35
spatelOK.. let me add snippet here and later we should cook it inside ansible playbook 04:35
spatelkayobe itself is different beast :) 04:36
mnasiadkaspatel: you can add a playbook under contrib/ directory as well04:37
spatelmnasiadka sounds good! let me first get it work right way in my production and then I can add in official doc. 04:39
spatelI would see if I can create systemd service to create all these interface and wire them with proper bridge 04:40
mnasiadkaspatel: it probably would be better to do it in NetworkManager/network-scripts in EL or networkd/whatever else in Debian world04:42
spatelHmm! I am using netplan for networking so may be that is other way to handle it 04:43
spatelmnasiadka question, what is the use of octavia_network_interface: ? 04:49
spatelwhat will happen if I don't specify this variable ?04:51
spatelnevermind, got it.. it use to setup controller_ip_port_list: 04:55
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add comprehensive checks for container restarts  https://review.opendev.org/c/openstack/kolla/+/89287805:33
spatelmnasiadka hey what is octavia-driver-agent container for?05:44
spatelI haven't seen it before05:44
mnasiadkaspatel: for other drivers than amphora (e.g. ovn)05:47
spateloh so for OVN we need that container correct?05:47
mnasiadkayes05:48
spatelcool05:48
opendevreviewMerged openstack/kolla master: Document KOLLA_UPGRADE_CHECK environment variable  https://review.opendev.org/c/openstack/kolla/+/89348405:49
opendevreviewMichal Nasiadka proposed openstack/kolla stable/xena: Add comprehensive checks for container restarts  https://review.opendev.org/c/openstack/kolla/+/89376607:07
opendevreviewMerged openstack/kolla master: Improve kolla-toolbox offline support.  https://review.opendev.org/c/openstack/kolla/+/88436807:21
opendevreviewNatz cheng proposed openstack/kolla-ansible master: Updated links in documentation to be safe links  https://review.opendev.org/c/openstack/kolla-ansible/+/89383607:51
opendevreviewNatz cheng proposed openstack/kolla master: Updated links in documentation to be safe links  https://review.opendev.org/c/openstack/kolla/+/89383807:58
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/xena: CI: completely drop upgrade jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/89379608:12
mmalchukmnasiadka is it ok now? completely drop upgrade jobs?08:13
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/xena: Use better default bind address for ironic-tftp  https://review.opendev.org/c/openstack/kolla-ansible/+/89342208:16
mnasiadkammalchuk: commented08:18
opendevreviewMerged openstack/kolla stable/2023.1: Add comprehensive checks for container restarts  https://review.opendev.org/c/openstack/kolla/+/89287608:19
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something  https://review.opendev.org/c/openstack/kolla-ansible/+/89360908:19
opendevreviewNatz cheng proposed openstack/kolla master: Use py3 as the default runtime for tox  https://review.opendev.org/c/openstack/kolla/+/89384008:26
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/xena: CI: Completely drop upgrade jobs from EOL Wallaby  https://review.opendev.org/c/openstack/kolla-ansible/+/89379608:28
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/xena: Use better default bind address for ironic-tftp  https://review.opendev.org/c/openstack/kolla-ansible/+/89342208:28
mmalchukmnasiadka done08:28
mmalchukmnasiadka could you also review gnocchi, its back : https://review.opendev.org/q/I3ca4e10508c26b752412789502ceb917ecb4dbeb08:34
opendevreviewMerged openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration  https://review.opendev.org/c/openstack/kolla-ansible/+/89317208:43
kevkomnasiadka: i found the issue with aarch64 podman 08:44
mnasiadkakevko: fantastic, how bad is it?08:45
mmalchukgreat work08:51
mmalchuksay it08:52
kevkowell, i am not sure why volumes are created under root:root08:55
kevkowhile on debian nonaarch64 it is fluentd:kolla08:55
kevko*with root:root perms08:56
opendevreviewMerged openstack/kolla master: openstack-base: add block for overriding global upper constraints  https://review.opendev.org/c/openstack/kolla/+/89360808:59
kevkoi also don't understand why volume is fluentd:kolla if user in fluentd is td-agent 09:01
kevkookay, now i got it ..i think we need to specify permissions for volume 09:08
kevkolet me try something 09:26
opendevreviewMaksim Malchuk proposed openstack/kolla stable/2023.1: Improve kolla-toolbox offline support.  https://review.opendev.org/c/openstack/kolla/+/89384909:26
opendevreviewMaksim Malchuk proposed openstack/kolla stable/zed: Improve kolla-toolbox offline support.  https://review.opendev.org/c/openstack/kolla/+/89385109:31
opendevreviewMaksim Malchuk proposed openstack/kolla stable/yoga: Improve kolla-toolbox offline support.  https://review.opendev.org/c/openstack/kolla/+/89385209:31
Fl1ntHi guys09:33
Fl1ntso, we do have an issue with this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/87827009:33
Fl1ntI've someone that keeps asking for the same question each round09:33
Fl1ntI do suspect this reviewer isn't using designate09:34
Fl1ntso how could we prevent such infinite back and forth on reviews?09:34
mnasiadkaPlease discuss that on the meeting, it's the venue for this.09:34
Fl1ntCan't make it at meeting times09:34
opendevreviewMaksim Malchuk proposed openstack/kolla stable/xena: Improve kolla-toolbox offline support.  https://review.opendev.org/c/openstack/kolla/+/89385409:35
Fl1ntI'm already having a meeting at community meeting time.09:35
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something  https://review.opendev.org/c/openstack/kolla-ansible/+/89360909:36
Fl1ntTBN: without this patch, the one that was merged: https://review.opendev.org/c/openstack/kolla-ansible/+/878063 will not work properly.09:36
mnasiadkaFl1nt: fine by me, do the discussion in Gerrit then09:39
mnasiadkaFl1nt: basically reviewers are pointing to the fact, it's not backwards compatible - and we can't break existing deployments in happy camper style.09:40
Fl1ntHow is there so many break from wallaby to xena that happened and then we can't fix a role that was improperly implemented? Designate simply just doesn't work in its current form.09:41
Fl1ntany upgrade of kolla do have breaking changes by nature.09:41
fricklerdesignate works perfectly fine in my deployment09:42
Fl1ntfrickler, how do you use it? Do you use it as DNSaaS with Openstack services native integration? Do you use it as DNSaaS solution only as public DNS provider? Do you use it with external delegation? Or are you just using it to publish FIP FQDN? 09:45
Fl1ntFirst of all, FIP FQDN can't work out of the box as neutron do not use the proper credentials to talk to designate API, so you can't get FIP FQDN to create automatically within designate without sink and notification topics, same things for automatic PTR creation and deletion, when trying to delete the FIP neutron just simply can't instruct designate to delete the recordset.09:47
Fl1ntThen by default the pools do use external IPs to instruct mDNS to update the DNS server which should not happens out of externally managed zones.09:48
Fl1ntetc09:48
Fl1ntout of the simplest DNSaaS scenario nothing works out of the box.09:49
fricklerwell it all works for me including records for FIPs09:49
Fl1ntso you're the only one not affected by the reported bugs.09:50
Fl1ntit can't work as kolla do not implement the role the way the upstream project intend it to be implemented.09:50
fricklerI'm also involved in designate upstream, so I doubt that to be true, either09:51
Fl1ntok fine, so everyone reporting bugs with kolla implementation is having it wrong right?09:52
fricklerall the bugs you reference are either incomplete or resolved according to what I see09:52
Fl1ntno they're not, this one is from the mDNS being improperly set on default kolla pool: https://bugs.launchpad.net/kolla-ansible/+bug/1773972 for instance. This one https://bugs.launchpad.net/kolla-ansible/+bug/1879557 is marked as fixed because I did fixed it but the fix can't work without the bigger patch. This one can't work as it needs multiple pools support on kolla that we currently don't have: 09:57
Fl1nthttps://bugs.launchpad.net/kolla-ansible/+bug/2008691. https://bugs.launchpad.net/kolla-ansible/+bug/2012292 this one is due to the way kolla improperly manage the pools.yaml file structure expected by designate-manage. etc 09:57
fricklerdid you even look at what I wrote on those bugs earlier?09:58
Fl1ntyes, why?09:58
fricklerbecause without logs, how do you know what is actually wrong in https://bugs.launchpad.net/kolla-ansible/+bug/1773972 for example?09:59
fricklerwhy do you keep denying that overriding pools.yaml with your own file isn't a valid, existing, implemented solution?09:59
Fl1ntBecause I've the exact same issue! I did create the patchset because I hitted every single listed bug report issue, I did deployed the role using plain fresh and vanilla configuration on a multinode deployment and it just don't work.10:00
Fl1ntfrickler, it doesn't work10:01
frickleryou may have an issue that looks the same, doesn't have to be identical. please create a bug reports with the relevant logs and configs, and we can discuss what is going wrong there10:01
Fl1ntI'll just abandon the patchset...10:03
opendevreviewGaĆ«l THEROND proposed openstack/kolla-ansible master: Revert "Fix improper designate-manage command usage."  https://review.opendev.org/c/openstack/kolla-ansible/+/89376910:07
opendevreviewMerged openstack/kolla-ansible master: ovn: Improve clustering  https://review.opendev.org/c/openstack/kolla-ansible/+/86892910:08
opendevreviewMerged openstack/kolla-ansible stable/yoga: Add documentation for migrating from CS8 to RL9  https://review.opendev.org/c/openstack/kolla-ansible/+/88485810:08
opendevreviewMerged openstack/kolla stable/2023.1: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89322010:08
opendevreviewMerged openstack/kolla stable/zed: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89324110:08
opendevreviewMerged openstack/kolla stable/yoga: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89324210:08
opendevreviewVerification of a change to openstack/kolla stable/xena failed: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89324310:08
kevkoI agree with Fl1nt, vanilla configuration not working 10:09
kevkofor example we need to config override domain name in neutron config ...and we have to provide pools.yaml 10:10
jangutter.... so would now be an awkward time to ask how to go about testing a major refactor of a role? (etcd) https://review.opendev.org/c/openstack/kolla-ansible/+/88801210:10
kevkofrickler: Fl1nt just gave us an option to configure kolla-ansible and let kolla render configs 10:11
kevko(with that fix)10:11
jangutterkevko: but I think it's usually more preferable to keep the code inside kolla-ansible simple, and use drop-in config for this.10:11
SvenKieskejangutter: xD10:11
kevkojangutter: simple and not working ? 10:12
kevkojangutter: Fl1nt patch is simple and working 10:12
SvenKieskemost of the designate changes seemed fine to me, that is, providing better defaults. the problem imho is that not all usecases are present in this approach and there's a mixup - imho - between "publicly" available dns servers (which don't need to be "public", only "reachable", as frickler put it)10:13
opendevreviewMerged openstack/kolla-ansible stable/zed: Drop useless DUMMY_ENVIRONMENT environment for cron container  https://review.opendev.org/c/openstack/kolla-ansible/+/89320910:15
SvenKieskeso I like the new templated pool yaml, it's imho much easier to tweak this way.10:15
kevkoand this is how we lose contributors :(10:15
kevkoSvenKieske it is not NEW  templated pools.yaml ...it's is fixed :D 10:16
SvenKieskeI guess both sides could do a slightly better job at communication here :) and the change is rather large, which is an issue on it's own. I spent much time reviewing this and still do, and I overlooked the public api being used for mdns axfr, which is really not a good design.10:17
opendevreviewMerged openstack/kolla-ansible stable/yoga: Drop useless DUMMY_ENVIRONMENT environment for cron container  https://review.opendev.org/c/openstack/kolla-ansible/+/89321010:18
opendevreviewMerged openstack/kolla-ansible stable/2023.1: Drop useless DUMMY_ENVIRONMENT environment for cron container  https://review.opendev.org/c/openstack/kolla-ansible/+/89320810:18
opendevreviewMerged openstack/kolla-ansible stable/xena: CI: Completely drop upgrade jobs from EOL Wallaby  https://review.opendev.org/c/openstack/kolla-ansible/+/89379610:18
opendevreviewMerged openstack/kolla stable/2023.1: docs: we moved UNBUILDABLE_IMAGES to separate file  https://review.opendev.org/c/openstack/kolla/+/89241010:18
SvenKieskejust because some setups might be fine with exposing mdns publicly it's not good to expose it by default for this scenario. there's a misunderstanding, that yes: mdns needs to reach the dns server via AXFR, but that can be done more securely by using e.g. a VPN, it should imho not be done via public inet10:18
opendevreviewMerged openstack/kolla stable/zed: docs: we moved UNBUILDABLE_IMAGES to separate file  https://review.opendev.org/c/openstack/kolla/+/89241110:18
kevkoit's not large ... 10:19
SvenKieskeI still think designate needs to be burned down and should be rebuilt with a sane architecture, but I guess I can't have nice things ;)10:19
SvenKieskeit's +299 -113 lines, so all in all 412 lines to review, at least, without considering context lines needed. so almost 500 lines. I'd consider this large.10:20
SvenKiesketbf there are some comments in there. yes I've seen larger changesets (looking at podman), but they tend to get worse, not better, with more lines.10:21
SvenKieskewhen the number of lines in a changeset approaches 1000 the probability that a bug is not catched during review approaches 1.10:22
SvenKieskethat's a quote adapted from "The odds of deployment failure approach 100% as the number of distinct change sets approaches seven." from: https://lwn.net/Articles/562333/10:23
mmalchukyou propose to split change to several reviews?10:29
SvenKieskefrickler did. I'm hesitant, because this has been worked on for a long time. It would've been better if this was brought up earlier (I think it wasn't? But I'm not sure). But it seems we have already lost this contributor.10:31
SvenKieskefrom a purity perspective it should really be done in separate changes, as indicated by the commit message. because imho a single commit should fix a single bug, not 5 (or whatever the number is)10:32
mmalchukI agree with kevko, I'm also still have changes in gerrit with 2+ years on review10:32
mmalchukhttps://review.opendev.org/c/openstack/kayobe/+/79369710:33
mmalchukfor example10:33
SvenKieskea single changeset should do a single atomic thing. imho this can be ignored for trivial fixes, which might get rolled up. but the dns stuff is really not trivial. but this is a rather theoretical perspective and we also already have other huge changesets.10:33
SvenKieskewe should strive to apply "rules" like these to all patchsets.10:33
mmalchukthe same issue... years ago there was 2 patchsets, then they were merged, then again shomeone ask to split10:34
SvenKieskemmalchuk: I feel your pain, I guess my first kolla change needed a year or so to get merged and there where constant changes being demanded. I also was close to giving up. So I really understand the perspective of new contributors, I think.10:34
mmalchukI'm still here10:35
mmalchukand try to push it10:35
SvenKieskewell, me too, somehow :D10:35
mmalchuklets help Fl1nt10:35
SvenKieskeso I really feel we should make it easier to contribute. if we want to enforce rules around changeset size that should be done immediately on patch submission or even before.10:36
SvenKieskeso people don't invest much time, and are then being told to rework everything.10:36
SvenKieskejust looked it up, my first patch took from october 2020 until march 202210:38
mmalchukthe change were abandoned and he leave irc10:41
mmalchukwhat can we do?10:41
fricklerlook at the bugs, try to reproduce the issue(s), make them actionable10:42
kevkoSvenKieske: from 500 lines, the bigger half ot that lines are docs :D ... so implementation is much much less10:42
SvenKieskekevko: sure, but the implementation does many different things at once. I had a fair share of questions on that merge request as well, just to really understand what was being done, because the submitted docs where also not very clear at the start.10:43
SvenKieskeand I also agree with frickler, that the bug reports, at least in part, where not really saying much what even was the problem10:44
SvenKieskee.g. https://bugs.launchpad.net/kolla-ansible/+bug/177397210:44
kevkokevko: many different things at once ? you meant it's fixing several not properly implemented mixed DNS stuff in designate role 10:45
SvenKieskeit has been some time, but last time I deployed this, it did work. so it would be really good to show what's actually broken, in the bug report. yes, that is a lot of work.10:45
kevkowhich is not working 10:45
kevkoSvenKieske: you probably had a simple ENV10:45
SvenKieskekevko: really it's nuanced: you can deploy designate in like at least 4 different scenarios, some of those don't work with kolla, like e.g. we only provide the bind9 backend.10:46
kevkoSvenKieske: did you use neutron dns and designate dnaas together ? 10:46
SvenKieskeit's not helpful to declare something "completely broken" when it works for some usecases.10:46
kevkoIt's also lucky that we have downstream git that we use instead of upstream and we have cherry-picked things that have been sitting for review for years and it works for us for several versions :)10:48
kevkoso, if we don't want to merge ...nevermind ..i've took that patches and we are using in downstream without problem :) 10:49
SvenKieskekevko: you have the same imprecise language as Flint imho. what does your question even mean? designate integration into neutron has several subfeatures which don't need to be used all at once, see: https://docs.openstack.org/designate/latest/user/neutron-integration.html so I'm not sure what I should answer to such broad questions, really.10:49
SvenKieskee.g. I did use neutron dns to provide automatic FQDN for hardware deployed by ironic.10:50
SvenKieskein general I like most of the rework of what flint did, I'd maybe rearrange some stuff, but I don't have the time for that, currently. So I would not oppose to merge this just based on the size of the changeset.10:51
SvenKieskethe one thing I'd definitely change is the exposition of mdns, because people can imho shoot themselves in the foot there from a security perspective.10:52
mmalchukfolks, please +W https://review.opendev.org/q/Iae8a937bb2c35d2eb83e2c4522d829a6cfe25f0d10:53
kevkowe have release notes for changes ...10:53
SvenKieskekevko: I'm not sure, who are you replying too?10:54
mmalchukmnasiadka this https://review.opendev.org/c/openstack/kolla-ansible/+/893250 should be abandoned10:54
mnasiadkammalchuk: are you sure? your comment is a bit contradictory ;)10:55
mmalchuksure10:55
mmalchukissue introduced in zed10:56
SvenKieskeI was just rereading that..yeah so is the bug present there, or not?10:56
SvenKieskeso not, it seems, fine then.10:56
SvenKieskeat least this project still has contributors you can argue with, not like some other projects who are rather abandoned.. oO10:57
mmalchukmnasiadka removed +1 from change) sorry...10:59
kevkoI've been arguing more and more lately11:00
mnasiadkawell, basically we need to have in mind that even if something is configured wrong atm in kolla-ansible, we need to have a path for users from point A to point B, not just claim this was broken, we fixed it, but now you need to reconfigure your whole designate approach11:03
mmalchukso, mnasiadka you agree to drop this huge change?11:05
mmalchukalmost 'huge' as kevko said)11:06
mnasiadkaWhatever the author wants, I agree we could support direct Neutron/Nova integration - but we should introduce that as an alternative (under some variable or something) and then deprecate the old approach11:06
mnasiadka(if needs deprecating)11:06
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something  https://review.opendev.org/c/openstack/kolla-ansible/+/89360911:06
mmalchukwho will do that? authore leaves11:06
kevkomnasiadka: if it is broken in kolla out-of-the box .. so it just means that majority of people using designate role is using config overrides ...so if we fix kolla ..they are ok ..they are using own config ... 11:13
kevkofor others witsh simpler config nothing will change ..just add reno that some vars need to be amended in kolla-ansible 11:14
mmalchuk+111:16
opendevreviewMerged openstack/kolla-ansible stable/zed: Fixes WEBSSO_KEYSTONE_URL Value  https://review.opendev.org/c/openstack/kolla-ansible/+/89321311:17
opendevreviewMerged openstack/kolla-ansible stable/2023.1: Fixes WEBSSO_KEYSTONE_URL Value  https://review.opendev.org/c/openstack/kolla-ansible/+/89321211:24
fricklerkevko: if it was broken out of the box, the CI would not work. if the CI is testing the wrong scenario in your opinion, start fixing that first. otherweise stop claiming it doesn't work when the CI says something different11:27
kevkofrickler: challenge accepted ..i will add to my todolist :) 11:29
mmalchukfrickler CI didn't check everything11:32
mmalchukit test simple scenario, not used by most people in production11:32
kevko+1 11:34
mmalchukmnasiadka lets build gnocchi : https://review.opendev.org/q/I3ca4e10508c26b752412789502ceb917ecb4dbeb11:48
mnasiadkakevko: as long as my pools.yaml is unchanged after this patch (and I donā€™t need to change any config) - Iā€™m happy11:51
mnasiadkaRewriting config from one var to another is not my type of sport11:52
mnasiadkaWe can discuss that on the meeting11:53
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something  https://review.opendev.org/c/openstack/kolla-ansible/+/89360912:14
mnasiadkamgoddard mnasiadka hrw  bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo - meeting in 9 minutes12:51
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: ovn: Improve clustering  https://review.opendev.org/c/openstack/kolla-ansible/+/89377012:57
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/zed: ovn: Improve clustering  https://review.opendev.org/c/openstack/kolla-ansible/+/89377112:57
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/yoga: ovn: Improve clustering  https://review.opendev.org/c/openstack/kolla-ansible/+/89377212:58
SvenKieskeI'm always amazed how a changes is being worked on for 8 months and then reviewed and merged in under two hours :) https://review.opendev.org/c/openstack/kolla-ansible/+/86892913:00
mnasiadka#startmeeting kolla13:01
opendevmeetMeeting started Wed Sep  6 13:01:26 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.13:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:01
opendevmeetThe meeting name has been set to 'kolla'13:01
mnasiadka#topic rollcall13:01
mnasiadkao/13:01
frickler\o13:01
janguttero/13:01
mhinero/13:02
SvenKieske\o/13:02
SvenKieskeo/13:02
mnasiadkaSvenKieske: well, it has been reviewed multiple times, it's just that this bug got a lot of attention in my company downstream ;)13:02
SvenKieskesure, but I'm getting robbed of my +1 stats ;)13:02
mnasiadka#topic agenda13:03
mnasiadka* CI status13:03
mnasiadka* Release tasks13:03
mnasiadka* Regular stable releases (first meeting in a month)13:03
mnasiadka* Current cycle planning13:03
mnasiadka* Additional agenda (from whiteboard)13:03
mnasiadka* Open discussion13:03
mnasiadka#topic CI status13:03
mnasiadkaSo, let's see13:03
mmalchuko/13:03
SvenKieskethere was the rocky infra breakage, but that's fixed, right?13:04
mnasiadkadeleted wallaby from EM branch status13:04
mnasiadkasince it's EOL13:04
mnasiadkayes, it was rocky infra breakage13:04
fricklerdo we need to mirror more repos?13:04
mnasiadkawell, rocky is not mirrored at all13:06
mmalchukthere was not only rocky issue13:06
mnasiadkasecond thing is we could mirror docker rpm from something that has smaller size13:06
mmalchukalso docker and two more13:06
mnasiadkagrafana is also usually broken every now and then13:06
mmalchukyep. docker and grafana13:06
mnasiadkafrickler: I was thinking on working on docker, just need to find proper place in system-config for that13:07
fricklermnasiadka: let me know if you need help, though I'd have to search for that myself13:08
mnasiadkafrickler: there's mirror-update role that uses rsync, I doubt docker exposes rsync13:08
mnasiadkaI'll have a look and start asking stupid questions on #opendev :)13:09
fricklermnasiadka: ah, right. I don't know if there is a different solution for rpms13:09
fricklerwe could by starting to mirror docker .debs though ;)13:09
mnasiadkafrickler: we do mirror them I think, we just don't use them in kolla13:09
fricklerhmm, o.k.13:10
bbezako/13:10
mnasiadkahmm, zed rocky9/debian periodics failed13:10
mnasiadka#link https://zuul.openstack.org/builds?project=openstack%2Fkolla&pipeline=periodic&skip=013:11
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something  https://review.opendev.org/c/openstack/kolla-ansible/+/89360913:11
opendevreviewMerged openstack/kolla stable/xena: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89324313:11
fricklerI also noticed there seem to still be rmq errors, like the latest failure in https://review.opendev.org/c/openstack/kolla/+/89372213:11
jsuazoo/13:11
SvenKieskekolla-publish-ubuntu-quay seems also to have issues?13:11
mnasiadkaSvenKieske: on zed? I don't think so13:13
mnasiadkaok, let's move on and see if it fails again13:13
mnasiadka#topic Release tasks13:13
mnasiadkaSo, this week we should post a patch with release highlights13:13
SvenKieskeit has stats here at least: https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=113:14
mnasiadkaAnybody wants to have a go at skimming what we could post as marketing highlights?13:14
mnasiadkascanning release notes is my usual go13:14
mnasiadkaSvenKieske: that dashboards surely needs love13:14
SvenKieskeI fixed what I knew how to fix13:15
SvenKieskeI still need to check what versions we can bump, but I guess it's not that much13:15
mnasiadkabbezak: want to have a go at release highlights?13:15
mnasiadkaSvenKieske: the whole prometheus stack would be happy :)13:16
mnasiadkaok, I'll do the release highlights :)13:17
mnasiadkaSo, when we are at highlights13:17
SvenKieskeah right, that are easy bumps, most of the time, did it in the past already13:17
mnasiadkaone thing is Let's Encrypt13:17
mnasiadkakevko: did you update that Kolla patch already?13:17
mnasiadkaseems not13:18
mnasiadkawould be nice to get it rolling sooner than later13:18
mnasiadkasecond thing is Podman13:18
mnasiadkamhiner: I assume you still have some arm related issues (but I think kevko is looking at them)?13:19
mhineryes, I am also trying to solve it but didnt have much time for that since the last meeting13:19
SvenKieskehere are also some unadressed podman comments left to do: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/85224013:19
mnasiadkaI see https://review.opendev.org/c/openstack/kolla-ansible/+/893187 is an easy one13:20
mnasiadkaSvenKieske: that one - I'm still puzzled why we need ternary and why it doesn't work the same way as for docker role?13:21
mhinerSvenKieske: I guess the debian repo comments can be closed and Im waiting on another opinion regarding the podman containers starting/restarting13:21
SvenKieskeI'm not sure on this ternary stuff. mhiner tested it and said it didn't work13:22
kevkomnasiadka: didn't have a time :( ...but i have it half reworked regarding kolla part 13:22
SvenKieskeif we had the time it would be cool to just use always venvs and be done with this ;)13:23
mnasiadkakevko: great, hopefully you will have time to finish it off ;)13:23
kevkoi am experimenting with podman on zuul ... i have some indices ..but still not confirmed13:23
mhinerit didnt work because the variable were set, but works when they are commented out like in https://review.opendev.org/c/openstack/ansible-collection-kolla/+/89299013:23
mnasiadkaok, I commented on the ansible-collection-kolla patch - sorry, but let's keep the changes only podman related13:23
mnasiadkaif you need to change anything in the docker role (and I don't know why would you) - it should be a separate patchset13:24
ihalomimnasiadka: its more like refactor since you pointed bad manners in code in podman patchset that were just copied from docker roles13:24
kevkobtw, it would be nice to merge kolla part of podman 13:24
mnasiadkaI pointed bad manners? I'm fine with copying bad manners :)13:25
mnasiadkakevko: I promised to have a look, should have time on Friday13:25
SvenKieskewell that's the consequence if you want to have the same code patterns everywhere, of course you have then unrelated refactors..that's quite some circular logic here..13:25
kevkobecause now when i needed to add cross-dependency from kolla-ansible -> kolla .... setup_gate is failing because kolla master don't know how to work with podman ... because not merged 13:25
ihalomimnasiadka: not directly you :D but you as reviewers :)13:25
mnasiadkaSvenKieske: well, adding new functionality should be separate from refactoring - it's a bit insane to do it in one step13:25
mnasiadkarefactor is refactor - it's not adding new functionality13:26
SvenKieskenot really, it's insane to replicate known wrong behaviour just because "we have done it this way since forever"13:26
mnasiadkawell, usually you do a refactor first, and then add new functionality on top of it13:26
kevko+113:27
mnasiadkaI just value small patchsets that do one thing, not change everything at the same time - because we have no clue what is old, what is new, how should it work, etc13:27
SvenKieskeyeah, this way you never refactor, because people with money always only pay for features, but not refactoring, so we end up where we are, old stuff rotting.13:27
SvenKieskeI see maybe 1% of commits doing refactoring and 99% being feature work, where it should be the reverse. these numbers are made up of course.13:28
mnasiadkaI don't know any people with money that want to invest in Kolla :)13:28
SvenKieskereally? who's paying you then? ;)13:28
kevko+1 :D 13:28
mnasiadkaand this is rather a sidetracking thread, we told Konstantin to refactor a lot of stuff in Kolla in order to add support for Podman, so we shouldn't change approaches for the same feature set in another role13:29
SvenKieskesorry for the rant, but it's still circular logic: people say: "don'13:29
Fl1ntmnasiadka, my company invest on our team that work with kolla and try to fix things when it break, they're paying time we pass on those patch.13:29
mnasiadkaSvenKieske: I know you're a fan of ranting, I'm Polish - we rant even when we sleep :)13:29
SvenKiesket do this -> people change their code -> now it's done differently than in an unrelated file -> separate your refactoring from feature work..13:29
*** hrww is now known as hrw13:29
mmalchukFl1nt me too13:30
SvenKieskehonestly all these issues in the last few weeks are always because we have no written down thing we all agreed upon, no?13:30
Fl1ntHere are my two cents13:30
Fl1ntsince a while now13:31
mnasiadkaCan we have those two cents in Open discussion section?13:31
mnasiadkanot now?13:31
Fl1ntsure13:31
mnasiadkathanks13:31
SvenKieskeso we are always arguing in circles, "do we need a release note for this?" "is this refactoring"? "should this patch be split up"? There should be no argument about this stuff during code review imho, it should be clear from coding guidelines or a linter.13:31
mnasiadkamhiner: let's discuss the a-c-k change in Gerrit, I'll try to do some testing and help with the code13:31
* SvenKieske will shut up for now :)13:31
mnasiadka#topic Regular stable releases (first meeting in a month)13:32
kevkoSvenKieske: feel free to write detailed developer docs and we can agree all together in review 13:32
mnasiadkafrickler: we should wait until the OVN clustering backports are merged, so let's revisit next week (I maybe wrongly assumed you'll want to do the releases)13:32
fricklerI think we should just stop doing stable releases13:33
mnasiadkaSvenKieske: yes, channel your ranting powers to a document and Gerrit patchset13:33
mmalchukmnasiadka can I add Xena backport of OVN?13:33
mnasiadkafrickler: at all?13:33
fricklerthey've become so irregular nobody can rely on them anyway13:33
mnasiadkammalchuk: sure13:33
frickleryes13:33
Fl1ntjust use tags13:34
mmalchukmnasiadka ok. it take some time to solve conflicts13:34
mnasiadkafrickler: so one release and we're done? this is also a bit weird for people still using pypi for kolla releases13:34
SvenKieskeI'll try, that will be a fun review I'm sure :)13:34
fricklerwell tags are releases for us13:34
fricklerstop publishing to pypi, too. issue solved13:34
mnasiadkawell, that was something I proposed on one PTG, nobody agreed :)13:35
Fl1ntTBH, we do only use tags are they ensure readability, discoverability and reproducibility (ish)13:35
SvenKieskeI agree, but does this need to be announced, for release engineering etc? I'm not sure on the openstack process here13:35
fricklerI think deployment project are special in that regard13:35
mnasiadka#link https://releases.openstack.org/reference/release_models.html13:35
mnasiadkathose are the release models we can use13:35
SvenKiesketags are nice, but the packages on pypi let people assume these are fresh, when they really are not13:35
fricklerbut I can try to find out. also I agree that that should be a PTG decision13:36
Fl1ntfrickler, why would you stop deploying to pypi, that's exactly the opposite of what people want13:36
mnasiadkacurrently we are cycle-with-rc13:36
frickler+trailing?13:36
SvenKieskeI mean we have regularly people asking about pypi packages, so from a user point of view it might be better to go in the other direction and publish more regularly.13:36
mnasiadkayes, +trailing13:37
mnasiadkabut if we go untagged, then it's probably worse mayhem13:37
SvenKieskeeither way, the current state is not good, I guess everybody agrees on that.13:37
fricklerI think people using pypi do not get what they expect13:37
SvenKieskecan the push to pypi not be automated? I really have no clue how much work it is13:37
mnasiadkatrue13:37
mnasiadkawell, we need to raise a patch in openstack/releases13:37
fricklerit is automated as part of tagging a release13:37
Fl1ntfrickler, quite the opposite, they exactly got a release semver that you know what is contained13:37
mnasiadkaquestion is why do they not get what they expect13:38
mnasiadkabecause if they expect bugs fixed in a certain bugfix release that gets released every month13:38
Fl1ntbecause kolla stable releases keeps moving13:38
mnasiadkathen I agree we would need to be better in bug squashing and versions management13:38
frickleryes, and there's nobody around to do that13:39
Fl1ntI could tell you what we're expecting at my scale, can't talk for everybody13:39
Fl1ntfrickler, I do13:39
SvenKieskewell we do regular bugfixes and backports, my understanding was, that these are a) not tagged(?) and b) not published to pypi13:39
SvenKieskeI'm sure I'm missing something, currently I understand that we miss a patch in openstack/releases, is that all to do? there must be more, no?13:40
mmalchukagree... as backported soo many fixes13:40
Fl1ntWe've 187 countries, 4 continents, 5 regions per continents, 3 AZ per regions worldwide global cluster, we do not want things to evolve from build to build of a certain release, so we base on snapshot and freeze and pypi13:40
Fl1ntso the workflow is as is13:41
fricklerSvenKieske: you always ask for written docs, now we have them here and you don't read them ;) https://docs.openstack.org/kolla/latest/contributor/release-management.html#stable-branch-lifecycle13:41
mnasiadkafrickler: that's true, we don't even have people to triage bugs and prioritise them, not speaking about closing them13:41
mmalchukwe have them. me? Sven?13:42
SvenKieskefrickler: well when it comes to openstack docs, I either ask, google, a dev on IRC or chatgpt, because navigating the docs site is...dangerous13:42
mnasiadkaAnd I agree it's a discussion for PTG - how to manage that going forwards13:42
Fl1ntWe build on xena-cos8-13.0.9 using pypi and local build, those build need to be reproducible 100% we can't have build A do diverge from build B, with current kolla structure, it's managable through tags, but yet we still sometimes have sleeps as some package on a tag aren't fixed, so to circumvent that we build against a specific repo snapshot for a specific release build13:42
mnasiadkabecause the current model is a bit broken13:42
SvenKieskeagreed on PTG discussion13:43
mnasiadkaok, let's go to additional agenda from whiteboard, because there's ~15 minutes left13:44
mnasiadka#topic Additional agenda (from whiteboard)13:44
SvenKieskebut thanks for the lifecycle link, but it seems we do not do what's documented there ;)13:44
mnasiadkafrickler: jobboard needs love :)13:44
frickleryes, needs time, too. next time ;)13:44
mnasiadkajsuazo - TaaS (tap-as-a-service)13:45
frickleryou can drop it from agenda for now I think13:45
mnasiadkaI commented today https://review.opendev.org/c/openstack/kolla/+/88515113:45
jsuazomnasiadka: Thanks! Btw we have tested both changes up to Zed, so everything should be working fine.13:46
mnasiadkaonce Kolla part is sort-of-solved - we can review kolla-ansible part13:46
jsuazonoted13:46
mnasiadkajsuazo: I'm not saying everything shouldn't be working fine, my question is why are we overriding the version from upper-constraints - and should we even be doing that13:48
jsuazomnasiadka: my comment wasn't related to yours, just mentioning. The version change was done because if left on the constraints, we were having dependency mismatch errors on neutron13:49
mnasiadkayes, because in the patch we're trying to install tap-as-a-service from the stable branch, not from pypi13:49
mnasiadkaquestion if we should13:49
mmalchukjsuazo bump version in requiremets?13:49
jsuazoOlder version didn't work when tested, had to up the version to get the expected results.13:50
fricklerdo we really need this in all of neutron? can't it be optional somehow?13:50
mnasiadka#link https://review.opendev.org/c/openstack/kolla-ansible/+/88541713:50
mnasiadkathis is the kolla-ansible part13:50
jsuazofrickler: I believe we included a variable to opt in and out, if not we could implement it13:51
mnasiadkabasically if we have a feature in kolla-ansible - it makes sense to install taas in the image13:51
mnasiadkaquestion why taas is in upper-contraints13:51
fricklerI'm not convinced it makes sense to include any possible feature in all neutron containers13:52
jsuazommalchuk: Would love to do so, didn't work out how to D; I'll take any pointers13:52
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something  https://review.opendev.org/c/openstack/kolla-ansible/+/89360913:52
Fl1ntisn't TAPaaS supposed to be a ghost town?13:52
fricklerseems it has been revived13:53
mmalchukjsuazo https://opendev.org/openstack/requirements.git13:53
mmalchukjsuazo just make change there13:53
mmalchukjsuazo bump or even drop13:53
Fl1ntfrickler, last patch is 3 months ago tho13:54
SvenKieskeI'll need to be in another meeting at 040013:54
jsuazoFl1nt: might be, but my company is pretty interested on these features, so we are working with one of the remaining members on how revive it13:54
Fl1ntjsuazo, ok, good to read that then :D13:54
jsuazowe are already working  on bug fixes and some new features for it13:54
mnasiadkawell, anyway - it has been added to u-c long time ago for networking-midonet13:55
jsuazommalchuk: Thanks!13:55
mnasiadkaI don't mind installing one lonely pypi package, but overriding u-c seems like an overkill for now13:55
mnasiadkaBut maybe it's just me13:56
mnasiadkalet's wait for an answer in the patchset13:56
mnasiadka#topic Open discussion13:56
mnasiadka(a.k.a. ranting time)13:56
fricklerI actually think we need more u-c overriding, but that's another topic for another day13:56
mmalchukGnocchi13:56
mmalchuklets build it13:56
mmalchukhttps://review.opendev.org/q/I3ca4e10508c26b752412789502ceb917ecb4dbeb13:56
mmalchukall fixed, CI passed13:57
fricklerthe version bump should merge on master first IMO13:57
fricklerhttps://review.opendev.org/c/openstack/kolla/+/89372213:57
mmalchuknot related13:57
jangutterjsuazo updating openstack/requirements/upper-constraints.txt means that non-kolla projects like devstack start testing the same thing too, rather than kolla being unique.13:57
mnasiadkafrickler: once it passes13:58
mmalchukfrickler master wes build from the master13:58
mmalchukwas*13:58
mmalchuknevermind13:58
mmalchukok13:58
mmalchuklack of Kayobe review for a week again:13:59
mmalchukhttps://review.opendev.org/c/openstack/kayobe/+/86139713:59
mmalchukhttps://review.opendev.org/c/openstack/kayobe/+/87955413:59
mmalchukbbezak ^13:59
mmalchukand new one https://review.opendev.org/c/openstack/kayobe/+/89368814:00
mnasiadkaok14:00
mnasiadkatime is up14:00
mnasiadkasee you next week14:00
mnasiadka#endmeeting14:00
opendevmeetMeeting ended Wed Sep  6 14:00:18 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:00
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-09-06-13.01.html14:00
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-09-06-13.01.txt14:00
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-09-06-13.01.log.html14:00
mmalchukI want done with https://storyboard.openstack.org/#!/story/2002009 on stable branches14:00
mmalchukmnasiadka thanks14:00
Fl1ntok guys, I'd like to thanks kevko SvenKieske mmalchuk for their understanding about the designate patch issue, I'll let you guys follow up as you want with this thing, if you want to restore it, I'm fine with that as there are still people/users that would probably benefit from it, but I won't work on it anymore neither do I will contribute back downstream patch from now on. Thanks a lot for all the hard work 14:00
Fl1ntand efforts you guys made, so long and thanks for the fish.14:00
Fl1ntI hope you'll find a way to manage those contribution issues soon.14:01
Fl1ntSee ya14:01
mmalchukFl1nt please restore changes14:01
Fl1ntmmalchuk, sure14:01
SvenKieskeyeah, only owner can restore AFAIK, had a problem with this in keystone as well14:01
mnasiadkaa core reviewer can restore as well14:02
Fl1ntah didn't knew, I though core were able to restore14:02
mmalchuknope14:02
Fl1ntok, well, it's restored14:02
mmalchukonly one? abandoned several14:02
Fl1nthope you guys will find appropriate way to manage that.14:03
Fl1ntmmalchuk, I've restored both patchs.14:03
mmalchukcool, thanks14:03
Fl1nttbn: splitting this patch in smaller chunks will create a dependency hell14:04
mmalchukimho we could find time to manage them14:04
Fl1ntit's up to you now ;-)14:04
Fl1ntsee ya everyone.14:04
mmalchukto team)14:04
mmalchukbye14:04
mnasiadkaok, so https://review.opendev.org/c/openstack/kolla-ansible/+/893769 (revert) should be merged ASAP?14:07
mmalchukfrickler https://review.opendev.org/c/openstack/kolla/+/89372214:27
mmalchukfrickler CI passed14:27
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something  https://review.opendev.org/c/openstack/kolla-ansible/+/89360915:06
opendevreviewVerification of a change to openstack/kayobe stable/2023.1 failed: Pass through kolla inventories as is  https://review.opendev.org/c/openstack/kayobe/+/88482215:40
fricklermnasiadka: I think that should not merge at all, but I need to doublecheck tomorrow15:46
mnasiadkafrickler: did w-115:47
mnasiadkawaiting for you :)15:47
fricklerthx15:47
opendevreviewWill Szumski proposed openstack/kolla-ansible master: Support exposing prometheus_server externally  https://review.opendev.org/c/openstack/kolla-ansible/+/82869516:27
opendevreviewVerification of a change to openstack/kolla master failed: Use latest tagged version from gnocchi again  https://review.opendev.org/c/openstack/kolla/+/89372216:55
fricklermeh :(16:59
opendevreviewMerged openstack/kayobe stable/2023.1: Pass through kolla inventories as is  https://review.opendev.org/c/openstack/kayobe/+/88482218:34
*** ralonsoh is now known as ralonsoh_away18:55
opendevreviewMerged openstack/kolla master: Use latest tagged version from gnocchi again  https://review.opendev.org/c/openstack/kolla/+/89372219:56

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!