Wednesday, 2025-05-14

opendevreviewOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/openstack-ansible/+/94969203:47
jrosserWirednull: the ovs scenario tests the ovs ml2 plugin. The default, if you don't specify anything is to use OVN, so by putting 'ovs' in your SCENARIO string you move away from that default.06:13
jrosserfor the other bridges its important to distinguish between using ovs on the controllers for the LXC bridges, or not, as you prefer06:14
jrosseryou can set that up either way, if you using dedicated network nodes might also influence where ovs is running, and also you might prefer to have ovs everywhere including controllers for consistency. But on the other hand you might want regular linux bridges for lxc on the controllers for simplicity.06:16
jrosseressentially, you can configure it whichever way you want (usual story with OSA!)06:17
noonedeadpunkgood morning06:46
opendevreviewDaniel Preussker proposed openstack/openstack-ansible-os_swift master: swift-object-expirer is its own distro package  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/94704807:00
WireLostjrosser, thank you!08:01
jrosserWireLost: unfortunately it is really very configurable and you have a lot of options08:05
jrossersticking with the defaults would give you OVN for neutron and linux bridges for the control plane LXC containers08:05
jrosserand there are times when those two worlds need to join properly, like octavia or ironic, which need some planning08:06
WireLostCool! Sounds good. I'm interested in a simpler OVS setup without OVN, which if I remember well would still use the Neutron agents, in a more similar architecture when with Linux Bridges. But if OVS+OVN is the default these days, I'll stick with it!08:33
noonedeadpunkWireLost: most of big companies using and providing openstack migrated most of their customers to OVN already08:35
noonedeadpunkas namespaces is actually the problem, despite them being easy to debug08:35
noonedeadpunkas when you have like thousands namespaces per net node, each l3 restart brings pain and suffer08:36
noonedeadpunkor well. I jsut don't like OVS.... I liked LXB, OVN is fine, but OVS is weird, imo...08:38
WireLostBut doesn't OVN depends on OVS? I also like LBR... =P08:39
noonedeadpunkYou don't get OVN benefits of resilient failovers and DVR, but you already introduce OVS complexity partially08:39
WireLostRight, yeah, I can see that...08:39
noonedeadpunkYes, it does. But it also uses OpenFlows rather then using OVS as a regular LXB08:39
WireLostHmm... Good to know! lol08:40
noonedeadpunkSo it's more cumbersome to debug and understand traffic flows in OVN08:41
WireLostMakes sense08:41
noonedeadpunkBut then you can make path way more efficient by serving FIPs from computes where VM is and not pass traffic around the cluster for no reason08:41
noonedeadpunkbut dunno. 08:42
noonedeadpunkOVS is way more alike to LXB indeed, so for ops people it will be easier to switch/learn08:43
WireLostBut is it simple to use OVS without OVN with OSA? Perhaps another SCENARIO?08:47
jrosserthat is what happens when you put 'ovs' in the scenario08:48
WireLostBut doesn't it also enables OVN?08:53
WireLostMaybe I should invert the question then... lol - How do enable OVN with OSA/AIO?  :-P08:54
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94971209:00
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94971209:01
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: remove xen info  https://review.opendev.org/c/openstack/openstack-ansible/+/94971209:02
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: remove XEN info  https://review.opendev.org/c/openstack/openstack-ansible/+/94971209:05
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: remove Xen info  https://review.opendev.org/c/openstack/openstack-ansible/+/94971209:05
jrosserWireLost: you don't explicitly enable it, its the default for the AIO09:07
jrosserSCENARIO=aio_lxc -> lxc control plane with OVN networking09:07
noonedeadpunkwe really should document somewhere that ovn is the default09:17
noonedeadpunkand reference to os_neutron docs for more details09:18
WireLostAhh ok, thanks!09:32
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: replace broken url to "Getting Started with MariaDB Galera Cluster"  https://review.opendev.org/c/openstack/openstack-ansible/+/94968310:22
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: fix for problem with duplicating the language switch  https://review.opendev.org/c/openstack/openstack-ansible/+/94941010:24
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94972711:07
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94972711:08
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: fix for small mistake in text  https://review.opendev.org/c/openstack/openstack-ansible/+/94972711:08
noonedeadpunkhuh. So I'm debugging the ovs lxc thing. and the VM seems to be getting just the wrong IP address11:19
opendevreviewMerged openstack/openstack-ansible master: Ensure /etc permissions in CI  https://review.opendev.org/c/openstack/openstack-ansible/+/94964711:19
noonedeadpunkso the port connected to the VM has IP 192.168.19.207 and then inside of the VM the IP is from 169.25411:19
jrosseroh thats link local11:21
jrosserso dhcp failure11:21
noonedeadpunkyeah, and it provided by DHCP, jsut tested with dhcpd -T11:21
noonedeadpunkor well11:21
jrosserdhclient -v11:22
noonedeadpunkeh. it's cirros11:22
jrosserblargh11:22
noonedeadpunkhttps://ibb.co/B2K6Rm2811:22
jrosseri usually see 169.254 anywhere on a system that was expecting to dhcp but for some reason that fails, and link local is the fallback11:23
noonedeadpunkright...11:23
noonedeadpunkok, let me try to assign the expected address as alias and check wtf with dhcp namespace is happening11:23
jrosserit could also be thats there just no L2 between the VM and wherever the dhcp should be11:25
jrosserlike tunnel fail or something11:25
noonedeadpunkI mean. It;s aio11:26
jrosseroh sure, but project network is geneve?11:26
noonedeadpunkvxlan, yeah11:26
jrosseroh sure - ovs11:26
noonedeadpunkbut it somehow easier and messier at the same time11:26
jrosserso it should be possible to be in the l3 agent namespace and ping the vm11:27
jrosser(once manually assigned IP)11:27
noonedeadpunkthere's no l3 as I didn't create it11:27
noonedeadpunkbut there's dhcp one :)11:27
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94972811:31
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94972811:33
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: fix for small mistake in text (with OpenStack)  https://review.opendev.org/c/openstack/openstack-ansible/+/94972711:33
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: fix for small mistake in text (with Open vSwitch)  https://review.opendev.org/c/openstack/openstack-ansible/+/94972811:33
noonedeadpunkjrosser: I think you;'re right as it seems that dnsmasq is not running in namespace11:34
jrosserah I wonder if we collide with the lxc one11:35
noonedeadpunkdnsmasq: cannot read /etc/neutron/dnsmasq-neutron.conf: Permission denied11:36
noonedeadpunkhttps://paste.openstack.org/show/bGjduWq7edtzo4JRfz4c/11:37
noonedeadpunkI wonder if that's actually something with apparmor11:39
noonedeadpunkdoes not look like it11:44
noonedeadpunkhm https://paste.openstack.org/show/bFmzYinmqeph5IekEMss/11:48
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94973211:52
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94973211:52
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: fix small mistake with :  https://review.opendev.org/c/openstack/openstack-ansible/+/94973211:52
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94973311:57
noonedeadpunkok, it's apparmor11:57
noonedeadpunktype=AVC msg=audit(1747223843.455:128039): apparmor="DENIED" operation="open" class="file" profile="dnsmasq" name="/openstack/venvs/neutron-31.0.0.0b2.dev92/etc/neutron/dnsmasq-neutron.conf" pid=392391 comm="dnsmasq" requested_mask="r" denied_mask="r" fsuid=0 ouid=0FSUID="root" OUID="root"11:57
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94973311:58
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: fix small mistake with MariaDB  https://review.opendev.org/c/openstack/openstack-ansible/+/94973311:58
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Admi  https://review.opendev.org/c/openstack/openstack-ansible/+/94973412:00
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: fix small mistake with MariaDB  https://review.opendev.org/c/openstack/openstack-ansible/+/94973312:05
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: fix small mistake with MariaDB and RabbitMQ  https://review.opendev.org/c/openstack/openstack-ansible/+/94973312:05
NeilHanlonplease do ping me if anything needs reviewing. i'll do my best to check but am gonna be pretty swamped with RH releasing 9.6 *and* 10.0 yesterday...12:05
noonedeadpunkwe have couple of things, so I will do a list and ping you explicitly :)12:10
noonedeadpunkso, I figured out the apparmor requirements...12:11
noonedeadpunkoh, well, we do have a code to just disable apparmor for dnsmasq12:16
opendevreviewMerged openstack/openstack-ansible master: docs: fix for problem with duplicating the language switch  https://review.opendev.org/c/openstack/openstack-ansible/+/94941012:22
opendevreviewMerged openstack/openstack-ansible master: docs: replace broken url to "Getting Started with MariaDB Galera Cluster"  https://review.opendev.org/c/openstack/openstack-ansible/+/94968312:22
noonedeadpunkok, so pretty much our disablement of apparmor does not work because of conflict error on lxc12:22
noonedeadpunkhttps://paste.openstack.org/show/bCO8TLSQCcdpETH2IV8q/12:22
noonedeadpunkhttps://bugs.launchpad.net/ubuntu/+source/lxc/+bug/211063512:40
noonedeadpunkdohy12:40
NeilHanlonsecurity was and remains to be a mistake12:42
noonedeadpunkI just kinda wonder how to workaround that....12:42
noonedeadpunkalso in fact we should configure rather then disable apparmor...12:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Path the usr.bin.lxc-copy apparmor profile  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/94974012:58
opendevreviewMerged openstack/openstack-ansible-os_swift master: swift-object-expirer is its own distro package  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/94704813:02
opendevreviewMerged openstack/openstack-ansible-os_swift master: Remove seemingly unsused mlocate cronjob  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/94705013:02
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: Move deploy-guide under docs for translation  https://review.opendev.org/c/openstack/openstack-ansible/+/94949713:04
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: Move deploy-guide under docs for translation  https://review.opendev.org/c/openstack/openstack-ansible/+/94949713:04
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: Move deploy-guide under docs for translation  https://review.opendev.org/c/openstack/openstack-ansible/+/94949713:04
opendevreviewMerged openstack/openstack-ansible-os_swift master: Migrate ring distribution to SSHCA  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/94699013:05
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Add OVS Ubuntu 24.04 jobs  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94870213:11
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Respect aa-disable exit code when disabling profiles  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94974313:11
noonedeadpunkAlso I kind of wonder what exactly should be backport to 2024.213:16
noonedeadpunkjust https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/949740 ?13:16
noonedeadpunkthen I probably need to writesome renos13:17
kleiniIs there already somewhere a guide for OSA how to migrate from OVS to OVN? I just read quickly through https://www.jimmdenton.com/migrating-lxb-to-ovn/ and it should be doable in a similar manner even without the database changes for LXB.13:22
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Admi  https://review.opendev.org/c/openstack/openstack-ansible/+/94973413:24
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Manage Cloud  https://review.opendev.org/c/openstack/openstack-ansible/+/94973413:25
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Manage Instances  https://review.opendev.org/c/openstack/openstack-ansible/+/94973413:25
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Manage Instances  https://review.opendev.org/c/openstack/openstack-ansible/+/94973413:26
noonedeadpunkkleini: unfortunatelly there's not so far13:30
noonedeadpunkwe'd gladly accept a contribution, but I don't think we digged deep into that so far13:31
noonedeadpunkI personally don't even a deployment with LXB on maintenance right now13:31
noonedeadpunkoh, wait, you asked from OVS :)13:32
noonedeadpunkOVS -> OVN I planned to check on in Q3 I guess, but I have no idea if I will be able to13:32
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Path the usr.bin.lxc-copy apparmor profile  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/94974013:38
jrossernoonedeadpunk: "Patch" not Path?13:40
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Manage Instances  https://review.opendev.org/c/openstack/openstack-ansible/+/94973413:40
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Patch the usr.bin.lxc-copy apparmor profile  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/94974013:40
noonedeadpunkright13:40
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Respect aa-disable exit code when disabling profiles  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94974313:43
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Manage Instances Page  https://review.opendev.org/c/openstack/openstack-ansible/+/94973413:44
noonedeadpunkactually this can explain a lot of weird intermittent things I saw with metadata on OVN as well...13:44
noonedeadpunkas haproxy also should be affected13:44
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Update uwsgi re-disable reno to contain bug ID  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94974913:51
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Manage Instances Page  https://review.opendev.org/c/openstack/openstack-ansible/+/94973414:04
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Simplify logic of including apparmor tasks  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94976014:17
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Simplify logic of including apparmor tasks  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94976014:20
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Manage Instances Page  https://review.opendev.org/c/openstack/openstack-ansible/+/94973414:23
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP docs: Update Scale Environment  https://review.opendev.org/c/openstack/openstack-ansible/+/94976314:30
noonedeadpunkjrosser: do you see anywhere `usr.sbin.haproxy` apparmor profile?14:37
jrosserwhereabouts?14:38
noonedeadpunkI'm just not sure if it's even a thing14:38
noonedeadpunkwe disable it here https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/neutron_apparmor.yml#L56-L5714:39
noonedeadpunkbut seems it kind of noop thing14:39
noonedeadpunkand commit was 7y ago14:40
jrosseri don't seem to have that in /etc/ or /usr/share14:41
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP docs: Update Scale Environment  https://review.opendev.org/c/openstack/openstack-ansible/+/94976314:42
noonedeadpunkah, actually found it in /usr/share/apparmor/extra-profiles/usr.sbin.haproxy14:42
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Manage Instances Page  https://review.opendev.org/c/openstack/openstack-ansible/+/94973414:50
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Manage Instances Page  https://review.opendev.org/c/openstack/openstack-ansible/+/94973415:44
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Troubleshooting  https://review.opendev.org/c/openstack/openstack-ansible/+/94977916:12
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Configure apparmor for dnsmasq  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94978016:20
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Configure apparmor for dnsmasq  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94978016:26
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Configure apparmor for haproxy  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94978116:29
noonedeadpunkI somehow hope that freezed roles won't cause troubles for us....16:34
noonedeadpunk(ie https://review.opendev.org/c/openstack/openstack-ansible/+/946281 )16:34
noonedeadpunkas we're not testing latest master roles right now16:34
noonedeadpunkbut also https://zuul.opendev.org/t/openstack/build/c3289a96dbb749378893110e84cffb62 is weired16:34
noonedeadpunkah, forget it16:35
noonedeadpunkI mixed up the job with https://zuul.opendev.org/t/openstack/build/44008bc78e30420890725917e9aa061516:35
noonedeadpunkNeilHanlon: btw. Another opportunity for the rant about https://zuul.opendev.org/t/openstack/build/c3289a96dbb749378893110e84cffb62 16:36
noonedeadpunkI'm really o_O how ceph folks don't care much to build backages not against EL, but against CentOS16:37
noonedeadpunkAnd also how RHEL is actually handling all these things....16:37
noonedeadpunkor they just don't care today about their distro at all?16:37
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Troubleshooting  https://review.opendev.org/c/openstack/openstack-ansible/+/94977916:44
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: docs: update Scaling Environment page grammar and console display  https://review.opendev.org/c/openstack/openstack-ansible/+/94976316:52
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: docs: update environment scaling page  https://review.opendev.org/c/openstack/openstack-ansible/+/94976316:52
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: docs: update environment scaling page  https://review.opendev.org/c/openstack/openstack-ansible/+/94976316:53
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: docs: update troubleshooting page  https://review.opendev.org/c/openstack/openstack-ansible/+/94977916:55
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: docs: Update Manage Instances Page  https://review.opendev.org/c/openstack/openstack-ansible/+/94973416:59
noonedeadpunkNeilHanlon: about reviews - there is this patch https://review.opendev.org/q/project:openstack/openstack-ansible-lxc_hosts and then pushed couple with fixes for Neutron https://review.opendev.org/q/project:openstack/openstack-ansible-os_neutron+status:open+label:Verified17:00
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: docs: update managing instances page  https://review.opendev.org/c/openstack/openstack-ansible/+/94973417:02
NeilHanlonthanks noonedeadpunk -- gave the first a review and will move along the other stack19:21
opendevreviewMerged openstack/openstack-ansible-os_swift master: Remove md5 checks of rings/builders  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/94702120:26
opendevreviewMerged openstack/openstack-ansible-os_neutron master: Respect aa-disable exit code when disabling profiles  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94974320:48
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: WIP: fix link  https://review.opendev.org/c/openstack/openstack-ansible/+/94980721:34
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: docs: fix broken links on Galera Cluster page  https://review.opendev.org/c/openstack/openstack-ansible/+/94980721:39
-opendevstatus- NOTICE: Setuptools 80.7.0 broke python package installs for many affecting CI jobs. That release has been yanked and it should be safe to recheck failed changes.21:59
opendevreviewMerged openstack/openstack-ansible-os_neutron master: Simplify logic of including apparmor tasks  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/94976022:50
opendevreviewMerged openstack/openstack-ansible-lxc_hosts master: Patch the usr.bin.lxc-copy apparmor profile  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/94974023:21

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!