Tuesday, 2025-05-13

opendevreviewOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/openstack-ansible/+/94955603:43
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: fix for mistakes with Ansible, OpenStack-Ansible and RabbitMQ  https://review.opendev.org/c/openstack/openstack-ansible/+/94953707:36
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94957007:40
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94957007:41
noonedeadpunkI think we need to figure out wtf is wrong with https://zuul.opendev.org/t/openstack/build/9199af1212034c2fbc4f086c0569416a07:43
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: update Project scope - remove duplicate phrase  https://review.opendev.org/c/openstack/openstack-ansible/+/94957007:55
opendevreviewMerged openstack/openstack-ansible master: Fix links to Ansible documentation  https://review.opendev.org/c/openstack/openstack-ansible/+/94953608:12
opendevreviewMerged openstack/openstack-ansible master: docs: fix for mistakes with Ansible, OpenStack-Ansible and RabbitMQ  https://review.opendev.org/c/openstack/openstack-ansible/+/94953708:12
opendevreviewMerged openstack/openstack-ansible master: docs: remove old note about bug for Ansible  https://review.opendev.org/c/openstack/openstack-ansible/+/94954108:12
opendevreviewMerged openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/openstack-ansible/+/94955608:16
opendevreviewMerged openstack/openstack-ansible-tests stable/2024.2: Remove retired qdrouterd repo from zuul jobs  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/94919308:34
opendevreviewMerged openstack/openstack-ansible-tests stable/2024.1: Remove retired qdrouterd repo from zuul jobs  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/94919408:40
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Include ceph_client role instead of importing  https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/94958009:17
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94958309:31
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: fix for issues when switching between pages using localization  https://review.opendev.org/c/openstack/openstack-ansible/+/94958309:44
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Use dynamic include instead of static imports for conditional tasks  https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/94958509:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Remove quotes from conditional statements  https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/94958809:57
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: docs: enable translation for deploy guide  https://review.opendev.org/c/openstack/openstack-ansible/+/94763410:08
opendevreviewDmitriy Chubinidze proposed openstack/openstack-ansible master: docs: enable translation for deploy guide  https://review.opendev.org/c/openstack/openstack-ansible/+/94763410:08
jpw_hey, i'm trying to get name based endpoints working and i'm following the instructions in https://docs.openstack.org/openstack-ansible/latest/user/prod/pretty_endpoint_naming.html#configuring-domain-based-endpoints-recommended. there seems to be a bit of wierdness here.10:18
jpw_if i have the name based haproxy configuration enabled during the first run it will fail since map files have not been created yet, yet if i try to create the maps using `openstack-ansible openstack.osa.keystone --tags haproxy-service-config` it will also fail on subsequent nodes because the haproxy configuration hasn't progressed to the point where /etc/haproxy/conf.d hasn been created.10:20
jpw_so my solution right now is to oscillate between running the two commands until everything is in place then run setup_infrastructure again. but it makes me feel like i'm doing something wrong. is there a better way?10:24
* noonedeadpunk on the meeting right now10:30
jpw_actually i think i've solved it. i was using wildcard DNS records, switching back to A records seems to have made the error go away10:53
* noonedeadpunk just finished11:21
noonedeadpunkjpw_: good that you've solved it though11:22
jpw_im not convinded i have tbh11:22
jpw_im just having another run through11:23
noonedeadpunkSo I don;t think it has anything to do with DNS records per say11:23
noonedeadpunkAs map file is created during openstack.osa.haproxy run11:24
jpw_im more just checking that i'm using this feature correctly. the doc's say that if those variables are modified to re run the parts of the playbook that they affect. 11:24
noonedeadpunkand then you pretty much extend regexpes for each service11:24
jpw_but those play's will be run as part of a general run right? so it should work from a deployment from scratch11:25
noonedeadpunkfrom scratch - yes, sure11:25
noonedeadpunkbut eventually the base regexp should be added regardless tbh11:26
noonedeadpunkas part of this: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/haproxy/haproxy.yml#L83-L8411:26
noonedeadpunkas we serve horizon/skyline/security through it already11:26
jrosserit would be very helpful to share whatever errors you get11:27
jpw_was I wrong to assume this configuration went in user_variables.yml?11:27
jpw_i'll make a paste11:27
noonedeadpunkit can go either to user_variables or to group_vars for a specific service11:28
jpw_https://paste.openstack.org/show/bmYHIKHP6RYIj23Diaim/11:31
jpw_the important bit of line #6 for me was `failed to parse sample expression <req.hdr(host),map_dom(/etc/haproxy/base_domain.map)]> : invalid args in converter 'map_dom' : failed to open pattern file </etc/haproxy/base_domain.map>`11:32
noonedeadpunkjpw_: huh11:43
noonedeadpunkjpw_: what was the result of previous handlers?11:46
noonedeadpunkas map files should have been generated here: https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/handlers/main.yml#L28-L3711:46
jpw_unfortunatly i'ts gone from my scrollback11:49
noonedeadpunkcan you jsut run openstack.osa.haproxy ?11:50
noonedeadpunkas context around is important11:51
jpw_yeah i've got it one moment11:51
noonedeadpunkI can assume that smth is off as logic is quite complex, but I don't see anything obvious11:52
jpw_https://paste.openstack.org/show/bFhzBSLff9wBHYggBQVg/11:52
noonedeadpunkdo you have anything in /etc/haproxy/map.conf.d/ ?11:52
noonedeadpunkhm11:53
jpw_drwxr-x--- 2 root    haproxy 4096 May 13 11:27 base_regex11:53
jpw_contains -rw-r----- 1 root haproxy   83 May 13 11:27 00-base.map11:53
noonedeadpunkI think I understand what is the issue now11:53
jpw_contains `#Regular expression map file - this comment is defined in the base frontend config`11:54
noonedeadpunkjpw_: what if you do like https://paste.openstack.org/show/bkNduoLbM2Rbm3uZOhY7/11:56
noonedeadpunkbtw11:56
noonedeadpunkyou also don't have to use base_domain either... this could be written through regex as well11:57
jpw_that seems happy now11:59
jpw_so what's the difference. the docs don't give a choice between domain/regex12:00
noonedeadpunkalso, I think you need to change the order of elemets in haproxy_maps for precedence to work properly... but not sure12:00
noonedeadpunkah, no, forget it12:00
noonedeadpunkit's correct now12:00
noonedeadpunkso map_dom should be slightly faster then map_reg12:01
noonedeadpunkbut you could also do `'use_backend %[req.hdr(host),map_reg(/etc/haproxy/base_regex.map)]'`12:01
noonedeadpunkand then entry can look like `"volume.cloud.* cinder_api-back"12:02
noonedeadpunkif you can help and patch the doc - that will be really appreciated12:02
jpw_oh i see, your fix ensures that files exists using haproxy_map_entries12:03
noonedeadpunkyeah12:04
noonedeadpunkas they are added only in later services, not in base one12:04
noonedeadpunkwhich raises the race condition12:04
jpw_sorry, where's the documentation repo?12:07
jpw_never mind, i just found it in the main repo12:08
noonedeadpunkyeah12:08
jrosseri think we should be able to test this?12:14
jrossertheres already some fake stuff for external fqdn for the stepca job12:14
jrosserso that could be adapted with some extra /etc/hosts entries for identity.blah volume.blah etc12:15
noonedeadpunkwe probably can. I'm a bit concerned about amount of jobs we already run though12:17
jrosserits probably enough for an infra job12:17
noonedeadpunkyeah, right12:17
noonedeadpunkor maybe add something for haproxy specifically....12:18
noonedeadpunkor even for molecule....12:18
jrosserindeed, there can be something specific here12:18
noonedeadpunkas that actually boils down to testing of haproxy maps I guess...12:19
noonedeadpunkbut not sure really12:19
*** tosky_ is now known as tosky12:27
jpw_this scm is interesting. im not sure what it's asking me to do. so i clone the repo, do i create a branch or modify main directly?12:32
noonedeadpunkyou can modify directly12:36
noonedeadpunkyou'd need to have git-review plugin though12:36
noonedeadpunkyou can also create the branch :)12:37
noonedeadpunkthe branch is used as a change topic12:37
noonedeadpunk(so you can group changes across multiple repos)12:38
noonedeadpunkyou also need an account https://review.opendev.org/, define a username, upload the ssh key12:38
opendevreviewJames Park-Watt proposed openstack/openstack-ansible master: doc: haproxy_base_service_overrides  https://review.opendev.org/c/openstack/openstack-ansible/+/94962512:46
jpw_phew, made it12:46
noonedeadpunkjpw_: thanks!13:07
jpw_yw13:07
opendevreviewMerged openstack/openstack-ansible master: doc: haproxy_base_service_overrides  https://review.opendev.org/c/openstack/openstack-ansible/+/94962513:44
noonedeadpunk#startmeeting openstack_ansible_meeting15:00
opendevmeetMeeting started Tue May 13 15:00:34 2025 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'openstack_ansible_meeting'15:00
noonedeadpunk#topic rollcall15:00
noonedeadpunko/15:00
David_Gomezo/15:00
jrossero/15:03
noonedeadpunk#topic office hours15:06
noonedeadpunkso I think we are super close today15:06
noonedeadpunkto get roles branched witrh 2025.115:06
noonedeadpunkI'm not sure we do have hard blockers actually15:07
NeilHanlono/15:07
jrosserit is a shame to miss the swift stuff15:08
noonedeadpunkthere are some Swift outstanding improvements, but I think we may be able to backport them after branching15:08
jrosserhah :)15:08
jrosser^ that15:08
jrosserits basically broken so this is bugfix work15:08
noonedeadpunktoday the blocker for them are 0775 permissions in /etc on Rocky15:08
noonedeadpunkwe can try to address it in ssh_keypairs role15:09
noonedeadpunkbut I feel weird changing permissions for etc15:09
jrossercan we do it in zuul pre task or something15:10
jrosserthats less wierd15:10
jrosseras its a CI specific fix, in a CI specific place15:10
NeilHanlonyeah.. i can get it fixed but it probably will take too long to be meaningful for us15:10
NeilHanloni have to build and test and publish containers, then nodepool would have to rebuild them... then propagate that out...15:10
noonedeadpunkI'm thinking if this can be done in CI for upgrade jobs...15:11
noonedeadpunkif in some pre_tasks15:11
noonedeadpunkit can work, yeah, I will try this out15:11
noonedeadpunkand the next confusing/broken thing is OVS for Noble.15:11
noonedeadpunk#link https://zuul.opendev.org/t/openstack/build/9199af1212034c2fbc4f086c0569416a15:12
noonedeadpunkand I was able to reproduce this, it seems15:12
noonedeadpunkso potentially this might be quite valid bug15:12
jrosserhmm i wonder if we have the right version15:13
jrosserlike assumption of ovs version vs distro repo vs UCA .....15:14
noonedeadpunkin AIO I got 3.5.015:14
noonedeadpunkbut also15:14
noonedeadpunkmetal is passing15:14
noonedeadpunkhttps://zuul.opendev.org/t/openstack/build/8f30d7d8d8e64d1ba25e6aa46f09f4cf15:14
jrosserhaving said all this - the other jobs are all OVN which uses OVS under the hood15:15
jrosserso that side of things is OK on noble15:16
noonedeadpunkwell. OVN is kinda different, as it's not neutron code that need to deal with OVS wiring and namespaces and etc15:16
noonedeadpunkits Neutron -> OVN -> OVS15:16
jrosserand i just catch up with this - it's the LXC version of this job which fails but metal is OK15:17
noonedeadpunkyeah, which is weird15:18
noonedeadpunkbut seems reproducible at very least15:18
noonedeadpunkI jsut did not check on logs in my AIO yet15:18
noonedeadpunkOther then that... I guess I'd love to land https://review.opendev.org/c/openstack/openstack-ansible/+/946281 regardless, before branching happens15:19
noonedeadpunkProbably it's a bit too late, but also I'm not sure what we are actually testing now15:19
noonedeadpunkthis should have been done right after the beta15:19
jrossertheres some errors about dropped packets on bt-int https://3e3b9bb51c4bbfcca1ce-b2ab9ab260082d15bc4e79c36fac49d2.ssl.cf5.rackcdn.com/openstack/9199af1212034c2fbc4f086c0569416a/logs/host/openvswitch/ovs-vswitchd.log.txt15:20
noonedeadpunkI have same log pretty much on aio, yes15:21
noonedeadpunkit can boil down to some brdige setup and veth15:24
noonedeadpunkas we connect br-provider in ovs with br-vlan in lxb thrtough veth eth1215:24
noonedeadpunkbut yeah - other then that things look pretty good15:30
jrosserit will be good to get the release done to work on new ansible and also cutting the number of jobs back where possible15:31
noonedeadpunkI don't think we will in fact drop anything15:32
noonedeadpunkas Debian 13 is pretty much released15:32
noonedeadpunkso I assume we'll need to look into it as well15:32
jrosseri have a few patches outstanding for zuul errors cleanup in stable and unmaintained15:33
jrosserand also a few that can't be merged15:34
noonedeadpunkwich are not passing15:34
noonedeadpunkyeah15:34
jrosseridk if we want to get them force merged or not15:34
jrosserbut there is a problem that the 2023.2 branch is deleted from the service repos15:34
jrosserbut not from ours15:34
jrosserso i think thats a big cause of the trouble15:35
noonedeadpunkoh... right...15:36
noonedeadpunkI think I need to look on EOM-ing the branch15:36
jrosseryep - there are several processes, EOM-ing, unmaintaining etc15:36
jrosserand of course we are somewhat offset from those today compared to the rest of the projects15:37
jrosserzed is messy right now in particular15:37
noonedeadpunkI frankly did not look into CI results except saw they're not feeling good15:38
noonedeadpunkok, so to sum up plan for this week - branch roles, propose rc1, EOM 2023.2, check what's up with Zed15:40
noonedeadpunkand try to figure out OVS/Swift15:43
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Ensure /etc permissions in CI  https://review.opendev.org/c/openstack/openstack-ansible/+/94964715:51
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Migrate ring distribution to SSHCA  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/94699015:51
noonedeadpunklet's see if that will help15:51
noonedeadpunkoh, one thing I wanted to discuss actually15:52
noonedeadpunkwhich is this patch15:53
noonedeadpunk#link https://review.opendev.org/c/openstack/openstack-ansible/+/94949715:53
noonedeadpunkand if we wanna go this route, or better to properly patch tooling and add support for deploy-guide folders import to Zanata instead15:53
noonedeadpunkas eventually - this patch is pretty much a workaround15:53
noonedeadpunkover stuff related to the translations overall15:54
noonedeadpunkso any input/ideas on how we prefer moving forward is really appreciated15:56
noonedeadpunkor we can try this way and revert if it does not work...15:59
noonedeadpunk#endmeeting15:59
opendevmeetMeeting ended Tue May 13 15:59:33 2025 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:59
opendevmeetMinutes:        https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-05-13-15.00.html15:59
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-05-13-15.00.txt15:59
opendevmeetLog:            https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-05-13-15.00.log.html15:59
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Use dynamic include instead of static imports for conditional tasks  https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/94965316:27
jpw_am i right that haproxy_user_ssl_cert,key,ca is the certificate applied to the external vip of haproxy?16:45
noonedeadpunkit might even for both, if you have tls for internal as well16:49
* jrosser hopes those are independent16:50
noonedeadpunkI don't think they are https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/vars/main.yml#L6516:50
jrosserfeels like an error16:50
* noonedeadpunk needs to refactor all that to enable indepenedent frontend configuration16:51
noonedeadpunka lot of legacy coming from our assumption of 2 frontend alway16:51
jrosseras likely use case would be user cert on the external vip and pki role on all internal things16:51
noonedeadpunkor no tls on internal...16:52
noonedeadpunkor do wildcard or SAN16:52
noonedeadpunkor rely on let's encrypt16:52
jrosserindeed, many options16:52
jrosseryes16:52
noonedeadpunkas let's encrypt on front and pki on back does work16:52
jrosserI think we never really explored anything but what you describe there16:52
noonedeadpunkbut haproxy_user_ssl_cert should be wildcard/SAN as of today16:52
noonedeadpunkif you want to cover internal with TLS16:53
noonedeadpunkyeah, we jsut do SANs16:53
jrosseryes this all needs a revisit16:53
jpw_yes it os both internal + external16:53
jrosseripv6 is ugly in this area too16:53
noonedeadpunkbut it is annoying in a way still16:53
jpw_im planning to spin up an acme server in due course. i just need something i can sign with an existing ca so not to have CA sprawl16:54
jrosseryou’ll just have to make sure all the services trust that CA16:55
jrosserif you use tls on the internal vip too16:55
jrosserwhich is optional16:55
noonedeadpunkor you can rely on OSA-issues CA16:55
jrosserwell not if the user supplied cert is on both sides?16:56
noonedeadpunkno, not then16:56
noonedeadpunkbut well16:56
jrosserjpw_: message is…. this can get complicated :/16:57
noonedeadpunkyou can pass custom root CA to OSA and it can generate an intermediate out of it16:57
jpw_so if i want custom certs on the external api's + horizon i need to override the whole of pki?16:57
jpw_eesh16:57
noonedeadpunkand then generate self-singed with the intermediate and add custom CA to trust store16:57
noonedeadpunkwe did that in quite some places16:57
noonedeadpunkjpw_: it all depends, really16:58
jpw_so generate an intermediate for the PKI playbooks to consume and use to sign services?16:58
jrosserif you want16:58
jrossersorry that’s not specific16:58
jpw_what would be ideal really16:58
jrosserin short the trust needs to exist in the right places16:59
jpw_s/what/that/16:59
noonedeadpunkLike - I'm using own self-signed CA for internal endpoints and then let's encrypt for public ones16:59
noonedeadpunkAnd then each region just issues an intermediate based of this root CA16:59
jpw_is there not a way to use self signed for internal and just use a user provided cert for publilc endpoints?17:00
noonedeadpunkand certs in each region are based of it17:00
noonedeadpunkI'm afraid not today17:00
jpw_ok that's fine17:00
jrosserjpw_: thats the one case you just need to take care with17:00
noonedeadpunkthen user provided should include domains for internal17:00
noonedeadpunkso it should be SAN/wildcard17:00
jrosserand your CA should be injected into all the hosts/containers17:01
noonedeadpunkyou can actually just define the variable and roles can do that injection for you17:01
jpw_if it's not possible i'll just go the self signed osa managed route it's easier and just a POC.17:01
jrossertbh this is a bit of an oversight - it should be easier than this17:02
jrosserwe do have an entirely self contained example in the AIO using pki inside and acme outside17:02
noonedeadpunkis `openstack_pki_install_ca: [{'name': 'MyRoot'}] and then place your root CA under openstack_deploy/pki/roots/MyRoot/certs/MyRoot.crt17:02
noonedeadpunkif you also place private key under openstack_deploy/pki/roots/MyRoot/private/MyRoot.key.pem ou can override an intermediate details to use it for all cert issuing17:04
noonedeadpunkjust set `openstack_pki_authorities` to this second part https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/ssl.yml#L49-L63 and set `signed_by: MyRoot`17:05
noonedeadpunkonce you run openstack.osa.certificate_authority - you should be able to drop the privbate key from openstack_deploy/pki/roots/MyRoot/certs/MyRoot.crt17:06
jpw_i'll take another look in due course. thanks for the info.17:13
jrosserif it’s just a poc I would avoid something you won’t do in prod and is complex17:16
jrossersimpler to have a local acme server to stand in for LE, maybe17:16
jpwi think that's what my plan is except since i'm working in greenfield I don't have anywhere to run vault17:36
jpwit's of so long as i can clearly communicate for x i need y that's good enough for me17:37
noonedeadpunknice - https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/946990 passes now18:15
opendevreviewDaniel Preussker proposed openstack/openstack-ansible-os_swift master: Remove md5 checks of rings/builders  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/94702118:15
opendevreviewDaniel Preussker proposed openstack/openstack-ansible-os_swift master: swift-object-expirer is its own distro package  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/94704818:15
opendevreviewDaniel Preussker proposed openstack/openstack-ansible-os_swift master: Remove seemingly unsused mlocate cronjob  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/94705018:15
jrosserthats great to see18:49
opendevreviewMerged openstack/openstack-ansible master: docs: fix for issues when switching between pages using localization  https://review.opendev.org/c/openstack/openstack-ansible/+/94958321:30
WirednullHi all! In an OSA AIO Dalmatian Ubuntu 24.04 (SCENARIO=aio_lxc_ovs) I still get Linux bridges like br-vxlan and br-vlan (br-mgmt/br-storage/lxcbr0 are fine) while only br-tun/br-provider/br-int are on OVS; shouldn’t the vxlan/vlan bridges be OVS too, or is that expected?21:43
WirednullOh, there are also the other bridges like br-bmaas, br-dbaas, br-lbaas... Shouldn't those also br with OVS from now on?21:48
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94968321:48
WirednullOne more question, with this SCENARIO, the OVN is not included. Correct?21:49
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94968321:49
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: docs: replace broken url to "Getting Started with MariaDB Galera Cluster"  https://review.opendev.org/c/openstack/openstack-ansible/+/94968321:51

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!