Tuesday, 2022-01-25

noonedeadpunkdoh, gerrit got upgraded again06:57
noonedeadpunkI almost get used to 3.0 :p06:58
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Set infra_lxc job to centos-8-stream  https://review.opendev.org/c/openstack/openstack-ansible/+/82613507:07
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd master: Add integrated linters test  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/79903807:13
noonedeadpunkmorning everyone \o/08:04
damiandabrowski[m]hey!08:04
jrossergood morning08:12
*** dviroel|out is now known as dviroel11:20
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd master: Add integrated linters test  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/79903811:29
*** sshnaidm|afk is now known as sshnaidm12:28
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-plugins master: Add ssh_keypairs role  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/82511313:00
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-plugins master: Add ssh_keypairs role  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/82511314:23
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Create ssh certificate authority  https://review.opendev.org/c/openstack/openstack-ansible/+/82529214:23
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Create ssh certificate authority  https://review.opendev.org/c/openstack/openstack-ansible/+/82529214:51
noonedeadpunk#startmeeting openstack_ansible_meeting15:00
opendevmeetMeeting started Tue Jan 25 15:00:28 2022 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'openstack_ansible_meeting'15:00
noonedeadpunk#topic rollcall15:00
noonedeadpunk\o/15:00
damiandabrowski[m]hey!15:00
*** sshnaidm is now known as sshnaidm|afk15:01
jrosserhello15:02
spatel\o/15:02
noonedeadpunk#topic bug triage15:04
noonedeadpunkwe have several new bugs lately 15:04
noonedeadpunkthey seem pretty simple though15:04
noonedeadpunkadmin16: sorry for stupid question, but aren't you admin0 for co-incedence ?:)15:05
noonedeadpunkas regarding https://bugs.launchpad.net/openstack-ansible/+bug/1955676 would be great to have a descision if it's go or no go15:06
noonedeadpunkOther then that, there's other interesting bug15:07
noonedeadpunk#link https://bugs.launchpad.net/openstack-ansible/+bug/195864515:07
noonedeadpunkI'm not sure I see why comma should be added there?15:07
noonedeadpunkas we put it inside horizon_local_settings, which is regular python code15:08
spatelseems typo 15:09
noonedeadpunkI'm not sure it does correct thing this way though https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/templates/horizon_local_settings.py.j2#L92-L9315:09
jrosserhuh https://opendev.org/openstack/horizon/src/branch/master/doc/source/configuration/settings.rst#openstack_keystone_domain_choices15:11
jrosserwhat is that all about15:11
noonedeadpunkI haven't used horizon for quite a while, but are choices even a required thing?15:11
noonedeadpunkI mean - that could be even avoided by default, or...?15:12
jrossersame thing appears to be the case for AVAILABLE_THEMES and another few things in there15:12
noonedeadpunk`option can be used`15:12
noonedeadpunkAnd if it's not defined - would all domains appear by default....15:13
noonedeadpunkI guess not, as default setting would prevent it I believe...15:13
jrosserwell - seems our code does not match the horizon docs either way15:13
noonedeadpunkwell that fully worked for me on T though...15:14
noonedeadpunkBut yes, agree, we need to change that15:15
jrosserhttps://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/horizon/templates/local_settings.j2#L10215:15
noonedeadpunkyeah, fair15:16
noonedeadpunkANother trivial bug 15:17
noonedeadpunk#link https://bugs.launchpad.net/openstack-ansible/+bug/195843415:17
noonedeadpunkI think that's also fair solution suggested?15:17
spatelhow does CI pass then? because we don't have that role for CI job?15:19
jrosserprobably becasue we do a metal deploy and it picks it up off the host15:19
noonedeadpunkYeah15:19
jrossertheres quite a few lxc<>metal bugs occur that way15:19
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Fix default multidomain choises  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/82631015:22
noonedeadpunk#topic office hours15:22
noonedeadpunkOh, backport candidate label has just been merged 15:23
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-os_aodh master: Ensure libxml2 is installed on debian systems  https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/82631115:24
jrosserdo we abandon these https://review.opendev.org/q/topic:reno-ocata ?15:24
noonedeadpunkyep15:25
noonedeadpunkit was really nasty side effect of EOLing integrated repo...15:25
jrosserandrewbonney: we need this for centos-8 removal https://review.opendev.org/c/openstack/openstack-ansible/+/82613515:25
noonedeadpunkSo back in days integrated and role repos were defined in single file in releases15:25
noonedeadpunkSo roles were eoled, and I decided to eol integrated as follow-up, once a-r-r will be switched to eol tag as well15:26
noonedeadpunkbut when adjusting list, CD decided to create branch on every repo, and then delete it....15:27
noonedeadpunkspawning tons of rubbish15:27
noonedeadpunkbtw returning back to horizon, there's one patch that is interesting I guess https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/82092815:28
noonedeadpunkI tested it in aio both for distro and source with some extra panel (like heat?). Source worked nicely, but for distro iirc panel wasn't added for $reason. It actually wasn't added even with current state of role, so might be issue with panel itself that is packaged...15:30
noonedeadpunkbut not sure15:30
noonedeadpunkAnd I wasn't able to take a look on CI stuff and how to re-use that efficiently during previous week15:31
jrosserno worries15:32
jrosseri have been out for a few days too15:32
jrosserjust managed to pick up the keypairs stuff again today15:32
noonedeadpunkOh, would be awesome to merge https://review.opendev.org/q/topic:integrated_lintersbtw15:32
jrosserandrewbonney: looks like more for you ^15:32
noonedeadpunkYeah, I saw you pushed changes, will hopefully review them tomorrow morning15:32
andrewbonney:)15:32
noonedeadpunkas well as damiandabrowski[m] to tempest15:33
jrosserso on the keypairs stuff - i *think* it will work for nova on a new deployment, though i've only tested AIO15:33
noonedeadpunkwill it for upgraded one ?:)15:33
jrosserthere is code to undo all the old config, but i need to add handling for existing sshd config which may conflict with what this new role wants to do15:33
jrosseri did add stuff to remove the old ssk keys, so it will probably work15:34
jrosserbut similar to SSL stuff you probably need to have run the play against all the hosts15:34
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Define variables for tempest plugins  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/82612115:36
*** dviroel is now known as dviroel|lunch15:37
noonedeadpunkI think it's quite fair to distribute across all computes.15:37
noonedeadpunkThankfully this should be done one time and could be run with limit afterwards?15:38
jrosseryes, i think with this method there is no coupling between the compute hosts at all15:38
jrosserwhere are we with the centos-8 removal15:38
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Define variables for tempest plugins  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/82612115:39
jrosserxena is broken on tempest vars?15:39
jrosservars/requirements15:39
noonedeadpunkX is broken with neutron plugin version15:40
noonedeadpunkEventually what I realized is that starting from X tempest is not in u-c anymore15:40
noonedeadpunkSo I got fed with that and decided that it's worth fixing plugins versions as well15:41
noonedeadpunkWith removal as far as I got, chain looks like - we merge https://review.opendev.org/c/openstack/openstack-ansible/+/826135, then we should able to merge https://review.opendev.org/c/openstack/openstack-ansible/+/824567 and https://review.opendev.org/c/openstack/openstack-ansible/+/824570 likely go last15:43
jrosseri need to look again at the centos-9 patches15:47
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest stable/xena: Define variables for tempest plugins  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/82606015:47
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-galera_server master: Convert xinetd clustercheck to systemd socket service  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/82404215:48
jrosser^ this is important if the proxysql stuff might take a while15:48
noonedeadpunkIt's in merge conflict now15:51
jrosserok no worries 15:51
jrosseri think i need to test it locally on a fresh AIO15:51
jrossersomething wierd is not working15:51
noonedeadpunkMain issue with proxysql, that I'm a bit struggling with, is that it requires users to be created for itself as well https://proxysql.com/documentation/users-configuration/ and all this layering of config that is applied...15:52
noonedeadpunkSo it's a bit fishy and would require big changes and some migration...15:52
noonedeadpunkBut honestly time is main thing that I'm struglling for real :)15:53
*** sshnaidm|afk is now known as sshnaidm15:55
noonedeadpunkAnd now I can finally work on updating dashboard to include BC15:59
spatelnoonedeadpunk I am working on W - X upgrade in my lab 16:01
noonedeadpunk#endmeeting16:01
opendevmeetMeeting ended Tue Jan 25 16:01:50 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:01
opendevmeetMinutes:        https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-01-25-15.00.html16:01
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-01-25-15.00.txt16:01
opendevmeetLog:            https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-01-25-15.00.log.html16:01
spatelShould i be worried of anything? like mysql SSL stuff?  (i am using F5 loadbalancer so thinking to disable it during upgrade otherwise it will break some stuff)16:02
noonedeadpunkI think no - it's lab after all :)16:03
noonedeadpunkbut as I said, there were some improvement from 24.0.0 that should have landed with https://review.opendev.org/c/openstack/openstack-ansible/+/82539116:04
spatelin lab i am not worried but i am going to create process from lab so in production i don't see any hiccups 16:04
spatelagreed about 24.1.0 but i am just testing upgrade process in multinode environment so make sure nothing missed 16:05
spatelnot worried about improvement 16:06
spatelwhat is the deal with mysql connection?16:07
spateli meant connection_recycle_time ? 16:08
opendevreviewMerged openstack/ansible-role-qdrouterd master: Use integrated linter jobs  https://review.opendev.org/c/openstack/ansible-role-qdrouterd/+/82583916:08
noonedeadpunkUm, and what's with it?16:09
noonedeadpunkyou mean that we implemented new variable to have better control over it?16:10
spatelI am asking what value i should be using in production? 16:10
spatelI have 300 compute nodes so its kinda large deployment 16:10
noonedeadpunkwell computes are not connecting to galera ;)16:11
spatelbut does it impact with message rate?16:11
noonedeadpunkBut it was only an issue when haproxy restarted a lot of times in a row, you could get up to max_connections easily16:11
spatelmy question is why does that setting came in picture ?16:11
spatelhmm! i have F5 ... 16:12
noonedeadpunkI'm not sure it matters as it's still l3 balancer?16:13
spatelyes16:13
spateli have seen lots of folks using very large value here like connection_recycle_time = 1500 16:13
noonedeadpunkyeah, so connections are not killed properly, and with previous default value if you restart haproxy like 5 times in an hour, you will catch stuck mysql because of not having spare connections16:14
noonedeadpunkBut it was always like that. So it's more an improvement16:14
spatelcopy that16:15
noonedeadpunkAnd giving controll over value in a nice way16:15
spatelso let me not touch it 16:15
spatelI am keeping SSL off for mysql just for safely 16:15
noonedeadpunkwhile generally it's decreased, you now can set `openstack_db_connection_recycle_time` and it would result if adjusting value for all services16:15
spatelnice universal variable :)16:16
jrosserspatel: at some point we might default to ssl everywhere16:16
jrosserand then start to make it the only option16:16
jrosserso changing the defaults maybe only stores up trouble for later16:17
jrosserlike with rabbitmq, you just have no choice now16:17
noonedeadpunk(it wasn't our call to have that said)16:17
noonedeadpunkeventually role still provides choice?16:18
jrosserno, but i am concerned about how we might support a mixture of SSL and non-SSL, and the migration between them in the long term for everything16:18
noonedeadpunkonce it's matter of including or not including pki role?16:19
noonedeadpunkah, yes, we have configs as well..16:19
jrosseryes, it's pretty widespread change16:19
noonedeadpunkbut yes, I agree that in general having mysql covered with TLS is a good thing overall16:19
jrosserthis may be a good PTG topic16:20
jrosserjames gibson has a spec for making the internal VIP https16:20
jrosserbut that is looking like it might be something that you do during an upgrade, as the intermediate http&https setup in parallel is pretty ugly16:21
spateljrosser :) ssl is good but bad for troubleshooting but i agreed for security 16:26
spatelI still need to work on OVN SSL stuff which i totally drop but i am back now 16:26
spatelOne more question currently if someone want to deploy OVN then they need to do export SCENARIO='aio_ovn_lxc'   can we create variable or something like neutron_agent_plugin=ovn  or openvswitch or linuxbridge ? 16:29
jrosserSCENARIO is only ever used for automatically configuring the AIO16:30
jrosserfor real deployments then variables need to be set16:30
spatelI am talking about production 16:30
spatel3 node 16:31
spatelwe don't have any good variable for that16:31
noonedeadpunkum, there's such variable `neutron_plugin_type: ml2.ovn`16:31
jrosserwe do16:31
jrosserhttps://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/templates/user_variables_neutron_ovn.yml.j216:31
jrosser^ that is what the ovn AIO scenario does, it applys that set of vars16:31
spatelhmm oh 16:32
jrossercheck out what is in that directory16:32
noonedeadpunkIt's even documented here https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html#openstack-ansible-user-variables16:32
jrosserall the different scenarios we test in CI just grab one of those files16:32
spateli misunderstood that :) sorry16:32
spatelCan we add small snippet about OVN in aio build? like if someone want to build OVN then do export SCENARIO='aio_ovn_lxc'   16:33
spatelexport SCENARIO='aio_ovn_lxc'  16:33
spatelsorry here - https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html16:34
spateli don't think other new folks know that we do have ovn support 16:35
noonedeadpunkwe have network scenarios listed here https://docs.openstack.org/openstack-ansible-os_neutron/latest/16:37
noonedeadpunkbut we probably want to drop this red warning and experimental support one day16:38
noonedeadpunk*about experimental16:38
*** dviroel|lunch is now known as dviroel16:38
spatelYes, if someone visit openstack-ansible and try to build AIO so just have option on AIO page that you can build with OVN for experimental 16:39
noonedeadpunkwell... we kind of need to maintain list of all scenarios then I'd say16:40
noonedeadpunkwhich I'm not sure how to achieve right now16:40
spatelotherwise we never every going to get bug report if folks not going to try :(16:40
noonedeadpunkI do agree here16:41
spatelall put all senario in one page and create symlink to AIO offical page16:42
spatellook like we need some cleanup on AIO page, because some of stuff they are just extra and old like this note  - configure: error: cannot run C compiled programs. 16:42
spatelI never ever get that error in AIO 16:43
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-plugins master: Add ssh_keypairs role  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/82511317:09
opendevreviewMerged openstack/ansible-role-systemd_networkd master: Add integrated linters test  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/79903817:13
opendevreviewMerged openstack/ansible-role-systemd_mount master: Add integrated linters test  https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/79903617:24
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use sysctl ini-like config file  https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/82633817:26
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use sysctl ini-like config file  https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/82633817:27
opendevreviewMerged openstack/openstack-ansible-os_horizon master: Refactor use of include_vars  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/82433117:30
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use journald logging for RabbitMQ  https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/82634517:40
opendevreviewMerged openstack/openstack-ansible-ceph_client master: Add integrated linters test  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/79881617:41
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not run rsyslog against RabbitMQ  https://review.opendev.org/c/openstack/openstack-ansible/+/82634717:42
opendevreviewMerged openstack/openstack-ansible stable/wallaby: Set infra_lxc job to centos-8-stream  https://review.opendev.org/c/openstack/openstack-ansible/+/82613517:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Define variables for tempest plugins  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/82612118:10
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest stable/xena: Define variables for tempest plugins  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/82606018:11
opendevreviewMerged openstack/openstack-ansible master: Fix definition of ssl_protocol  https://review.opendev.org/c/openstack/openstack-ansible/+/82399018:19
opendevreviewMerged openstack/openstack-ansible master: Add defaults for TLS 1.3 and rename TLS <= 1.2 variable  https://review.opendev.org/c/openstack/openstack-ansible/+/82394318:19
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Change default mode while creating directories  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/82477418:21
jamesdentondoes OSA have the ability to specify an FQDN for external_lb_vip_address that is compatible with a NAT?18:27
jamesdentonmaybe haproxy_bind_external_lb_vip_address is the trick18:28
jrosseri think i have done this18:42
jrosserhmm18:42
jrosseryes, haproxy_bind_external_lb_vip_address was the natted address18:45
jrosserand external_lb_vip_address was the floating ip18:45
jamesdentonyep, or in my case, the FQDN that points to the public IP18:46
jrosser(this is an AIO on a private network in an openstack project with a neutron floating IP)18:46
jamesdentonhaproxy was trying to bind to that18:46
jamesdentoni see18:46
jrosserit even managed to get itself an LE certificate18:46
jamesdentonhaproxy_bind_external_lb_vip_address worked, thank you18:46
jamesdentonreeeeeally18:46
jrosseryup18:47
jrosserreal dns entry required somewhere, but yes it worked18:47
jamesdentonaye18:47
jamesdentondo you know if we have the ability to drop root CAs onto the host? for corp-signed certs?18:48
jrosserhttps://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/defaults/main.yml#L171-L17518:48
jamesdentonmy hero, thank you18:49
jrossernp18:49
opendevreviewMerged openstack/openstack-ansible-os_tempest master: Deprecate congress and lxd tempest plugins  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/82611118:56
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-galera_server master: Convert xinetd clustercheck to systemd socket service  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/82404219:08
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-galera_server master: Convert xinetd clustercheck to systemd socket service  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/82404221:12
*** dviroel is now known as dviroel|afk21:38
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-os_horizon master: Fix default multidomain choices  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/82631022:28

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!