Tuesday, 2024-11-19

opendevreviewMerged openstack/openstack-ansible-ops master: Fix Grafana deployment  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/93552010:07
opendevreviewMerged openstack/openstack-ansible-os_nova master: Change ordering of /etc/ operations to improve upgrades  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/93494111:39
opendevreviewMerged openstack/openstack-ansible stable/2023.1: Deny access to any paths including /. for console proxies.  https://review.opendev.org/c/openstack/openstack-ansible/+/93551611:47
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Re-enable uWSGI as default for Neutron  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/93564212:04
opendevreviewMerged openstack/openstack-ansible-os_keystone stable/2023.2: federation: ensure cloud credentials are found on utility host  https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/93453412:14
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.2: Bump SHAs for 2023.2  https://review.opendev.org/c/openstack/openstack-ansible/+/93564712:32
* jrosser looks at the ironic role12:44
jrosseri wonder if the effort for doing nginx->apache there is really so different from adding a skeleton apache role to the plugins repo12:45
jrossernoonedeadpunk: did you think about this apache role at all? there seems a ton of stuff with all the install and mpm setup that some sort of "apache base" code would be helpful12:46
noonedeadpunkI tried to look for something suitable in galaxy but eventually failed to find one that would support multiple quite different vhosts12:48
noonedeadpunkand I'd say we should do the role in it's own repo as well....12:49
jrosserperhaps we can start with something simple, just to do the install and all the boilerplate12:49
jrosserits like a wtf number of lines of code just to take what we put into the repo server and transplant it into the ironic role12:49
noonedeadpunkI beleive we can extract some base thing from ... repo or skyline...12:49
jrosserfor simply just serving12:50
jrosser  /httpboot12:50
jrosseri'm unsure what the most effective thing to do is here.....12:51
noonedeadpunkfor my worklog apache role is kind of "for later this cycle"12:51
noonedeadpunkcommon role makes total sense12:51
noonedeadpunkI tried to use this one and push some proposals, but as you can see - it didn't go well https://github.com/geerlingguy/ansible-role-apache/pull/25612:52
noonedeadpunkbut I was actually thinking of taking it as "base" of a sort12:52
noonedeadpunkbut quite some refactoring would be needed if to do from scratch12:53
jrosserso you think that taking the starting point from the todays repo server may not be the right starting point?12:54
noonedeadpunkand right now I'm quite urged to finalize bgp & ovn thing I left almost year ago :(12:54
jrosser:)12:54
jrosserso i was motivated to fix up the ironic role, and when i look its just a bit 8-O12:54
noonedeadpunkWe need to have a way of supplying very arbitrary vhost configs12:54
jrosserwell, thats why i was thinking to just ignore that totally to start with12:55
jrosserand do the basic setup and install12:55
noonedeadpunkyeah, same when I tried to deal with mpm stuff - each role does things in a very unique way12:55
noonedeadpunkalso EL vs DEB is quite a challenging thing...12:56
jrosseryeah i saw, which was another reason to factor it out as it's just complexity++12:56
noonedeadpunkbut yeah, maybe factoring out bits from repo would be enough...12:57
jrossermaybe we need eventually seperate apache_base and apache_vhost roles or something12:57
noonedeadpunkI thnk we should be able to deal with tasks_from12:58
jrosseryeah of course12:58
jrosserlet me see if i can take the common parts from repo_server12:58
noonedeadpunkas eventually it's gonna be quite alike to haproxy12:58
noonedeadpunkI can ask for a repository for the role today12:59
jrosserok13:00
noonedeadpunkalso good opprtunity to work on murano/senlin deprecation....13:14
kleiniI am stumbling now several times over the same issue with Galera re-deployment. The normal galera-install.yml fails and keeps failing with database out of sync state. I fix that, by placing a .my.cnf with the root database password inside the Galera container. Is my solution right? Is the galera-install.yml supposed to place that .my.cnf file? How should that work correctly?13:53
noonedeadpunkkleini: so eventually we stopped handling `root` password through my.cnf in favor of `admin` user as mariadb dev recommended to not use `root` and leave it alone13:56
noonedeadpunkso it's assumed that `root` should be allowed through socket auth rather then password13:56
noonedeadpunkbut we didn't make any upgrade script to restore root user from password to socket auth13:56
noonedeadpunkyou can also set `galera_root_user: root` instead of current default of `admin`13:57
kleiniokay, then that is the issue. Mysql wants a password for root when using the unix socket13:58
noonedeadpunkthat indeed is the case for upgraded envs....13:59
noonedeadpunkjrosser: ok, I think I got now to understanding about your pointer regarding capi proxy. As now I got that it's needed to access kubernetes API when loadbalancer does not have floating IP assigned, right?14:04
kleinimysql_native_password must be auth_socket14:04
noonedeadpunkI can't recall the command altering user tbh... but yeah, default for root is not `mysql_native_password` for mariadb nowadays14:05
jrossernoonedeadpunk: yes - the capi proxy lets the control plane k8s call to the workload k8s API endpoint when either no fip, or isolated control plane14:06
jrosserand also there is different deployment needed for OVN / not-OVN14:07
jrosserit must be on network nodes for not-OVN, and all computes for OVN14:07
noonedeadpunkbut how the heck does it work? As  worker cluster has it's own network14:07
noonedeadpunkaha....14:07
jrosserif you look in the generated haproxy config, the backend server configs are put into the right nertwork namespaces14:08
* noonedeadpunk playing in ovn env there14:08
jrosserthere are also env vars, to allow you to always use the proxy, if you want that behaviour instead of ever using the fip14:09
jrossersee here for OVN, you can override the playbook target https://github.com/openstack/openstack-ansible-ops/blob/master/mcapi_vexxhost/playbooks/mcapi_proxy.yml#L2314:10
kleinisorry, it must not be auth_socket but unix_socket14:10
noonedeadpunkyeah, trying to find in code how it does connect to networks in OVN14:11
jrosserright so afaik on the compute node that has the vm, you do have the namespace you need14:11
jrosserfor dhcp or something14:11
noonedeadpunkthe octavia VM? Or it doesn't go through the LB then?14:12
jrosserthis is nothing to do with octavia14:12
noonedeadpunkiirc there's only metadata namespaces....14:12
noonedeadpunkaha14:12
jrosserright, so this leverages those14:12
noonedeadpunkso it doesn't spawn own namespaces, but rely on existing ones14:12
jrossercorrect14:12
noonedeadpunkok, that does make sense14:12
noonedeadpunkjsut trying to asses if I'd prefer to hide in horizon menu to spawn without fip or rolling out proxy is safe enough :D14:13
jrosserwell do consider that allowed_cidr in the cluster template is ignored14:14
jrosserwe patched magnum-cluster-api to fail creation if a FIP was enabled14:14
jrosserhttps://github.com/vexxhost/magnum-cluster-api/issues/45414:15
noonedeadpunkalso AZs are not respected....14:15
noonedeadpunkwhen there's more then 1 AZ supplied one in Horizon is simply ignored as well14:15
jrosseri would say that for use cases that are not private, FIP is not safe14:15
jrosserbut that depends on local circumstances i guess14:15
noonedeadpunkyeah. true14:15
jrosserso we have ours set to always proxy14:15
noonedeadpunkhow you do that? as I don't see variables in the role...14:17
noonedeadpunk`os.getenv("PROXY_ALWAYS", 0) == 0`14:17
noonedeadpunkfound it14:17
jrosserhttps://paste.opendev.org/show/bSsnR74uaiydKGEKzHFA/14:18
noonedeadpunkonce I get it working, will try to document it :)14:18
noonedeadpunkand move that var to defaults....14:19
noonedeadpunkokay, perfect, thanks!14:19
jrosserit might be that POD_IP is mandatory and i have forgotton it from the code/vars14:19
jrosserthis origianted from a service running in k8s of course so it needed a little work to be OK outside that14:20
jrosserthen we fixed the port to allow iptables to be possible, rather than random port selection14:20
jrosserhttps://github.com/bbc/magnum-cluster-api/commit/5fdc755aca57d0dc662ff994b74367d99c6874ae14:23
kleinimany thanks, by changing the root user in Galera I don't have issues with galera-install.yml anymore14:24
jrosserkleini: this all came about because the old code adjusted the permissions of the root user, and with some new release of mariadb that had really strange side effects resulting in brokenness14:29
jrosserthe advice was to never touch the root user, which is how the `admin` user came to exist now14:30
jrosserit does sound like we miss a step somewhere in an upgrade script though?14:31
kleiniALTER USER 'root'@'localhost' IDENTIFIED WITH unix_socket; GRANT ALL ON *.* TO 'root'@'localhost' WITH GRANT OPTION; DROP USER 'root'@'127.0.0.1'; DROP USER 'root'@'::1'; DROP USER 'root'@'%';14:44
kleinithat was necessary for my database initially created with Stein or Train.14:44
noonedeadpunkyeah, I had same issue back then, but already can't recall when it was. I think on 2023.1 when upgrading ubuntu from 20.04 to 22.0414:56
noonedeadpunkso yes, I think we've missed covering that back then14:56
noonedeadpunkbut also I probably won't dare to adjust root user with playbook anyway14:56
noonedeadpunkas who knows if there were some other usecases in deployment with it...14:57
noonedeadpunk#startmeeting openstack_ansible_meeting15:00
opendevmeetMeeting started Tue Nov 19 15:00:27 2024 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'openstack_ansible_meeting'15:00
noonedeadpunk#topic rollcall15:00
NeilHanlono/15:00
noonedeadpunko/15:00
NeilHanlonhello from SuperComputing 202415:00
noonedeadpunkwow, sounds quite thrilling15:00
noonedeadpunk#topic office hours15:02
noonedeadpunkso, we have landed a CI change to address intermittent rocky issues15:02
noonedeadpunkuntil infra mirrors will be in place15:02
noonedeadpunkso instead of mirrorlist we're using a specific mirror15:03
* jrosser has other meeting :(15:04
noonedeadpunkhttp://dl.rockylinux.org/ to be specific15:04
noonedeadpunkNeilHanlon: that's smth we can trust and bother someone from your team if it's desynced, right?:)15:05
noonedeadpunkalso it seems that with switch to mirror.mariadb.org centos is broken15:07
noonedeadpunk#link https://zuul.opendev.org/t/openstack/build/5d488593758d4945b8e3efeaa9625ab3/log/logs/etc/host/yum.repos.d/MariaDB.repo.txt15:07
noonedeadpunkas repo doesn't have `centos-9` path - https://mirror.mariadb.org/yum/11.4/15:08
noonedeadpunkit's centos/9 instead....15:08
noonedeadpunkand then it's not amd64 but x86_64 /o\15:08
mgariepyhello15:08
noonedeadpunko/15:09
noonedeadpunkeventually all distros except rocky and alma support both ways15:10
noonedeadpunkand I wonder if we can do just `rhel` for both centos and rocky....15:11
noonedeadpunkso smth to look at at least...15:11
noonedeadpunkwe've discussed today shortly need for apache role and repo for it15:11
noonedeadpunkwith this I was having a question - how to name it?15:12
noonedeadpunkansible-role-apache?15:12
noonedeadpunkansible-role-httpd?15:12
noonedeadpunkI'm inclined to the second option as apache is confusing/broad name15:12
noonedeadpunkbut not sure15:12
mgariepyeither suits me not much to say for or against.15:13
noonedeadpunkjrosser: any opinion?15:14
noonedeadpunkon ansible-role-(apache|httpd)15:14
noonedeadpunkwith asking the new repo will also work on sunsetting murano/senlin. and probably reviving watcher15:15
noonedeadpunkToday also looked/briefly tested Neutron uWSGI mode, which seems to be finally fixed for 2024.215:17
noonedeadpunkand also for OVN15:17
noonedeadpunk#link https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/93564215:17
noonedeadpunkit doesn't seem to work for distro path though....15:18
noonedeadpunkthere're simply no such services so far: https://zuul.opendev.org/t/openstack/build/26fc4af095b34c6eaa8d47772bfc3af4/log/logs/host/neutron-ovn-maintenance-worker.service.journal-12-56-05.log.txt15:20
noonedeadpunkcould be missing some packages....15:20
noonedeadpunkbut well... jammy is also having old packages iirc....15:21
noonedeadpunkit's still 2024.1 or so15:21
noonedeadpunk#link https://zuul.opendev.org/t/openstack/build/26fc4af095b34c6eaa8d47772bfc3af4/log/logs/etc/host/apt/sources.list.d/uca.sources.txt#3415:21
noonedeadpunklikely we need to add the noble distro job as well15:22
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Re-enable uWSGI as default for Neutron  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/93564215:25
noonedeadpunkeventually... distro method is marked as unsupported for 2024.2 for jammy anyway...15:25
noonedeadpunkbut will double-check on that15:28
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Switch Ubuntu distro jobs to 24.04  https://review.opendev.org/c/openstack/openstack-ansible/+/93566415:34
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Re-enable uWSGI as default for Neutron  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/93564215:34
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Use rhel packages for all EL-based distros  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/93566515:41
noonedeadpunkI am also failing to work on rabbitmq force_bootstrap flag and further refactoring of haproxy role15:41
noonedeadpunkwith that time for branching 2024.2 is coming really rapidly15:42
jrosseransible-role-httpd is fine15:42
noonedeadpunkso I guess I would suggest to land what we have today and branch. and backport whatever we decide we need after branching15:42
noonedeadpunkas I don't see too much of backports, except if you wanna land apache/httpd role this cycle jrosser15:43
jrosserif we can get something simple for a base config i think that would be beneficial15:48
jrosseri have already cut/paste from the repo server into a temporary repo and have some time in a AIO now to integrate it15:49
noonedeadpunkaha, ok... then question is - how fast repo will be created...15:49
jrossereven if we leave generation of vhost till some later time15:49
noonedeadpunkok, let's try15:49
noonedeadpunkwe heave release deadline in 2 weeks15:50
jrosserhmmm this is maybe somewhat optimistic15:50
noonedeadpunkyeah15:50
jrosserwhat is for sure is that ironic role is broken for metal deploys15:50
jrosseron rocky/centos15:50
jrosserso perhaps there is some quick fix there15:51
jrosseras an alternative15:51
noonedeadpunkquick fix sounds like what we need now....15:51
jrosserexample is here https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/93301015:52
jrosserand fairly obvious fail to re-use port 80 https://zuul.opendev.org/t/openstack/build/52b7f07374cb4f22832cf22fe74c3593/log/logs/host/nginx.service.journal-14-26-18.log.txt15:53
noonedeadpunkI guess that's could be just a different path? https://opendev.org/openstack/openstack-ansible-os_ironic/src/branch/master/tasks/ironic_conductor_post_install.yml#L102-L10815:56
noonedeadpunkah https://zuul.opendev.org/t/openstack/build/196b2de293234c00a9aadba5868a0afc/log/logs/etc/host/nginx/nginx.conf.txt#39-4015:56
noonedeadpunkmaybe some kind of lineinfile to replace 80 with management_address....15:57
noonedeadpunkubuntu simply doesn't contain server{} section in nginx.conf15:58
jrosserhttps://github.com/openstack/openstack-ansible-os_ironic/blob/8980eacd6736a0df3f6259b03a17793494ecb2cb/tasks/ironic_conductor_post_install.yml#L102-L10815:58
noonedeadpunkit works for ubuntu, yeah. but on EL default is part of /etc/nginx/nginx.conf15:59
jrosseris this that we need to handle the path to the conf file differently on ubuntu/rh?15:59
jrosseroh you mean that the conf file is merged16:00
noonedeadpunkyeah16:00
noonedeadpunk#endmeeting16:01
opendevmeetMeeting ended Tue Nov 19 16:01:54 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:01
opendevmeetMinutes:        https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-11-19-15.00.html16:01
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-11-19-15.00.txt16:01
opendevmeetLog:            https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-11-19-15.00.log.html16:01
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Retire murano/senlin/sahara  https://review.opendev.org/c/openstack/openstack-ansible/+/93567016:12
jrosserthat really is made hard to not have the default server on rh nginx16:23
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Retire Senlin role  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/93567316:24
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Retire Murano role  https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/93567416:26
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_sahara master: Retire Sahara role  https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/93567516:27
noonedeadpunkthey've tried their best16:35
noonedeadpunkactually, I think it's about same in apache?16:35
noonedeadpunkah, no16:36
jrossermaybe we could place a systemd dropin to override with `ExecStart=/usr/sbin/nginx -c /path/to/our/own/config`17:30
noonedeadpunkhehe, we can do that, sure17:31
noonedeadpunkwon't actually some kind of `replace` work? or issue is that they do have multiple records for v4 and v6?17:31
jrosserthere is two records yes, but we could replace17:31
jrosseris that safe when the package is upgraded?17:32
noonedeadpunkshouldn't a current version be kept by default if upgraded with ansible?17:33
noonedeadpunkdunno how it's in rhel though...17:33
noonedeadpunkhm. we've discussed apache role on some ptg, didn't we?17:51
jrosserperhaps we did18:04
jrosserfor now i am making a rocky9 aio and i think that the systemd drop in will be the most robust18:04
noonedeadpunkI've proposed patches for makingapache repo in the meanwhile18:05
noonedeadpunkhm, seems smth is off even with http://dl.rockylinux.org/18:08
jrosseri'm also just having my second odd failure here doing a manual aio18:09
noonedeadpunkhttps://zuul.opendev.org/t/openstack/build/6cf414ee1e344890af64f38ece47fb9c18:09
noonedeadpunkand https://zuul.opendev.org/t/openstack/build/3d849db128914961b2a641ec1cd03d1618:09
jrosseroh dear18:09
noonedeadpunkhere is netcat https://zuul.opendev.org/t/openstack/build/7fa360a9baf2409eb934650caf1d514718:13
noonedeadpunkNeilHanlon: any guess wtf going on with rocky mirrors?:)18:13
jrosserthis is not right https://paste.opendev.org/show/bprusvHe3Jg7Yp8SZhSL/18:15
noonedeadpunkare you testing with https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934060 ?18:17
noonedeadpunkso somehow this condition is not passing?18:20
jrosseri just am trying master18:20
noonedeadpunkbut we don't have 4.0 merged there....18:20
noonedeadpunk`stdout': '4.0.3'`18:21
noonedeadpunko_O18:21
jrosserright18:21
noonedeadpunkum...18:22
jrosserhttps://paste.opendev.org/show/bqtANv0pXO6J5oVPwJRu/18:25
jrosserthats almost like only the contents of the system repo were known at the point it installed rabbitmq-server18:32
jrosseri just removed the package, re-ran the rabbitmq playbook and the version is now as it should be18:33
noonedeadpunkso we have some kind of order issue?18:38
noonedeadpunkI'm also amazed how bravely on "stable" release they've bumped rabbitmq to 4.018:39
noonedeadpunkas happening this to me on production - I would be very-very-very unhappy and confused....18:39
jrosseri booted a rocky 9 vm - perhaps not the newest image18:42
jrosserjust ran setup-everything18:42
jrosserthat thats what i got18:42
jrosseri'm perhaps not knowing enough rocky stuff to understand if the older 9.x image is a reasonable thing to use18:43
noonedeadpunkyeah, dunno what to say. that's indeed super confusing18:43
noonedeadpunkI think it shouldn't matter if it doesn't have rabbit pre-installed.18:43
noonedeadpunkbut still it's likely being installed inside the role18:44
noonedeadpunkcould be as some dependency....18:44
jrosseryep - i can try again tomorrow but for now i fixed it and carry on18:44
noonedeadpunkwould be reely good to understand if this is reproducible or not...18:53
noonedeadpunkas https://zuul.opendev.org/t/openstack/build/72a3705c1352473bb8582a29a3572eea/log/logs/openstack/aio1-rabbit-mq-container-0199806c/dnf.rpm.log.txt says it should be fine.....18:54

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!