| *** spotz is now known as Guest2040 | 01:32 | |
| *** spotz_ is now known as spotz | 01:32 | |
| opendevreview | Neil Hanlon proposed openstack/openstack-ansible master: WIP - Add support for running on Rocky Linux https://review.opendev.org/c/openstack/openstack-ansible/+/823573 | 04:29 |
|---|---|---|
| jrosser | good morning | 07:42 |
| jrosser | new dashboard looks tidy | 07:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_container_create stable/victoria: Remove legacy centos-8 jobs https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/827960 | 07:45 |
| noonedeadpunk | \o/ | 08:03 |
| noonedeadpunk | well the only issue with backports candidates is that I dunno how to hide ones that already got cherry-picked | 08:03 |
| noonedeadpunk | the nice thing is that with infra hint I was able to re-arrange all osa repos as "child" projects in gerrit, so now `parentproject:openstack/openstack-ansible` works as a charm and no need in nasty stuff | 08:05 |
| jrosser | that makes it much easier | 08:05 |
| jrosser | i think we are only one or two patches from finishing the centos-8 remove | 08:06 |
| noonedeadpunk | huh? I thought we merged everything... | 08:15 |
| noonedeadpunk | Ah, we just missed topic on them :) | 08:15 |
| jrosser | you can check on here also whats left https://zuul.openstack.org/jobs | 08:20 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Convert infra-journal-remote playbook to role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/824731 | 09:12 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Move infra-journal-remote logic to its role https://review.opendev.org/c/openstack/openstack-ansible/+/824734 | 09:28 |
| opendevreview | Merged openstack/openstack-ansible-lxc_container_create stable/victoria: Remove legacy centos-8 jobs https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/827960 | 09:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/xena: Replace CentOS 8 with Stream jobs https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828095 | 09:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/wallaby: Replace CentOS 8 with Stream jobs https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/827966 | 09:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/victoria: Drop CentOS 8 jobs https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828096 | 09:43 |
| noonedeadpunk | seems these are last ones ^ | 09:43 |
| jrosser | oh dear | 10:04 |
| jrosser | widespread centos-8-stream failure fatal: [aio1]: FAILED! => {"changed": false, "cmd": "ifup lxcbr0", "msg": "[Errno 2] No such file or directory: b'ifup': b'ifup'", "rc": 2, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} | 10:05 |
| opendevreview | Merged openstack/openstack-ansible-os_gnocchi master: Remove apache to uwsgi migration tasks and variables https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/827439 | 10:29 |
| *** dviroel|out is now known as dviroel|river | 10:58 | |
| *** dviroel|river is now known as dviroel|rover | 10:58 | |
| *** dviroel|rover is now known as dviroel|ruck | 11:01 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Ensure that the legacy network-scripts package is present https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828113 | 11:32 |
| jrosser | ^ guesswork - lets see what happens..... | 11:35 |
| noonedeadpunk | I wonder if it's easier just to replace that with `ip link set {{ lxc_net_bridge } up` | 11:38 |
| noonedeadpunk | As ifupdown is non default for any OS as of today | 11:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Clean up bionic variables https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828114 | 11:44 |
| noonedeadpunk | btw would be awesome to get this rebased and merged https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/731228 | 11:46 |
| jrosser | oh right yes there really is some old stuff to clean up there | 12:03 |
| *** dviroel|ruck is now known as dviroel|ruck|afk | 12:23 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Replace ifupdown with native ip-link https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828120 | 12:30 |
| jonher | i just deployed barbican in wallaby and got some errors loading the horizon panel, so i asked in #openstack-barbican and apparently the current upstream code for barbican-ui is just a template and not something that's funcitoning, should we remove barbican-ui from OSA to avoid confusion for others? | 13:16 |
| jonher | apparently there has been a review open for quite some time now: https://review.opendev.org/c/openstack/barbican-ui/+/702399 to add basic functionality, but no reviewers that know enough horizon | 13:17 |
| jrosser | andrewbonney: https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828113 | 13:27 |
| noonedeadpunk | jonher: mayabe we just set `horizon_enable_barbican_ui: False` here regardless of barbican group length? https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/defaults/main.yml#L298 | 13:40 |
| noonedeadpunk | with comment why we're doing this | 13:41 |
| jonher | sure, that seems reasonable | 13:42 |
| noonedeadpunk | want to push a patch for that?:) | 13:43 |
| jonher | will do :) | 13:43 |
| *** dviroel|ruck|afk is now known as dviroel|ruck | 14:03 | |
| opendevreview | Jonathan Herlin proposed openstack/openstack-ansible-os_horizon master: Disable barbican-ui dashboard https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/828154 | 14:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Clean up bionic variables https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828114 | 14:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Replace ifupdown with native ip-link https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828120 | 14:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Replace ifupdown with native ip-link https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828120 | 14:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Replace ifupdown with native ip-link https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828120 | 14:44 |
| NeilHanlon | hooo.. that was a weekend of fun getting an AIO running on Rocky linux lol | 14:44 |
| noonedeadpunk | was it that bad? | 14:45 |
| NeilHanlon | i don't even know why i hit so many roadblocks, somehow I feel it's something about me.. I kept running into missing repositories inside containers, but I think much of that was from the removal of the centos-8 data from openstack-ansible-lxc_hosts / lxc_container_create | 14:46 |
| NeilHanlon | some of it is definitely just learning pains of me getting more comfortable with all the pieces, which is necessary, ultimately | 14:49 |
| noonedeadpunk | you could spawn metal aio only, not to care about containers | 14:50 |
| noonedeadpunk | we usually get metal working before even starting LXC for new distros | 14:50 |
| noonedeadpunk | and it's all about jsut replacing s/lxc/metal/ in $SCENARIO | 14:51 |
| NeilHanlon | i'll spin one up in metal mode today, see if that flies any better | 14:51 |
| spatel | NeilHanlon do we have LXC offical image for rocky ? | 14:52 |
| spatel | not lxc but disk image to spin up lxc container | 14:52 |
| NeilHanlon | cannonical builds ones here: https://us.lxd.images.canonical.com/ | 14:53 |
| NeilHanlon | i'm awaiting this to be merged for DIB https://review.opendev.org/c/openstack/diskimage-builder/+/825957 | 14:53 |
| noonedeadpunk | spatel: we don't actually need this? as we build using dnf nowadays | 14:54 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/vars/redhat.yml#L16 | 14:55 |
| noonedeadpunk | Rocky totally needs some different command | 14:56 |
| NeilHanlon | so that's an interesting thing, too... that command isn't run until after it makes images 🤔 | 14:56 |
| spatel | noonedeadpunk really? dnf building image.. how.. | 14:58 |
| NeilHanlon | it's been possible for a long time, basically installing into a chroot or jail | 14:58 |
| NeilHanlon | you can "technically" do it on bare metal, too. but like.. probably dont :) | 14:59 |
| spatel | i know you can install package in chroot jail using yum/dnf but i didn't know we are also doing that now. good to know. | 15:00 |
| noonedeadpunk | it was always a pita to maintain and retrieve from lxc and issues with these images performance raised this and then | 15:03 |
| noonedeadpunk | So it's more reliable to build chroot, considering we need the most lightweight thing possilbe | 15:04 |
| *** dviroel|ruck is now known as dviroel|ruck|lunch | 15:13 | |
| spatel | +1 | 15:24 |
| jrosser | NeilHanlon: do you want to work in parallel on this - there must be other patches to roles outside openstack-ansible repo that are needed? | 15:27 |
| jrosser | i can follow along in a rocky vm? | 15:27 |
| NeilHanlon | Sure that'd be great, I appreciate it. I'm doing a run through now with metal instead of LXC to see how that goes | 15:33 |
| NeilHanlon | but yes, I think it will be a couple of different projects that need coordination and I'd appreciate the assist on that | 15:33 |
| jrosser | sure, i'm just grabbing the rocky-8 cloud image and will play-along | 15:37 |
| NeilHanlon | The major things I (think) I had to do were: Add PowerTools back in lxc_hosts' lxc_install_dnf [1] ; remove futures & pypowervm from os_nova's constraints [2]; and change some LXC settings [3]. As well as compiling python3-lxc for python3.8. | 15:39 |
| NeilHanlon | [1] http://git.shrug.pw:3000/advancedla/infra/src/branch/main/ansible/playbooks/files/lxc_install_dnf.yml#L32-L39 ; [2] http://git.shrug.pw:3000/advancedla/infra/src/branch/main/ansible/playbooks/files/os_nova.patch ; [3] http://git.shrug.pw:3000/advancedla/infra/src/branch/main/ansible/playbooks/files/user_lxc.yml | 15:39 |
| jrosser | have you got an approach yet for how to do rocky specific vars? {role}/vars/rocky-8.yml ? | 15:42 |
| jrosser | oh interesting on the compile lxc for python3.8 | 15:43 |
| jrosser | there are some new interesting features in ansible which might make that not needed | 15:43 |
| *** akahat|rover is now known as akahat|dinner | 15:43 | |
| NeilHanlon | not exactly, i was hoping most of them would be able to just end up being 'redhat' vars, since they should theoretically be the same anyways | 15:52 |
| jrosser | there is probably some consolidation we can do here as well | 15:54 |
| *** dviroel|ruck|lunch is now known as dviroel|ruck | 16:09 | |
| jrosser | NeilHanlon: did you have this go wrong? https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_aio_config.yml#L116-L121 | 16:24 |
| jrosser | i had to add vars: ansible_python_interpreter='/usr/bin/python3' to that task in order to make it find the selinux python bindings | 16:24 |
| jrosser | which is a bit hacky, but also we have this in progress https://review.opendev.org/c/openstack/openstack-ansible/+/827556 | 16:25 |
| NeilHanlon | i disable selinux outright first, so I guess I'm not hitting that bit | 16:26 |
| jrosser | also https://paste.opendev.org/show/812560/ | 16:27 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-galera_server master: Improve incremental backups rotation in mariabackup script https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/828170 | 16:31 |
| opendevreview | Damian DÄ…browski proposed openstack/openstack-ansible-galera_server master: Improve incremental backups rotation in mariabackup script https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/828170 | 16:33 |
| jrosser | NeilHanlon: did you have to fix anything about ansible not knowing the default python interpreter for rocky? | 16:54 |
| NeilHanlon | ah, probably need https://review.opendev.org/c/openstack/openstack-ansible/+/823573 | 16:55 |
| jrosser | yes i have that | 16:56 |
| jrosser | but i'm getting "msg": "The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required.", during installing the database | 16:57 |
| NeilHanlon | yeah, that's what i'm getting on a metal install now, too | 16:57 |
| jrosser | ans thats bevause "ansible_python_interpreter": "/opt/ansible-runtime/bin/python3.8" | 16:57 |
| jrosser | oh hmm hold on | 16:58 |
| jrosser | yeah if you run any of the playbooks with -vvvv then it looks like it's using /usr/bin/python3.8 as the interpreter | 17:01 |
| jrosser | i think i showed this before, where we had ansible not knowing about Debian Buster | 17:02 |
| jrosser | https://github.com/openstack/openstack-ansible/blob/stable/victoria/scripts/bootstrap-ansible.sh#L78-L80 | 17:02 |
| NeilHanlon | okay, yeah.. i think I had experimented with that a bit, setting it to 3.6, originally. But i'm not sure if I did it as explicitly to 3.8 | 17:04 |
| * jrosser tests | 17:05 | |
| * NeilHanlon 2 | 17:05 | |
| jrosser | ok i think we need this https://paste.opendev.org/show/812562/ | 17:06 |
| jrosser | ansible itself needs to run with python3.8 (becasue new ansible requires is), so we use PYTHON_EXEC_PATH="$(which python3.8)" | 17:07 |
| jrosser | but ansible ssh to the host and runs a python command, which should use OSA_ANSIBLE_PYTHON_INTERPRETER="/usr/bin/python3" because of things like the bindings for selinux and lxc | 17:07 |
| jrosser | and also the mysql bindings | 17:08 |
| *** akahat|dinner is now known as akaha|rover | 17:08 | |
| NeilHanlon | ok, I see | 17:10 |
| jrosser | this is a bit subtle, that the python environment that ansible-playbook runs in is quite seperate from the one that the tasks run on the targets in | 17:11 |
| jrosser | and the requirements for those are somewhat different | 17:11 |
| NeilHanlon | yeah, I am beginning to recognize it can make quite a difference | 17:12 |
| jrosser | there is code now in ansible for it to be able to try a number of different interpreters squentially on the target without failing | 17:13 |
| jrosser | specifically to cater for "import foobar" only being valid for a specific interpreter | 17:13 |
| jrosser | the apt and dnf modules already do this | 17:13 |
| jrosser | but that has to be implemented on a module by module basis, and isn't fixed globally in the core of ansible itself | 17:14 |
| NeilHanlon | didn't know that about ansible now looking for multiple interpreters, very cool | 17:18 |
| jrosser | ok next thing looks like this https://opendev.org/openstack/openstack-ansible-os_cinder/src/branch/master/vars/redhat.yml#L34 | 17:22 |
| NeilHanlon | that comes from PowerTools | 17:23 |
| jrosser | ok sure, we need to put that back | 17:23 |
| NeilHanlon | i went looking for where it was removed but wasn't sure.. for LXC i was doing http://git.shrug.pw:3000/advancedla/infra/src/branch/main/ansible/playbooks/files/lxc_install_dnf.yml#L32-L39 | 17:23 |
| jrosser | but lxc_hosts would be the wrong place as this is failing in a metal deploy | 17:23 |
| NeilHanlon | right | 17:23 |
| jrosser | i expect that the conditional on here fails https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/tasks/openstack_hosts_configure_dnf.yml#L114-L119 | 17:26 |
| NeilHanlon | if i'm reading https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/defaults/main.yml#L163 right, I think that probably that can be loosened to just enable powertools, or maybe enable only if EPEL is also enabled | 17:50 |
| NeilHanlon | fwiw, my test AIO/LXC just finished and appears to have worked 🤔 | 17:52 |
| noonedeadpunk | o_O | 17:53 |
| noonedeadpunk | that sounds to good to be true :) | 17:53 |
| NeilHanlon | i know! https://drop1.neilhanlon.me/irc/uploads/1465254a508de23e/image.png | 17:56 |
| jrosser | noonedeadpunk: how do you think this gets installed? https://zuul.opendev.org/t/openstack/build/996c9111c1154e389f03e74b9f2c53b8/log/logs/redhat-rpm-list-installed-host-09-52-10.txt#240 | 18:42 |
| jrosser | when we've got this https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/defaults/main.yml#L163 | 18:43 |
| jrosser | i'm confused :) | 18:43 |
| noonedeadpunk | maybe powetools doesn't need enablement nowadays... | 18:45 |
| noonedeadpunk | why we actually ask for it? https://opendev.org/openstack/openstack-ansible-os_cinder/src/branch/master/vars/redhat.yml#L34 | 18:45 |
| jrosser | cinder seems to need it for this https://github.com/openstack/openstack-ansible-os_cinder/blob/master/vars/redhat.yml#L34 | 18:45 |
| jrosser | snap :) | 18:46 |
| jrosser | seems like kind of minimal use generally https://codesearch.opendev.org/?q=kmod-devel | 18:46 |
| noonedeadpunk | but we don't enable it anywhere? | 18:46 |
| noonedeadpunk | I mean - I dunno how cinder role gets it, or how supposed to, but powertools needed to be explicitly enabled one day... So likely nowadays with stream it's not the case anymore | 18:47 |
| jrosser | `The kmod-devel package provides header files used for development of applications that wish to load or unload Linux kernel modules` | 18:47 |
| jrosser | oh well this is my rocky vm | 18:48 |
| jrosser | and it fails to install it | 18:48 |
| jrosser | ^ and it skips installing powertools repo, as the enabled flag is false | 18:48 |
| jrosser | then cinder fails | 18:48 |
| noonedeadpunk | huh... and we dunno if regular centos-8 was failing with it lately or not... | 18:48 |
| jrosser | and i wonder how/why this happens in CI | 18:48 |
| noonedeadpunk | I'd say not | 18:49 |
| noonedeadpunk | no idea tbh.... | 18:49 |
| noonedeadpunk | hm... also - it should be installed only inside repo container I guess? | 18:50 |
| jrosser | oh interesting i think that zuul sets it up | 18:51 |
| jrosser | https://zuul.opendev.org/t/openstack/build/996c9111c1154e389f03e74b9f2c53b8/log/job-output.txt#3561 | 18:52 |
| noonedeadpunk | damn it | 18:52 |
| noonedeadpunk | we need to clean this all up.... | 18:52 |
| noonedeadpunk | I bet I can recall HighAvailability passing as OK... | 18:53 |
| jrosser | we could easily merge wrong stuff like this | 18:53 |
| noonedeadpunk | I bet we even dropped HighAvailability enablement... | 18:54 |
| noonedeadpunk | ah, well, it's only pacemaker-corosync go from there | 18:55 |
| noonedeadpunk | which is the case for masakari only | 18:55 |
| jrosser | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/tasks/mirror/CentOS-8.yaml | 18:57 |
| noonedeadpunk | wow, epel | 18:58 |
| noonedeadpunk | damn it | 18:58 |
| noonedeadpunk | I'm not really sure how we should keep track of such things... | 19:01 |
| NeilHanlon | i have an idea but it involves like 30 interns | 19:06 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Ensure that the legacy network-scripts package is present https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/828113 | 19:14 |
| noonedeadpunk | :D hehe | 19:14 |
| jrosser | noonedeadpunk: maybe we add a var to that zuul role and set it in our base job | 19:18 |
| jrosser | install_extra_repos: False | 19:18 |
| noonedeadpunk | I like this idea the way more then 30 interns :D | 19:19 |
| noonedeadpunk | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/tasks/mirror/Debian.yaml#L12 that's not good as well I believe | 19:20 |
| jrosser | idk if that’s possible to pass settings to those base roles from our job definitions | 19:20 |
| noonedeadpunk | I bet it is https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.vars | 19:23 |
| *** noonedeadpunk_ is now known as noonedeadpunk | 19:34 | |
| *** dviroel|ruck is now known as dviroel|out | 22:41 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
