| *** ysandeep|out is now known as ysandeep|rover | 05:08 | |
| jrosser | good morning | 07:40 |
|---|---|---|
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 08:02 | |
| noonedeadpunk | mornings! | 08:20 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Add upgrade path from lsyncd to shared filesystem. https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/839411 | 09:31 |
| damiandabrowski[m] | hey! | 09:48 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Do not adjust libvirtd sysconfig for centos-9 https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/841862 | 09:59 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Do not adjust libvirtd sysconfig for centos-9 https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/841862 | 10:01 |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 10:01 | |
| jrosser | ok so i get a tempest pass on centos-9 metal locally here | 10:03 |
| jrosser | noonedeadpunk: ^^ next thing would be how we get an LXC install | 10:08 |
| noonedeadpunk | oh | 10:22 |
| noonedeadpunk | I'm trying to figure out wtf happened with centos-nfv-openvswitch package.... | 10:23 |
| noonedeadpunk | alse seems like infra scenario is broken? | 10:23 |
| noonedeadpunk | `aodhclient 2.4.1 depends on pbr>=1.4 The user requested (constraint) pbr===5.8.1` ugh | 10:25 |
| jrosser | hrrm | 10:28 |
| jrosser | noonedeadpunk: do you have a link to a patch which breaks like that? | 10:28 |
| jrosser | https://opendev.org/openstack/python-aodhclient/src/branch/master/requirements.txt#L5 | 10:29 |
| jrosser | a lot of infra jobs were broken because of this https://review.opendev.org/c/openstack/openstack-ansible/+/841595 | 10:34 |
| jrosser | is there something else too? | 10:34 |
| noonedeadpunk | I'm not really sure why it's happened now. As requirement was there for a while and they're not contraversary | 10:35 |
| noonedeadpunk | * contradictory | 10:35 |
| jrosser | where is that breaking? | 10:36 |
| noonedeadpunk | IN your centos 9 patches, is https://zuul.opendev.org/t/openstack/build/487c4b5f64e043c8916a29b0d660bb64/log/job-output.txt#8470 | 10:37 |
| jrosser | oh wow | 10:41 |
| jrosser | looking back through the #opendev logs, looks like there was a bunch of pypi CDN trouble on thursday | 11:01 |
| jrosser | also mentioning trouble with pbr | 11:01 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Pass valid cert regen variable to pki role https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/841617 | 11:01 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Remove all code for lsync, rsync and ssh https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837588 | 11:03 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Clean up legacy lsycnd, rsync and ssh key config https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837859 | 11:03 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Use the same vars file for all versions of centos https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/841618 | 11:03 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Use distro packages for nginx on centos. https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/841619 | 11:03 |
| *** dviroel|out is now known as dviroel | 11:21 | |
| jrosser | how do i debug which version of config_template we are using | 11:47 |
| jrosser | well, which actual config_template gets called, i mean | 11:48 |
| jrosser | confusion between ceph_ansible config_template vs plain config_template vs openstack.config_tempalate.config_template | 11:48 |
| *** spotz_ is now known as spotz | 11:55 | |
| noonedeadpunk | I can recall debugging this one... On master it's not ours for sure. And ours plugin/collection is exactly same thing | 12:41 |
| noonedeadpunk | (in terms of codebase) | 12:41 |
| noonedeadpunk | regarding ceph_ansible it's tougher. and it's likely the one that being used by default atm. | 12:42 |
| noonedeadpunk | I merged PR to ceph-ansible master (before deprecation) that replaces their config_template with our collection | 12:43 |
| noonedeadpunk | but that will never be released now... | 12:43 |
| *** ysandeep|rover is now known as ysandeep|rover|afk | 12:45 | |
| *** ysandeep|rover|afk is now known as ysandeep|rover | 13:11 | |
| jrosser | noonedeadpunk: i looked at making config template understand properties files https://paste.opendev.org/show/beItOWLGjuuCJUmOt88e/ | 13:32 |
| jrosser | not sure if that is a giant hack or not...... | 13:32 |
| jrosser | it treats the properties as if they were already in the default section of an ini file | 13:33 |
| noonedeadpunk | jrosser: I wonder if calling them sysctl would make sense to anybody except me... | 13:38 |
| noonedeadpunk | but having "default" section hack I saw somewhere on stackoverflow as a solution :D | 13:39 |
| jrosser | indeed - thats exactly what i copied | 13:39 |
| jrosser | i was also wondering if the large code we have in there for --diff actually works at all | 13:40 |
| jrosser | it doesnt appear to report any changes at all from what i see in diff mode | 13:43 |
| noonedeadpunk | I can imagine it being borked. I can recall spotting that when ara was missing diffs for it. | 13:44 |
| noonedeadpunk | there was some tricky thing to fix it... | 13:46 |
| *** ysandeep|rover is now known as ysandeep|out | 15:02 | |
| *** dviroel is now known as dviroel|lunch | 15:25 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Add upgrade path from lsyncd to shared filesystem. https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/839411 | 15:30 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Remove all code for lsync, rsync and ssh https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837588 | 15:30 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Clean up legacy lsycnd, rsync and ssh key config https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837859 | 15:30 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Use the same vars file for all versions of centos https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/841618 | 15:30 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Use distro packages for nginx on centos. https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/841619 | 15:31 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Use distro packages for nginx on centos. https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/841619 | 15:32 |
| *** dviroel|lunch is now known as dviroel | 16:24 | |
| spatel | jrosser do you know what is going here? - https://paste.opendev.org/show/bmRI8tFkmylUMiAXptYN/ | 16:28 |
| spatel | I reboot one of infra node and noticed that error | 16:29 |
| jrosser | bad superblock means that something is not formatted | 16:32 |
| spatel | hmm | 16:32 |
| jrosser | i have /var/lib/machines.raw on /var/lib/machines type ext4 (rw,noatime,nodiratime) | 16:33 |
| spatel | mount| grep machines | 16:33 |
| spatel | nothing mounted | 16:33 |
| spatel | I have installed 3 infra node last week and today reboot 2 and noticed failed service on 2 rebooted node | 16:34 |
| spatel | 1 node which i didn't reboot yet showing mounted /var/lib/machines | 16:35 |
| spatel | I am 100% it will failed if i reboot last node | 16:35 |
| jrosser | `file /var/lib/machines.raw` ? | 16:36 |
| spatel | /var/lib/machines.raw: Linux rev 1.0 ext4 filesystem data, UUID=b2c7accf-c3bb-4ab0-9301-371a099de544 (extents) (64bit) (large files) (huge files) | 16:36 |
| spatel | Type=btrfs in /usr/lib/systemd/system/var-lib-machines.mount ? I am running ubuntu 20.04 | 16:38 |
| spatel | why its saying ext4 then? | 16:38 |
| jrosser | i dont know | 16:38 |
| spatel | I just confirmed this on other production deployment and found service failed over there too | 16:41 |
| jrosser | sorry i'm just on the phone | 16:43 |
| spatel | np, i am trying to figure out what is wrong | 17:03 |
| spatel | how critical its if that filesystem not mounted | 17:04 |
| spatel | jrosser found issue and it was related Type= https://paste.opendev.org/show/b1XI4qKoDks3u7A6XKNS/ | 17:07 |
| spatel | we need to patch it | 17:07 |
| jrosser | do you see a bug anywhere? https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/tasks/lxc_cache_rootfs.yml#L36 | 17:08 |
| jrosser | that looks hardwired to use ext4 | 17:09 |
| spatel | hmm very interesting | 17:10 |
| spatel | look like bug somewhere.. may be we have override somewhere.. just guessing | 17:11 |
| jrosser | there is no variable there that i see to override | 17:14 |
| spatel | I am seeing same behavior on all my production deployment running ubuntu | 17:15 |
| spatel | if you are not seeing in your deployment then i would be shocked :) | 17:17 |
| jrosser | it is the same on one here | 17:17 |
| jrosser | i don't yet see how anything except ext4 should be in that file though | 17:17 |
| jrosser | oh hold on | 17:20 |
| spatel | https://paste.opendev.org/show/by0761bd7XSyaIactsJx/ | 17:21 |
| spatel | chasing this now... | 17:21 |
| jrosser | spatel: wait | 17:21 |
| jrosser | you are looking at code for master branch there | 17:21 |
| spatel | i am sure you found something :) | 17:21 |
| jrosser | you're running Xena? | 17:21 |
| spatel | I am running 23.3.0 | 17:22 |
| spatel | https://paste.opendev.org/show/bfsoT63IuYAjM2E0W9SD/ | 17:22 |
| prometheanfire | twice now, I've gone from 24.0.0 to 24.2.0 and running setup-everything (specifically galera) has caused the galera cluster to fall on it's face | 17:23 |
| spatel | I am seeing this issue in every single tags of wallaby | 17:23 |
| jrosser | for wallabty/xena systemd_mount role is not used | 17:23 |
| spatel | oh ok | 17:23 |
| jrosser | spatel: it is super trivial https://github.com/openstack/openstack-ansible-lxc_hosts/blob/stable/wallaby/tasks/lxc_cache_rootfs.yml#L35-L39 | 17:24 |
| jrosser | and the file is not even a template, so there is even less way for btrfs to end up in there https://github.com/openstack/openstack-ansible-lxc_hosts/blame/stable/wallaby/files/var-lib-machines.mount | 17:24 |
| spatel | you are correct.. | 17:25 |
| spatel | look at full content of file and look like its coming from somewhere else.. https://paste.opendev.org/show/bJabhynjHMXd5wTt6Y3o/ | 17:27 |
| jrosser | spatel: and this is a fresh install of wallaby? | 17:27 |
| spatel | Yes last week i deployed this environment | 17:27 |
| spatel | first i use 23.2.2 and then you suggested to go with 23.3.0 to fix FD leak for oslo so i did minor upgrade | 17:28 |
| jrosser | this is unrelated to that | 17:29 |
| jrosser | we have accidentally used a file that is shipped with systemd https://packages.ubuntu.com/search?suite=focal&arch=any&mode=exactfilename&searchon=contents&keywords=var-lib-machines.mount | 17:29 |
| jrosser | noonedeadpunk: ^ | 17:29 |
| spatel | jrosser you are correct, its not part of OSA deployment | 17:30 |
| jrosser | prometheanfire: you may have to be a bit more specific with some debugging info...... | 17:30 |
| jrosser | spatel: maybe best to make a bug on launchpad for this, i don't have a quick idea of what the best fix is | 17:33 |
| noonedeadpunk | I wonder how accidentally it was | 17:34 |
| noonedeadpunk | considering it's part of systemd-container that I assume is nspawn | 17:34 |
| jrosser | not sure, i think the btrfs stuff all has roots in nspawn | 17:34 |
| jrosser | yes | 17:34 |
| jrosser | but we can't do btrfs any more anyway | 17:34 |
| jrosser | i wonder if it's possible to make a systemd drop-in for a mount to change type= | 17:35 |
| noonedeadpunk | I wonder why it's not in overrides though... as like we intentionally override that | 17:35 |
| spatel | why OSA didn't override that file with this file. or may be it order issue - https://github.com/openstack/openstack-ansible-lxc_hosts/blob/stable/wallaby/tasks/lxc_cache_rootfs.yml#L35-L39 | 17:35 |
| jrosser | if you update systemd then this file could get overwitten | 17:36 |
| jrosser | or if systemd-container is installed later, perhaps | 17:36 |
| noonedeadpunk | Hmm. I wonder if https://github.com/openstack/openstack-ansible-lxc_hosts/commit/337ddf87802c7597b7c51a071eb9829a4b87defa might be wrong then | 17:36 |
| noonedeadpunk | if we just need an override.... | 17:36 |
| jrosser | well, read the file though | 17:37 |
| jrosser | # This unit is required for pre-240 versions of systemd...... | 17:37 |
| jrosser | so the behaviour is not always the same, depending on systemd version | 17:37 |
| prometheanfire | jrosser: just recovered, running the playbook again, will provide logs if it occurs again | 17:38 |
| prometheanfire | worked this time | 17:41 |
| * prometheanfire shrugs | 17:41 | |
| *** dviroel is now known as dviroel|out | 20:51 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!