| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon stable/2023.2: Deploy default policy files https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/913815 | 04:31 |
|---|---|---|
| farbod | Hello, I get an error for deployments others than stable/2023.2 on bootstraping deployment host on task Install collection requirements with ansible galaxy. Is it normal? | 07:26 |
| jrosser | farbod: can you paste the output you get, maybe at paste.opendev.org so we can see what goes wrong? | 07:45 |
| noonedeadpunk | I don't think we have any issues with 2023.1 bootstrap - our CI's doing that look healthy | 09:17 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon master: Do not change mode of files recursively https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/911420 | 09:28 |
| jero | heya | 09:38 |
| jero | iam trying to upgrade from 2023.1 to 2023.2 via ./scripts/run-upgrade.sh | 09:39 |
| jero | giving me following fatal | 09:39 |
| jero | https://paste.opendev.org/show/baKix3QrmtEdV8V6a6JR/ | 09:39 |
| jero | any idea to further debug this? | 09:39 |
| noonedeadpunk | jero: hey | 09:45 |
| noonedeadpunk | I think I know the reason for it | 09:45 |
| noonedeadpunk | which is basically a gnocchi issue | 09:45 |
| noonedeadpunk | but huh, I thought it should be affecting only master... | 09:46 |
| noonedeadpunk | basically talking about https://github.com/gnocchixyz/gnocchi/pull/1379 | 09:46 |
| noonedeadpunk | jero: but about debug - executing /openstack/venvs/ceilometer-28.0.2.dev7/bin/ceilometer-upgrade does write output to journalctl | 09:47 |
| jero | thanks for pointing in the journalctl direction - not thought about that :x | 09:52 |
| jero | seems like the upgrade is not the only problem | 09:52 |
| jero | https://paste.opendev.org/show/bbczJvN3t5OyJfOj98ru/ | 09:52 |
| noonedeadpunk | jero: yeah, exactly why I told about gnocchi | 09:53 |
| noonedeadpunk | I faced same issue like previous week, but the PR for gnocchi was solving issue iirc. | 09:54 |
| noonedeadpunk | but sandbox was on master | 09:54 |
| noonedeadpunk | so worth checking gnocchi-api logs and I assume it's in a restart loop or smth | 09:54 |
| noonedeadpunk | ah... and another issue were policies | 09:55 |
| noonedeadpunk | ie https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/913244 | 09:56 |
| noonedeadpunk | but that was also only for master I guess | 09:56 |
| noonedeadpunk | jero: so checking gnocchi journal can be helpful to see what's the exact issue | 10:01 |
| jero | Mar 22 10:00:50 infra1-gnocchi-container-4abde65d gnocchi-api[450494]: 2024-03-22 10:00:50,987 [450494] ERROR oslo_policy._cache_handler: Config file not found /etc/gnocchi/policy.yaml: No such file or directory | 10:01 |
| noonedeadpunk | ok, yeah | 10:01 |
| * noonedeadpunk guessed | 10:01 | |
| jero | yes,you was right | 10:02 |
| noonedeadpunk | jero: out of interest - do smth like rm /etc/gnocchi/policy.* | 10:02 |
| noonedeadpunk | and restart service | 10:02 |
| noonedeadpunk | as I've seen this issue but failed to reliably replicate it | 10:03 |
| noonedeadpunk | like services should not need presence of policy.yaml but they might not like it just vanishing during runtime | 10:03 |
| noonedeadpunk | as we can potentially just miss just handler trigger here: https://opendev.org/openstack/openstack-ansible-os_gnocchi/src/branch/master/tasks/gnocchi_post_install.yml#L52-L59 | 10:05 |
| jero | rm /etc/gnocchi/policy.* and systemctl restart gnocchi-api.service - no more errors | 10:14 |
| jero | but still, when running /openstack/venvs/ceilometer-28.0.2.dev7/bin/ceilometer-upgrade throws the same rror | 10:15 |
| noonedeadpunk | huh? | 10:15 |
| noonedeadpunk | hav you done that on all containers? | 10:16 |
| noonedeadpunk | and another thing - can you interact with gnocchi through cli? | 10:18 |
| jero | i did now | 10:24 |
| jero | no more errors | 10:24 |
| jero | :) | 10:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Switch SHAs to EOM https://review.opendev.org/c/openstack/openstack-ansible/+/913413 | 10:27 |
| noonedeadpunk | jero: I will push a patch now to restart service on policy file removal - I find it as most likely issue | 10:28 |
| noonedeadpunk | unless you wanna test it more and push the patch as well :) | 10:28 |
| jero | thanks for pointing me in the right direction | 10:29 |
| jero | ill rerun the upgrade-script | 10:29 |
| noonedeadpunk | jero: um, I'm not sure you can re-run it | 10:30 |
| noonedeadpunk | or it's worth doing | 10:30 |
| noonedeadpunk | (maybe you can though...) | 10:30 |
| noonedeadpunk | but best would be to run remaining playbooks or just setup-openstack.yml | 10:30 |
| noonedeadpunk | as ceilometer is really pretty much close to the end of list | 10:31 |
| jero | i see, oke lets see :) | 10:31 |
| noonedeadpunk | meaning most services are already upgraded | 10:31 |
| noonedeadpunk | when run-upgrade.sh fails it throws out "left" things | 10:31 |
| jero | oh, yes i see | 10:33 |
| jero | good to know | 10:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Switch SHAs to EOM https://review.opendev.org/c/openstack/openstack-ansible/+/913414 | 10:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Switch SHAs to EOM https://review.opendev.org/c/openstack/openstack-ansible/+/913418 | 10:46 |
| jero | there was a second error when running the upgrade-script: https://paste.opendev.org/show/bIwDsFo2gM6iPLJ6PKT5/ | 10:56 |
| jero | fatal: [compute1]: FAILED! => {"msg": "Unexpected failure during module execution: 'mtime'", "stdout": ""} | 10:56 |
| opendevreview | Jonathan Herlin proposed openstack/openstack-ansible master: Fix inventory defenition for Cloudkitty https://review.opendev.org/c/openstack/openstack-ansible/+/912269 | 10:57 |
| noonedeadpunk | jero: ok, this one is slightly less trivial... | 11:08 |
| noonedeadpunk | sounds like some file that is expected to be in repo is missing... | 11:08 |
| gokhan__ | Hello folks, I have problems on ceph cluster. ceph -s gives no response. I checked mons everthing seems ok and quorum is correct. Cluster status is warning for some osds.on ceph dashboard I can monitor ceph cluster. Do you have any ideas? How can we troubleshoot this problem? | 11:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Do not resolve all host_vars when haproxy_backend_node is a mapping https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/913927 | 11:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Switch SHAs to EOM https://review.opendev.org/c/openstack/openstack-ansible/+/913413 | 11:38 |
| noonedeadpunk | gokhan__: ceph -s from ... compute? | 11:44 |
| noonedeadpunk | this can be smth like mtu issue | 11:44 |
| noonedeadpunk | or firewall | 11:45 |
| noonedeadpunk | we had that couple of times, but always that was smth related to networking | 11:45 |
| gokhan__ | noonedeadpunk, it has worked before but after changing mon daemons it can't work again. I checked monmap it is ok | 11:46 |
| noonedeadpunk | and new daemons are in ceph.conf I assume? | 11:48 |
| gokhan__ | I am trying on both compute and infra nodes. | 11:48 |
| noonedeadpunk | I guess I'm not sure what is meant under changing daemons | 11:49 |
| noonedeadpunk | as then it's kinda same questions - mtu and firewall... | 11:49 |
| noonedeadpunk | ok, crap, seems we need to merge unmaintained patches without any real testing | 11:50 |
| noonedeadpunk | as branches are dropped now, so they're being checked-out to master.... | 11:50 |
| gokhan_ | noonedeadpunk, I am trying distribution upgrade. I had to change nodes which mons are running on. | 11:50 |
| noonedeadpunk | well, again - check that ceph.conf is updated if mon IPs changed. then that you generally can reach monitors through telnet or smth, and that storage interfaces have same mtus | 11:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Switch SHAs to EOM https://review.opendev.org/c/openstack/openstack-ansible/+/913413 | 12:03 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon stable/2023.2: Deploy default policy files https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/913815 | 12:08 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.2: Determine if upgrade source branch is stable/ or unmaintained/ https://review.opendev.org/c/openstack/openstack-ansible/+/912434 | 14:38 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Restart OVN on certificate changes https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/912762 | 14:42 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Imporove Jinja indentation for service templates https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/913925 | 15:00 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!