| hamidlotfi_ | I found a conflict in one file with the name `socket_util.py` in two different locations that have a bug. | 07:24 |
|---|---|---|
| hamidlotfi_ | first ` /usr/lib/python3/dist-packages/ovs/socket_util.py` | 07:24 |
| hamidlotfi_ | second: `/openstack/venvs/neutron-26.1.1/lib/python3.10/site-packages/ovs/socket_util.py` | 07:24 |
| hamidlotfi_ | in the first location with no bug (means fixed the bug) and in the second location has a bug that is not fixed. | 07:24 |
| hamidlotfi_ | and it is in the `neutron_server` container | 07:24 |
| hamidlotfi_ | @jrosser @noonedeadpunk | 07:24 |
| noonedeadpunk | mornings | 08:03 |
| noonedeadpunk | hamidlotfi_: would be much appreciuated if you could provide `/usr/bin/pip3 list` and `/openstack/venvs/neutron-26.1.1/bin/pip list` | 08:06 |
| noonedeadpunk | as it's defenitely difference in python3-ovs package versions, but kinda wonder which are used | 08:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Add quorum queues support for the service https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/887849 | 08:17 |
| hamidlotfi_ | noonedeadpunk: I don't have globally pip3 in `/usr/bin/pip3` in the `neutron_server` container | 09:17 |
| noonedeadpunk | Melonia: it's passed with --user-data key | 09:17 |
| hamidlotfi_ | https://www.irccloud.com/pastebin/l9lHvw68/ | 09:17 |
| hamidlotfi_ | @noonedeadpunk | 09:18 |
| noonedeadpunk | hamidlotfi_: you're running OVN? | 09:20 |
| noonedeadpunk | can you then run `python3 -c "import importlib.metadata; print(importlib.metadata.version('ovs'))"`? | 09:23 |
| hamidlotfi_ | noonedeadpunk: 3.0.3 | 09:30 |
| hamidlotfi_ | In the OVN env | 09:30 |
| noonedeadpunk | well, that would explain it | 09:41 |
| noonedeadpunk | bug is fixed in 2.17.3 | 09:45 |
| noonedeadpunk | I'm not sure how to workaround that though, since version comes from upper-constraints file | 09:53 |
| hamidlotfi_ | I can just replace the healthy file with the bug version, right? | 10:17 |
| hamidlotfi_ | Can this issue exist for other files? | 10:18 |
| noonedeadpunk | hamidlotfi_: the problem that it's defined here: https://opendev.org/openstack/requirements/src/branch/master/upper-constraints.txt#L190 | 10:22 |
| noonedeadpunk | eventually, we cache that file on a repo container, so you technically should be able to adjust version there | 10:22 |
| noonedeadpunk | or fork the repo and override requirements_git_repo | 10:24 |
| noonedeadpunk | I've also proposed patches to the requirements repo | 10:25 |
| noonedeadpunk | https://review.opendev.org/c/openstack/requirements/+/892892 | 10:26 |
| hamidlotfi_ | noonedeadpunk: Thanks. | 10:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/zed: Release constraint for OVS/OVN versions for EL https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/892563 | 10:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update Senlin SHA https://review.opendev.org/c/openstack/openstack-ansible/+/892910 | 11:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/890131 | 11:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/888607 | 11:26 |
| noonedeadpunk | We are super close to finish ansible-core 2.15 topic: https://review.opendev.org/q/topic:osa/core-2.15+status:open | 11:41 |
| noonedeadpunk | I'm trying to figure out what's actually wrong with adjutant, as it weirdly different for lxc/metal | 11:42 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts stable/yoga: Refactor LXC image expiration https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/892313 | 12:33 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts stable/2023.1: Refactor LXC image expiration https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/892311 | 13:07 |
| ayush | hi team, can any body guide me in red hat open stack platform using triple-o heat template | 14:27 |
| noonedeadpunk | I would ask that in #tripleo | 14:29 |
| ayush | ok | 14:29 |
| mgariepy | quite interesting : https://www.youtube.com/watch?v=ZRWSZRPhJUs | 14:39 |
| opendevreview | Merged openstack/openstack-ansible-plugins stable/zed: Skip updating service password by default https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/888153 | 14:40 |
| Bico | Hi. How can I decommission a faulty infra node? | 14:43 |
| noonedeadpunk | Bico: is it already dead? | 14:47 |
| *** Guest535 is now known as starkis | 14:49 | |
| ayush | hi Team | 14:51 |
| ayush | all of a sudden my all compute node went down | 14:51 |
| ayush | it shows this in error | 14:51 |
| ayush | Aug 28 14:50:36 c1w-rabbit-mq-container-ec1dd9ed rabbitmq-server[2582905]: closing AMQP connection <0.11279.52> (172.29.239.223:58786 -> 172.29.239.190:5671 - uwsgi:272367:5a5216d7-eab3-45d3-b830-4c7c1fb82afc): missed heartbeats from client, timeout: 60s Aug 28 14:50:36 c1w-rabbit-mq-container-ec1dd9ed rabbitmq-server[2582905]: closing AMQP connection <0.11266.52> (172.29.2 | 14:51 |
| ayush | in the rabbit mq side | 14:51 |
| Bico | noonedeadpunk: nope. Still running. | 14:55 |
| Bico | noonedeadpunk: but /var/lib/ partition is *crashed* | 14:58 |
| noonedeadpunk | ayush: what's on nova compute side? Also is networking is fine? | 14:59 |
| opendevreview | Merged openstack/openstack-ansible-os_placement master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/888603 | 14:59 |
| noonedeadpunk | Bico: so basically containers are dead? | 15:00 |
| Bico | noonedeadpunk: containers are running as the result of command lxc-ls -f inside that infra node. | 15:01 |
| Bico | noonedeadpunk: I suppose because of /openstack is ok | 15:01 |
| noonedeadpunk | just trying to understand if it's worth just killing it or worth adding new one and gracefully shutting down old one | 15:01 |
| noonedeadpunk | well, rootfs of containers is in aio1_glance_container-edbe54fe/ | 15:01 |
| Bico | In the documentation I saw how to remove a compute node but not an infra one. | 15:04 |
| noonedeadpunk | Yeah. because it's slightly less trivial | 15:04 |
| noonedeadpunk | or well. depending on level of disturbance :D | 15:05 |
| noonedeadpunk | eventually, you can just do openstack-ansible lxc-containers-destroy.yml --limit <control>-host_containers | 15:06 |
| noonedeadpunk | then remove infra node from /etc/openstack_deploy/openstack_user_config.yml | 15:07 |
| noonedeadpunk | and remove from inventory /opt/openstack-ansible/scripts/inventory-manage.py -r <control> | 15:07 |
| noonedeadpunk | But I'd probably won't go that way, as if rabbitmq is alive - it might affect services | 15:08 |
| noonedeadpunk | So I'd tried to run `cd /opt/openstack-ansible; ansible -m command -a "rabbitmqctl cluster_status" rabbitmq_all` to check if rabbit is fine | 15:09 |
| noonedeadpunk | if it is - then it would make sense to do things slightly more gracefully | 15:09 |
| Bico | I will check rabbit and galera first | 15:10 |
| noonedeadpunk | if rabbit has already fallen from cluster - I think you indeed can just go on and drop the host from inventory | 15:21 |
| noonedeadpunk | btw, once you add new host and new member to rabbit cluster, you will need to run smth like openstack-ansible setup-openstack.yml to re-write service configurations and replace rabbitmq host there with new one | 15:23 |
| Bico | noonedeadpunk: I think the containers are unresponsive. Checking the ansible commands. | 15:40 |
| noonedeadpunk | yeah... | 15:40 |
| noonedeadpunk | then it's basically matter of cleaning up inventory and openstack_user_config | 15:40 |
| noonedeadpunk | oh, btw | 15:41 |
| Bico | noonedeadpunk: but on the UI and via API I get some errors, probably trying to connect to these containers. | 15:41 |
| noonedeadpunk | haproxy should have disabled these backends | 15:41 |
| noonedeadpunk | you can disable them explicitly using this playbook: https://paste.openstack.org/show/bGOGEGAkhvpJ23Yc2MYd/ | 15:43 |
| Bico | noonedeadpunk: cool, will take a look. Thank you! | 15:46 |
| noonedeadpunk | ie `openstack-ansible play.yml -e group_to_disable=controlXX-host_containers` | 15:48 |
| noonedeadpunk | fwiw, there're some fresh articles on superuser regarding OSA scaling: https://superuser.openinfra.dev/articles/author/dmitriyrabotyagov/ | 17:04 |
| noonedeadpunk | Will need to adopt these and porpose to docs later on... | 17:04 |
| jamesdenton | nice! | 18:30 |
| Bico | noonedeadpunk: meh, the destroy container playbook doesn't work because it wants ssh to the broken container. | 23:57 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!