| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Switch trove-api to wsgi running under apache. https://review.opendev.org/c/openstack/kolla-ansible/+/854759 | 01:33 |
|---|---|---|
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Switch trove-api to wsgi running under apache. https://review.opendev.org/c/openstack/kolla-ansible/+/854759 | 01:42 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Switch trove-api to wsgi running under apache. https://review.opendev.org/c/openstack/kolla-ansible/+/854759 | 01:46 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Implement automatic deploy of trove https://review.opendev.org/c/openstack/kolla-ansible/+/863321 | 02:06 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: CI: Trove: Create a database instance https://review.opendev.org/c/openstack/kolla-ansible/+/863521 | 02:06 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Modernize the swift role https://review.opendev.org/c/openstack/kolla-ansible/+/797498 | 02:55 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Implement automatic deploy of trove https://review.opendev.org/c/openstack/kolla-ansible/+/863321 | 03:02 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: CI: Trove: Create a database instance https://review.opendev.org/c/openstack/kolla-ansible/+/863521 | 03:08 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Modernize the swift role https://review.opendev.org/c/openstack/kolla-ansible/+/797498 | 06:52 |
| mnasiadka | EugenMayer4: haproxy expects everything in one file | 08:16 |
| bbezak | guesswhat: for vlan transparency you would need to add vlan_transparent variable in neutron.conf - https://docs.openstack.org/neutron/latest/configuration/neutron.html#DEFAULT.vlan_transparent via https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html#openstack-service-configuration-in-kolla, and also alter mtu for that - | 09:31 |
| bbezak | https://docs.openstack.org/neutron/latest/admin/config-mtu.html#networks-with-enabled-vlan-transparency | 09:31 |
| guesswhat | bbezak: thanks, I am not sure if this would help my use case tho. ( i have attached external vm to the os network ; vlan trunk; but its not getting dhcp address, unless I create explictly port with MAC, but even then port has down status, not sure how its working , maybe provider, externale network would be better solution ) | 09:38 |
| guesswhat | MAC of that VM | 09:38 |
| guesswhat | when will be images builded from master ( https://review.opendev.org/c/openstack/kolla/+/868565/3/kolla/common/config.py ) published? | 09:56 |
| mnasiadka | they were, today | 10:06 |
| mnasiadka | (I assume you mean published) | 10:06 |
| EugenMayer4 | i upgraded from 13.2.0 to 13.7.0 today (to prepare for a bigger jump to 14.x) - it seems like my instances can no longer access the metadata service via `curl -s http://169.254.169.254` - is there anything in particular i could check? | 10:07 |
| EugenMayer4 | i already restarted the controller / computes once | 10:07 |
| mnasiadka | ML2/OVS or ML2/OVN? | 10:07 |
| EugenMayer4 | mnasiadka was that for me? | 10:07 |
| mnasiadka | EugenMayer4: yes | 10:07 |
| EugenMayer4 | Trying to check / remember what this installation was. Assuming OVN, trying to double check | 10:09 |
| EugenMayer4 | mnasiadka: neutron_plugin_agent: 'ovn' - so OVN | 10:13 |
| mnasiadka | EugenMayer4: so check neutron_ovn_metadata_agent logs | 10:13 |
| EugenMayer4 | neither on the compute nor on the controller i can find those logs under /var/log/kolla - do you mean the stdout from the container? | 10:15 |
| mnasiadka | no, they should be under /var/log/kolla/neutron | 10:16 |
| mnasiadka | on the computes | 10:16 |
| EugenMayer4 | https://gist.github.com/EugenMayer/e4e6d807972f2443ef42d135b68747ab those are the container logs at least, checking the others now | 10:17 |
| EugenMayer4 | mnasiadka ok on the compute we have an issue https://gist.github.com/EugenMayer/2b600b0c24974b672616c746e3dcb6a7 | 10:18 |
| EugenMayer4 | 2022-12-30 11:17:31.048 7 CRITICAL neutron [-] Unhandled error: neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface tap415ba715-d1, namespace ovnmeta-415ba715-dc0b-4a5e-beb9-43f71b0666a2. | 10:19 |
| mnasiadka | I would stop the agent, clean up the netns, and start it again | 10:19 |
| EugenMayer4 | can you point me to a doc on how to cleanup the namespaces? | 10:20 |
| EugenMayer4 | i cannot say that iam ovn bulletproof at all | 10:20 |
| mnasiadka | ip netns ls, ip netns delete <name> | 10:25 |
| EugenMayer4 | i assume on all coputes? | 10:26 |
| EugenMayer4 | computes | 10:26 |
| EugenMayer4 | so what i did: stopped the `neutron_ovn_metadata_agent` container, then removed the netns (one) and started the container again. One netns has been recreated due to that | 10:29 |
| EugenMayer4 | uppon starting the container, i see that the logs still throw: | 10:30 |
| EugenMayer4 | 2022-12-30 11:29:49.505 7 CRITICAL neutron [-] Unhandled error: neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface tap415ba715-d1, namespace ovnmeta-415ba715-dc0b-4a5e-beb9-43f71b0666a2. | 10:30 |
| EugenMayer4 | mnasiadka reading up on https://docs.openstack.org/releasenotes/kolla/xena.html ones again to see if i missed anything on the upgrade from 13.2 to 13.7 | 10:42 |
| EugenMayer4 | also checked https://docs.openstack.org/releasenotes/kolla-ansible/xena.html and di find nothing | 10:45 |
| EugenMayer4 | i assume just trying the planned 14.0 upgrade will not be a chance to fix this by accident (desp. move) | 10:46 |
| EugenMayer4 | found something similar https://www.mail-archive.com/yahoo-eng-team@lists.launchpad.net/msg89710.html | 10:52 |
| EugenMayer4 | which endeded up as https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1995735 | 10:53 |
| EugenMayer4 | running ip netns exec ovnmeta-415ba715-dc0b-4a5e-beb9-43f71b0666a2 ip a i can see the tap intarface that seems to be problematic | 10:55 |
| EugenMayer4 | it seems like https://github.com/svinota/pyroute2/issues/923 is the fix for this issue. But this would mean the docker image running metadata (some python app i assume) has an outdated pyroute2 ? | 10:57 |
| EugenMayer4 | currently deployed with kolla: | 11:00 |
| EugenMayer4 | pip3 list | grep kolla | 11:00 |
| EugenMayer4 | kolla 13.7.0 | 11:00 |
| EugenMayer4 | kolla-ansible 13.7.0 | 11:00 |
| mnasiadka | check inside the metadata agent container image - I mean the pyroute2 version | 11:06 |
| mnasiadka | it's mandated by what is in openstack requirements - https://github.com/openstack/requirements/blob/master/upper-constraints.txt (check proper branch) | 11:07 |
| EugenMayer4 | mnasiadka not sure how to check this inside the image. Is the system python used? | 11:14 |
| mnasiadka | yes, docker exec -it neutron_ovn_metadata_agent pip3 freeze | grep pyroute | 11:15 |
| EugenMayer4 | very interesting, runnin `docker pull quay.io/openstack.kolla/ubuntu-source-neutron-metadata-agent:xena` does actually pull a newer image when running this on the compute | 11:17 |
| EugenMayer4 | pip3 freeze | grep pyroute | 11:18 |
| EugenMayer4 | pyroute2==0.6.6 | 11:18 |
| EugenMayer4 | AFAICS this is the actual problematic version | 11:18 |
| EugenMayer4 | seems like in yoga 0.6.6 is still used | 11:19 |
| EugenMayer4 | it seems like kolla did not pull the images during the upgrade to 13.7 at all | 11:21 |
| EugenMayer4 | when i manually pull xena i have a more recent version then the ones i see on the computes that did not not pull | 11:22 |
| EugenMayer4 | mnasiadka entirely stuck here, cannot get anything up. Any help would be awesome | 12:01 |
| EugenMayer4 | the question is if i somehow can work-arround it by manually removing the IP | 12:04 |
| EugenMayer4 | i have manually removed the IP from each of the tap interfaces and could bring up the metadata service this way. I'am not sure how temporary this fix is | 12:40 |
| mnasiadka | EugenMayer4: if you're using published container images - it might make sense to run kolla-ansible pull to get latest images | 13:21 |
| EugenMayer4 | mnasiadka i did all that | 13:21 |
| EugenMayer4 | mnasiadka i was running the latest images. But the pyroute package is even too old in yoga, so it did not change anything | 13:21 |
| mnasiadka | yeah, I see that, it seems the fix is in 0.6.10 | 13:24 |
| mnasiadka | but upper-constraints.txt in xena and yoga is 0.6.6 | 13:25 |
| mnasiadka | https://bugs.launchpad.net/neutron/+bug/1991501 - here is a similar bug | 13:27 |
| EugenMayer4 | mnasiadka do you think updating to ZED could fix that? | 13:44 |
| mnasiadka | https://github.com/openstack/requirements/blob/fc7e2105e81c352602085bd2928a706d0ab8a80d/upper-constraints.txt#L151 - seems like it | 13:44 |
| EugenMayer4 | but there are not zed releases yet AFAICs | 13:45 |
| EugenMayer4 | quay.io/openstack.kolla/ubuntu-source-neutron-metadata-agent:zed is not available | 13:45 |
| EugenMayer4 | i guess zed is "master" https://docs.openstack.org/releasenotes/kolla-ansible/zed.html - so not released yet | 13:46 |
| EugenMayer4 | seems like the openstack release at least has been done already https://releases.openstack.org/ | 13:47 |
| mnasiadka | we renamed container images in Zed release | 13:58 |
| mnasiadka | there is no binary, so there is no source | 13:58 |
| mnasiadka | and yes, we did a release | 13:59 |
| mnasiadka | https://quay.io/repository/openstack.kolla/neutron-metadata-agent?tab=tags&tag=latest | 13:59 |
| mnasiadka | it's there | 13:59 |
| EugenMayer4 | ah interesting | 14:02 |
| EugenMayer4 | mnasiadka do you assume i can go from 13 to 15 directly or should i go from 13.7 to 14.7 and then to 15? | 14:03 |
| mnasiadka | you need to go from xena to yoga and then to zed | 14:03 |
| EugenMayer4 | I see | 14:04 |
| EugenMayer4 | Thank you! | 14:04 |
| EugenMayer4 | mnasiadka i guess i can go from 13.7 to 14.7 direct, i do not need to go to 14.0 first? | 14:04 |
| mnasiadka | yes, you can | 14:05 |
| EugenMayer4 | Thank you a lot, i guess i finish the k8s 1.25 upgrrade first here, since the rke2 cluster is freaking out a bit too due to PSP issues, then i try to go to yoga, i assume i will have the same issue again, fixing it and then going on. | 14:06 |
| EugenMayer4 | mnasiadka: I did check the upgrade notes for yoga but there was way less to consider compared to the update to xena or even "within xena" - anything you would warn to look at? | 14:07 |
| EugenMayer4 | one thing i'am not sure about yet is the change with nova_enable_external_metadata | 14:07 |
| mnasiadka | EugenMayer4: I think the major change is switch to Ubuntu Jammy - if you're using Ubuntu :) | 14:08 |
| mnasiadka | EugenMayer4: normally nobody needs nova metadata exposed to the internet | 14:08 |
| EugenMayer4 | thanks | 14:08 |
| EugenMayer4 | you mean the images did switch? Or is the host os expected to be jammy? | 14:08 |
| EugenMayer4 | asking because i use debian bullseye as a host os | 14:09 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/zed: Revert "[release] Use Zed images in stable/zed" https://review.opendev.org/c/openstack/kolla-ansible/+/868774 | 14:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/zed: Revert "[release] Update ansible-collection-kolla to stable/zed" https://review.opendev.org/c/openstack/kolla-ansible/+/868775 | 14:13 |
| EugenMayer4 | mnasiadka sorry for disturbing again. But could you shortly hint if the upgrade on jammy was related to the docker image os or what is expected to run as a host os? | 14:30 |
| mnasiadka | EugenMayer4: it's expected to be a host os, but I think we support Focal in Zed as a ,,transitional'' OS - https://github.com/openstack/kolla-ansible/blob/stable/zed/ansible/roles/prechecks/vars/main.yml | 14:32 |
| frickler | but we don't support ubuntu on debian or vice versa. for sure that's completely untested at least | 14:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering https://review.opendev.org/c/openstack/kolla-ansible/+/868929 | 15:20 |
| EugenMayer4 | how is that even a dependency? I mean, since everything is running on the containers frickler mnasiadka? Since i run debian bullseye, i could use the bullseye zed images too to match my host os, if that is such a requirement | 15:43 |
| EugenMayer4 | i did not intentionally "pick" ubuntu images, i assume that was just the default (iam yet also not aware where to configure this flavour / image type) | 15:44 |
| EugenMayer4 | i ran the prechecks today, so i assume it should have complained if i run ubuntu focal containers on a debian-bullseye? | 15:47 |
| EugenMayer4 | Just checked on docker container on the controller as an example | 15:49 |
| EugenMayer4 | cat /etc/debian_version | 15:49 |
| EugenMayer4 | bullseye/sid | 15:49 |
| EugenMayer4 | This means that quay.io/openstack.kolla/ubuntu-source-nova-compute:xena actually point to debian based images by default | 15:50 |
| mnasiadka | Ubuntu is debian based, so it’s less of a problem to mix, but we don’t support that. Maybe we should add some warning to prechecks. | 15:58 |
| EugenMayer4 | Do i mix at all? running bullseye as host and seeing that :xena is referenced, seems to be a match? :xena points to images based on debian:bullseye like quay.io/openstack.kolla/ubuntu-source-nova-compute:xena | 16:51 |
| EugenMayer4 | I'am kind of confused here, that you guys mention that i mix OS. It either is auto-selected, thus since i use debia bullseye as host os, kolla selects :xena and thus a bullseye image, or it just happens to be the default by accidenet and i should have actually "set it to ensure" | 16:52 |
| opendevreview | Dawud proposed openstack/kolla-ansible master: Add filter to label groups for prometheus https://review.opendev.org/c/openstack/kolla-ansible/+/868930 | 17:04 |
| opendevreview | Dawud proposed openstack/kolla-ansible master: Add filter to label groups for prometheus https://review.opendev.org/c/openstack/kolla-ansible/+/868930 | 19:03 |
| *** Guest689 is now known as darkkilla | 20:22 | |
| *** EugenMayer48 is now known as EugenMayer4 | 20:34 | |
| opendevreview | Dawud proposed openstack/kolla-ansible master: Add filter to label groups for prometheus https://review.opendev.org/c/openstack/kolla-ansible/+/868930 | 20:37 |
| dking_tmp | This isn't specifically a kolla question, but is there anybody around who might be able to help me troubleshoot why a I can't seem to ping a floating IP on a very basic new kolla-ansible deploy? | 21:54 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!