| mnasiadka | frickler, bbezak, kevko: https://review.opendev.org/c/openstack/kolla-ansible/+/892322 - let's merge this, for now I left bullseye support on master (for the upgrade job to pass) - once backported to antelope - I'll remove support and amend docs. | 06:30 |
|---|---|---|
| mnasiadka | frickler, kevko: I just noticed most of cephadm jobs failures is basically because of rabbitmq - and I see cephadm jobs are green in https://review.opendev.org/c/openstack/kolla-ansible/+/882825 - should we merge that? | 07:03 |
| frickler | mnasiadka: comment about the reno, but otherwise I'm fine with that | 07:10 |
| mnasiadka | frickler: responded | 07:11 |
| mnasiadka | frickler: thanks | 07:17 |
| mnasiadka | frickler: forgot the depends-on - https://review.opendev.org/c/openstack/kolla-ansible/+/890391/4 - can you have a look as well? | 07:37 |
| SvenKieske | o/ | 07:38 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add command to force reset the state of RabbitMQ https://review.opendev.org/c/openstack/kolla-ansible/+/890391 | 07:49 |
| frickler | mnasiadka: so that's how you sneak in review requests? ;) but I admit I simply missed that stack, too | 07:49 |
| mnasiadka | fixed the nit, let's see if the cephadm-upgrade jobs get better | 07:50 |
| mnasiadka | frickler: I thought sneaking in review requests is my job ;) | 07:54 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible master: Enable RabbitMQ HA queues by default https://review.opendev.org/c/openstack/kolla-ansible/+/882825 | 07:59 |
| frickler | mnasiadka: it is and it seems you're good at it :) | 08:05 |
| opendevreview | Merged openstack/kolla-ansible master: debian: Add Bookworm Host OS support https://review.opendev.org/c/openstack/kolla-ansible/+/892322 | 08:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support https://review.opendev.org/c/openstack/kolla-ansible/+/892744 | 08:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support https://review.opendev.org/c/openstack/kolla-ansible/+/892744 | 08:29 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support https://review.opendev.org/c/openstack/kolla-ansible/+/892744 | 08:32 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add support of podman deployment https://review.opendev.org/c/openstack/kolla-ansible/+/799229 | 08:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support https://review.opendev.org/c/openstack/kolla-ansible/+/892744 | 08:51 |
| opendevreview | Merged openstack/kayobe stable/2023.1: CI: fetch systemd configuration including kolla-generated unit files https://review.opendev.org/c/openstack/kayobe/+/892589 | 08:55 |
| opendevreview | Merged openstack/kayobe stable/zed: CI: fetch systemd configuration including kolla-generated unit files https://review.opendev.org/c/openstack/kayobe/+/892590 | 08:55 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add command to force reset the state of RabbitMQ https://review.opendev.org/c/openstack/kolla-ansible/+/890391 | 10:09 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Enable RabbitMQ HA queues by default https://review.opendev.org/c/openstack/kolla-ansible/+/882825 | 10:10 |
| mnasiadka | (was not rebased on top of Bookworm patch) | 10:10 |
| SvenKieske | i feel we really should write something down to reach consensus on how we do reviews, a coding guideline, or something in the way. | 10:45 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Configure OVN to emit "need to frag" packets in case of MTU mismatch https://review.opendev.org/c/openstack/kolla-ansible/+/892777 | 10:57 |
| mnasiadka | SvenKieske: I would like to understand how many people read docs at all ;) | 11:12 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Remove after-Zed TODOs https://review.opendev.org/c/openstack/kolla-ansible/+/892783 | 11:29 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Remove after-Zed TODOs https://review.opendev.org/c/openstack/kolla-ansible/+/892783 | 11:32 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add support of podman deployment https://review.opendev.org/c/openstack/kolla-ansible/+/799229 | 11:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Remove after-Zed TODOs https://review.opendev.org/c/openstack/kolla-ansible/+/892783 | 11:55 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Configure OVN to emit "need to frag" packets in case of MTU mismatch https://review.opendev.org/c/openstack/kolla-ansible/+/892777 | 12:07 |
| SvenKieske | mnasiadka: why do you want to understand that? it doesn't change the number of people who did. ;) but at least you can point people somewhere on how to do things without having to retype the same explanations each time. it also helps to build and document consensus in the community. | 12:17 |
| mnasiadka | SvenKieske: Even I don't remember half of the things that are documented, so pointing people is sometimes complicated as well ;) | 12:20 |
| frickler | well we do have https://docs.openstack.org/kolla/latest/contributor/code-reviews.html and https://docs.openstack.org/kolla/latest/contributor/kolla-design-philosophy.html , feel free to submit patches | 12:23 |
| kevko | for me documentation is grep -ri something kolla-ansible-git/ :D | 12:25 |
| kevko | btw, folks ..we see this in logs 2023-08-23 07:25:44.088 698 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-82870e77-074a-4bc8-b248-f3faf65b54f7 - - - - -] Refusing to bind port 283e51ae-ac59-4d8c-8d1c-d10d4d341d3d to dead agent: <neutron.plugins.ml2.drivers.ovn.agent.neutron_agent.ControllerGatewayAgent object at | 12:26 |
| kevko | 0x7f4fddade700> | 12:26 |
| kevko | openstack yoga and ovn | 12:26 |
| kevko | did anyone see this ? | 12:26 |
| mnasiadka | is your agent dead? | 12:26 |
| kevko | network agent list not showing dead ... | 12:27 |
| kevko | but restart neutron-ovn-metadata agent helps | 12:27 |
| kevko | metadata-agent just not logging | 12:28 |
| kevko | but i have informations from customer ...not directly ... probably i need to study deeper | 12:28 |
| kevko | but first i asked here :P | 12:28 |
| kevko | i've get this also from them | 12:28 |
| kevko | 2023-08-11 16:09:51.539 7 INFO neutron.agent.ovn.metadata.agent [-] Cleaning up ovnmeta-5a7e40cb-fd88-4f5c-a44f-7686622b955d namespace which is not needed anymore | 12:28 |
| kevko | 2023-08-14 08:55:30.564 699 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.157.106.71:6642: receive error: Connection reset by peer | 12:28 |
| kevko | 2023-08-14 08:55:30.567 699 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.157.106.71:6642: connection dropped (Connection reset by peer) | 12:28 |
| kevko | 2023-08-14 08:55:30.568 699 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.157.106.6:6642: connecting... | 12:28 |
| kevko | 2023-08-14 08:55:30.569 699 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.157.106.6:6642: connected | 12:28 |
| mnasiadka | not really, haven't seen that, but I see you've already raised a backport for a related patch in Neutron ;-) | 12:29 |
| kevko | yep :D | 12:29 |
| mnasiadka | but the cause might not be the same | 12:29 |
| kevko | but just to see if tests are passing | 12:29 |
| kevko | yeah ..as a hotfix i think the dead_agent_interval (or similar config in neurton - don't remember ) can fix the issue ...but that's not proper fix i think | 12:30 |
| mnasiadka | well, maybe your OVN DBs are busy doing something else ;) | 12:30 |
| kevko | btw, what about LE ? | 12:31 |
| mnasiadka | needs reviews, will try next week, now I'm busy with something more priority downstream | 12:32 |
| kevko | mnasiadka: what about this ..did you see ? https://paste.openstack.org/show/bdC30SL8yp7dAD49RnUk/ | 12:41 |
| mnasiadka | kevko: no, you might want to check ovn-nb-db logs :) | 12:42 |
| opendevreview | Merged openstack/kayobe stable/yoga: Fix for seed-containers being unable to use password protected registry https://review.opendev.org/c/openstack/kayobe/+/892479 | 12:42 |
| opendevreview | Martin Hiner proposed openstack/ansible-collection-kolla master: Adding podman support https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240 | 12:42 |
| opendevreview | Merged openstack/kayobe stable/zed: Fix for seed-containers being unable to use password protected registry https://review.opendev.org/c/openstack/kayobe/+/892478 | 12:42 |
| mnasiadka | kevko: erm, sb - it's 6642 so it's SB | 12:42 |
| opendevreview | Merged openstack/kayobe stable/2023.1: Fix for seed-containers being unable to use password protected registry https://review.opendev.org/c/openstack/kayobe/+/892480 | 12:46 |
| kevko | btw, mnasiadka how did you notice my neutron patches so quick ? :D | 12:52 |
| opendevreview | Merged openstack/kayobe stable/xena: Fix for seed-containers being unable to use password protected registry https://review.opendev.org/c/openstack/kayobe/+/892501 | 12:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Remove after-Zed TODOs https://review.opendev.org/c/openstack/kolla-ansible/+/892783 | 13:14 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering https://review.opendev.org/c/openstack/kolla-ansible/+/868929 | 13:28 |
| mnasiadka | frickler: seems rax-ord is having a bad day - all gating jobs for 890391 just failed with repository issues on bootstrap-servers | 13:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ansible: bump min version to 2.14 and max to 2.15 https://review.opendev.org/c/openstack/kolla-ansible/+/887227 | 13:35 |
| mnasiadka | kevko: regarding LE - did you update the Kolla patch after hrw's comments? you were to write some functions or something? | 13:36 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support https://review.opendev.org/c/openstack/kolla-ansible/+/892744 | 13:39 |
| frickler | mnasiadka: that looks like our mirror configuration for bookworm may be incomplete, iirc it is correct that the mirrored repo isn't signed | 13:52 |
| mnasiadka | frickler: rocky9 job failed on docker repo, but bookworm did not fail in the past | 13:53 |
| mnasiadka | so something is fishy | 13:53 |
| frickler | but we didn't really run on bookworm nodes earlier, did we? rocky9 failure may be unrelated. I'll check for other issues anyway | 13:55 |
| mnasiadka | well, we did - we merged the bookworm patch in master :) | 13:57 |
| frickler | hmm, o.k., let's take this to #opendev then | 13:59 |
| frickler | the 4th job did not run on rax-ord and still failed | 14:01 |
| opendevreview | Merged openstack/kayobe stable/yoga: CI: fetch systemd configuration including kolla-generated unit files https://review.opendev.org/c/openstack/kayobe/+/892591 | 14:05 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Add command to force reset the state of RabbitMQ https://review.opendev.org/c/openstack/kolla-ansible/+/890391 | 14:05 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Enable RabbitMQ HA queues by default https://review.opendev.org/c/openstack/kolla-ansible/+/882825 | 14:05 |
| opendevreview | Martin Hiner proposed openstack/ansible-collection-kolla master: Adding podman support https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240 | 14:34 |
| kevko | mnasiadka yep, i reworked little bit kolla patch but didn't send for review | 14:41 |
| kevko | probably i will send on monday | 14:41 |
| kevko | but kolla-ansible part will not be changed | 14:41 |
| fungi | frickler: mnasiadka: see further comments in #opendev, but after i figured out how to pull apart kolla's logs, the errors pretty clearly indicate issues downloading packages from docker.com directly (and this happened in more than one of our donor providers, so seems to be globally impacting) | 14:47 |
| fungi | we do generate reprepro mirrors of docker.com's ubuntu packages, but it doesn't look like we currently mirror anything from there for non-ubuntu distros | 14:52 |
| fungi | we supply caching reverse web proxies for download.docker.com, but that's not going to solve problems with the contents of that site | 14:54 |
| fungi | so anyway, it seems like you should be looking to the docker.com staff, it doesn't appear to be anything within our sphere of control | 14:56 |
| fungi | seems like that would also explain why your ubuntu jobs succeeded while your debian and rocky9 jobs failed | 14:57 |
| fungi | assuming you're using our ubuntu mirror of those anyway (we enforce mirror coherency for the ubuntu packages by mirroring them with reprepro) | 14:58 |
| mnasiadka | fungi: thanks, will check again on Monday - 5pm here | 15:04 |
| mnasiadka | fungi: are you guys open to mirroring download.docker.com content or not really? | 15:05 |
| mnasiadka | fungi: ah, caching reverse proxies, that’s usually enough ;) | 15:08 |
| spatel | mnasiadka I am seeing very strange behavior with rbd_secret_uuid: (let me explain you) | 15:08 |
| spatel | I deployed openstack without ceph and rbd_secret_uuid was some random foobar | 15:09 |
| spatel | later I have integrate external ceph with openstack and now my nova-compute throwing error when I attach volume, its saying secret not found. after tracking I found nova-compute looking for ceph fsid UUID | 15:10 |
| fungi | mnasiadka: we're clearly open to mirroring things from there since we do it for their ubuntu packages. we could easily add similar mirrors for the debian packages. for rhel-based distros though, we're not aware of available tools for enforcing mirror coherency so it probably wouldn't help situations like this one where the contents of their package repository are bad | 15:10 |
| spatel | I have change default generated rbd_secret_uuid in passwords.yml to ceph fsid | 15:10 |
| spatel | is this how kolla-ansible handle rbd_secret_uuid ? | 15:10 |
| spatel | This issue only show up in zed release. I haven't seen in past | 15:11 |
| frickler | I think we do not use the ubuntu docker mirror yet. I agree we should add debian there and then make kolla CI use both | 15:12 |
| spatel | is kolla-ansible now using opensearch for central logging? | 15:52 |
| rockey | spatel: yes | 15:54 |
| spatel | so no more ELK ? | 15:55 |
| rockey | EFK has turned into FOO-D :)) | 15:55 |
| rockey | FluentD, Opensearch, Opensearch-Dashboard | 15:56 |
| spatel | oh so its Licensing issue with ELK | 15:56 |
| rockey | yeah | 15:56 |
| rockey | it is elastisearch fork from version 7.11 | 15:56 |
| spatel | Does Zed release support opensearch or its still not ready for production? | 15:56 |
| rockey | it is in zed and you can build/run it | 15:58 |
| spatel | I can just download docker image and push to my local repository and give it a shot | 15:59 |
| rockey | spatel: headsup tho, in the upgrade scenario, it migrates elasticsearch docker volume into opensearch | 16:00 |
| rockey | have a look in .../roles/opensearch/tasks/upgrade.yml | 16:00 |
| spatel | This is new deployment so assuming it won't create any issue | 16:01 |
| rockey | then go nuts | 16:02 |
| spatel | haha | 16:03 |
| SvenKieske | as always: read the release notes :) there's a specific mention of the opensearch stuff | 16:15 |
| spatel | rockey do I need to do anything in multinode inventory to tell deploy it to X server? | 16:16 |
| spatel | SvenKieske 100% agreed :) | 16:16 |
| spatel | I have dedicated server for logging and I want it just use that for centralize logging. | 16:17 |
| SvenKieske | https://docs.openstack.org/releasenotes/kolla/zed.html#upgrade-notes | 16:19 |
| SvenKieske | "elasticsearch, kibana and logstash images have been dropped. Zed release brings in support for opensearch and opensearch-dashboards but there’s no equivalent for logstash." | 16:19 |
| SvenKieske | have a nice weekend all o/ | 16:19 |
| SvenKieske | oh btw, terraform is forked: | 16:20 |
| SvenKieske | https://opentf.org/announcement | 16:20 |
| spatel | SvenKieske Thanks!! | 16:21 |
| SvenKieske | you're welcome :) | 16:22 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance with S3 backend support https://review.opendev.org/c/openstack/kolla-ansible/+/844614 | 16:36 |
| spatel | rockey I have install opensearch and following doc which suggest create index but I am trying to do but next step is disable and not moving forward - https://ibb.co/KVjJTz0 | 16:42 |
| spatel | am I missing something? | 16:43 |
| rockey[m] | <spatel> "rockey I have install opensearch..." <- This might suck, but in matrix client now. | 17:11 |
| rockey[m] | Create an index named `flog-*` and im fairly sure you will be good | 17:11 |
| spatel | Its not allowing me to create :( | 17:11 |
| spatel | the button is grayed and not able to move | 17:11 |
| rockey[m] | Alright | 17:12 |
| spatel | is there a different way to create index? | 17:12 |
| rockey[m] | You'll have to try, currently boozing up on an afterwork | 17:12 |
| rockey[m] | Hopefully you'll fix it | 17:13 |
| spatel | hmm let me see and try to google it out | 17:15 |
| opendevreview | Maximilian Sesterhenn proposed openstack/kolla-ansible master: WIP: Add ovn-bgp-agent / FRR / neutron-bgpvpn / Horizon BGPVPN dashboard support https://review.opendev.org/c/openstack/kolla-ansible/+/891622 | 17:31 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Add command to force reset the state of RabbitMQ https://review.opendev.org/c/openstack/kolla-ansible/+/890391 | 18:43 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Enable RabbitMQ HA queues by default https://review.opendev.org/c/openstack/kolla-ansible/+/882825 | 19:13 |
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/kolla-ansible master: DNM: testing designate https://review.opendev.org/c/openstack/kolla-ansible/+/892278 | 19:42 |
| opendevreview | Merged openstack/kolla-ansible master: Add command to force reset the state of RabbitMQ https://review.opendev.org/c/openstack/kolla-ansible/+/890391 | 20:43 |
| opendevreview | Merged openstack/kolla-ansible master: Enable RabbitMQ HA queues by default https://review.opendev.org/c/openstack/kolla-ansible/+/882825 | 21:15 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!