Friday, 2023-08-25

mnasiadkafrickler, bbezak, kevko: https://review.opendev.org/c/openstack/kolla-ansible/+/892322 - let's merge this, for now I left bullseye support on master (for the upgrade job to pass) - once backported to antelope - I'll remove support and amend docs.06:30
mnasiadkafrickler, kevko: I just noticed most of cephadm jobs failures is basically because of rabbitmq - and I see cephadm jobs are green in https://review.opendev.org/c/openstack/kolla-ansible/+/882825 - should we merge that?07:03
fricklermnasiadka: comment about the reno, but otherwise I'm fine with that07:10
mnasiadkafrickler: responded07:11
mnasiadkafrickler: thanks07:17
mnasiadkafrickler: forgot the depends-on - https://review.opendev.org/c/openstack/kolla-ansible/+/890391/4 - can you have a look as well?07:37
SvenKieskeo/07:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Add command to force reset the state of RabbitMQ  https://review.opendev.org/c/openstack/kolla-ansible/+/89039107:49
fricklermnasiadka: so that's how you sneak in review requests? ;) but I admit I simply missed that stack, too07:49
mnasiadkafixed the nit, let's see if the cephadm-upgrade jobs get better07:50
mnasiadkafrickler: I thought sneaking in review requests is my job ;)07:54
opendevreviewDr. Jens Harbott proposed openstack/kolla-ansible master: Enable RabbitMQ HA queues by default  https://review.opendev.org/c/openstack/kolla-ansible/+/88282507:59
fricklermnasiadka: it is and it seems you're good at it :)08:05
opendevreviewMerged openstack/kolla-ansible master: debian: Add Bookworm Host OS support  https://review.opendev.org/c/openstack/kolla-ansible/+/89232208:25
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support  https://review.opendev.org/c/openstack/kolla-ansible/+/89274408:28
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support  https://review.opendev.org/c/openstack/kolla-ansible/+/89274408:29
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support  https://review.opendev.org/c/openstack/kolla-ansible/+/89274408:32
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Add support of podman deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/79922908:46
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support  https://review.opendev.org/c/openstack/kolla-ansible/+/89274408:51
opendevreviewMerged openstack/kayobe stable/2023.1: CI: fetch systemd configuration including kolla-generated unit files  https://review.opendev.org/c/openstack/kayobe/+/89258908:55
opendevreviewMerged openstack/kayobe stable/zed: CI: fetch systemd configuration including kolla-generated unit files  https://review.opendev.org/c/openstack/kayobe/+/89259008:55
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Add command to force reset the state of RabbitMQ  https://review.opendev.org/c/openstack/kolla-ansible/+/89039110:09
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Enable RabbitMQ HA queues by default  https://review.opendev.org/c/openstack/kolla-ansible/+/88282510:10
mnasiadka(was not rebased on top of Bookworm patch)10:10
SvenKieskei feel we really should write something down to reach consensus on how we do reviews, a coding guideline, or something in the way.10:45
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Configure OVN to emit "need to frag" packets in case of MTU mismatch  https://review.opendev.org/c/openstack/kolla-ansible/+/89277710:57
mnasiadkaSvenKieske: I would like to understand how many people read docs at all ;)11:12
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Remove after-Zed TODOs  https://review.opendev.org/c/openstack/kolla-ansible/+/89278311:29
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Remove after-Zed TODOs  https://review.opendev.org/c/openstack/kolla-ansible/+/89278311:32
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Add support of podman deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/79922911:41
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Remove after-Zed TODOs  https://review.opendev.org/c/openstack/kolla-ansible/+/89278311:55
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: Configure OVN to emit "need to frag" packets in case of MTU mismatch  https://review.opendev.org/c/openstack/kolla-ansible/+/89277712:07
SvenKieskemnasiadka: why do you want to understand that? it doesn't change the number of people who did. ;) but at least you can point people somewhere on how to do things without having to retype the same explanations each time. it also helps to build and document consensus in the community.12:17
mnasiadkaSvenKieske: Even I don't remember half of the things that are documented, so pointing people is sometimes complicated as well ;)12:20
fricklerwell we do have https://docs.openstack.org/kolla/latest/contributor/code-reviews.html and https://docs.openstack.org/kolla/latest/contributor/kolla-design-philosophy.html , feel free to submit patches12:23
kevkofor me documentation is grep -ri something kolla-ansible-git/ :D12:25
kevkobtw, folks ..we see this in logs 2023-08-23 07:25:44.088 698 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-82870e77-074a-4bc8-b248-f3faf65b54f7 - - - - -] Refusing to bind port 283e51ae-ac59-4d8c-8d1c-d10d4d341d3d to dead agent:  <neutron.plugins.ml2.drivers.ovn.agent.neutron_agent.ControllerGatewayAgent object at12:26
kevko0x7f4fddade700>12:26
kevkoopenstack yoga and ovn 12:26
kevkodid anyone see this ? 12:26
mnasiadkais your agent dead?12:26
kevkonetwork agent list not showing dead ...12:27
kevkobut restart neutron-ovn-metadata agent helps 12:27
kevkometadata-agent just not logging 12:28
kevkobut i have informations from customer ...not directly ... probably i need to study deeper 12:28
kevkobut first i asked here :P 12:28
kevkoi've get this also from them 12:28
kevko2023-08-11 16:09:51.539 7 INFO neutron.agent.ovn.metadata.agent [-] Cleaning up ovnmeta-5a7e40cb-fd88-4f5c-a44f-7686622b955d namespace which is not needed anymore12:28
kevko2023-08-14 08:55:30.564 699 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.157.106.71:6642: receive error: Connection reset by peer12:28
kevko2023-08-14 08:55:30.567 699 WARNING ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.157.106.71:6642: connection dropped (Connection reset by peer)12:28
kevko2023-08-14 08:55:30.568 699 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.157.106.6:6642: connecting...12:28
kevko2023-08-14 08:55:30.569 699 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:10.157.106.6:6642: connected12:28
mnasiadkanot really, haven't seen that, but I see you've already raised a backport for a related patch in Neutron ;-)12:29
kevkoyep :D 12:29
mnasiadkabut the cause might not be the same12:29
kevkobut just to see if tests are passing 12:29
kevkoyeah ..as a hotfix i think the dead_agent_interval (or similar config in neurton - don't remember ) can fix the issue ...but that's not proper fix i think 12:30
mnasiadkawell, maybe your OVN DBs are busy doing something else ;)12:30
kevkobtw, what about LE ? 12:31
mnasiadkaneeds reviews, will try next week, now I'm busy with something more priority downstream12:32
kevkomnasiadka: what about this ..did you see ? https://paste.openstack.org/show/bdC30SL8yp7dAD49RnUk/12:41
mnasiadkakevko: no, you might want to check ovn-nb-db logs :)12:42
opendevreviewMerged openstack/kayobe stable/yoga: Fix for seed-containers being unable to use password protected registry  https://review.opendev.org/c/openstack/kayobe/+/89247912:42
opendevreviewMartin Hiner proposed openstack/ansible-collection-kolla master: Adding podman support  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/85224012:42
opendevreviewMerged openstack/kayobe stable/zed: Fix for seed-containers being unable to use password protected registry  https://review.opendev.org/c/openstack/kayobe/+/89247812:42
mnasiadkakevko: erm, sb - it's 6642 so it's SB12:42
opendevreviewMerged openstack/kayobe stable/2023.1: Fix for seed-containers being unable to use password protected registry  https://review.opendev.org/c/openstack/kayobe/+/89248012:46
kevkobtw, mnasiadka how did you notice my neutron patches so quick ? :D 12:52
opendevreviewMerged openstack/kayobe stable/xena: Fix for seed-containers being unable to use password protected registry  https://review.opendev.org/c/openstack/kayobe/+/89250112:56
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Remove after-Zed TODOs  https://review.opendev.org/c/openstack/kolla-ansible/+/89278313:14
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering  https://review.opendev.org/c/openstack/kolla-ansible/+/86892913:28
mnasiadkafrickler: seems rax-ord is having a bad day - all gating jobs for 890391 just failed with repository issues on bootstrap-servers13:32
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ansible: bump min version to 2.14 and max to 2.15  https://review.opendev.org/c/openstack/kolla-ansible/+/88722713:35
mnasiadkakevko: regarding LE - did you update the Kolla patch after hrw's comments? you were to write some functions or something?13:36
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support  https://review.opendev.org/c/openstack/kolla-ansible/+/89274413:39
fricklermnasiadka: that looks like our mirror configuration for bookworm may be incomplete, iirc it is correct that the mirrored repo isn't signed13:52
mnasiadkafrickler: rocky9 job failed on docker repo, but bookworm did not fail in the past13:53
mnasiadkaso something is fishy13:53
fricklerbut we didn't really run on bookworm nodes earlier, did we? rocky9 failure may be unrelated. I'll check for other issues anyway13:55
mnasiadkawell, we did - we merged the bookworm patch in master :)13:57
fricklerhmm, o.k., let's take this to #opendev then13:59
fricklerthe 4th job did not run on rax-ord and still failed14:01
opendevreviewMerged openstack/kayobe stable/yoga: CI: fetch systemd configuration including kolla-generated unit files  https://review.opendev.org/c/openstack/kayobe/+/89259114:05
opendevreviewVerification of a change to openstack/kolla-ansible master failed: Add command to force reset the state of RabbitMQ  https://review.opendev.org/c/openstack/kolla-ansible/+/89039114:05
opendevreviewVerification of a change to openstack/kolla-ansible master failed: Enable RabbitMQ HA queues by default  https://review.opendev.org/c/openstack/kolla-ansible/+/88282514:05
opendevreviewMartin Hiner proposed openstack/ansible-collection-kolla master: Adding podman support  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/85224014:34
kevkomnasiadka yep, i reworked little bit kolla patch but didn't send for review 14:41
kevkoprobably i will send on monday 14:41
kevkobut kolla-ansible part will not be changed 14:41
fungifrickler: mnasiadka: see further comments in #opendev, but after i figured out how to pull apart kolla's logs, the errors pretty clearly indicate issues downloading packages from docker.com directly (and this happened in more than one of our donor providers, so seems to be globally impacting)14:47
fungiwe do generate reprepro mirrors of docker.com's ubuntu packages, but it doesn't look like we currently mirror anything from there for non-ubuntu distros14:52
fungiwe supply caching reverse web proxies for download.docker.com, but that's not going to solve problems with the contents of that site14:54
fungiso anyway, it seems like you should be looking to the docker.com staff, it doesn't appear to be anything within our sphere of control14:56
fungiseems like that would also explain why your ubuntu jobs succeeded while your debian and rocky9 jobs failed14:57
fungiassuming you're using our ubuntu mirror of those anyway (we enforce mirror coherency for the ubuntu packages by mirroring them with reprepro)14:58
mnasiadkafungi: thanks, will check again on Monday - 5pm here15:04
mnasiadkafungi: are you guys open to mirroring download.docker.com content or not really?15:05
mnasiadkafungi: ah, caching reverse proxies, that’s usually enough ;)15:08
spatelmnasiadka I am seeing very strange behavior with rbd_secret_uuid:  (let me explain you) 15:08
spatelI deployed openstack without ceph and rbd_secret_uuid was some random foobar 15:09
spatellater I have integrate external ceph with openstack and now my nova-compute throwing error when I attach volume, its saying secret not found. after tracking I found nova-compute looking for ceph fsid UUID 15:10
fungimnasiadka: we're clearly open to mirroring things from there since we do it for their ubuntu packages. we could easily add similar mirrors for the debian packages. for rhel-based distros though, we're not aware of available tools for enforcing mirror coherency so it probably wouldn't help situations like this one where the contents of their package repository are bad15:10
spatelI have change default generated rbd_secret_uuid in passwords.yml to ceph fsid 15:10
spatelis this how kolla-ansible handle rbd_secret_uuid ?15:10
spatelThis issue only show up in zed release. I haven't seen in past 15:11
fricklerI think we do not use the ubuntu docker mirror yet. I agree we should add debian there and then make kolla CI use both15:12
spatelis kolla-ansible now using opensearch for central logging?15:52
rockeyspatel: yes15:54
spatelso no more ELK ?15:55
rockeyEFK has turned into FOO-D :))15:55
rockeyFluentD, Opensearch, Opensearch-Dashboard15:56
spateloh so its Licensing issue with ELK15:56
rockeyyeah15:56
rockeyit is elastisearch fork from version 7.1115:56
spatelDoes Zed release support opensearch or its still not ready for production? 15:56
rockeyit is in zed and you can build/run it15:58
spatelI can just download docker image and push to my local repository and give it a shot 15:59
rockeyspatel: headsup tho, in the upgrade scenario, it migrates elasticsearch docker volume into opensearch16:00
rockeyhave a look in .../roles/opensearch/tasks/upgrade.yml16:00
spatelThis is new deployment so assuming it won't create any issue 16:01
rockeythen go nuts16:02
spatelhaha16:03
SvenKieskeas always: read the release notes :) there's a specific mention of the opensearch stuff16:15
spatelrockey do I need to do anything in multinode inventory to tell deploy it to X server?16:16
spatelSvenKieske 100% agreed :) 16:16
spatelI have dedicated server for logging and I want it just use that for centralize logging. 16:17
SvenKieskehttps://docs.openstack.org/releasenotes/kolla/zed.html#upgrade-notes16:19
SvenKieske"elasticsearch, kibana and logstash images have been dropped. Zed release brings in support for opensearch and opensearch-dashboards but there’s no equivalent for logstash."16:19
SvenKieskehave a nice weekend all o/16:19
SvenKieskeoh btw, terraform is forked:16:20
SvenKieskehttps://opentf.org/announcement16:20
spatelSvenKieske Thanks!! 16:21
SvenKieskeyou're welcome :)16:22
opendevreviewJuan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance with S3 backend support  https://review.opendev.org/c/openstack/kolla-ansible/+/84461416:36
spatelrockey I have install opensearch and following doc which suggest create index but I am trying to do but next step is disable and not moving forward - https://ibb.co/KVjJTz016:42
spatelam I missing something?16:43
rockey[m]<spatel> "rockey I have install opensearch..." <- This might suck, but in matrix client now.17:11
rockey[m]Create an index named `flog-*` and im fairly sure you will be good 17:11
spatelIts not allowing me to create :( 17:11
spatelthe button is grayed and not able to move 17:11
rockey[m]Alright17:12
spatelis there a different way to create index?17:12
rockey[m]You'll have to try, currently boozing up on an afterwork17:12
rockey[m]Hopefully you'll fix it17:13
spatelhmm let me see and try to google it out17:15
opendevreviewMaximilian Sesterhenn proposed openstack/kolla-ansible master: WIP: Add ovn-bgp-agent / FRR / neutron-bgpvpn / Horizon BGPVPN dashboard support  https://review.opendev.org/c/openstack/kolla-ansible/+/89162217:31
opendevreviewVerification of a change to openstack/kolla-ansible master failed: Add command to force reset the state of RabbitMQ  https://review.opendev.org/c/openstack/kolla-ansible/+/89039118:43
opendevreviewVerification of a change to openstack/kolla-ansible master failed: Enable RabbitMQ HA queues by default  https://review.opendev.org/c/openstack/kolla-ansible/+/88282519:13
opendevreviewErik Olof Gunnar Andersson proposed openstack/kolla-ansible master: DNM: testing designate  https://review.opendev.org/c/openstack/kolla-ansible/+/89227819:42
opendevreviewMerged openstack/kolla-ansible master: Add command to force reset the state of RabbitMQ  https://review.opendev.org/c/openstack/kolla-ansible/+/89039120:43
opendevreviewMerged openstack/kolla-ansible master: Enable RabbitMQ HA queues by default  https://review.opendev.org/c/openstack/kolla-ansible/+/88282521:15

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!