Monday, 2024-08-26

opendevreviewRoman Krček proposed openstack/kolla-ansible master: Rewrite kolla-ansible CLI to python  https://review.opendev.org/c/openstack/kolla-ansible/+/92311005:49
opendevreviewRoman Krček proposed openstack/kolla-ansible master: Don't notify handlers during copy-cert  https://review.opendev.org/c/openstack/kolla-ansible/+/92465106:00
opendevreviewRoman Krček proposed openstack/kolla-ansible master: Performance: Don't notify handlers during config  https://review.opendev.org/c/openstack/kolla-ansible/+/74516406:00
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: switch to use mariadb flavored commands  https://review.opendev.org/c/openstack/kolla-ansible/+/92703906:49
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: use ironic_tftp_listen_address in dnsmasq.conf  https://review.opendev.org/c/openstack/kolla-ansible/+/92707907:04
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add certificates for RabbitMQ internode TLS  https://review.opendev.org/c/openstack/kolla-ansible/+/92138007:28
opendevreviewGrzegorz Bialas proposed openstack/kolla-ansible master: Fix links in docs for unmaintained releases  https://review.opendev.org/c/openstack/kolla-ansible/+/92702208:21
opendevreviewSven Kieske proposed openstack/kolla-ansible master: harden haproxy TLS configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/91540308:30
SvenKieskecan some core look at https://review.opendev.org/c/openstack/kolla-ansible/+/898307 ? :)08:39
opendevreviewSven Kieske proposed openstack/kolla-ansible stable/2024.1: CI: improve get_logs.sh  https://review.opendev.org/c/openstack/kolla-ansible/+/92708908:41
opendevreviewSven Kieske proposed openstack/kolla-ansible stable/2023.2: CI: improve get_logs.sh  https://review.opendev.org/c/openstack/kolla-ansible/+/92709008:41
opendevreviewSven Kieske proposed openstack/kolla-ansible stable/2023.1: CI: improve get_logs.sh  https://review.opendev.org/c/openstack/kolla-ansible/+/92709108:41
opendevreviewGrzegorz Bialas proposed openstack/kolla-ansible master: Fix links in docs for unmaintained releases  https://review.opendev.org/c/openstack/kolla-ansible/+/92702208:49
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Use py3 as the default runtime for tox  https://review.opendev.org/c/openstack/kolla-ansible/+/89830709:12
mnasiadkaI did have a look, my OCD won with that whitespace09:13
SvenKieske:D that's bothering me for weeks already09:16
opendevreviewGrzegorz Bialas proposed openstack/kolla-ansible master: Fix links in docs for unmaintained releases  https://review.opendev.org/c/openstack/kolla-ansible/+/92702209:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default.  https://review.opendev.org/c/openstack/kolla-ansible/+/90495910:06
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Disable wsrep_provider plugin  https://review.opendev.org/c/openstack/kolla-ansible/+/92709610:07
opendevreviewPierre Riteau proposed openstack/kayobe master: [DNM] Test seed-vm job with Ubuntu image without EFI changes  https://review.opendev.org/c/openstack/kayobe/+/92710010:39
opendevreviewMerged openstack/kolla-ansible master: Use py3 as the default runtime for tox  https://review.opendev.org/c/openstack/kolla-ansible/+/89830710:46
frickler.oO(one small space for gerrit, a giant leap for kolla reviewers) :D11:06
mnasiadkahaha11:07
opendevreviewPierre Riteau proposed openstack/kayobe master: [DNM] Test seed-vm job with Ubuntu image without EFI changes  https://review.opendev.org/c/openstack/kayobe/+/92710011:14
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Implement TLS for Redis  https://review.opendev.org/c/openstack/kolla-ansible/+/90918811:32
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add caching options  https://review.opendev.org/c/openstack/kolla-ansible/+/92710411:32
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add Redis as caching backend for Keystone  https://review.opendev.org/c/openstack/kolla-ansible/+/90920111:33
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add Redis as caching backend for Nova  https://review.opendev.org/c/openstack/kolla-ansible/+/90920311:34
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add Redis as caching backend for Placement  https://review.opendev.org/c/openstack/kolla-ansible/+/90922211:35
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add Redis as caching backend for Heat  https://review.opendev.org/c/openstack/kolla-ansible/+/90922411:35
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add Redis as caching backend for Ceilometer  https://review.opendev.org/c/openstack/kolla-ansible/+/92672111:36
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Fix upgrade jobs failing on adding new certificates  https://review.opendev.org/c/openstack/kolla-ansible/+/92628411:53
SvenKieskemnasiadka: wondering if the processes inside neutron_ovn_agent container run in the correct pid namespace when I look at the logs? https://review.opendev.org/c/openstack/kolla-ansible/+/904959 ps output is either empty or cut off11:54
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default.  https://review.opendev.org/c/openstack/kolla-ansible/+/90495911:55
mnasiadkaSvenKieske: I just love debugging in CI ;)11:55
SvenKieskewho doesn't? :) ah there was a stray pipe :D11:56
SvenKieskethe ovn status checks below that seem also rather brittle tbh11:57
mnasiadkafor now it's a WIP with debugging needed ;)11:58
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Fix upgrade jobs failing on adding new certificates  https://review.opendev.org/c/openstack/kolla-ansible/+/92628412:08
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Fix upgrade jobs failing on adding new certificates  https://review.opendev.org/c/openstack/kolla-ansible/+/92628412:09
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: CI: Fix upgrade jobs failing on adding new certificates  https://review.opendev.org/c/openstack/kolla-ansible/+/92628412:10
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add certificates for RabbitMQ internode TLS  https://review.opendev.org/c/openstack/kolla-ansible/+/92138012:11
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: CI: Fix upgrade jobs failing on adding new certificates  https://review.opendev.org/c/openstack/kolla-ansible/+/92628412:11
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add support for RabbitMQ internode tls  https://review.opendev.org/c/openstack/kolla-ansible/+/92138112:11
opendevreviewMichal Nasiadka proposed openstack/kolla master: mariadb: Bump to 11.4  https://review.opendev.org/c/openstack/kolla/+/92698112:14
opendevreviewMerged openstack/kolla-ansible stable/2024.1: Fix prometheus.yml templating  https://review.opendev.org/c/openstack/kolla-ansible/+/92661812:17
mnasiadkakevko: can you please have a look here: https://review.opendev.org/c/openstack/kolla-ansible/+/925971 ?12:21
opendevreviewMerged openstack/kolla-ansible master: Add REQUESTS_CA_BUNDLE to kolla-toolbox container  https://review.opendev.org/c/openstack/kolla-ansible/+/92310512:24
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Enable internal/backend TLS in LE scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/92621012:26
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Enable internal/backend TLS in LE scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/92621012:27
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: CI: Fix upgrade jobs failing on adding new certificates  https://review.opendev.org/c/openstack/kolla-ansible/+/92628413:04
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Switch mariadb's loadbalancer from HAProxy to ProxySQL  https://review.opendev.org/c/openstack/kolla-ansible/+/91372413:04
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add backend TLS between MariaDB and ProxySQL  https://review.opendev.org/c/openstack/kolla-ansible/+/90991213:04
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add frontend TLS ability to ProxySQL  https://review.opendev.org/c/openstack/kolla-ansible/+/92550013:04
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add frontend database TLS for Keystone  https://review.opendev.org/c/openstack/kolla-ansible/+/92550713:04
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add caching options  https://review.opendev.org/c/openstack/kolla-ansible/+/92710413:12
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Disable wsrep_provider plugin  https://review.opendev.org/c/openstack/kolla-ansible/+/92709614:04
opendevreviewMichal Nasiadka proposed openstack/kolla master: DNM: Add some debug to healthcheck_port  https://review.opendev.org/c/openstack/kolla/+/92714914:46
opendevreviewMichal Nasiadka proposed openstack/kolla master: mariadb: Add label before upgrade to 11  https://review.opendev.org/c/openstack/kolla/+/92715815:03
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: switch to use mariadb flavored commands  https://review.opendev.org/c/openstack/kolla-ansible/+/92703915:04
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Disable wsrep_provider plugin  https://review.opendev.org/c/openstack/kolla-ansible/+/92709615:12
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Disable wsrep_provider plugin  https://review.opendev.org/c/openstack/kolla-ansible/+/92709615:12
opendevreviewMichal Nasiadka proposed openstack/kolla master: mariadb: Bump to 11.4  https://review.opendev.org/c/openstack/kolla/+/92698115:13
opendevreviewMichal Nasiadka proposed openstack/kolla master: mariadb: Bump to 11.4  https://review.opendev.org/c/openstack/kolla/+/92698115:13
opendevreviewMichal Nasiadka proposed openstack/kolla master: CI: Disable CentOS/Rocky/Ubuntu aarch64 jobs  https://review.opendev.org/c/openstack/kolla/+/92716315:17
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Enable internal/backend TLS in LE scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/92621015:20
mnasiadkafrickler: https://review.opendev.org/c/openstack/kolla/+/927163 - posted that to ease the strain on OSUOSL a bit - might be we'll need to disable the Debian jobs as well until situation stabilises... (cc: fungi)15:22
fricklermnasiadka: ack, +6 ;) this morning when I saw the arm pipeline was empty, I was hoping that things had recovered, sadly it doesn't look like that15:25
mnasiadkawell, basically the debian build job finishes in 2hr 59minutes and then it's success - or we get over it and it gets timed out15:26
mnasiadkacomparing to 30 minutes in the past - I don't think we should bump up the time limit...15:26
fricklerack15:28
samcat116Hi all, does someone know the rationale about setting the rpc workers this low: https://github.com/openstack/kolla-ansible/blob/ab4a79d552e3d382bd31e8e08a5931c035db2c69/ansible/group_vars/all.yml#L68916:33
samcat116with modern servers I can't imagine a scenario where the number of cpus calculated there would be used rather than the hard coded values, resulting in a pretty small number of workers16:33
fricklersamcat116: iiuc the idea is to have a default that works well for the limited resources in CI or poc deployments. for your production setup you can tune it as needed16:40
samcat116Is there any docs or guidance on that tuning?16:40
samcat116We’re seeing some intermittent neutron 504 timeouts and I’m wondering if this is partially responsible16:41
samcat116On a cloud of around 1500vms and 3000-4000 networks16:41
samcat116I get that’s likely on the neutron side and not the kills side obviously16:42
opendevreviewMerged openstack/kolla master: CI: Disable CentOS/Rocky/Ubuntu aarch64 jobs  https://review.opendev.org/c/openstack/kolla/+/92716316:54
fricklersamcat116: that's a pretty high network/vm ratio I'd say. I'm not aware of any specific tuning guide, but maybe https://docs.openstack.org/large-scale/index.html can be interesting for you17:26
mnasiadkasamcat116: with ML2/OVS most probably RMQ is more problematic than the number of workers - not speaking about the storm that will happen when you restart l3-agent/openvswitch-agent17:43
samcat116we are running OVN, so after reading more I guess these aren't even useful17:43
mnasiadkaWell, then maybe it would be good to start a thread on #openstack-neutron to find what those 504s are really linked to17:45
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default.  https://review.opendev.org/c/openstack/kolla-ansible/+/90495917:49
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Disable wsrep_provider plugin  https://review.opendev.org/c/openstack/kolla-ansible/+/92709617:52
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Disable wsrep_provider plugin  https://review.opendev.org/c/openstack/kolla-ansible/+/92709617:52
opendevreviewMichal Nasiadka proposed openstack/kolla master: DNM: Run fio tests  https://review.opendev.org/c/openstack/kolla/+/92721020:06
opendevreviewMichal Nasiadka proposed openstack/kolla master: DNM: Run fio tests  https://review.opendev.org/c/openstack/kolla/+/92721020:09
opendevreviewMichal Nasiadka proposed openstack/kolla master: DNM: Run fio tests  https://review.opendev.org/c/openstack/kolla/+/92721020:14
opendevreviewMichal Nasiadka proposed openstack/kolla master: DNM: Run fio tests  https://review.opendev.org/c/openstack/kolla/+/92721020:18

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!