Friday, 2022-02-11

opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134000:25
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134000:46
opendevreviewconna proposed openstack/kolla-ansible master: Skyline: Add skyline role  https://review.opendev.org/c/openstack/kolla-ansible/+/82846401:36
opendevreviewMichal Nasiadka proposed openstack/kolla stable/wallaby: base: Drop usage of Ceph Nautilus from RDO  https://review.opendev.org/c/openstack/kolla/+/82822806:04
Fl1ntHi everyone!08:43
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/wallaby: Fix Apparmor libvirt profile removal  https://review.opendev.org/c/openstack/kolla-ansible/+/82883009:19
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/victoria: Fix Apparmor libvirt profile removal  https://review.opendev.org/c/openstack/kolla-ansible/+/82883109:19
opendevreviewPierre Riteau proposed openstack/kayobe master: CI: test fact caching  https://review.opendev.org/c/openstack/kayobe/+/80821809:28
Fl1ntPeople, is anyone experimenting the infamous RabbitMQ Connections timeout issue where RMQ Close connection to OS Services because they missed Heartbeat ? I'm having this issue on one of our relatively small platform and it drive me crazy09:29
Fl1ntThe network is OK, no latency, no congestion, no dropped packets.09:30
Fl1ntThis platform doesn't get any VMs running yet09:31
Fl1ntbut yet Horizon can't reach Services APIs from time to time (especially nova-api-wsgi) because HAProxy see its backend server as DOWN for no reason, then 3/4h later everything went back to normal09:32
Fl1ntand then this issue raise again from time to time09:32
Fl1ntI've of course check oslo.messaging rmq timeout, raise them, check for thread=1 on WSGI congfiguration all is OK09:33
Fl1ntbut the issue still arise09:33
Fl1ntThis platform is based on CentOS 8.5 and Kolla/Kolla-ansible 11.3.009:34
Fl1ntour monitoring shows that the servers literally are idle, services do nothing, no errors on either RMQ or MariaDB, only HAProxy saw a bunch of services DOWN and that went away as soon as we restarted the docker container.09:36
Fl1ntbut we still can catch RMQ/Services message about either the service missing Heartbeat and/or the RMQ Server being unreachable.09:36
Fl1ntwhich isn't on par with what the services monitoring probes themselves or manual inspection report.09:37
Fl1ntWe currently have a cluster of 3 RMQ Servers with each 1TB of RAM and 96 CPUs.09:38
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Refactor fluentd syslog logging  https://review.opendev.org/c/openstack/kolla-ansible/+/74511109:56
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Fix fluentd v1 buffer syntax issue  https://review.opendev.org/c/openstack/kolla-ansible/+/78062909:56
opendevreviewMark Goddard proposed openstack/kolla-ansible master: docs: add information about development libvirt TLS certs  https://review.opendev.org/c/openstack/kolla-ansible/+/82877209:57
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: haproxy: support different internal/external ports  https://review.opendev.org/c/openstack/kolla-ansible/+/82339509:58
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: haproxy: support fqdn based single frontend  https://review.opendev.org/c/openstack/kolla-ansible/+/82431209:59
opendevreviewMerged openstack/kolla-ansible master: [CI] Filter fluentd errors more  https://review.opendev.org/c/openstack/kolla-ansible/+/82859410:02
opendevreviewMerged openstack/kolla-ansible stable/xena: neutron: fix placement endpoint type configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/82875310:10
opendevreviewMerged openstack/kolla-ansible stable/wallaby: neutron: fix placement endpoint type configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/82875410:10
opendevreviewMerged openstack/kolla-ansible stable/victoria: neutron: fix placement endpoint type configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/82875510:10
opendevreviewMerged openstack/kolla-ansible stable/ussuri: neutron: fix placement endpoint type configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/82875610:10
jingvarFl1nt: Haproxy timeouts for backends?10:14
jingvartimeout client 3m10:15
jingvarqueue, server the same10:16
Fl1ntjingvar, ?10:26
Fl1ntjingvar, Yes it seems there was few backends that get there check failed10:28
jingvarFl1nt: /etc/kolla/haproxy/* find and increase timeout values10:34
Fl1ntjingvar, No I mean, I know that ^^10:34
jingvarok10:34
opendevreviewMerged openstack/kolla-ansible stable/xena: Fix Apparmor libvirt profile removal  https://review.opendev.org/c/openstack/kolla-ansible/+/82869210:35
Fl1ntI wanted to know if anyone was facing such issue with an empty/almost empty platform and if so which values did you put and why.10:35
jingvarwe plays with OVN, it fails on 3 controlers , but it works on 1 controller deployment (tempest 60 threads).  I'll try to place OVNs DBs behind haproxy and VIP10:39
mnasiadkajingvar: why?10:40
jingvarmetadata agent walks through avalible dbs endpoints  for leader10:43
jingvarlost connection, reconnet, wait, reconect etc10:43
mnasiadkayes, that's how it works10:43
mnasiadkahow do you plan to check which OVN DB instance is master?10:44
jingvarit is pain10:44
mnasiadkapain in what?10:44
Fl1ntin the ass ? :D10:44
mnasiadkaOVN DBs are doing periodic snapshots, they will change leader every 24h10:44
mnasiadkaand all services using it, need to be connected to the leader10:45
jingvarI can check who is leader only from ovns container, not from haproxy10:45
mnasiadkaand it's a TCP service, and all code was written with direct connection in mind, not through a load balancer10:46
jingvardances around dbs happens until tempest run (30min)10:46
opendevreviewMark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/78227310:46
mnasiadkajingvar: until? or during?10:47
opendevreviewMark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/78227310:47
jingvarsorry, durring10:48
mnasiadkajingvar: make sure you have this in your Neutron code: https://review.opendev.org/c/openstack/neutron/+/82232810:48
jingvarthank will check, I used wallaby source images 1 week age as I remember10:51
Fl1ntmnasiadka, which version of OS enforce OVN usage?10:52
mnasiadkaFl1nt: none10:52
Fl1ntOo oh? Ok, thought there will be a version that enforce that, nice to know it won't :D10:53
mnasiadkajingvar: I would rather encourage you to find the issue, raise bugs in Neutron and follow up in #openstack-neutron, instead of trying to LB a service that should not be load balanced (think of OVN DBs like RabbitMQ)10:53
mnasiadkaFl1nt: ML2/OVS hasn't been deprecated, there are no plans for it (yet) - but probably those will show up once feature parity is there.10:53
Fl1ntI'll to dig a bit on OVN at some point but we currently have so much things to update that I can't keep it up with all those new things involved.10:54
opendevreviewMark Goddard proposed openstack/kolla-ansible master: libvirt: make it possible to run libvirt on the host  https://review.opendev.org/c/openstack/kolla-ansible/+/82535710:58
opendevreviewMerged openstack/kolla master: rabbitmq: update to 3.9  https://review.opendev.org/c/openstack/kolla/+/82765611:03
jingvarmnasiadka: yep, but we've finished our investiagtions few hours ago11:04
opendevreviewMerged openstack/kayobe stable/xena: Set requirements branch for IPA build  https://review.opendev.org/c/openstack/kayobe/+/82865811:23
opendevreviewMerged openstack/kayobe stable/wallaby: Set requirements branch for IPA build  https://review.opendev.org/c/openstack/kayobe/+/82865911:23
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Refactor fluentd syslog logging  https://review.opendev.org/c/openstack/kolla-ansible/+/74511111:33
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Fix fluentd v1 buffer syntax issue  https://review.opendev.org/c/openstack/kolla-ansible/+/78062911:33
opendevreviewRadosÅ‚aw Piliszek proposed openstack/kolla-ansible master: [DNM] [CI] Run all jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/82886411:52
jingvarmnasiadka: what about RH and OVN behind peacemaker?12:36
mnasiadkajingvar: that is active/passive12:36
jingvaryes, but OVN can be behind a balancer12:37
jingvarbrobaly it is my case https://bugs.launchpad.net/networking-ovn/+bug/195359112:38
SvenKieskenot really kolla-ansible related, but my correctly configured ssh key for gerrit/review.opendev.org stopped working12:48
SvenKieskeany idea who I could contact about that12:49
mnasiadkajingvar: networking-ovn is not maintained anymore, add that bug to neutron as well ;-)12:55
opendevreviewSven Kieske proposed openstack/kolla-ansible master: re-add rabbitmq config for interfaces  https://review.opendev.org/c/openstack/kolla-ansible/+/75857613:04
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Fix Apparmor libvirt profile removal  https://review.opendev.org/c/openstack/kolla-ansible/+/82883013:32
opendevreviewKonstantin Yarovoy proposed openstack/kolla master: Add podman for image building  https://review.opendev.org/c/openstack/kolla/+/82579114:21
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: haproxy: support different internal/external ports  https://review.opendev.org/c/openstack/kolla-ansible/+/82339514:22
opendevreviewMichal Arbet proposed openstack/kolla master: Refactor image module and transition to DockerClient  https://review.opendev.org/c/openstack/kolla/+/82578614:29
opendevreviewMichal Arbet proposed openstack/kolla master: Add podman for image building  https://review.opendev.org/c/openstack/kolla/+/82579114:29
opendevreviewKonstantin Yarovoy proposed openstack/kolla master: Add podman for image building  https://review.opendev.org/c/openstack/kolla/+/82579114:46
opendevreviewKonstantin Yarovoy proposed openstack/kolla master: Add podman for image building  https://review.opendev.org/c/openstack/kolla/+/82579114:48
opendevreviewKonstantin Yarovoy proposed openstack/kolla master: Transition to DockerClient and docker 3.0.0+  https://review.opendev.org/c/openstack/kolla/+/82578615:01
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: haproxy: support different internal/external ports  https://review.opendev.org/c/openstack/kolla-ansible/+/82339515:12
opendevreviewMerged openstack/kolla-ansible stable/xena: Fix log rotation for fluentd created files  https://review.opendev.org/c/openstack/kolla-ansible/+/82840415:13
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Fix log rotation for fluentd created files  https://review.opendev.org/c/openstack/kolla-ansible/+/82840515:13
opendevreviewMerged openstack/kolla-ansible stable/victoria: Fix log rotation for fluentd created files  https://review.opendev.org/c/openstack/kolla-ansible/+/82840615:13
opendevreviewPierre Riteau proposed openstack/kolla-ansible master: Use Nova API 2.87 in openstack-exporter queries  https://review.opendev.org/c/openstack/kolla-ansible/+/82890215:59
opendevreviewMark Goddard proposed openstack/kayobe master: libvirt: deploy libvirt on the host  https://review.opendev.org/c/openstack/kayobe/+/82535916:01
opendevreviewMark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/78227316:09
opendevreviewMark Goddard proposed openstack/kayobe master: DNM: molecule debug  https://review.opendev.org/c/openstack/kayobe/+/82692116:10
opendevreviewMark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/78227316:16
jairHello all I am wondering if anyone has run into this: https://bugs.launchpad.net/kolla-ansible/+bug/196057716:27
jairany suggestion will be appreciated16:28
jairHello all I am wondering if anyone has run into this: https://bugs.launchpad.net/kolla-ansible/+bug/196057716:31
jairI think I sent the message and disappeared ?16:32
opendevreviewSven Kieske proposed openstack/kolla-ansible master: re-add rabbitmq config for interfaces  https://review.opendev.org/c/openstack/kolla-ansible/+/75857617:26
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134017:44
opendevreviewWill Szumski proposed openstack/kolla-ansible master: Adds prometheus_scrape_interval  https://review.opendev.org/c/openstack/kolla-ansible/+/82891618:39
opendevreviewMark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/78227318:48
opendevreviewMark Goddard proposed openstack/kayobe master: libvirt: deploy libvirt on the host  https://review.opendev.org/c/openstack/kayobe/+/82535919:00
opendevreviewMark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/78227320:11
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134020:14
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134022:11
jairhttps://bugs.launchpad.net/kolla-ansible/+bug/1960577 any idea?23:18

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!