| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 00:25 |
|---|---|---|
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 00:46 |
| opendevreview | conna proposed openstack/kolla-ansible master: Skyline: Add skyline role https://review.opendev.org/c/openstack/kolla-ansible/+/828464 | 01:36 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/wallaby: base: Drop usage of Ceph Nautilus from RDO https://review.opendev.org/c/openstack/kolla/+/828228 | 06:04 |
| Fl1nt | Hi everyone! | 08:43 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/wallaby: Fix Apparmor libvirt profile removal https://review.opendev.org/c/openstack/kolla-ansible/+/828830 | 09:19 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/victoria: Fix Apparmor libvirt profile removal https://review.opendev.org/c/openstack/kolla-ansible/+/828831 | 09:19 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: test fact caching https://review.opendev.org/c/openstack/kayobe/+/808218 | 09:28 |
| Fl1nt | People, is anyone experimenting the infamous RabbitMQ Connections timeout issue where RMQ Close connection to OS Services because they missed Heartbeat ? I'm having this issue on one of our relatively small platform and it drive me crazy | 09:29 |
| Fl1nt | The network is OK, no latency, no congestion, no dropped packets. | 09:30 |
| Fl1nt | This platform doesn't get any VMs running yet | 09:31 |
| Fl1nt | but yet Horizon can't reach Services APIs from time to time (especially nova-api-wsgi) because HAProxy see its backend server as DOWN for no reason, then 3/4h later everything went back to normal | 09:32 |
| Fl1nt | and then this issue raise again from time to time | 09:32 |
| Fl1nt | I've of course check oslo.messaging rmq timeout, raise them, check for thread=1 on WSGI congfiguration all is OK | 09:33 |
| Fl1nt | but the issue still arise | 09:33 |
| Fl1nt | This platform is based on CentOS 8.5 and Kolla/Kolla-ansible 11.3.0 | 09:34 |
| Fl1nt | our monitoring shows that the servers literally are idle, services do nothing, no errors on either RMQ or MariaDB, only HAProxy saw a bunch of services DOWN and that went away as soon as we restarted the docker container. | 09:36 |
| Fl1nt | but we still can catch RMQ/Services message about either the service missing Heartbeat and/or the RMQ Server being unreachable. | 09:36 |
| Fl1nt | which isn't on par with what the services monitoring probes themselves or manual inspection report. | 09:37 |
| Fl1nt | We currently have a cluster of 3 RMQ Servers with each 1TB of RAM and 96 CPUs. | 09:38 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Refactor fluentd syslog logging https://review.opendev.org/c/openstack/kolla-ansible/+/745111 | 09:56 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Fix fluentd v1 buffer syntax issue https://review.opendev.org/c/openstack/kolla-ansible/+/780629 | 09:56 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: docs: add information about development libvirt TLS certs https://review.opendev.org/c/openstack/kolla-ansible/+/828772 | 09:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: haproxy: support different internal/external ports https://review.opendev.org/c/openstack/kolla-ansible/+/823395 | 09:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: haproxy: support fqdn based single frontend https://review.opendev.org/c/openstack/kolla-ansible/+/824312 | 09:59 |
| opendevreview | Merged openstack/kolla-ansible master: [CI] Filter fluentd errors more https://review.opendev.org/c/openstack/kolla-ansible/+/828594 | 10:02 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: neutron: fix placement endpoint type configuration https://review.opendev.org/c/openstack/kolla-ansible/+/828753 | 10:10 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: neutron: fix placement endpoint type configuration https://review.opendev.org/c/openstack/kolla-ansible/+/828754 | 10:10 |
| opendevreview | Merged openstack/kolla-ansible stable/victoria: neutron: fix placement endpoint type configuration https://review.opendev.org/c/openstack/kolla-ansible/+/828755 | 10:10 |
| opendevreview | Merged openstack/kolla-ansible stable/ussuri: neutron: fix placement endpoint type configuration https://review.opendev.org/c/openstack/kolla-ansible/+/828756 | 10:10 |
| jingvar | Fl1nt: Haproxy timeouts for backends? | 10:14 |
| jingvar | timeout client 3m | 10:15 |
| jingvar | queue, server the same | 10:16 |
| Fl1nt | jingvar, ? | 10:26 |
| Fl1nt | jingvar, Yes it seems there was few backends that get there check failed | 10:28 |
| jingvar | Fl1nt: /etc/kolla/haproxy/* find and increase timeout values | 10:34 |
| Fl1nt | jingvar, No I mean, I know that ^^ | 10:34 |
| jingvar | ok | 10:34 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: Fix Apparmor libvirt profile removal https://review.opendev.org/c/openstack/kolla-ansible/+/828692 | 10:35 |
| Fl1nt | I wanted to know if anyone was facing such issue with an empty/almost empty platform and if so which values did you put and why. | 10:35 |
| jingvar | we plays with OVN, it fails on 3 controlers , but it works on 1 controller deployment (tempest 60 threads). I'll try to place OVNs DBs behind haproxy and VIP | 10:39 |
| mnasiadka | jingvar: why? | 10:40 |
| jingvar | metadata agent walks through avalible dbs endpoints for leader | 10:43 |
| jingvar | lost connection, reconnet, wait, reconect etc | 10:43 |
| mnasiadka | yes, that's how it works | 10:43 |
| mnasiadka | how do you plan to check which OVN DB instance is master? | 10:44 |
| jingvar | it is pain | 10:44 |
| mnasiadka | pain in what? | 10:44 |
| Fl1nt | in the ass ? :D | 10:44 |
| mnasiadka | OVN DBs are doing periodic snapshots, they will change leader every 24h | 10:44 |
| mnasiadka | and all services using it, need to be connected to the leader | 10:45 |
| jingvar | I can check who is leader only from ovns container, not from haproxy | 10:45 |
| mnasiadka | and it's a TCP service, and all code was written with direct connection in mind, not through a load balancer | 10:46 |
| jingvar | dances around dbs happens until tempest run (30min) | 10:46 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu https://review.opendev.org/c/openstack/kayobe/+/782273 | 10:46 |
| mnasiadka | jingvar: until? or during? | 10:47 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu https://review.opendev.org/c/openstack/kayobe/+/782273 | 10:47 |
| jingvar | sorry, durring | 10:48 |
| mnasiadka | jingvar: make sure you have this in your Neutron code: https://review.opendev.org/c/openstack/neutron/+/822328 | 10:48 |
| jingvar | thank will check, I used wallaby source images 1 week age as I remember | 10:51 |
| Fl1nt | mnasiadka, which version of OS enforce OVN usage? | 10:52 |
| mnasiadka | Fl1nt: none | 10:52 |
| Fl1nt | Oo oh? Ok, thought there will be a version that enforce that, nice to know it won't :D | 10:53 |
| mnasiadka | jingvar: I would rather encourage you to find the issue, raise bugs in Neutron and follow up in #openstack-neutron, instead of trying to LB a service that should not be load balanced (think of OVN DBs like RabbitMQ) | 10:53 |
| mnasiadka | Fl1nt: ML2/OVS hasn't been deprecated, there are no plans for it (yet) - but probably those will show up once feature parity is there. | 10:53 |
| Fl1nt | I'll to dig a bit on OVN at some point but we currently have so much things to update that I can't keep it up with all those new things involved. | 10:54 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: libvirt: make it possible to run libvirt on the host https://review.opendev.org/c/openstack/kolla-ansible/+/825357 | 10:58 |
| opendevreview | Merged openstack/kolla master: rabbitmq: update to 3.9 https://review.opendev.org/c/openstack/kolla/+/827656 | 11:03 |
| jingvar | mnasiadka: yep, but we've finished our investiagtions few hours ago | 11:04 |
| opendevreview | Merged openstack/kayobe stable/xena: Set requirements branch for IPA build https://review.opendev.org/c/openstack/kayobe/+/828658 | 11:23 |
| opendevreview | Merged openstack/kayobe stable/wallaby: Set requirements branch for IPA build https://review.opendev.org/c/openstack/kayobe/+/828659 | 11:23 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Refactor fluentd syslog logging https://review.opendev.org/c/openstack/kolla-ansible/+/745111 | 11:33 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Fix fluentd v1 buffer syntax issue https://review.opendev.org/c/openstack/kolla-ansible/+/780629 | 11:33 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] [CI] Run all jobs https://review.opendev.org/c/openstack/kolla-ansible/+/828864 | 11:52 |
| jingvar | mnasiadka: what about RH and OVN behind peacemaker? | 12:36 |
| mnasiadka | jingvar: that is active/passive | 12:36 |
| jingvar | yes, but OVN can be behind a balancer | 12:37 |
| jingvar | brobaly it is my case https://bugs.launchpad.net/networking-ovn/+bug/1953591 | 12:38 |
| SvenKieske | not really kolla-ansible related, but my correctly configured ssh key for gerrit/review.opendev.org stopped working | 12:48 |
| SvenKieske | any idea who I could contact about that | 12:49 |
| mnasiadka | jingvar: networking-ovn is not maintained anymore, add that bug to neutron as well ;-) | 12:55 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible master: re-add rabbitmq config for interfaces https://review.opendev.org/c/openstack/kolla-ansible/+/758576 | 13:04 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Fix Apparmor libvirt profile removal https://review.opendev.org/c/openstack/kolla-ansible/+/828830 | 13:32 |
| opendevreview | Konstantin Yarovoy proposed openstack/kolla master: Add podman for image building https://review.opendev.org/c/openstack/kolla/+/825791 | 14:21 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: haproxy: support different internal/external ports https://review.opendev.org/c/openstack/kolla-ansible/+/823395 | 14:22 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Refactor image module and transition to DockerClient https://review.opendev.org/c/openstack/kolla/+/825786 | 14:29 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Add podman for image building https://review.opendev.org/c/openstack/kolla/+/825791 | 14:29 |
| opendevreview | Konstantin Yarovoy proposed openstack/kolla master: Add podman for image building https://review.opendev.org/c/openstack/kolla/+/825791 | 14:46 |
| opendevreview | Konstantin Yarovoy proposed openstack/kolla master: Add podman for image building https://review.opendev.org/c/openstack/kolla/+/825791 | 14:48 |
| opendevreview | Konstantin Yarovoy proposed openstack/kolla master: Transition to DockerClient and docker 3.0.0+ https://review.opendev.org/c/openstack/kolla/+/825786 | 15:01 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: haproxy: support different internal/external ports https://review.opendev.org/c/openstack/kolla-ansible/+/823395 | 15:12 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: Fix log rotation for fluentd created files https://review.opendev.org/c/openstack/kolla-ansible/+/828404 | 15:13 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Fix log rotation for fluentd created files https://review.opendev.org/c/openstack/kolla-ansible/+/828405 | 15:13 |
| opendevreview | Merged openstack/kolla-ansible stable/victoria: Fix log rotation for fluentd created files https://review.opendev.org/c/openstack/kolla-ansible/+/828406 | 15:13 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: Use Nova API 2.87 in openstack-exporter queries https://review.opendev.org/c/openstack/kolla-ansible/+/828902 | 15:59 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: libvirt: deploy libvirt on the host https://review.opendev.org/c/openstack/kayobe/+/825359 | 16:01 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu https://review.opendev.org/c/openstack/kayobe/+/782273 | 16:09 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: DNM: molecule debug https://review.opendev.org/c/openstack/kayobe/+/826921 | 16:10 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu https://review.opendev.org/c/openstack/kayobe/+/782273 | 16:16 |
| jair | Hello all I am wondering if anyone has run into this: https://bugs.launchpad.net/kolla-ansible/+bug/1960577 | 16:27 |
| jair | any suggestion will be appreciated | 16:28 |
| jair | Hello all I am wondering if anyone has run into this: https://bugs.launchpad.net/kolla-ansible/+bug/1960577 | 16:31 |
| jair | I think I sent the message and disappeared ? | 16:32 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible master: re-add rabbitmq config for interfaces https://review.opendev.org/c/openstack/kolla-ansible/+/758576 | 17:26 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 17:44 |
| opendevreview | Will Szumski proposed openstack/kolla-ansible master: Adds prometheus_scrape_interval https://review.opendev.org/c/openstack/kolla-ansible/+/828916 | 18:39 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu https://review.opendev.org/c/openstack/kayobe/+/782273 | 18:48 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: libvirt: deploy libvirt on the host https://review.opendev.org/c/openstack/kayobe/+/825359 | 19:00 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu https://review.opendev.org/c/openstack/kayobe/+/782273 | 20:11 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 20:14 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 22:11 |
| jair | https://bugs.launchpad.net/kolla-ansible/+bug/1960577 any idea? | 23:18 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!