| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: placement: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/937448 | 07:40 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: cinder: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/937449 | 07:40 |
| mnasiadka | morning | 07:41 |
| mnasiadka | frickler: https://review.opendev.org/c/openstack/kolla-ansible/+/936425 - addressed your comments, should be good to go | 07:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2024.1: Add support for Ubuntu 24.04 LTS https://review.opendev.org/c/openstack/kolla/+/932386 | 07:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2024.1: Add support for Ubuntu Noble Numbat (24.04 LTS) https://review.opendev.org/c/openstack/kolla-ansible/+/932539 | 07:51 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2024.1: Add support for Ubuntu Noble Numbat (24.04 LTS) https://review.opendev.org/c/openstack/kolla-ansible/+/932539 | 07:53 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2024.1: Add support for Ubuntu Noble Numbat (24.04 LTS) https://review.opendev.org/c/openstack/kolla-ansible/+/932539 | 07:55 |
| frickler | mnasiadka: thx, +2. did you look at the ovn-upgrade failures yet? I think those are unrelated | 08:01 |
| mnasiadka | frickler: looks like some bug in Octavia - https://acbecf2c4485a49c5074-a45826960d45b13f019a0388ffa6602d.ssl.cf1.rackcdn.com/936425/15/check/kolla-ansible-debian-upgrade-ovn/447615d/primary/logs/kolla/octavia/octavia-health-manager.txt | 08:03 |
| mnasiadka | unrelated surely | 08:03 |
| mnasiadka | oh, timed out connection to database | 08:03 |
| mnasiadka | a lot of OFFLINE_HARD statuses in proxysql - surely unrelated | 08:04 |
| mnasiadka | but interested | 08:04 |
| mnasiadka | interesting | 08:04 |
| frickler | hmm, that makes me think of https://review.opendev.org/c/openstack/kolla-ansible/+/937671, but we didn't merge that yet | 08:06 |
| mnasiadka | yes, that's my thought exactly now :) | 08:06 |
| mnasiadka | I'll analyse later on | 08:07 |
| frickler | but also kind of bad error handling in octavia, if a db conn error causes such a traceback | 08:07 |
| frickler | ty | 08:07 |
| mnasiadka | 2024-12-17 21:39:22 MySQL_Monitor.cpp:2267:monitor_galera_thread(): [ERROR] Error on Galera check for 192.0.2.3:3306 after 0ms. Unable to create a connection. If the server is overload, increase mysql-monitor_connect_timeout. Error: timeout or error in creating new connection: Can't connect to server on '192.0.2.3' (115). | 08:08 |
| mnasiadka | it might be we're not setting monitor_connect_timeout | 08:08 |
| mnasiadka | ah, actually I added it in the unmerged change | 08:09 |
| mnasiadka | I'll break it out so we can backport I guess | 08:09 |
| mnasiadka | no, I'm blind | 08:09 |
| mnasiadka | it's there, but not configurable | 08:10 |
| frickler | well in fact the jobs passed earlier for 936425, so might have been just bad luck | 08:11 |
| mnasiadka | I don't like bad luck in multinode jobs :) | 08:16 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: nova: Split out metadata to a separate container https://review.opendev.org/c/openstack/kolla-ansible/+/936425 | 08:55 |
| mnasiadka | frickler: had to do a last minute fix for nova-metadata logging - we eneded up with logs in /var/log/kolla/nova/.log ;-) | 08:56 |
| mnasiadka | but only in uwsgi case, interesting - anyway it's better that it's in the metadata split patch | 09:01 |
| opendevreview | Mickael Razzouk proposed openstack/ansible-collection-kolla master: add docker_apt_key_env variable to docker role https://review.opendev.org/c/openstack/ansible-collection-kolla/+/934126 | 09:05 |
| opendevreview | Mickael Razzouk proposed openstack/ansible-collection-kolla master: add docker_apt_key_env variable to docker role https://review.opendev.org/c/openstack/ansible-collection-kolla/+/934126 | 09:08 |
| kevko | mnasiadka: proxysql offline_hard is normal when backend is down ... in one-node deployment you can't do anything .as you actually lost only one backend ... | 09:10 |
| kevko | mnasiadka: this looks like some bug in octavia >> Connection to database failed. Retrying in 10 seconds.: AttributeError: 'Engine' object has no attribute 'scalar' << nothing with proxysql ... even if prxysql has offline_hard ..mysql client getting error "Proxysql can't reach hostgroup 0 in 10000 ms " or similar ... mysql client - oslo.db will | 09:12 |
| kevko | recover connection after some time ... ..this is something different | 09:12 |
| frickler | mnasiadka: ah, good catch | 09:12 |
| kevko | or it looks like | 09:14 |
| opendevreview | Merged openstack/kolla-ansible master: Revert "CI: Disable SLURP jobs for D cycle" https://review.opendev.org/c/openstack/kolla-ansible/+/937864 | 09:17 |
| mnasiadka | kevko: using the opportunity you're online - would be nice to get some reviews on https://review.opendev.org/c/openstack/kolla-ansible/+/937671 - I'd like to stop owning a lot of defaults ;-) | 09:33 |
| kevko | mnasiadka: will check | 09:56 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: ironic: Make ironic-inspector optional https://review.opendev.org/c/openstack/kolla-ansible/+/923948 | 10:04 |
| kevko | mnasiadka: some comments added ... | 10:17 |
| mnasiadka | kevko: will have a look later, thanks - I see basically mariadb_monitor_connect_interval is problematic, we could own that default I guess - I'll rework | 10:22 |
| kevko | mnasiadka: well, proxysql has much more mechanisms to check if NODE is OK | 10:26 |
| kevko | mnasiadka: mysql ping, mysql monitor, galera monitor, and the most important is max_failed queries .... | 10:27 |
| kevko | mnasiadka: biggest problem is the last one i think | 10:27 |
| kevko | mnasiadka: because it's correctly shunning the node if not available ...and send error to clients ...which is actually correct ..every service is able to recover i think from this state ... | 10:28 |
| kevko | mnasiadka: but in overall it's correct ..because if there is an issue with backend ...proxysql will kill the connections ...and create new | 10:29 |
| kevko | mnasiadka: maybe you can check relation chain https://review.opendev.org/c/openstack/kolla-ansible/+/936922/16 << it's almost done ...just some cosmetic stuff as renos and friends | 10:29 |
| mnasiadka | EAB is on the bottom, I'd like to get that templated out script to be multiline if possible - but I'm fine if that's a followup | 10:36 |
| mnasiadka | kevko: can you +2 the EAB patch if you're fine with it as it is now? | 10:37 |
| kevko | mnasiadka: well, i fixed EAB because it was not working on my tests :D ... i can do a followup to make it multiline | 10:38 |
| kevko | mnasiadka: i wanted to mainly rework TLS | 10:38 |
| mnasiadka | ok then, merging | 10:54 |
| kevko | mnasiadka: thx | 10:55 |
| mnasiadka | frickler: https://zuul.opendev.org/t/openstack/build/7179eed086784244aad23ed1b5fdd70a/log/primary/logs/kolla/nova/nova-compute.txt#10106 - this started to show up recently in our Noble jobs, Nova has a broken calculator? ;-) | 11:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Make ironic-inspector optional https://review.opendev.org/c/openstack/kolla-ansible/+/923948 | 11:22 |
| opendevreview | Jakub Darmach proposed openstack/kolla-ansible master: Exclude NFS backend mount from ownership change https://review.opendev.org/c/openstack/kolla-ansible/+/923945 | 11:26 |
| opendevreview | Merged openstack/kolla-ansible stable/2024.1: Fix nova-libvirt's secrets - device or resource is busy https://review.opendev.org/c/openstack/kolla-ansible/+/936176 | 11:41 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: Fix nova-libvirt's secrets - device or resource is busy https://review.opendev.org/c/openstack/kolla-ansible/+/936175 | 11:48 |
| opendevreview | Merged openstack/kolla-ansible master: Add Let's Encrypt EAB support https://review.opendev.org/c/openstack/kolla-ansible/+/935656 | 12:20 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: proxysql: Revert to upstream defaults for most settings https://review.opendev.org/c/openstack/kolla-ansible/+/937671 | 12:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add service-uwsgi-config role https://review.opendev.org/c/openstack/kolla-ansible/+/936646 | 13:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: nova: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/935975 | 13:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: placement: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/937448 | 13:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: cinder: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/937449 | 13:11 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [letsencrypt] Rename script for updating certificates https://review.opendev.org/c/openstack/kolla/+/936919 | 13:13 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [letsencrypt] Add ProxySQL support for certificate renewal https://review.opendev.org/c/openstack/kolla/+/936920 | 13:14 |
| opendevreview | Jakub Darmach proposed openstack/kolla-ansible master: Docs update - kolla-ansible cli parameters https://review.opendev.org/c/openstack/kolla-ansible/+/937985 | 13:17 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: cinder: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/937449 | 13:17 |
| opendevreview | Jakub Darmach proposed openstack/kolla-ansible master: Docs update - kolla-ansible cli parameters https://review.opendev.org/c/openstack/kolla-ansible/+/937985 | 13:20 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: WIP: we want to track github releases easier https://review.opendev.org/c/openstack/kolla/+/880154 | 13:28 |
| opendevreview | Jakub Darmach proposed openstack/kolla-ansible master: Docs update - kolla-ansible cli parameters https://review.opendev.org/c/openstack/kolla-ansible/+/937985 | 13:31 |
| mnasiadka | 10 minutes meeting reminder - mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees dougszu darmach | 13:51 |
| opendevreview | Alex Welsh proposed openstack/kolla-ansible master: Use file service discovery in Prometheus https://review.opendev.org/c/openstack/kolla-ansible/+/937894 | 13:51 |
| mnasiadka | #startmeeting kolla | 14:03 |
| opendevmeet | Meeting started Wed Dec 18 14:03:08 2024 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:03 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:03 |
| opendevmeet | The meeting name has been set to 'kolla' | 14:03 |
| mnasiadka | #topic rollcall | 14:03 |
| mnasiadka | o/ | 14:03 |
| frickler | \o | 14:03 |
| frickler | I was just about to wake you up ;) | 14:03 |
| frickler | hmm, looks like small audience anyway | 14:05 |
| mmalchuk | o/ | 14:05 |
| mnasiadka | yup | 14:05 |
| mnasiadka | #topic agenda | 14:05 |
| mnasiadka | * CI status | 14:05 |
| mnasiadka | * Release tasks | 14:05 |
| mnasiadka | * Current cycle planning | 14:06 |
| mnasiadka | * Additional agenda (from whiteboard) | 14:06 |
| mnasiadka | * Open discussion | 14:06 |
| mnasiadka | #topic CI status | 14:06 |
| mnasiadka | well, new Jammy kernel broke Neutron, but thanks to frickler it looks good now | 14:06 |
| mnasiadka | After some work with proxysql settings I think we're getting multinode testing stability back | 14:06 |
| mnasiadka | #topic Release tasks | 14:07 |
| mnasiadka | I don't think we have any | 14:07 |
| mnasiadka | #topic Current cycle planning | 14:07 |
| mnasiadka | I've been progressing with uWSGI stuff, thanks to frickler and bbezak for reviews, will continue with the rest of services | 14:08 |
| mnasiadka | I've seen kevko is working on finalising the database internal TLS, so we might finally get better in terms of backend/internal TLS | 14:08 |
| frickler | yeah, thanks for that, keep pinging me if I don't respond fast enough | 14:08 |
| mnasiadka | I think for keystone and horizon - I'll just switch from apache+mod_wsgi to apache+mod_uwsgi - we have users that use mod_auth_mellon and existing stack for federation, so I doubt we'll ever want to get rid of apache | 14:10 |
| mnasiadka | #topic Additional agenda | 14:11 |
| frickler | apache+mod_proxy + external uwsgi, like devstack does, might be an alternative? not sure if it is better, though | 14:11 |
| mnasiadka | yeah, right - I think we want to do what devstack does | 14:11 |
| mnasiadka | so we don't find out weird issues by ourselves | 14:11 |
| mnasiadka | (mattcrees): What do people think on backporting this patch? https://review.opendev.org/c/openstack/kolla-ansible/+/937670 We may not want to, as people who only ever run from one location will get an unnecessary config change. We should find out if anyone will be hit by that first. | 14:11 |
| mnasiadka | Matt C is not here, but he wanted us to discuss that | 14:12 |
| mnasiadka | frickler: I'm probably fine with backporting this, but need a second core to approve :) | 14:12 |
| frickler | hmm, the backport would also cause a restart in itself, right? so spontaneously I'd rather not do this | 14:13 |
| mnasiadka | Ok then, we'll backport that downstream | 14:16 |
| mnasiadka | #topic Open discussion | 14:17 |
| mnasiadka | Anybody anything? | 14:17 |
| frickler | not from me | 14:19 |
| mnasiadka | then I guess we're done :) | 14:21 |
| mnasiadka | Thanks for coming frickler and mmalchuk | 14:21 |
| mnasiadka | #endmeeting | 14:21 |
| opendevmeet | Meeting ended Wed Dec 18 14:21:12 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:21 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-12-18-14.03.html | 14:21 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-12-18-14.03.txt | 14:21 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-12-18-14.03.log.html | 14:21 |
| mmalchuk | thanks mnasiadka | 14:21 |
| frickler | o/ | 14:21 |
| *** darmach5 is now known as darmach | 14:24 | |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: nova: Split out metadata to a separate container https://review.opendev.org/c/openstack/kolla-ansible/+/936425 | 14:48 |
| frickler | mnasiadka: do you know if matt is away for the holidays already? if yes, I'd likely propose an update to https://review.opendev.org/c/openstack/kolla-ansible/+/937038 myself | 15:00 |
| mnasiadka | frickler: yes he is, feel free to update :) | 15:00 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: ironic: Add kolla_ironic_inspection_network https://review.opendev.org/c/openstack/kayobe/+/937996 | 15:38 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible master: Cinder: move to block-storage in service catalog https://review.opendev.org/c/openstack/kolla-ansible/+/937038 | 15:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Retry Ansible Galaxy calls https://review.opendev.org/c/openstack/kolla-ansible/+/936757 | 15:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Mark slurp jobs as non voting for now https://review.opendev.org/c/openstack/kolla-ansible/+/938003 | 16:33 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Mark slurp jobs as non voting for now https://review.opendev.org/c/openstack/kolla-ansible/+/938003 | 16:40 |
| opendevreview | Merged openstack/kolla-ansible master: nova: Split out metadata to a separate container https://review.opendev.org/c/openstack/kolla-ansible/+/936425 | 18:56 |
| opendevreview | Merged openstack/kolla-ansible master: Add service-uwsgi-config role https://review.opendev.org/c/openstack/kolla-ansible/+/936646 | 20:04 |
| opendevreview | Merged openstack/kolla-ansible master: CI: Mark slurp jobs as non voting for now https://review.opendev.org/c/openstack/kolla-ansible/+/938003 | 20:23 |
| opendevreview | Merged openstack/kolla-ansible master: Generate system scoped public-openrc https://review.opendev.org/c/openstack/kolla-ansible/+/937636 | 20:32 |
| mnasiadka | frickler: https://review.opendev.org/c/openstack/kolla/+/935974 - that one would be good to merge regarding uwsgi - it’s rather trivial | 20:39 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/2024.2: Generate system scoped public-openrc https://review.opendev.org/c/openstack/kolla-ansible/+/938020 | 22:42 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/2024.1: Generate system scoped public-openrc https://review.opendev.org/c/openstack/kolla-ansible/+/938021 | 22:42 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!