| opendevreview | Merged openstack/kolla-ansible master: Disable new defaults and scope for Ironic (RBAC) https://review.opendev.org/c/openstack/kolla-ansible/+/906858 | 08:05 |
|---|---|---|
| mnasiadka | SvenKieske: oh boy, can you add that to topics for next weekly meeting? | 08:09 |
| kevko | mnasiadka: do you mean service tokens ? | 08:10 |
| mnasiadka | yup | 08:10 |
| SvenKieske | already did :) | 08:12 |
| SvenKieske | that specific bug at least | 08:12 |
| SvenKieske | but maybe we should also take a look at other openstack projects and their use of service tokens in k-a? | 08:12 |
| kevko | it's only nova,cinder if i am correct | 08:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Run SLURP upgrade job https://review.opendev.org/c/openstack/kolla-ansible/+/905322 | 08:49 |
| atmark | hello, I'm hitting an issue upgrading MariaDB Galera from Wallaby to Xena. ATM, all containers on Xena except MariaDB-*. Whenever I upgrade mariadb, it blows up and I have to do full restore to recover. | 08:56 |
| atmark | full db restore* | 08:57 |
| atmark | I'm using this image https://quay.io/repository/openstack.kolla/ubuntu-source-mariadb:xena | 08:57 |
| atmark | Wallaby is running mariadb 10.3 while Xena is 10.5 | 08:59 |
| atmark | I'm using xena-eol branch | 09:00 |
| SvenKieske | did you check mariadb logs, what exactly blows up there? I had once an issue with a genuine bug in galera code which triggered on openstack queries. solution was to downgrade mariadb. | 09:02 |
| atmark | I will provide the logs | 09:05 |
| atmark | one sec | 09:05 |
| atmark | SvenKieske: which version did you downgrade to? | 09:06 |
| atmark | I tried 10.5 and 10.6 but it did not help | 09:06 |
| SvenKieske | that was way before xena I believe. I have to check if I still got my subscription to that bugreport. I doubt it though, as this was at a different company. | 09:07 |
| SvenKieske | atmark: that was the crash, there where actually multiple mariadb versions affected, but you shouldn't run on this old release I guess: https://jira.mariadb.org/browse/MDEV-24978 | 09:09 |
| SvenKieske | if you see clear signs of mariadb crashing I'd encourage you to write a bugreport over at mariadb.org though, they are really responsive and fixing stuff rather fast. | 09:10 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/zed: Deprecate devicemapper due to removal in Docker https://review.opendev.org/c/openstack/kayobe/+/907293 | 09:11 |
| atmark | kolla upgrade task - https://clbin.com/KX5Vz | 09:12 |
| *** darmach1 is now known as darmach | 09:12 | |
| atmark | first ctrl node -https://clbin.com/5SdMh | 09:12 |
| atmark | second ctrl - https://clbin.com/o1hs4 | 09:12 |
| atmark | third ctrl - https://clbin.com/wxgOc | 09:12 |
| atmark | (mariadb)[mysql@c2ostack01c /]$ dpkg -l | egrep 'galera|mariadb-server' | 09:14 |
| atmark | ii galera-4 26.4.14-ubu2004 amd64 Replication framework for transactional applications | 09:14 |
| atmark | ii mariadb-server 1:10.5.22+maria~ubu2004 all MariaDB database server binaries (metapackage depessgalera-4 26.4.14-ubu2004galera-4 26.4.14-ubu2004sssusdssssss/ccc | 09:14 |
| SvenKieske | 2024-02-01 9:09:04 0 [ERROR] WSREP: /home/buildbot/buildbot/build/gcs/src/gcs_state_msg.cpp:gcs_state_msg_get_quorum():947: Failed to establish quorum. | 09:15 |
| SvenKieske | you seem to have a problem establishing quorum for some reason. | 09:15 |
| SvenKieske | are the clocks on these servers synchronized? like do you have a robust NTP setup? | 09:15 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/yoga: Deprecate devicemapper due to removal in Docker https://review.opendev.org/c/openstack/kayobe/+/907294 | 09:17 |
| atmark | They are sycned. I have systemd-timesyncd to a local NTP server | 09:17 |
| atmark | ran `date` at same time https://paste.openstack.org/show/bp36wTy4R8N89jqLgG4E/ | 09:20 |
| atmark | I googled that WSREP error and I found this bug report which doesn't have a fix https://jira.mariadb.org/browse/MDEV-26316 | 09:21 |
| SvenKieske | that bugreport has a statement: "This can be closed (deleted). It was caused by our livenessprobe to kick too soon." | 09:29 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Remove docker devicemapper support https://review.opendev.org/c/openstack/kayobe/+/906386 | 09:31 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/2023.2: Deprecate devicemapper due to removal in Docker https://review.opendev.org/c/openstack/kayobe/+/907064 | 09:32 |
| SvenKieske | atmark: the liveness probe for mariadb in k-a is here: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/mariadb/handlers/main.yml#L23 you could try with a different timeout there and see if that helps? you could also check your network though, as you seem to be the only person currently with problems in this area. so I suspect a local issue. | 09:33 |
| SvenKieske | atmark: regarding your ntp: you are saying your systemd-timesyncd is synced to a single local ntp server? best practice is to actually run 3 or 5. regardless, a single server should at least offer synchronity, until it completely fails. | 09:34 |
| atmark | I have to two local NTP serers. | 09:38 |
| atmark | I doubt it's something to do with networking but I'll double check | 09:39 |
| atmark | I don't have any firewall rule that would blocking the replication | 09:41 |
| SvenKieske | I meant more in the direction of actual errors on interfaces. gbic transceivers die all the time. hardware has faults. it's normal :) always worth to double check. you could also reopen the above bugreport at mariadb if you are certain it's a bug there. | 09:42 |
| atmark | if I restore the wallaby mariadb image and ran mariadb_recovery, it comes back online | 09:42 |
| SvenKieske | so if you have no hardware or setup errors I would really get in contact with the mariadb folks then, I guess, if it was my machine. | 09:43 |
| SvenKieske | you could also try different mariadb patch releases, to pinpoint the faulty release. | 09:44 |
| atmark | I will try your suggestions | 09:48 |
| atmark | if it still doesn't work, i'll reach out to #maria in libera | 09:49 |
| atmark | thanks | 09:49 |
| SvenKieske | atmark: for reference: we are currently running mariadb 10.6.15 and not experiencing these problems. HTH :) | 09:49 |
| atmark | which openstack release? | 09:50 |
| atmark | I tried this image quay.io/openstack.kolla/ubuntu-source-mariadb-server:yoga | 09:51 |
| atmark | same issue | 09:51 |
| atmark | it has mariadb 10.6.16 | 09:51 |
| SvenKieske | we are on antelope (2023.1) | 09:53 |
| atmark | Wallaby to Xena upgrades mariadb package from 10.3 to 10.5 and galera-3 to galera-4 | 09:53 |
| *** ravlew is now known as Guest1201 | 10:42 | |
| *** ravlew1 is now known as ravlew | 10:43 | |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Deprecate devicemapper due to removal in Docker https://review.opendev.org/c/openstack/kayobe/+/907292 | 12:14 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/zed: Deprecate devicemapper due to removal in Docker https://review.opendev.org/c/openstack/kayobe/+/907293 | 12:29 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/yoga: Deprecate devicemapper due to removal in Docker https://review.opendev.org/c/openstack/kayobe/+/907294 | 12:31 |
| jovial[m] | The kayobe-seed-images-rocky9 job is failing with: invalid tag 'mirror-int.iad.rax.opendev.org:4447/openstack.kolla/base:master-rocky-9'. Didn't we see that elsewhere? What is the fix? | 12:40 |
| mnasiadka | docker-py version? | 13:15 |
| opendevreview | Jake Hutchinson proposed openstack/kolla-ansible master: Ironic parameter rework and default NTP server https://review.opendev.org/c/openstack/kolla-ansible/+/893031 | 13:23 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration https://review.opendev.org/c/openstack/kayobe/+/895199 | 13:26 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration https://review.opendev.org/c/openstack/kayobe/+/895199 | 13:31 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration https://review.opendev.org/c/openstack/kayobe/+/895199 | 13:36 |
| opendevreview | Jake Hutchinson proposed openstack/kolla-ansible master: Ironic parameter rework and default NTP server https://review.opendev.org/c/openstack/kolla-ansible/+/893031 | 13:50 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Remove docker devicemapper support https://review.opendev.org/c/openstack/kayobe/+/906386 | 14:10 |
| opendevreview | Merged openstack/kayobe stable/2023.2: Deprecate devicemapper due to removal in Docker https://review.opendev.org/c/openstack/kayobe/+/907064 | 14:19 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Refactor of getting list of containers that will also prepare function for migration scenario https://review.opendev.org/c/openstack/kolla-ansible/+/905837 | 14:20 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add container engine migration scenario https://review.opendev.org/c/openstack/kolla-ansible/+/836941 | 14:20 |
| opendevreview | Jake Hutchinson proposed openstack/kolla-ansible master: Ironic parameter rework and default NTP server https://review.opendev.org/c/openstack/kolla-ansible/+/893031 | 16:27 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration https://review.opendev.org/c/openstack/kayobe/+/895199 | 16:31 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration https://review.opendev.org/c/openstack/kayobe/+/895199 | 16:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Run SLURP upgrade job https://review.opendev.org/c/openstack/kolla-ansible/+/905322 | 16:38 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Run SLURP upgrade job https://review.opendev.org/c/openstack/kolla-ansible/+/905322 | 16:39 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add octavia to openstack core (a.k.a. enable by default) https://review.opendev.org/c/openstack/kolla-ansible/+/907456 | 16:49 |
| opendevreview | Merged openstack/kayobe stable/2023.1: Deprecate devicemapper due to removal in Docker https://review.opendev.org/c/openstack/kayobe/+/907292 | 17:30 |
| opendevreview | Merged openstack/kayobe stable/zed: Deprecate devicemapper due to removal in Docker https://review.opendev.org/c/openstack/kayobe/+/907293 | 17:30 |
| opendevreview | Merged openstack/kayobe stable/yoga: Deprecate devicemapper due to removal in Docker https://review.opendev.org/c/openstack/kayobe/+/907294 | 17:30 |
| opendevreview | Merged openstack/kayobe master: Remove docker devicemapper support https://review.opendev.org/c/openstack/kayobe/+/906386 | 19:31 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Fix 'cinder-backup' service when Swift with TLS enabled https://review.opendev.org/c/openstack/kolla-ansible/+/907495 | 20:27 |
| opendevreview | Merged openstack/kayobe stable/yoga: Switch IPA builds to CentOS Stream 9 for yoga https://review.opendev.org/c/openstack/kayobe/+/903242 | 23:42 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!