Thursday, 2024-02-01

opendevreviewMerged openstack/kolla-ansible master: Disable new defaults and scope for Ironic (RBAC)  https://review.opendev.org/c/openstack/kolla-ansible/+/90685808:05
mnasiadkaSvenKieske: oh boy, can you add that to topics for next weekly meeting?08:09
kevkomnasiadka: do you mean service tokens ? 08:10
mnasiadkayup08:10
SvenKieskealready did :)08:12
SvenKieskethat specific bug at least08:12
SvenKieskebut maybe we should also take a look at other openstack projects and their use of service tokens in k-a?08:12
kevkoit's only nova,cinder if i am correct08:13
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Run SLURP upgrade job  https://review.opendev.org/c/openstack/kolla-ansible/+/90532208:49
atmarkhello, I'm hitting an issue upgrading MariaDB Galera from Wallaby to Xena. ATM, all containers on Xena except MariaDB-*. Whenever I upgrade mariadb, it blows up and I have to do full restore to recover.08:56
atmarkfull db restore*08:57
atmarkI'm using this image https://quay.io/repository/openstack.kolla/ubuntu-source-mariadb:xena08:57
atmarkWallaby is running  mariadb 10.3 while Xena is 10.508:59
atmarkI'm using xena-eol branch09:00
SvenKieskedid you check mariadb logs, what exactly blows up there? I had once an issue with a genuine bug in galera code which triggered on openstack queries. solution was to downgrade mariadb.09:02
atmarkI will provide the logs09:05
atmarkone sec09:05
atmarkSvenKieske: which version did you downgrade to?09:06
atmarkI tried 10.5 and 10.6 but it did not help 09:06
SvenKieskethat was way before xena I believe. I have to check if I still got my subscription to that bugreport. I doubt it though, as this was at a different company.09:07
SvenKieskeatmark: that was the crash, there where actually multiple mariadb versions affected, but you shouldn't run on this old release I guess: https://jira.mariadb.org/browse/MDEV-2497809:09
SvenKieskeif you see clear signs of mariadb crashing I'd encourage you to write a bugreport over at mariadb.org though, they are really responsive and fixing stuff rather fast.09:10
opendevreviewPierre Riteau proposed openstack/kayobe stable/zed: Deprecate devicemapper due to removal in Docker  https://review.opendev.org/c/openstack/kayobe/+/90729309:11
atmarkkolla upgrade task - https://clbin.com/KX5Vz 09:12
*** darmach1 is now known as darmach09:12
atmarkfirst ctrl node -https://clbin.com/5SdMh09:12
atmarksecond ctrl - https://clbin.com/o1hs409:12
atmarkthird ctrl - https://clbin.com/wxgOc09:12
atmark(mariadb)[mysql@c2ostack01c /]$ dpkg -l | egrep 'galera|mariadb-server'09:14
atmarkii  galera-4                       26.4.14-ubu2004                   amd64        Replication framework for transactional applications09:14
atmarkii  mariadb-server                 1:10.5.22+maria~ubu2004           all          MariaDB database server binaries (metapackage depessgalera-4                       26.4.14-ubu2004galera-4                       26.4.14-ubu2004sssusdssssss/ccc09:14
SvenKieske2024-02-01  9:09:04 0 [ERROR] WSREP: /home/buildbot/buildbot/build/gcs/src/gcs_state_msg.cpp:gcs_state_msg_get_quorum():947: Failed to establish quorum. 09:15
SvenKieskeyou seem to have a problem establishing quorum for some reason.09:15
SvenKieskeare the clocks on these servers synchronized? like do you have a robust NTP setup?09:15
opendevreviewPierre Riteau proposed openstack/kayobe stable/yoga: Deprecate devicemapper due to removal in Docker  https://review.opendev.org/c/openstack/kayobe/+/90729409:17
atmarkThey are sycned. I have systemd-timesyncd  to a local NTP server09:17
atmarkran `date` at same time https://paste.openstack.org/show/bp36wTy4R8N89jqLgG4E/09:20
atmarkI googled that WSREP error and I found this bug report which doesn't have a fix https://jira.mariadb.org/browse/MDEV-26316 09:21
SvenKieskethat bugreport has a statement: "This can be closed (deleted). It was caused by our livenessprobe to kick too soon."09:29
opendevreviewMark Goddard proposed openstack/kayobe master: Remove docker devicemapper support  https://review.opendev.org/c/openstack/kayobe/+/90638609:31
opendevreviewMark Goddard proposed openstack/kayobe stable/2023.2: Deprecate devicemapper due to removal in Docker  https://review.opendev.org/c/openstack/kayobe/+/90706409:32
SvenKieskeatmark: the liveness probe for mariadb in k-a is here: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/mariadb/handlers/main.yml#L23 you could try with a different timeout there and see if that helps? you could also check your network though, as you seem to be the only person currently with problems in this area. so I suspect a local issue.09:33
SvenKieskeatmark: regarding your ntp: you are saying your systemd-timesyncd is synced to a single local ntp server? best practice is to actually run 3 or 5. regardless, a single server should at least offer synchronity, until it completely fails.09:34
atmarkI have to two local NTP serers. 09:38
atmarkI doubt it's something to do with networking but I'll double check 09:39
atmarkI don't have any firewall rule that would blocking the replication09:41
SvenKieskeI meant more in the direction of actual errors on interfaces. gbic transceivers die all the time. hardware has faults. it's normal :) always worth to double check. you could also reopen the above bugreport at mariadb if you are certain it's a bug there.09:42
atmarkif I restore the wallaby mariadb image and ran mariadb_recovery, it comes back online 09:42
SvenKieskeso if you have no hardware or setup errors I would really get in contact with the mariadb folks then, I guess, if it was my machine.09:43
SvenKieskeyou could also try different mariadb patch releases, to pinpoint the faulty release.09:44
atmarkI will try your suggestions 09:48
atmarkif it still doesn't work, i'll reach out to #maria in libera 09:49
atmarkthanks 09:49
SvenKieskeatmark: for reference: we are currently running mariadb 10.6.15 and not experiencing these problems. HTH :)09:49
atmarkwhich openstack release?09:50
atmarkI tried this image  quay.io/openstack.kolla/ubuntu-source-mariadb-server:yoga09:51
atmarksame issue09:51
atmarkit has mariadb 10.6.1609:51
SvenKieskewe are on antelope (2023.1)09:53
atmarkWallaby to Xena upgrades mariadb package from 10.3 to 10.5 and galera-3 to galera-4 09:53
*** ravlew is now known as Guest120110:42
*** ravlew1 is now known as ravlew10:43
opendevreviewPierre Riteau proposed openstack/kayobe stable/2023.1: Deprecate devicemapper due to removal in Docker  https://review.opendev.org/c/openstack/kayobe/+/90729212:14
opendevreviewPierre Riteau proposed openstack/kayobe stable/zed: Deprecate devicemapper due to removal in Docker  https://review.opendev.org/c/openstack/kayobe/+/90729312:29
opendevreviewPierre Riteau proposed openstack/kayobe stable/yoga: Deprecate devicemapper due to removal in Docker  https://review.opendev.org/c/openstack/kayobe/+/90729412:31
jovial[m]The kayobe-seed-images-rocky9 job is failing with: invalid tag 'mirror-int.iad.rax.opendev.org:4447/openstack.kolla/base:master-rocky-9'. Didn't we see that elsewhere? What is the fix?12:40
mnasiadkadocker-py version?13:15
opendevreviewJake Hutchinson proposed openstack/kolla-ansible master: Ironic parameter rework and default NTP server  https://review.opendev.org/c/openstack/kolla-ansible/+/89303113:23
opendevreviewJake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration  https://review.opendev.org/c/openstack/kayobe/+/89519913:26
opendevreviewJake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration  https://review.opendev.org/c/openstack/kayobe/+/89519913:31
opendevreviewJake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration  https://review.opendev.org/c/openstack/kayobe/+/89519913:36
opendevreviewJake Hutchinson proposed openstack/kolla-ansible master: Ironic parameter rework and default NTP server  https://review.opendev.org/c/openstack/kolla-ansible/+/89303113:50
opendevreviewMark Goddard proposed openstack/kayobe master: Remove docker devicemapper support  https://review.opendev.org/c/openstack/kayobe/+/90638614:10
opendevreviewMerged openstack/kayobe stable/2023.2: Deprecate devicemapper due to removal in Docker  https://review.opendev.org/c/openstack/kayobe/+/90706414:19
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Refactor of getting list of containers that will also prepare function for migration scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/90583714:20
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Add container engine migration scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/83694114:20
opendevreviewJake Hutchinson proposed openstack/kolla-ansible master: Ironic parameter rework and default NTP server  https://review.opendev.org/c/openstack/kolla-ansible/+/89303116:27
opendevreviewJake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration  https://review.opendev.org/c/openstack/kayobe/+/89519916:31
opendevreviewJake Hutchinson proposed openstack/kayobe master: Add NTP parameter configuration  https://review.opendev.org/c/openstack/kayobe/+/89519916:32
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Run SLURP upgrade job  https://review.opendev.org/c/openstack/kolla-ansible/+/90532216:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Run SLURP upgrade job  https://review.opendev.org/c/openstack/kolla-ansible/+/90532216:39
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Add octavia to openstack core (a.k.a. enable by default)  https://review.opendev.org/c/openstack/kolla-ansible/+/90745616:49
opendevreviewMerged openstack/kayobe stable/2023.1: Deprecate devicemapper due to removal in Docker  https://review.opendev.org/c/openstack/kayobe/+/90729217:30
opendevreviewMerged openstack/kayobe stable/zed: Deprecate devicemapper due to removal in Docker  https://review.opendev.org/c/openstack/kayobe/+/90729317:30
opendevreviewMerged openstack/kayobe stable/yoga: Deprecate devicemapper due to removal in Docker  https://review.opendev.org/c/openstack/kayobe/+/90729417:30
opendevreviewMerged openstack/kayobe master: Remove docker devicemapper support  https://review.opendev.org/c/openstack/kayobe/+/90638619:31
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible master: Fix 'cinder-backup' service when Swift with TLS enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/90749520:27
opendevreviewMerged openstack/kayobe stable/yoga: Switch IPA builds to CentOS Stream 9 for yoga  https://review.opendev.org/c/openstack/kayobe/+/90324223:42

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!