| *** f0o_ is now known as f0o | 06:21 | |
| gokhan_ | Good morning folks, first member of the rabbitmq hosts is failed and can not be added to cluster again. I am trying to solve this with rabbitmq-upgrade=true. is it possible to change bootstrap host from member 1 to member 2 ? | 07:07 |
|---|---|---|
| noonedeadpunk | gokhan_: yes, sure that's possible | 08:48 |
| noonedeadpunk | there's a variable rabbitmq_primary_cluster_node https://opendev.org/openstack/openstack-ansible-rabbitmq_server/src/branch/master/defaults/main.yml#L43 | 08:49 |
| noonedeadpunk | but I guess I'd try to recover manually there, by running `rabbitmqctl forget_cluster_node` | 08:51 |
| noonedeadpunk | and then on failed node `rabbitmqctl join_cluster` | 08:51 |
| noonedeadpunk | potentially, you'd need to drop mnesia db on failed node before joining the cluster | 08:51 |
| gokhan_ | thanks noonedeadpunk I tried manually to solve issues but it seems all nodes are crashed. I rebuild rabbitmq containers again and solved issues. | 10:55 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Nov 12 15:00:38 2024 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| mgariepy | hey | 15:01 |
| mgariepy | been a while | 15:01 |
| noonedeadpunk | indeed! hope all is well | 15:01 |
| mgariepy | yep everything is going well :) | 15:02 |
| mgariepy | how about you? | 15:02 |
| noonedeadpunk | a bit /o\ with everything going on | 15:03 |
| noonedeadpunk | but overall things are moving at least | 15:03 |
| jrosser | o/ hello | 15:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Move healthcheck playbooks to collection https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/933610 | 15:04 |
| noonedeadpunk | #topic office hours | 15:04 |
| noonedeadpunk | So, mariadb 11.4 is pretty much ready | 15:05 |
| noonedeadpunk | I still had to disable TLS verification, but it affects only mariabackup | 15:05 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/922377/12/templates/cluster.cnf.j2 | 15:05 |
| noonedeadpunk | regarding rabbitmq 4.0 - I somehow thought that we're using 3.13 for 2024.1 :( | 15:06 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934060 | 15:07 |
| noonedeadpunk | but it's indeed 3.12 :( | 15:07 |
| noonedeadpunk | I was thinking it was part of 2024.1 but it's not https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/922378 | 15:08 |
| noonedeadpunk | so I wonder what are we gonna do with that... | 15:08 |
| noonedeadpunk | as this means - we can't have 4.0 for 2025.1 then as well | 15:08 |
| andrewbonney | I hadn't thought about SLURP impact on RMQ version jumps. That's not ideal | 15:11 |
| mgariepy | 3.13 is EOL as of now for community release | 15:12 |
| noonedeadpunk | well, community is supporting always only current release nowdays | 15:12 |
| noonedeadpunk | so we can't always be on latest for stable branches anyway | 15:13 |
| noonedeadpunk | So eventually we either need to wait for 2025.2 for 4.0, or backport 3.13 to 2024.1 and write a release note with requirement of minor upgrade before proceeding | 15:13 |
| noonedeadpunk | neither are ideal to say the least | 15:13 |
| andrewbonney | If we wait for 4.0 it feels like there's a risk of getting way behind, subject to RMQ's release cadence | 15:14 |
| mgariepy | A pre-step to upgrade rabbit on 2024.1 for SLURP to 2025.1 isn't too bad IMO. | 15:18 |
| noonedeadpunk | probably it's a lower risk indeed... | 15:18 |
| noonedeadpunk | bad we realized this after getting 29.1.0 out though | 15:19 |
| noonedeadpunk | but we pushed it out not too far ago | 15:19 |
| andrewbonney | Could we add any automation to step through major versions from current to target, or does that feel awkward? Feels like this may come up again in future | 15:19 |
| noonedeadpunk | I'd say it's indeed quite awkward... While it could be possible, I'm not sure we want to have such complexity in code | 15:20 |
| andrewbonney | Fair enough | 15:21 |
| noonedeadpunk | as we have a check for current rabbitmq version. so we can add another upgrade loop somehow... somewhere... | 15:21 |
| noonedeadpunk | but it should be on playbook level anyway I assume | 15:21 |
| noonedeadpunk | not on a role level | 15:22 |
| noonedeadpunk | so it will be not neat at all | 15:22 |
| noonedeadpunk | but I think indeed that bumping rabbitmq version for 2024.1 might be best option now. | 15:26 |
| noonedeadpunk | but then there will be 2 concenring things with 4.0 | 15:27 |
| noonedeadpunk | first - quorum queues requirement, and second 3.13 requirement. | 15:27 |
| noonedeadpunk | none of them are critical to have that said | 15:27 |
| noonedeadpunk | I will push patch to 2024.1 I guess and some release notes supporting this requirement | 15:28 |
| noonedeadpunk | I also wrote some doc around how I was doing pretty endpoint names | 15:28 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible/+/934536 | 15:28 |
| andrewbonney | noonedeadpunk: I think if backporting 3.13 we'll also need Idb5c02109458771853e0fdbc7f6bb27beaa731b4, otherwise broken experimental feature flags will get enabled | 15:29 |
| andrewbonney | See 'khepri_db' in https://www.rabbitmq.com/docs/upgrade | 15:30 |
| NeilHanlon | o/ way late, sry. triple double booked today because of timezone fun 🙃 | 15:32 |
| noonedeadpunk | andrewbonney: I thought you wanted to link another patch, but I see what you mean | 15:33 |
| noonedeadpunk | I thought that's also in 2024.1 though :D | 15:33 |
| mgariepy | we also need to make sure that the rabbitmq validate the version before upgrade in case older branch of 2024.1 was deployed and not upgraded before the SLURP to 2025.1 | 15:36 |
| noonedeadpunk | that is a very good point | 15:37 |
| noonedeadpunk | and we can fail with meaningfull error if we can't | 15:44 |
| noonedeadpunk | upgrade | 15:44 |
| mgariepy | before the cluster is screwed :D hahha | 15:48 |
| noonedeadpunk | preferably "D | 15:49 |
| noonedeadpunk | and we're less then 1 month until final release | 15:49 |
| noonedeadpunk | we also have quite some outstanding reviews | 15:50 |
| noonedeadpunk | #link http://bit.ly/osa-review-board-v5 | 16:00 |
| noonedeadpunk | #endmeeting | 16:00 |
| opendevmeet | Meeting ended Tue Nov 12 16:00:22 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-11-12-15.00.html | 16:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-11-12-15.00.txt | 16:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-11-12-15.00.log.html | 16:00 |
| opendevreview | Merged openstack/openstack-ansible-os_rally master: Add retries for UC fetching over HTTP https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/934144 | 17:20 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Do not add vip['address'] so SAN if haproxy is binded to interface https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/934549 | 17:37 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally stable/2024.1: Add retries for UC fetching over HTTP https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/934815 | 17:44 |
| mgariepy | why so much post_failure ? | 17:54 |
| frickler | infra issues | 17:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Get exact version of installed rabbitmq https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934822 | 18:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Bump RabbitMQ version to 4.0 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934060 | 18:40 |
| noonedeadpunk | andrewbonney: I'm actually a bit surprised that feater flags are not enabled on post-upgrade. As I would expect they should be according to my understanding | 18:41 |
| noonedeadpunk | as in case of cluster is healthy - this should execute anyway? https://opendev.org/openstack/openstack-ansible-rabbitmq_server/src/branch/master/tasks/main.yml#L85-L111 | 18:41 |
| noonedeadpunk | maybe some kind of race condition.... | 18:42 |
| andrewbonney | I'll take another look tomorrow. After a run in an AIO that was my finding, but I did have to back out of the 4.0 upgrade, so possible something was in an unusual state | 18:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server stable/2024.1: Upgrade RabbitMQ to 3.13 for 2024.1 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934826 | 19:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Document requirement for RabbitMQ upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/934828 | 19:28 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Generate PKI SAN as a list https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/934550 | 20:00 |
| opendevreview | Merged openstack/openstack-ansible-ceph_client stable/2024.1: Fix a list of packages allowed from epel https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/934532 | 22:38 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!