| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Set default for octavia_barbican_enabled https://review.opendev.org/c/openstack/openstack-ansible/+/769179 | 08:15 |
|---|---|---|
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: kata: disable installation on debian bullseye https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/817714 | 08:22 |
| noonedeadpunk | mgariepy: https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 looks veeery good, wdyt? | 08:25 |
| noonedeadpunk | Look, we're waiting for 6 seconds here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_566/817385/1/check/openstack-ansible-deploy-aio_metal-ubuntu-focal/566c36a/logs/ara-report/results/561.html | 08:25 |
| noonedeadpunk | which is exactly the thing we were missing | 08:25 |
| noonedeadpunk | I think we need to split this patch into 2 to actually backport fix of mysql_upgrade | 08:26 |
| noonedeadpunk | because that things do work is more like a lucky co-incidence | 08:26 |
| noonedeadpunk | yes, this still does not solve server being disabled during upgrade... | 08:27 |
| noonedeadpunk | but I'd say that for this let's wait for descision on your bug report | 08:28 |
| noonedeadpunk | will run rechecks and try to catch a timeout but I bet we won't now. | 08:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Fix mysql_upgrade possible race condition https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/819079 | 09:10 |
| masterpe[m] | Hi, I'm troubleshooting why our rabbitmq cluster is using 500% usage and only sending 300 messages per second. I try to reproduce it by using perf-test. But I'm only able to reproduce it by using fan-out and 500 producers. And limit message rate on 0,125 | 10:57 |
| masterpe[m] | The cluster is installed with OSA. | 10:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 11:24 |
| mgariepy | noonedeadpunk, yep i think we should at least wait for the upgrade to be finished. | 13:06 |
| mgariepy | and it's reasonable to wait to see what is the next step from mariadb on the bug resolution. | 13:08 |
| noonedeadpunk | but I'm not sure if it's blocker now for 10.6? As the rest potential issues will be the same for 10.5 and 10.6 | 13:09 |
| mgariepy | did we saw any other issues on 10.6. when waiting for mariadb_upgrade ? | 13:12 |
| mgariepy | the hang we had on debian last week we have not idea what did happen.. and i haven't seen any other timeout | 13:12 |
| mgariepy | as for the db lock how many time did we had the issue but didn't realize because it wasn't happening too often ? | 13:15 |
| mgariepy | So we do go with 10.6.5 after all ? | 13:31 |
| *** odyssey4me is now known as Guest6697 | 13:33 | |
| noonedeadpunk | I saw the only issue as for now and it was only during user creation or when we launched another upgrade | 13:50 |
| noonedeadpunk | We were having same hang for 20.5 btw, but we supposed it was because of root user usage, so we moved to admin | 13:51 |
| noonedeadpunk | but we hang at exact same place | 13:51 |
| noonedeadpunk | *10.5 | 13:51 |
| mgariepy | ok. | 13:52 |
| mgariepy | let's go with 10.6.5 then ? | 13:52 |
| mgariepy | i'm happy with that. and i guess 10.6.8 will have a fix for this issue | 13:52 |
| noonedeadpunk | yeah, I'd say let's go with 10.6.8 | 14:26 |
| noonedeadpunk | *10.6.5 lol | 14:26 |
| noonedeadpunk | we can do minor bump afterwards | 14:26 |
| mgariepy | yep. ok | 14:30 |
| noonedeadpunk | but we won't do major one... | 14:31 |
| mgariepy | sure i don't think it's really buggy anyway :D at least for now. :D | 14:33 |
| mgariepy | and on the plus side monty is really keen to help on issues. i don't think we will have much issue with it. | 14:34 |
| noonedeadpunk | that is true, yes | 14:34 |
| spatel | 😃 at least for now. 😃 @mgariepy that is good one | 14:44 |
| spatel | noonedeadpunk i am testing RabbitMQ without HA and i really link it. | 14:45 |
| spatel | now i am testing some disaster scenario, like node down/partition etc.. | 14:46 |
| mgariepy | lol | 14:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Ensure prettytable is present in the ansible-runtime virtualenv https://review.opendev.org/c/openstack/openstack-ansible/+/777061 | 15:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Pin uWSGI version https://review.opendev.org/c/openstack/openstack-ansible/+/819133 | 15:45 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_cinder master: WIP: test fix for zun volume tempest failures https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/819141 | 16:08 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: kata: disable installation on debian bullseye https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/817714 | 16:09 |
| *** akahat|rover is now known as akahat | 16:33 | |
| opendevreview | Merged openstack/openstack-ansible-os_tempest master: Update URL for documentation of python-tempestconf https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/818869 | 17:23 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Fix mysql_upgrade possible race condition https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/819079 | 18:13 |
| spatel | all and all rabbitMQ without HA running super neat and clean | 18:19 |
| spatel | I did test network partition etc.. and noticed failed RPC call but then in 1 minute it got recover so not a big deal | 18:20 |
| spatel | I am going to push all my production to non-HA queue | 18:20 |
| noonedeadpunk | well, great news is that it's configurable relatively easily) | 18:38 |
| noonedeadpunk | and in days back in R non-HA was disaster | 18:38 |
| noonedeadpunk | every controller outage lead to rabbit failure | 18:39 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 18:39 |
| mgariepy | i remember some pain when upgrading from 14.04 > 16.04.. | 18:44 |
| opendevreview | Merged openstack/ansible-config_template master: Move config_template to plugins/action https://review.opendev.org/c/openstack/ansible-config_template/+/819010 | 18:46 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: keypair: copy key to deploy host rather than setup host https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/816997 | 18:57 |
| spatel | noonedeadpunk i just deleted HA policy in rabbitMQ UI and automatically it removed mirror | 19:16 |
| spatel | i have tried all possible way to break RabbitMQ but it works | 19:16 |
| spatel | next week i will try non-HA on my production cloud so will see how it goes :) | 19:17 |
| spatel | if you google rabbit issue. every single person suggesting don't do HA for RPC which is not worth it.. these are short live messages which is ok to get lost | 19:18 |
| spatel | only billing related data required HA otherwise not worth | 19:18 |
| opendevreview | Merged openstack/openstack-ansible stable/ussuri: Backported fixes for healthcheck playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/803926 | 19:25 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!