| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Remove cinder v2 references https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/934593 | 00:03 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Do not define TLS for frontend with `mode tcp` https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/934551 | 02:15 |
| cnilesh | is vpnaas is supported | 06:15 |
| noonedeadpunk | yeah, it kind of is and works nicely for us | 08:50 |
| cnilesh | noonedeadpunk++ thank you so much | 08:59 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Add ability to extend TLS SAN per VIP https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/934552 | 10:11 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-rabbitmq_server master: Ensure that first/last host detection is deterministic https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/935273 | 10:26 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-plugins master: Ensure that first/last host detection is deterministic https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/935274 | 10:29 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_senlin master: Ensure that first/last host detection is deterministic https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/935275 | 10:34 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_skyline master: Ensure that first/last host detection is deterministic https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/935277 | 10:39 |
| noonedeadpunk | andrewbonney: do we actually need to backport these ^ ? | 11:47 |
| noonedeadpunk | as I don't think we did backported rest? | 11:47 |
| noonedeadpunk | *did backport | 11:47 |
| andrewbonney | Oh, I thought these were merged much longer ago, my mistake | 11:52 |
| noonedeadpunk | ath the very beginning of this cycle | 11:55 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible stable/2024.1: Deny access to any paths including /. for console proxies. https://review.opendev.org/c/openstack/openstack-ansible/+/934686 | 13:27 |
| mgariepy | finally ! | 13:28 |
| noonedeadpunk | ┳━┳ ヽ(ಠل͜ಠ)ﾉ | 13:29 |
| mgariepy | do you have any toolling that help check all theses fails ? | 13:30 |
| mgariepy | like asking chatgpt to dig for you .. lol | 13:31 |
| noonedeadpunk | no, not really | 13:46 |
| noonedeadpunk | jsut wasting own time for now | 13:46 |
| mgariepy | would need a few intern | 13:47 |
| mgariepy | haha | 13:47 |
| noonedeadpunk | sounds like really worthy idea :D | 13:50 |
| mgariepy | i beleive, auto recheck for sporadic/random fails would be worth it as well. | 13:51 |
| mgariepy | it would reduce the number or cpu cycle needed, as well as the storage of logs needed, patch would merge a bit quicker (at least for this week) | 13:54 |
| noonedeadpunk | well, I mean, it should somehow understand what is random vs non-random | 13:58 |
| noonedeadpunk | and that's where requirement for comment for recheck is coming from | 13:59 |
| noonedeadpunk | so that instead of wasting limited CI resources teams were trying to understand on how to reduce failures | 13:59 |
| noonedeadpunk | and were spending time on that | 13:59 |
| mgariepy | yeah | 14:07 |
| mgariepy | but still this week was fail all over the place. | 14:08 |
| noonedeadpunk | it's better now then it was couple of weeks ago... | 14:08 |
| mgariepy | and the recheck on 20 jobs takes a lot more than 1 job redone on a failire. | 14:08 |
| noonedeadpunk | yeah, true | 14:09 |
| noonedeadpunk | we had really big issues with feting u-c from gitea for upgrade jobs | 14:10 |
| noonedeadpunk | as there were no retries and timeout was quite low | 14:10 |
| mgariepy | in some case the job do recheck for some type of failure. but i guess it's just like scheduling of the job that fails. | 14:10 |
| noonedeadpunk | it's if it failed in pre-run iirc | 14:11 |
| noonedeadpunk | and we've moved bootstrap-aio to pre-run already | 14:12 |
| noonedeadpunk | so some basic "health" of the node is "tested" there | 14:13 |
| mgariepy | python3 again for rocky .. | 14:13 |
| noonedeadpunk | some mirror is de-synced for sure | 14:13 |
| mgariepy | well all the other containers passed.. | 14:13 |
| mgariepy | it's surely not using a different mirror. | 14:14 |
| noonedeadpunk | but I guess it's randomly passed through LB or smth | 14:14 |
| noonedeadpunk | as there're no infra mirrors for rocky | 14:14 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/03853dbb3937458d96b37fbe9656d576/log/logs/etc/host/yum.repos.d/rocky.repo.txt | 14:15 |
| noonedeadpunk | this can return different things for containers | 14:15 |
| noonedeadpunk | https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=BaseOS-9 | 14:15 |
| noonedeadpunk | and I guess each container just picks and sticks to smth... | 14:16 |
| mgariepy | hmm we don't have containers logs ? | 14:17 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/94f46fa36e9649efba153e0c65295129/logs | 14:17 |
| noonedeadpunk | huh | 14:18 |
| noonedeadpunk | seems they've broke... | 14:18 |
| mgariepy | fun | 14:18 |
| mgariepy | it broke on failure.. | 14:20 |
| mgariepy | a passed one seems to be working fine | 14:21 |
| mgariepy | at least this one is : https://zuul.opendev.org/t/openstack/build/719acd0878b74a268ba126419615629d | 14:21 |
| noonedeadpunk | but there /etc is not stored | 14:22 |
| mgariepy | indeed but at least the logs are. | 14:23 |
| noonedeadpunk | I wonder if we should just repalce mirrorlist with some specific mirror for rocky | 14:26 |
| noonedeadpunk | and that would help at least for now | 14:26 |
| noonedeadpunk | especially if it would be default http://dl.rockylinux.org/ | 14:27 |
| mgariepy | how many forks in parallels for ansible tasks ? | 14:31 |
| noonedeadpunk | it depends | 14:33 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/scripts-library.sh#L45-L52 | 14:33 |
| mgariepy | ok | 14:35 |
| mgariepy | hmm https://zuul.opendev.org/t/openstack/build/8a7175b16e7548ffa59d4dcfa8adb8f7/logs | 14:37 |
| mgariepy | galera is special in that case | 14:38 |
| noonedeadpunk | could be something about package being missing or smth... | 14:43 |
| mgariepy | maybe | 14:50 |
| mgariepy | shall we ask for a hold on a vm ? | 14:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Replace usage of mirrorlist with an official Rocky mirror https://review.opendev.org/c/openstack/openstack-ansible/+/935362 | 15:05 |
| noonedeadpunk | I wonder if that would help for rocky issue with python ^ | 15:05 |
| noonedeadpunk | I believe this should help us to pass CI more sucessfully | 15:14 |
| gillesMo | Hello ! I'm on 2023.1, deployed by OSA 27.5.1, and I'm facing a live migration error. The bug is known at nova : https://bugs.launchpad.net/nova/+bug/2039803 | 15:52 |
| gillesMo | => ERROR oslo_messaging.rpc.server nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. | 15:52 |
| gillesMo | But, I have the same compute nodes, same CPU... | 15:53 |
| noonedeadpunk | don't you accidentally have old Xeon E5 or smth like that? | 15:54 |
| andrewbonney | We have that issue and reverted an older Nova change to work around it, but the config workarounds in the bug report should do the same thing | 15:56 |
| gillesMo | Yes : Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz | 16:01 |
| noonedeadpunk | mhm | 16:01 |
| gillesMo | I'm trying the workaround with skip_cpu_compare_at_startup and skip_cpu_compare_at_dest | 16:01 |
| noonedeadpunk | so, I have catched a regression in linux kernel between 5.13 and 5.17 iirc. | 16:01 |
| noonedeadpunk | where kernel injects a CPU flag arbitrary which is not present on CPU | 16:02 |
| noonedeadpunk | and I was able to fix that only by upgrading to LTS kernel | 16:02 |
| noonedeadpunk | there was a kernel patch, I tried to build it with kpatch, but somehow failed :( | 16:02 |
| gillesMo | That's remind me something, I had to revert from HWE Ubuntu kernel to stndard. I'm now using 5.4.0-198-generic | 16:02 |
| noonedeadpunk | let me try to find the bug report | 16:02 |
| noonedeadpunk | huh | 16:03 |
| noonedeadpunk | so you're on 20.04? | 16:03 |
| gillesMo | Yes 2023.1 on Ubuntu 20.04 | 16:03 |
| noonedeadpunk | ok, so we saw that on 2023.1 after upgrade to 22.04 | 16:03 |
| gillesMo | We've just finished upgrades from Wallaby 18.04... | 16:03 |
| gillesMo | Rhaaa, does not work, even with the workaround... Unless I made a typo | 16:25 |
| gillesMo | Typo... I put a section workaround instead of workarounds, retry | 16:33 |
| noonedeadpunk | hehe | 16:34 |
| gillesMo | Workarounds are working ! | 16:42 |
| gillesMo | Thanks ! I can go home in peace. Bye ! | 16:44 |
| mgariepy | lol | 16:49 |
| noonedeadpunk | o/ | 16:49 |
| mgariepy | Path rocky.repo does not exist ! | 16:51 |
| noonedeadpunk | doh | 16:51 |
| mgariepy | probablyu need the full path.. | 16:51 |
| mgariepy | haah | 16:51 |
| noonedeadpunk | oh damn | 16:52 |
| noonedeadpunk | lol | 16:52 |
| mgariepy | we should run the full tests in pre-run.. so we have retries | 16:52 |
| mgariepy | lol | 16:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Replace usage of mirrorlist with an official Rocky mirror https://review.opendev.org/c/openstack/openstack-ansible/+/935362 | 16:53 |
| noonedeadpunk | mgariepy: they're limited time-wise more then runs | 16:53 |
| mgariepy | i'll be back a bit later. | 17:01 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Rename haproxy_tls_vip_binds to haproxy_vip_binds https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/934553 | 17:12 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Use mirror.mariadb.org to install packages from https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/924354 | 17:55 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible stable/2024.1: Deny access to any paths including /. for console proxies. https://review.opendev.org/c/openstack/openstack-ansible/+/934686 | 20:02 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible stable/2024.1: Deny access to any paths including /. for console proxies. https://review.opendev.org/c/openstack/openstack-ansible/+/934686 | 20:02 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Update mariadb to 11.4.4 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/922377 | 23:08 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!