| *** tosky_ is now known as tosky | 00:24 | |
| semantic | Hello! Could you please look into https://bugs.launchpad.net/openstack-ansible/+bug/2046223 ? I went into this when trying to estimate changes we will need to adopt when switching to new openstack-ansible, which I suppose is to be released soon. | 09:29 |
|---|---|---|
| noonedeadpunk | semantic: hey. sure | 09:34 |
| noonedeadpunk | semantic: so basically if service is dead, haproxy simple does not detects it, right? | 09:36 |
| noonedeadpunk | and yeah, we should have been released previous week, but reviews of release suggestions somhow very slow this time... | 09:37 |
| noonedeadpunk | oh, it's actually just L4 check, not L7 | 09:39 |
| semantic | Yeah, if service is dead (or, for example in case of repo server, there is no file 'upper_constraints_cached.txt') haproxy still redirects requests to it, because of health probe not working actually. option httpchk (without additional parameters i suppose) should be added before http-check options. It can be tested by validating haproxy config, like this haproxy -f /etc/haproxy/haproxy.cfg -c | 09:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Add httpchk option when httpcheck_options are defined https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/903463 | 09:51 |
| noonedeadpunk | semantic: this should fix it ^ | 09:51 |
| semantic | Thank you! | 09:53 |
| noonedeadpunk | would be very helpful if you could test it and confirm it works :) | 09:55 |
| semantic | It actually does. :) But reveals another bug: in https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/nova_all/haproxy_service.yml we have this: haproxy_backend_httpcheck_options: "{{ haproxy_nova_console_http_mode | ternary(['expect status 200', 'send ' ~ haproxy_nova_healthcheck_hdr ~ ' meth HEAD uri /vnc.html'], []) }}", which ultimately results in situation when http-check expect goes in config BEFORE | 10:11 |
| semantic | http-check send. Haproxy fails on it. Should be after. | 10:11 |
| semantic | This problem appears in three services in https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/nova_all/haproxy_service.yml as far as i can see. Also in https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/murano_all/haproxy_service.yml and in https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/zun_all/haproxy_service.yml | 10:26 |
| * noonedeadpunk trying to process | 10:35 | |
| jrosser | me too :/ | 10:35 |
| jrosser | theres an obvious typo in the murano file where `haproxy_backend_httpcheck_options` is defined twice | 10:35 |
| noonedeadpunk | yeah | 10:36 |
| noonedeadpunk | ah, order is wrong.... | 10:36 |
| noonedeadpunk | let me check haproxy docs... | 10:37 |
| noonedeadpunk | actually, looking here is not stating order anywhere... https://www.haproxy.com/documentation/aloha/latest/load-balancing/health-checks/http/ | 10:40 |
| semantic | I actually couldn't find clear explanations in docs, but the order is important. I suppose it is related to the way haproxy generates implicit options (https://github.com/haproxy/haproxy/blob/f8e095b05828b8c82a01e37a18ea33e581e55431/src/tcpcheck.c#L3620) when you do not configure explicit ruleset http-check connect, http-check send, http-check expect. Otherwise it fails with sorta confusing error ''http-check send' : unable to add http-check | 11:10 |
| semantic | send rule at step 2 (missing connect rule)..' | 11:10 |
| noonedeadpunk | yeah, that's really unbvious from docs. You can only guess based on examples... | 11:14 |
| noonedeadpunk | I have also found issue in blazar simmilar to murano | 11:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix http-check ordering for services https://review.opendev.org/c/openstack/openstack-ansible/+/903488 | 11:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Add httpchk option when httpcheck_options are defined https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/903463 | 11:20 |
| noonedeadpunk | semantic: I think it should be better now | 11:20 |
| noonedeadpunk | also, are you aware of SLURP releases? Meaning, that OpenStack starting with Antelope does support jumping between SLURP releases? Ie, you will be able to upgrade for 2023.1 to 2024.1 directly without 2023.2? | 11:22 |
| noonedeadpunk | It's not that I discourage you to upgrade more like if you aware that this is a thing now | 11:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix http-check ordering for services https://review.opendev.org/c/openstack/openstack-ansible/+/903488 | 11:46 |
| jrosser | noonedeadpunk: how is it that we did not see these haproxy config errors - https://github.com/openstack/openstack-ansible-haproxy_server/blob/2cc2fceaf6369ccbdda048d8e26a276e90f818a1/handlers/main.yml#L43 | 12:27 |
| noonedeadpunk | we didn't load option httpchk - so it was not jsut used | 12:27 |
| noonedeadpunk | all http-check options are simply ignored | 12:27 |
| noonedeadpunk | and backends are up becuase it falls back to L4 check | 12:28 |
| jrosser | oh dear | 12:28 |
| jrosser | sorry about that it's my changes | 12:28 |
| noonedeadpunk | nah. no worries | 12:28 |
| noonedeadpunk | I wish it was discovered like ... previous week.. so we don't have to backport, but whatever | 12:28 |
| noonedeadpunk | but it's really fun clash of bugs | 12:29 |
| opendevreview | OpenStack Release Bot proposed openstack/openstack-ansible stable/2023.2: Update .gitreview for stable/2023.2 https://review.opendev.org/c/openstack/openstack-ansible/+/903496 | 12:34 |
| opendevreview | OpenStack Release Bot proposed openstack/openstack-ansible master: Update master for stable/2023.2 https://review.opendev.org/c/openstack/openstack-ansible/+/903497 | 12:34 |
| noonedeadpunk | finally | 12:41 |
| NeilHanlon | 🥳 | 13:25 |
| NeilHanlon | I wanted to ask, have rocky jobs seemed better the last few days? | 13:25 |
| noonedeadpunk | it looks better for me, but we didn't have too much jobs either | 13:27 |
| NeilHanlon | yeah, true. | 13:28 |
| NeilHanlon | i removed some mirrors that were just... bad | 13:28 |
| NeilHanlon | a couple of them synced the initial 9.3 release but nothing further--which I *think* is what was causing the failures, but it's not clear to me why our mirrrormanager software thought that it was OK to keep them in the rotation | 13:28 |
| NeilHanlon | anyways, I will work this week/next on a change to use the canonical baseurl instead of mirrorlist for our jobs | 13:29 |
| NeilHanlon | (and also clean up some old centos mirror variables which are unused) | 13:29 |
| opendevreview | Merged openstack/openstack-ansible master: Update master for stable/2023.2 https://review.opendev.org/c/openstack/openstack-ansible/+/903497 | 13:33 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.2: Update .gitreview for stable/2023.2 https://review.opendev.org/c/openstack/openstack-ansible/+/903496 | 13:45 |
| NeilHanlon | noonedeadpunk: meeting today? | 15:04 |
| noonedeadpunk | ugh | 15:05 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:05 |
| opendevmeet | Meeting started Tue Dec 12 15:05:23 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:05 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:05 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:05 |
| noonedeadpunk | #topic rollcall | 15:05 |
| noonedeadpunk | o/ | 15:05 |
| noonedeadpunk | thanks Neil and sorry all | 15:05 |
| NeilHanlon | sorry <3 | 15:05 |
| damiandabrowski | hi! | 15:05 |
| noonedeadpunk | I got my reminder but somehow closed it and got distracted right away | 15:06 |
| NeilHanlon | hehe i've been there | 15:07 |
| NeilHanlon | i got distracted for afew minutes myself lol | 15:07 |
| jrosser | o/ hi | 15:07 |
| opendevreview | Merged openstack/openstack-ansible-plugins stable/2023.1: Add no_log to setup_roles inlcude https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/902068 | 15:09 |
| noonedeadpunk | #topic office hours | 15:10 |
| noonedeadpunk | We have 1 interesting bug that I can't confirm | 15:10 |
| noonedeadpunk | I pinged you NeilHanlon about it yesterday :) | 15:10 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2046172 | 15:11 |
| noonedeadpunk | But I guess I need a help here | 15:12 |
| noonedeadpunk | I've spawned couple of hosts today from the cloud image, but they all had curl-minimal | 15:12 |
| NeilHanlon | yeah. i didn't get around to it yesterday but it's on my short list for today to figure out | 15:13 |
| noonedeadpunk | And reporter claims that issue can be reproduced by downloading latest ISO (which I haven't done as don't have hardware handy to test) | 15:13 |
| NeilHanlon | it seems it happens when curl is installed instead of curl-minimal. which leads me to believe something is resolving /usr/bin/curl to curl-minimal, even though curl satisfies the requirement | 15:13 |
| jrosser | is there a risk that the cloud image is a bit different from an install from ISO (or however you do baremetal) | 15:13 |
| jrosser | hah snap | 15:13 |
| NeilHanlon | we can repro in the cloud image by using `dnf swap curl-minimal curl` which will replace the minimal version with the full-fat curl | 15:14 |
| noonedeadpunk | Aha | 15:14 |
| noonedeadpunk | didn't know about that | 15:14 |
| noonedeadpunk | Will try it out | 15:14 |
| noonedeadpunk | Do you know if it's intended difference of curl vs curl-minimal in builds? | 15:15 |
| NeilHanlon | it's intended for DIB builds | 15:15 |
| NeilHanlon | as we build nodepool images from the rocky container images, which ship with curl-minimal | 15:16 |
| noonedeadpunk | ok then we need to see wtf causing it.... | 15:16 |
| jrosser | we have run into trouble in the past when the nodepool images diverged from "real" installations | 15:16 |
| NeilHanlon | #link https://review.opendev.org/c/openstack/diskimage-builder/+/871212 | 15:16 |
| noonedeadpunk | yeah, I can recall how we patched to use /usr/bin/curl | 15:17 |
| noonedeadpunk | Which I believe worked back then... | 15:17 |
| NeilHanlon | yeah, and it should; dnf knows that both these things are '/usr/bin/curl' but it seems something _else_ is resolving the dependency and deciding it should be the other one (?) | 15:18 |
| NeilHanlon | but that implies something changed in ansible-core which is now preresolving things rather than passing directly to dnf | 15:19 |
| jrosser | that would not be a surprise, ansible re-does all the work that apt would have done | 15:20 |
| jrosser | mostly correctly :/ | 15:20 |
| NeilHanlon | https://paste.opendev.org/show/bHG1os0z0LlVlHX631XE/ | 15:20 |
| NeilHanlon | here's my thought: we add some logic to bootstrap script which will figure out which curl is installed, and just force whatever version _we_ want, onto the system. | 15:22 |
| NeilHanlon | i.e., we choose either curl or curl-minimal; then we modify the bootstrap script to check which is installed, and if it's the wrong one, run the swap command | 15:22 |
| jrosser | can we issue the swap command regardless - i.e it doesnt blow up badly if theres nothing to do? | 15:23 |
| NeilHanlon | looks like you can, yes | 15:24 |
| NeilHanlon | running it twice gives exit 0 both times | 15:24 |
| NeilHanlon | just says 'nothing to do' basically | 15:24 |
| NeilHanlon | there's also a `dnf shell` command, which can be used to run multiple commands in a transaction-like shell | 15:25 |
| NeilHanlon | https://github.com/rocky-linux/rocky-tools/blob/main/migrate2rocky/migrate2rocky.sh#L980-L990 | 15:25 |
| damiandabrowski | sorry i'm semi off, but maybe this change has something to do with curl issues https://review.opendev.org/c/openstack/openstack-ansible/+/872896 | 15:39 |
| NeilHanlon | yeah that was the one that we introduced to get around this issue initially | 15:40 |
| noonedeadpunk | ugh, I got fully distracted again :( | 15:43 |
| NeilHanlon | i only got partially distracted | 15:44 |
| noonedeadpunk | Yeah, I just not sure if we can do a swap in any not completely disgusting way with ansible. | 15:45 |
| noonedeadpunk | Other then that - 2023.2 got branched and 28.0.0 should be out | 15:45 |
| noonedeadpunk | We had another bug report about haproxy today, which was not included into the release | 15:45 |
| noonedeadpunk | slightly nasty one | 15:46 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2046223 | 15:46 |
| noonedeadpunk | So I've already proposed patches to cover that - would be nice to land them shortly | 15:46 |
| noonedeadpunk | I'm also thinking that it might be nice to release new minor releases for stable branches this year | 15:47 |
| noonedeadpunk | I will take a look at SHA bumps and to unfreeze master after branching | 15:47 |
| noonedeadpunk | I will be also away during next week and not sure if will be able to make to the meeting | 15:48 |
| noonedeadpunk | and then 26 is kinda Christmas and then 2nd January is also right after NY... | 15:49 |
| noonedeadpunk | So I wonder if we should like cancel couple of meetings wrt | 15:49 |
| noonedeadpunk | #endmeeting | 15:57 |
| opendevmeet | Meeting ended Tue Dec 12 15:57:40 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:57 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-12-12-15.05.html | 15:57 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-12-12-15.05.txt | 15:57 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-12-12-15.05.log.html | 15:57 |
| NeilHanlon | i think canceling a couple meetings make sense. i can run next week's meeting | 16:02 |
| noonedeadpunk | ok, that would be sweet. And then let's cancel 26th of December and 2nd of January | 16:12 |
| NeilHanlon | sounds like a plan | 16:12 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/2023.2: Update deploy guide copyright years to be current https://review.opendev.org/c/openstack/openstack-ansible/+/903523 | 16:52 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Run deploy guide publish jobs when .gitreview changes https://review.opendev.org/c/openstack/openstack-ansible/+/903524 | 16:54 |
| jrosser | to fix deploy guide publishing ^^ | 16:54 |
| jrosser | first patch is a trivial update which should publish the 2023.2 deploy guide | 16:55 |
| jrosser | second one should make it automatic next time | 16:55 |
| noonedeadpunk | makes sense | 16:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Update deploy guide copyright years to be current https://review.opendev.org/c/openstack/openstack-ansible/+/903541 | 16:56 |
| NeilHanlon | hehe anndd i wish i could go back in time to remove my comment lol | 16:56 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.2: Update deploy guide copyright years to be current https://review.opendev.org/c/openstack/openstack-ansible/+/903523 | 17:31 |
| opendevreview | Merged openstack/openstack-ansible master: Update deploy guide copyright years to be current https://review.opendev.org/c/openstack/openstack-ansible/+/903541 | 17:40 |
| opendevreview | Merged openstack/openstack-ansible master: Run deploy guide publish jobs when .gitreview changes https://review.opendev.org/c/openstack/openstack-ansible/+/903524 | 21:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!