| opendevreview | likui proposed openstack/openstack-ansible-plugins master: Replace base64.encodestring with base64.encodebytes https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/895755 | 03:16 |
|---|---|---|
| semantic | noonedeadpunk, thanks for supporting the feature! Still, we have to evaluate if the option would be really needed in production. Overall this cpu utilization based on busy waiting seems like BEAM responsivness optimization and it tends to lower with lowering available cpu cores by the way. | 06:28 |
| semantic | jrosser, yes, I did check dashboard, as I said this is pretty much almost empty openstack deployment and there's no backlog of messages, so rabbitmq is close to idling | 06:29 |
| noonedeadpunk | mornings | 07:05 |
| noonedeadpunk | oh, jrosser, you have already started looking into debian 12? :) | 07:06 |
| jrosser | noonedeadpunk: yes! up to the poo t it’s broken….. | 07:06 |
| jrosser | it delays and tempest fails with my patches | 07:07 |
| noonedeadpunk | `Please, ensure that you have at least 1 host in repo_all group with the following distribution: debian 12.1 x86_64` | 07:07 |
| jrosser | no idea why yet | 07:07 |
| noonedeadpunk | 12.1 sounds wrong for debian.... | 07:07 |
| jrosser | argh autocorrect | 07:08 |
| jrosser | yes - I didn’t find why it does that | 07:08 |
| noonedeadpunk | `Found venv_build_targets: {'debian-12-x86_64': 'aio1_repo_container-9e6db7be'` | 07:08 |
| jrosser | the version values come up different in the repo vs. the host | 07:08 |
| noonedeadpunk | and target looks correct.... | 07:08 |
| jrosser | look back in irc history, there was some chatter last week about this | 07:09 |
| noonedeadpunk | maybe we have some condition somewhere for debian 11 | 07:09 |
| jrosser | I have a vm somewhere I think here | 07:09 |
| noonedeadpunk | I was just going to have a look at debian 12 during the week as well :) | 07:09 |
| jrosser | if you’re going to make an AIO it would be good to get more eyes on this version wierdness | 07:11 |
| jrosser | and I didn’t yet figure out why tempest fails | 07:12 |
| jrosser | also version of some repos we use might be wrong, perhaps | 07:12 |
| noonedeadpunk | ++ | 07:12 |
| jrosser | we are lucky that mariadb already have bookworm packages though | 07:12 |
| noonedeadpunk | we also kinda need to close that before merging anything else I guess https://review.opendev.org/q/topic:osa/core-2.15+status:open | 07:13 |
| noonedeadpunk | as we keep breaking linters with new patches... | 07:14 |
| noonedeadpunk | s/we/I/ | 07:14 |
| jrosser | hmm some wierd sphinx thing too | 07:32 |
| noonedeadpunk | Yeah, I'm on it | 07:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Fix deployment guide renderring https://review.opendev.org/c/openstack/openstack-ansible/+/895769 | 07:54 |
| noonedeadpunk | this should fix it ^ | 07:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible-core to 2.15.3 and ansible-lint https://review.opendev.org/c/openstack/openstack-ansible/+/892371 | 07:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible collection versions https://review.opendev.org/c/openstack/openstack-ansible/+/892373 | 08:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible collection versions https://review.opendev.org/c/openstack/openstack-ansible/+/892373 | 08:12 |
| jrosser | noonedeadpunk: yes https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/894740 is really wierd failure | 09:01 |
| jrosser | i did the same and looked in the apt history and it makes no sense | 09:02 |
| noonedeadpunk | It looked like there's simply no 10.11.5 in repos | 09:02 |
| noonedeadpunk | and then apt jsut installs from the default repo | 09:02 |
| noonedeadpunk | as we're using infra cache | 09:03 |
| noonedeadpunk | but it's there: http://mirror.iad3.inmotion.opendev.org:8080/MariaDB/mariadb-10.11.5/repo/ubuntu/dists/jammy/main/binary-amd64/ | 09:04 |
| noonedeadpunk | and then - it's not failing for non-upgrade.... | 09:05 |
| noonedeadpunk | Also regarding mariadb - we've spotted some difference in how incremental backups are made - while it was couple of seconds previously when backup made member to have syncing state (and causing haproxy to mark backend as down), now it's couple of minutes | 09:07 |
| noonedeadpunk | Also according to incremental filesize, it looks like incremental backup is made from last full always, rather then from previous incremental... But it could be some change to the script actually, rather then mariabackup itself | 09:08 |
| jrosser | the version string is `1:10.11.5+maria~ubu2204` | 09:20 |
| jrosser | i guess 10.6. beats 10.11. in alphanumeric sort | 09:22 |
| jrosser | if there is a broken version compare somewhere (/me looks at mad code in ansible apt module) | 09:23 |
| noonedeadpunk | Maybe that won't be a thing with core 2.15?:) | 09:24 |
| noonedeadpunk | but that would be.... doh | 09:24 |
| noonedeadpunk | I believe they use python3-apt for comparing versions | 09:25 |
| jrosser | perhaps this needs running locally and just seeing what apt-policy says | 09:26 |
| noonedeadpunk | yeah | 09:26 |
| noonedeadpunk | if that is possible to reproduce at all | 09:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Fix deployment guide renderring https://review.opendev.org/c/openstack/openstack-ansible/+/895769 | 09:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Update deploy guide with relevant roles https://review.opendev.org/c/openstack/openstack-ansible/+/895779 | 09:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible-core to 2.15.3 and ansible-lint https://review.opendev.org/c/openstack/openstack-ansible/+/892371 | 10:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible collection versions https://review.opendev.org/c/openstack/openstack-ansible/+/892373 | 10:11 |
| noonedeadpunk | deploy guide should be fixed | 10:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Update RabbitMQ to 3.12 series https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/895804 | 10:59 |
| noonedeadpunk | fwiw, it seems I don't catch same issue with debian 12 as we see in CI | 11:12 |
| noonedeadpunk | wheels are built and stored properly | 11:12 |
| noonedeadpunk | also weird, that wheels build has failed only on Cinder but not before in CI | 11:13 |
| jrosser | noonedeadpunk: you mean the ansible version number thing? | 11:15 |
| noonedeadpunk | yup | 11:15 |
| jrosser | i think i have a held node for this | 11:15 |
| jrosser | though i have forgotton....... | 11:16 |
| jrosser | frickler: you did a held node for me last week - can you remind me what it's set on? | 11:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Switch classic queues to version 2 https://review.opendev.org/c/openstack/openstack-ansible/+/895806 | 11:17 |
| jrosser | oh hmm i wonder if thats no longer existing https://zuul.opendev.org/t/openstack/autoholds | 11:17 |
| noonedeadpunk | oh... there's a new authold tab in zuull... that's nice | 11:17 |
| noonedeadpunk | so you alse didn't saw anything weird wrt debian 12 locally? | 11:19 |
| noonedeadpunk | * also didn't see | 11:19 |
| frickler | jrosser: iirc that was for a patch which then stopped failing? anyway, let me know if you need anything new set up | 11:19 |
| noonedeadpunk | should ave been for this one I guess https://review.opendev.org/c/openstack/openstack-ansible/+/894561 | 11:20 |
| noonedeadpunk | mmmm... regarding tempest... | 11:24 |
| noonedeadpunk | I _think_ Debian still has unpatches version of OVS that doesn't respect hostname | 11:24 |
| noonedeadpunk | openvswitch-common=3.1.0-2 | 11:24 |
| noonedeadpunk | I mean this one https://bugs.launchpad.net/cloud-archive/+bug/2017757 | 11:26 |
| noonedeadpunk | zigo: hey! do you know if it's possible to minorly update ovs on bookworm to include bugfix of this ^ ? | 11:26 |
| noonedeadpunk | basically that's the patch https://mail.openvswitch.org/pipermail/ovs-dev/2023-March/403255.html | 11:27 |
| noonedeadpunk | As I would kinda hate to return workaround specifically for debian 12 | 11:28 |
| noonedeadpunk | (https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/891453) | 11:29 |
| zigo | noonedeadpunk: I can backport patches, but hardly can package a new version. What patch do you need in? | 11:30 |
| zigo | I can easily put a patched version in my unofficial debian.net backport repos though... | 11:31 |
| noonedeadpunk | basically this one https://github.com/openvswitch/ovs/commit/037e2d916146140d14f12dbd34070a07755eaa3e | 11:31 |
| noonedeadpunk | it's already in ovs 3.1.1 iirc | 11:32 |
| noonedeadpunk | yup, it's in 3.1.1 | 11:32 |
| noonedeadpunk | because it makes append command to overwrite things | 11:33 |
| noonedeadpunk | which is really nasty, especially when it resets node hostnames on service restart, for example | 11:34 |
| noonedeadpunk | But I'm pretty sure there're more unexpected behaviour | 11:34 |
| noonedeadpunk | both ubuntu and redhat already patched/backported fix for that just in case :) | 11:37 |
| jrosser | noonedeadpunk: locally my debian12 got as far as tempest | 11:40 |
| * jrosser looks for the vm | 11:40 | |
| jrosser | noonedeadpunk: https://paste.opendev.org/show/b212irj0QBWH1qyBoPGw/ | 11:44 |
| noonedeadpunk | yup, same for me. And on tempest it fails due to buggy ovs | 11:50 |
| noonedeadpunk | same bug that was on ubuntu and centos | 11:50 |
| noonedeadpunk | that's really weird. I wonder if dib has smth to do with that | 11:51 |
| noonedeadpunk | Though I've reproduced mariadb upgrade failure | 11:55 |
| noonedeadpunk | https://paste.openstack.org/show/bGr77uEc35xBCmfumSny/ | 11:56 |
| noonedeadpunk | And I'm pretty much confused when/how 10.6 got installed | 11:56 |
| jrosser | i wonder what `TASK [galera_server : Add galera repo]` actually did | 11:57 |
| jamesdenton | good morning | 13:03 |
| teresa | Hi | 13:07 |
| jrosser | o/ hello | 13:07 |
| teresa | Is there any way to get rid of 'macaddress' in the instances launched on ubuntu cloud image? It makes difficulty when you go to exchange interfaces on the customer instance | 13:08 |
| teresa | I mean: https://i.imgur.com/KkCrH64.png | 13:08 |
| teresa | the "match" section is generated by cloud-init and if you manually delete it, it will work correctly. | 13:09 |
| jamesdenton | that mac address corresponds to the neutron port assigned to the VIF | 13:13 |
| jamesdenton | i imagine if you're swapping interfaces/ports out that you'll need to somehow update the netplan file accordingly | 13:13 |
| teresa | It makes difficulty when you go to exchange interfaces on the customer instance "as the macaddress is changed and then you lost the connection to the server!" | 13:14 |
| teresa | jamesdenton: but the server continue to work well if I delete this lines: match, macaddress, set_name | 13:15 |
| jamesdenton | what's the reason for changing the interface of the instance? just curious | 13:15 |
| jamesdenton | you can try creating user-data for cloud-init that changes the netplan template to not include the mac address. for a single-homed instance it's probably OK. | 13:17 |
| teresa | to change the network attached to instance. We purchacse a tenant to the customer. They change interfaces (deleteing and adding interfaces) time to time for different reason, especially in their dev env | 13:17 |
| teresa | jamesdenton: "mac address corresponds to the neutron port assigned to the VIF" --> does it have any security reason? I mean is it ok to remove macaddress from the point of security? | 13:19 |
| jamesdenton | i only ask because there may be better ways of accomplishing the end goal. But if that is the way they want to go about it, you'll likely want to launch the instance with custom network definition via metadata. See https://cloudinit.readthedocs.io/en/22.3/topics/network-config-format-v2.html on the format | 13:20 |
| jamesdenton | "I mean is it ok to remove macaddress from the point of security?" -> Yes, it's OK to remove the MAC address inside the netplan config. The purpose is to match the MAC address to that particular interface name (ie. eno3). If you had multiple interfaces it would make more sense, for consistency. For single NIC instance it's not really necessary | 13:21 |
| jamesdenton | In terms of security, the instance is limited to sending traffic as the MAC address of the port/VIF connected to it. This is called 'port security' and there are default rules | 13:22 |
| teresa | Thanks a lot | 13:25 |
| jamesdenton | anyone here using the elk_metrics_7x repo? | 13:53 |
| andrewbonney | We are, although we tend to update it very infrequently | 13:59 |
| jrosser | we tend to have a huge bash at it every time we do some upgrade work on our elk stuff | 14:02 |
| andrewbonney | Yes, bash is probably a good way to describe it... | 14:03 |
| jrosser | though i think we did get someone else here in irc using it fairly recently, i think i patched some stuff which had rotted a bit | 14:03 |
| jrosser | like versions of things to download being a bit 404 and the like | 14:03 |
| jrosser | jamesdenton: to clarify, we use it for logs specifically | 14:04 |
| jrosser | the rest, not so much | 14:04 |
| jrosser | as much as elastic would tell you a document database can store a time series, reality says a bit different | 14:04 |
| jamesdenton | i've seen the commits to it over time but wasn't sure how it was used. thanbks | 14:12 |
| noonedeadpunk | I got my laptop dead. Can somebody start the meeting?:) | 15:00 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:05 |
| opendevmeet | Meeting started Tue Sep 19 15:05:06 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:05 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:05 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:05 |
| noonedeadpunk | #topic rollcall | 15:05 |
| noonedeadpunk | o/ | 15:05 |
| mgariepy | hey | 15:05 |
| mgariepy | half there. | 15:05 |
| noonedeadpunk | Sorry folks, I got my laptop dead couple of hours ago, so will be pretty much away until will get a new one | 15:05 |
| noonedeadpunk | In the meanwhile - we can catch up on what's going on | 15:06 |
| jamesdenton | o/ | 15:07 |
| jrosser | o/ hello | 15:07 |
| noonedeadpunk | # topic office hours | 15:09 |
| noonedeadpunk | Ugh | 15:09 |
| noonedeadpunk | #topic office hours | 15:09 |
| noonedeadpunk | So, we had good progress on merging ansible-core 2.15, but I've pushed patches that break linters again | 15:10 |
| noonedeadpunk | So some more reviews are needed | 15:10 |
| noonedeadpunk | And maybe we should prevent from merging other things until then? | 15:10 |
| noonedeadpunk | As it starts to be annoying... | 15:11 |
| jrosser | it is these https://review.opendev.org/q/topic:osa%252Fcore-2.15 | 15:13 |
| noonedeadpunk | Yes | 15:13 |
| noonedeadpunk | We also have deploy guide job broken now | 15:13 |
| noonedeadpunk | But, seems the fix works | 15:14 |
| noonedeadpunk | Aha, it's already in gates | 15:15 |
| noonedeadpunk | Then we need to decide on PTG time | 15:15 |
| noonedeadpunk | I'd suggest October 24 | 15:16 |
| noonedeadpunk | (it's Tuesday) | 15:17 |
| mgariepy | would work for me. | 15:18 |
| noonedeadpunk | Smth like 14-17 UTC? | 15:18 |
| noonedeadpunk | Or 15-18 | 15:18 |
| jrosser | oct 24 looks ok | 15:18 |
| mgariepy | either of the time are ok with me. | 15:19 |
| noonedeadpunk | Awesome | 15:21 |
| noonedeadpunk | I will book a time slot then | 15:21 |
| noonedeadpunk | Do we want to have an operator hour? As this seems to not work in fact | 15:21 |
| noonedeadpunk | Will also create an etherpad for topics | 15:22 |
| noonedeadpunk | #action noodeadpunk to book ptg slot and create an etherpad | 15:22 |
| noonedeadpunk | Right before my laptop went down, I was trying to come up with cycle highlights | 15:23 |
| noonedeadpunk | I was about to mention: Debian 12, Ansible core 2.15, rabbitmq 3.12 and quorum queues | 15:24 |
| noonedeadpunk | Was thinking about skyline, but I put zero effort there as of today | 15:25 |
| noonedeadpunk | Anything else worth to mention? | 15:25 |
| noonedeadpunk | I hate timing when we should push these, but we have what we have... | 15:25 |
| jrosser | feels somehow like we spend most of this cycle stabilising 2023.1 | 15:26 |
| noonedeadpunk | Yeah | 15:26 |
| noonedeadpunk | And it was only partially our fault | 15:26 |
| opendevreview | Merged openstack/openstack-ansible master: [doc] Fix deployment guide renderring https://review.opendev.org/c/openstack/openstack-ansible/+/895769 | 15:27 |
| noonedeadpunk | A lot of keystone regressions as well | 15:27 |
| noonedeadpunk | Though, might be this time we release in time... | 15:27 |
| jrosser | quorum queues is probably the big thing to get completed | 15:28 |
| jrosser | there is also some test improvements, i fixed a load of mess in os_tempest for more complex scenarios | 15:29 |
| jrosser | not sure that is all merged | 15:29 |
| noonedeadpunk | Yeah, not sure it fits in highlights... | 15:29 |
| noonedeadpunk | As not sure plenty of ppl care about that | 15:29 |
| jrosser | likley not | 15:30 |
| noonedeadpunk | Will add skyline I guess... still plenty of time to work on that 😅 | 15:30 |
| jrosser | good luck | 15:32 |
| jrosser | i think there is 'interesting' stuff to unpick there | 15:32 |
| noonedeadpunk | Yeah... I can imagine... | 15:32 |
| noonedeadpunk | But it was asked quite a lot | 15:32 |
| noonedeadpunk | So worth trying | 15:33 |
| jrosser | though actually we might have made it all a bit easier with haproxy maps now | 15:33 |
| noonedeadpunk | Ah. Also, do we want to enable quorum queues by default now? | 15:33 |
| noonedeadpunk | As this is smth that teases me | 15:33 |
| jrosser | if we can make that work on 3.x then the upgrade to 4.x would be smooth | 15:33 |
| jrosser | like switch to quorum independantly of the major version upgrade of rabbitmq | 15:34 |
| noonedeadpunk | Yeah, that's true. I guess it works, in general, but not sure about defaults | 15:34 |
| noonedeadpunk | They're quite some improvements to oslo regarding that from OVH | 15:34 |
| noonedeadpunk | That are not merged yet | 15:34 |
| noonedeadpunk | https://review.opendev.org/q/topic:bug-2031497 | 15:36 |
| noonedeadpunk | And they should make quite a difference imo | 15:37 |
| noonedeadpunk | So maybe we should disable quorum by default once we merge the topic and add another job to check that? | 15:37 |
| noonedeadpunk | Or dunno | 15:38 |
| noonedeadpunk | Really afraid after OVN | 15:38 |
| jrosser | that would be lower risk | 15:39 |
| jrosser | we could also switch to quorum pretty quick after we release | 15:40 |
| noonedeadpunk | Yeah, that as well | 15:40 |
| jrosser | like get the code and default setting merged before release | 15:40 |
| jrosser | then switch the default afterwards | 15:40 |
| noonedeadpunk | Well, not sure how much lower, given that rabbit is a nasty thing on it's own | 15:40 |
| noonedeadpunk | And you can understand if it's working or not only at scale at day2 ops | 15:41 |
| noonedeadpunk | Ok, sounds like a plan then | 15:42 |
| noonedeadpunk | That's all for me, does anybody has smth to discuss? | 15:42 |
| jrosser | we have some bugs | 15:45 |
| jrosser | like https://bugs.launchpad.net/openstack-ansible/+bug/2034714 | 15:46 |
| jrosser | i can only think that they have overridden `memcached_servers` there to point to the internal VIP | 15:50 |
| noonedeadpunk | Yeah, we have doc for that for "balancing" memcached through haproxy | 15:53 |
| noonedeadpunk | No idea why that would cause such issue.. | 15:54 |
| noonedeadpunk | But maybe it relies on a cache in a way... | 15:54 |
| noonedeadpunk | Worth checking that for sure | 15:54 |
| noonedeadpunk | Or well, asking about that at least | 15:54 |
| noonedeadpunk | #endmeeting | 15:59 |
| opendevmeet | Meeting ended Tue Sep 19 15:59:22 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-09-19-15.05.html | 15:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-09-19-15.05.txt | 15:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-09-19-15.05.log.html | 15:59 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Define mode for journald configuration file https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/895681 | 17:31 |
| Karni | "As default behaviour is that traffic is being processed by 1 core, Multi-queue virtio-net do release this. It provides an approach that scales the network performance as the number of vCPUs increases, by allowing them to transfer packets through more than one virtqueue pair at a time. (Ref: RedHat Docs)" | 21:22 |
| Karni | By setting either hw_vif_multiqueue_enabled per image in its properties or hw:vif_multiqueue_enabled per flavor (Yoga+). But is there any way to apply on an existing instanc?. | 21:22 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
