Tuesday, 2023-10-03

*** kopecmartin|off is now known as kopecmartin07:04
clarkbAlmost time for our team meeting18:58
clarkb#startmeeting infra19:01
opendevmeetMeeting started Tue Oct  3 19:01:01 2023 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
opendevmeetThe meeting name has been set to 'infra'19:01
clarkb#link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/OTYE3H5MGJHG2LMHKB6DYOED4HVGO3JL/ Our Agenda19:01
clarkb#topic Announcements19:01
clarkbOpenStack is finalizing the bobcat release this week (tomorrow if all goes according to plan)19:01
clarkbThen just under two weeks from now the PTG will be held19:03
clarkbWe aren't hosting any PTG meetings ourselves but we should expect people to use meetpad and keep on eye on that19:03
clarkbas well as etherpad19:03
clarkbThings to be aware of as we are making changes to the systems19:04
clarkb#topic Mailman 319:04
clarkbWe are down to our last mailman site migration for lists.openstack.org19:04
clarkbThe current plan is to perform that migration on October 12, 2023 around 1530 UTC iirc19:05
clarkbfungi: ^ anything to add to that?19:05
funginothing to add19:05
fungisorry, tc meeting was distracting and wrapped up late19:05
clarkbdid you want to bring up the mailserver configuration changes you have proposed?19:05
clarkbIn particular I think we can go ahead and add the nordix list and bitbucket the mailman address19:05
clarkbI've +2'd both changes19:06
fungitrying to organize my thoughts...19:06
fungiso some of the proposed changes are driven by the recent exim and libspf2 vulnerabilities which were announced19:07
fungii guess it's worth noting that there's at least one unfixed (from our perspective) buffer underrun in libspf2 and our listservs are the only places that's potentially exploitable by remote connection19:08
fungi#link https://review.opendev.org/897078 Temporarily disable SPF checking on ML servers19:08
fungiup for discussion whether that's something we want to do, it seems like exploiting that would require a malicious recursive resolver, but also there's not even any clear agreement on whether the suspected underrun is the one zdi was claiming to be able to exploit (the author of the fix for the publicly known underrun didn't succeed in finding a way to exploit it)19:09
fungiwe've also got a configuration change for something that was probably just missed in the switch of our server configs to adapt for mm3:19:10
fungi#link https://review.opendev.org/897086 Blackhole deliveries for Mailman v3 local user19:10
fungithat should be safe, and would silence a lot of errors in the mta log on the new server as well as freeing up a lot of cruft in its deferral queue19:11
fungifinally there's a proposal to add a new mailing list:19:11
fungi#link https://review.opendev.org/897234 Add mailing list for Nordix environment19:11
funginotable as it would be the first addition of a mailing list to our production mm3 server through continuous deployment automation of our config not associated with a migration effort19:12
fungii expect it will "just work" but we should check on it after deployment19:12
fungias for the upcoming lists.openstack.org migration, i still need to flesh out the migration plan and write the config changes for that19:13
clarkbya our testing does create mailing lists so it should have coverage but good to confirm19:13
fungiwell, and we19:13
fungi've created "new" lists on the server before each new migration19:14
fungisince they're created by ansible before we run the imports19:14
fungifor the final maintenance, i'll also send a one-week reminder to the openstack-discuss ml on thursday19:15
clarkbthanks. Anything else?19:16
funginothing on my end, no19:16
clarkb#topic InMotion/OpenMetal Cloud Replacement19:17
clarkbI haven't followed up to yuriy again to restart the conversation. part of the reason for the delay is a realization that yuriy would probably just prefer to have a phone call (on meetpad)19:18
clarkbI'm thinking I should propose something for later this week. Does Friday morning pacific time seem reasonable? I've also got some stuff at home so may have to jump around19:18
fungii can make that19:19
clarkbcool I'll try to propose that later today19:20
clarkb#topic Zuul PCRE Deprecation19:21
clarkbI don't see tempest in the list of warnings zuul provides19:21
clarkbstill plenty of warnings, but I don't think there is anything special we need to do at this point? Any concerns with dropping this off of our agenda?19:21
fungii have no concerns with that19:22
tonybShould we (I) propose some patches to fix the errors?19:22
tonybor do we want the teams to do that?19:22
clarkbtonyb: frickler has been driving it so far I would coordinate with him. In general I think we're hoping projects become a bit more responsive. It also isn't super urgent19:22
tonybKind of a "here's an attempt, go nuts"19:23
tonybOkay19:23
clarkbpart of the struggle in the past has been we write a bunch of changes then no one reviews them so it feels like wasted effort19:23
clarkbbetter if they write the patches knowing they can review them and we just trim projects out of zuul when their configs are invalid. But there is a balancing act19:23
tonybOkay19:24
fungii have changes along those lines that are still open since 201519:24
fungii take that back, i have a change that's still open from 201419:24
tonybhehe Okay19:24
clarkbin general though the zuul features to workaround the loss of regex functionality seem to be sufficient for us so far so we don't need to feedback to zuul on the change19:25
clarkbjust a matter of getting projects to update their configs19:25
clarkb#topic Python Container Updates19:26
clarkb#link https://review.opendev.org/q/(+topic:bookworm-python3.11+OR+hashtag:bookworm+)status:open19:26
clarkbAs mentioned last week zuul-registry relies on openssl 1.x and breaks under bookworm. There are however a few other things we can update under that topic if you have time for review19:26
clarkbEarly next week I would like to land the Gerrit java 17 + bookworm update as well19:26
clarkbas that is the last major one still pending19:26
clarkb#topic Etherpad 1.9.3 Upgrade19:28
clarkbI haven't heard from frickelr on whether or not there was time to test this19:28
clarkbtl;dr is I noticed some weirdness that appaered to be due to browser caching because switching to incognito mode and 1.9.3 was fine19:28
clarkbwe were hopign to get more data points from people testing against the held 1.9.3 node before we upgraded19:28
clarkbtonyb: ^ if you'd like to do that we can dig up the IP address. YOu ahve to edit /etc/hosts to point etherpad.opendev.org at it due to redirects but then you'll talk to the test server with its bad cert19:29
fungiif we're going to update it, soon would be preferable as we're approaching another ptg and we want to stabilize the deployment19:29
tonybYes please19:29
fungiip address is in my comment on the upgrade change19:29
clarkbfungi: yup, I'm hoping we can do ti this week19:29
clarkb#link https://review.opendev.org/c/opendev/system-config/+/896454 Change to upgrade has test node details in comments19:29
clarkblet us know if you test it and what you find. Then we can decide on upgrading or not19:31
tonybWill do19:31
clarkb#topic Gitea 1.2119:31
clarkbThere was timeline discussion in the gitea discord room the other day and they indicated it was ~3 weeks away19:31
clarkbWe also got a 1.20.5 update recently. We should upgrade to that before we continue 1.21 planning/testing19:32
clarkb#link https://review.opendev.org/c/opendev/system-config/+/897244 update to 1.20.5 first19:32
fungihas a changelog for 1.21 appeared yet?19:32
clarkbThe 1.20.5 changelog looks very straightforward. We can probably merge that right now if we want, but can also delay until after the openstack reelase to be careful19:32
clarkbfungi: I have not seen a 1.21 changelog yet19:33
clarkb#topic Gerrit replication leaked task files19:34
clarkb#link https://gerrit-review.googlesource.com/c/plugins/replication/+/387314 Clarkb wrote and pushed a fix upstream19:34
clarkbThe bulk of my time last week was spent around this. I was asked to write unittests but I couldn't even get gerrit and its plugins to build successfully locally which meant I couldn't run tests (also there are no docs on how to run plugin tests)19:35
clarkbI pivoted to learning how to make that work first. Turns out every version of Gerrit needs a different version of Bazel to build. If you don't use the correct version of Bazel then the linters will fail (and possibly other things).19:35
clarkbTo address this apparently everyone using bazel uses Bazelisk to run bazel19:35
clarkbonce I figured this out and installed bazelisk I was able to build Gerrit and run tests. I pushed a docs update upstream to have bazelisk info and examples which is in review. Then iterated on tests until I had some that fail on the old code and pass in my update19:36
clarkbIf anyone else wants to dive into Gerrit things I'm happy to help more directly as well. Hopefully the docs updates land at some point though19:37
clarkbAll that to say I'm hopeful we can fix this issue before we upgrade to 3.8 or maybe as part of the 3.8 upgrade depending on how things mereg and get applied to stable branches19:37
clarkbonce the bookworm upgrade is done and the plugin is fixed that will be the next gerrit task on my todo list19:38
clarkbwe already have some testing of the 3.7 to 3.8 upgrade buit we'll need to read release notes and probably hold nodes for testing particularly of the rollback19:38
clarkb#topic Open Discussion19:39
clarkbThat was all I had written in the agenda19:39
clarkbOne thing I noticed today is that our arm64 image builds all seem to be failing19:39
clarkbI was going to say I bet the builder filled its disk. I decied to just check really quickly and that is indeed the issue19:39
clarkbso we'll want to stop services, clean out the dib tmp dir and remove any leaked image build files, reboot for good measure (clears mount tables) and restart things19:40
clarkboh! the other gerrit thing I wanted to mention is their community meeting is happening at 8am Pacific Time Thursday October 5 on Discord19:41
tonybWhen I was doing DIB work I had a tool that'd clear out the mount tables without a reboot, I can try and dig that up for next time19:41
clarkbI'm going to try and be at the gerrit meeting. I joined their discord. The only gotcha is I have to drop the kids off at school that day. I can probably get them there early and make it back in time to make the start of the meeting19:42
clarkbpreviously there were issues with that meeting beause no one from google would join to let us into the google meet room19:42
clarkbthey solved that by hosting it in discord I guess19:42
clarkbI'm going to try and be involved upstream a bit more since I've gone through the process of learning some of their tooling I may as well fix the occaisional bug19:43
tonybYou could also do discord on your phone ;P19:43
clarkbthat is a good idea actually then I can walk home at a normal pace :)19:43
clarkbLast call anything else to discuss?19:44
tonybnope.19:44
clarkbthank you everyone for you time today and in general helping to run opendev19:47
clarkbwe'll be back here same time and location next week19:47
clarkb#endmeeting19:47
opendevmeetMeeting ended Tue Oct  3 19:47:14 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:47
opendevmeetMinutes:        https://meetings.opendev.org/meetings/infra/2023/infra.2023-10-03-19.01.html19:47
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-10-03-19.01.txt19:47
opendevmeetLog:            https://meetings.opendev.org/meetings/infra/2023/infra.2023-10-03-19.01.log.html19:47

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!