Tuesday, 2024-04-23

clarkbalmost meeting time18:57
fungiindeed18:58
clarkb#startmeeting infra19:00
opendevmeetMeeting started Tue Apr 23 19:00:09 2024 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:00
opendevmeetThe meeting name has been set to 'infra'19:00
clarkb#link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/WBD5BLMI6ZPKUB6FOSZ65P7P5RR7CHFS/ Our Agenda19:00
clarkb#topic Announcements19:00
clarkbI have no announcements. Did I miss anything announcement worthy?19:00
fungii don't think so19:01
clarkb#topic Upgrading Old Servers19:01
clarkbhaven't seen anything on this subject over the last week. I did have a mini panic when I thought Focal was EOL this month but it has another 12 months before that happens19:02
fungiit's been brought up that wiki.o.o is still very far behind and its ua/esm subscription expired a while ago19:02
fungii'm trying to work out where that discussion with canonical stalled19:03
clarkbthanks. I think the best case short term there is to renew the esm sub if possible and then contineu to look at a replacement19:03
fungibut rebuilding it on a newer system, if anyone has time, would address both that and the impending ssl cert expiration for it19:03
clarkbya though I'm happy to deal with the cert in a few weeks (which is on the agenda for later)19:04
clarkb#topic MariaDB Upgrades19:05
clarkbEtherpad and mailman have new shiny upgraded mariadbs now. Gerrit and Gitea are the remaining services that need similar treatment19:06
clarkbAfter thinking about it a bit due to the ease of upgrades for everything else I'm kidna thinking we may get away with just sending it for gitea19:06
clarkbCI should check general compatibility with newer mariadb and if that checks out then the upgrade process itself has been pretty reliable so far and we can probably expect it to continue to be reliable19:07
clarkbI'll push up a change for gitea and if anyone is concerned about that we can post those concerns in review19:07
clarkbOn the gerrit side of the house we will have to upgrade the db manually since the current ansible is very hands off with container management (and we want it to be)19:07
clarkbBut I do think we can just pick a time for gerrit and get it done since gerrit's db isn't very critical and the actual downtime should be quite short19:08
fungii agree19:08
clarkbin that case I guess I should also push up a change to reflect the update that will ultimately be somewhat manual in gerrit as well19:09
clarkb#topic AFS Mirror Cleanups19:09
clarkbI haven't made any new progress on Xenial cleanup. I think the scope of it is daunting enough that I'm happy to keep kicking it down the road if I can :) but I should just dive in and see what I learn19:10
clarkb#topic Building Ubuntu Noble Nodes19:11
clarkbhttps://review.opendev.org/c/opendev/glean/+/915907 merged which adds python3.12 support to glean19:11
fungiofficial release day for noble is thursday this week, right?19:11
clarkbI believe that our image builds use glean from releases by default though19:11
fricklerfungi: yes19:12
clarkbThis means we should plan to make a glean release. I can do that probably on Thursday ish as well19:12
clarkbonce we have a release we then need to monitor our next image builds but we have decent coverage of glean in CI so not too worried19:12
clarkb#link https://review.opendev.org/c/zuul/nodepool/+/916053 Update Nodepool debootstrap to support Noble19:12
clarkbseparately we need to update Nodepool images to add support to debootstrap for noble19:13
clarkbcorvus: ^ not sure if you have any input on that, but I think it does have the votes it needs so maybe we just go ahead and approve it now19:13
corvussgtm; i'll add a general +2 (i haven't reviewed the details)19:14
clarkback thanks19:14
clarkbGood news is there is solid progress here. Thank you frickler for testing it out and finding where the initial issues are19:14
frickler3rd patch needed will be https://review.opendev.org/c/openstack/project-config/+/91605019:14
clarkbonce we've got these three items sorted out we should be able to start building images and/or adding mirroring19:16
fungilooks like those changes are approved now19:16
fungiso just need the glean tag (assuming things merge/promote successfully)19:16
clarkbyup and I've got that on my todo list now19:17
fungiawesome, thanks19:17
clarkbI'm not committing to doing it immediately because I'm not sure what i need to do to get my gpg key out of cold storage and usable again19:17
fricklerthere is also a testing patch in dib fwiw https://review.opendev.org/c/openstack/diskimage-builder/+/91591519:17
fricklerI'll update that once the glean release is done19:18
clarkbsounds good19:18
clarkb#topic Etherpad 2.0.x Upgrade19:18
clarkbI can't remember the state of this during our last meeting, but I got the unreleased state of Etherpad to work with new api auth methods and added a bunch of testing to ensure that our admin tasks don't regress with the new setup19:19
clarkbEverything seems to be working which means that as soon as etherpad makes a release we should be read to upgrade. I don't want to upgrade until a release is done just in case anything else changes19:19
clarkb#link https://review.opendev.org/c/opendev/system-config/+/914119 Etherpad updates if interested19:20
clarkbthat chagne has all the new testing and docs and so on if you are interested in taking an early look at it19:20
clarkb#topic Add reference to the project-team-guide in the fail ci msg19:20
clarkb#link https://review.opendev.org/91418919:21
clarkbfungi: you added this topic want to fill us in?19:21
fungithis was more of a concern with blurring lines between opendev and openstack19:21
fungisee my comments on the linked review19:21
clarkbgotcha, I think I'm ok with adding that to the openstack tenant and having those of us still in the openstack tenant that probably shouldn't deal with it19:22
fungibasically, are we (collectively) okay having failure results in the openstack zuul tenant linking to openstack-specific guidance/documentation?19:22
clarkband if anyone has questions or concerns we can point them in the direction of using a different tenant19:22
clarkb(just to be clear that I don't think we should add all the possible debug doc links to the openstack tenant, more that we're in the openstack tenant so openstack things are probably ok)19:23
funginormally i'd say yes, but the openstack tenant was our original/default tenant and is still used by a lot of non-openstack projects as a result19:23
fricklerI agree that this may be a good opportunity to progress with tenant separation if needed19:23
clarkbits also something the vast majority of people are unlikely to read unfornately19:23
corvusi agree with clarkb and frickler19:24
clarkbI know I click the zuul summary button myself :)19:24
fungii'm okay approving it, but wanted to make sure there weren't major objections19:24
fricklerthe openstack guidelines don't sound too unreasonable for other projects I'd say19:24
fungiright, they're pretty general and not bad19:24
fungii'm just wary of setting new precedent without further consultation19:24
clarkbI'll +2 it after the meeting to make my stance clear in review (and to record it better)19:25
fricklerthx for mentioning the patch, though, I had missed it somehow19:25
fungisounds good, thanks19:25
clarkb#topic Gerrit 3.9 Upgrade Planning19:25
clarkbWe're in a position now where I think we could upgrade Gerrit tomorrow if we wanted to19:26
clarkbGerrit 3.9 reindexing should work, our images are updated to pull that fix in, etc19:26
clarkb#link https://www.gerritcodereview.com/3.9.html Release Notes19:26
clarkbthere are some intentional breaking changes though as noted in the release notes. None of them really impact us from what I can tell.19:26
clarkbThere is also a very straightforward downgrade path back to 3.8 if necessary19:27
fungii'm up for a tomorrow upgrade ;)19:27
clarkbAll that to say our next steps here should be to write up a document to track things like downgrade testing and breaking change review. Then we can schedule a day for the upgrade and get it done. I'll volunteer to get that document started and we can coordinate from there19:27
clarkbto be clear I don't think it will happen tomorrow :) just that the major known issues have been addressed :)19:28
fungiyeah, got it19:28
clarkbI also want us to keep in mind that we should start thinking about a gerrit server upgrade. Historically we've tried to give people some notice of these in order to update their firewall rules. THough honestly with the number of old clients attempting to authenticate and failing it may be a good idea to make it a surprise :)19:28
fungialso my availability for the first half of next month is pretty limited, so if i'm going to help it will either be next week or several weeks out19:28
clarkbseveral weeks out is probably fine19:29
clarkbsince there is prep work to get done and plenty of other stuff to do as well19:29
clarkbTalking out loud here I think our longer term planning should be something like upgrade to 3.9 in Mayish, upgrade the underlying server to noble during the summer sometime, then upgrade to 3.10 in November/December/January time frame19:30
clarkbOne other thing to note that doesn't really affect us since we've already done the work is JGit just proposed that they will drop java 11 support19:30
clarkbWe're already on java 17 so that is fine for us19:30
fricklerwould we consider moving away from vexxhost with the server? like we still don't have feedback about the unplanned shutdown yet? also recurring network issues19:31
clarkbConsidering the alternative options I'm not sure there is a clear cut winner in hosting. However, that is a good question and somethign we should keep in mind as we plan for server replacement19:31
clarkbvexxhost in particular lets us run a large server which the jvm running gerrit seems to appreciate19:32
fricklerI would be seeing ovh as possible option, maybe we can the the flavor question with them if there is interest in that19:33
clarkbLet's bring that back up after we've upgraded to 3.9 and start planning the server replacement. Ideally we'd be able to work with vexxhost to address the concerns19:33
fricklers/the/address/19:33
clarkb#topic Wiki Cert Renewal19:35
clarkbIn the past I've renewed this cert when there has been a week or less of validity19:35
clarkbthe current cert expires May 18, 2024. Which means renewing it sometime after May 1119:35
clarkbI'm happy to do that, but didn't want others worried it was ignored as we get the daily warning email19:36
fungiftr i'll be out of town, though it's a straight-up file replacement and apache2 restart19:36
fricklerjust noticing that iirc I still have no login for that server, so could not help right away if needed19:37
clarkbfrickler: thats a good call out. I suspect tonyb doesn't either. I can probably manually address that too when dealing with the cert19:37
tonybI do. fwiw19:38
clarkbah ok19:38
fungii'm happy to add/update ssh keys in a few minutes19:38
clarkbfungi: thanks!19:38
fungii can just copy/merge from a properly config-managed server, i expect19:38
tonybI'm wondering if we can build a new server in that opendev domain to get LE so this becomes less of a problem19:38
fricklernote there's a review with a new key up19:38
tonyb..... as part of the server upgrades away from bionic19:38
fungitonyb: the domain isn't the problem, just need ansible working19:39
tonybahh okay.  nevermind 19:39
fungibut yes, if we can get it onto a newer ubuntu version, easily solved19:39
tonybwe'll get that "soon"19:39
clarkbIn any case I wanted to make sure that others know the deadline here is known and that I intend on addressing it manually19:39
frickler+119:40
clarkbNormally I would ask for volunteers but it involves paying money and filign an expense report and I don't expect others to do that here19:40
clarkbside note: The cost of a cert and the cost of a beer have converged and in the wrong direction19:40
fungifingers crossed this is the last year you expense a cert on domaincheap19:41
clarkb#topic Open Discussion19:41
fungiso you can focus on expensing beer19:41
clarkbAnything else?19:41
tonybif we can get a new machine before may 18 donee need to do the expense dance?19:41
fungias mentioned above, i'll be travelling may 2-13 and not available to help with things19:42
fricklerregarding the ceph reef mirror change, do we want to/can we move all the repos onto a single volume, so we don't need a new one for each release?19:42
clarkbtonyb: probably not, but I wouldn't worry about it19:42
fungii could do the cert replacement on may 14 but that's cutting it close19:42
tonybfwiw I'll be in the US from Sunday 28/April to mid June (maybe the 17th)19:42
frickleralso I'll be away this Fri and Wed-Fri next week19:43
clarkbtonyb: basically renewing the cert is cheap enough we shouldn't consider it a hard deadlien thing19:43
tonybclarkb: okay.19:43
fungioh, in other news, i've started in on a dns record audit for openstack.org as mentioned in last week's meeting19:44
clarkbI hope you all manage to enjoy your time off19:44
clarkbfungi: manage to delete any records yet?19:44
fungiit's complicated by the domain being shared jurisdiction with foundation staff and webdev contractors, so all three parties err on the side of caution in not deleting records we've collectively forgotten why we added19:45
fungii see quite a few i'm pretty sure can be deleted, but i'll float a list first19:45
clarkbsounds good, thank you for looking at that19:46
clarkbI'll give it a few more minutes if anyone else has somethign to share otherwise we can probably end about 10 minutes early today19:46
fungiideally, we'll clean up what we know can go, then provide the other two parties with a list of records we don't have any interest in and let them separately filter the remainder19:46
fungicurrent record count is close to 300, so it's nontrivial to untangle19:48
tonybwow that's much more than I would have guessed.19:48
fungithere are still records in there pointing to sites that contained openstack project policy board (tc predecessor) election results19:49
fungieasily irrelevant for over a decade19:49
tonybfair enough 19:50
clarkbsounds like that was everything. Thank you for your time today and help running OpenDev. We'll be back here same time and location next week.19:50
clarkb#endmeeting19:50
opendevmeetMeeting ended Tue Apr 23 19:50:19 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:50
opendevmeetMinutes:        https://meetings.opendev.org/meetings/infra/2024/infra.2024-04-23-19.00.html19:50
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/infra/2024/infra.2024-04-23-19.00.txt19:50
opendevmeetLog:            https://meetings.opendev.org/meetings/infra/2024/infra.2024-04-23-19.00.log.html19:50
fungithanks clarkb!19:50
clarkband now I can go find some food. I think I have leftover garlic noodles19:50

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!