| clarkb | Meeting time | 18:59 |
|---|---|---|
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Dec 5 19:00:08 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/JBOF56GLO37SHVBZEXWSP4N3CIRIDQGG/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | I mentioned this previously but I won't be able to make our meeting time next week | 19:01 |
| clarkb | I've put on the agenda a proposal to go ahead and skip the meeting on these days due to holidays: December 12, 26 and January 2 | 19:01 |
| fungi | sgtm | 19:01 |
| fungi | thanks! | 19:01 |
| * tonyb will be traveling back to AU this weekend so basically absent | 19:02 | |
| clarkb | That means our last meeting of the year would be December 19 and we'll be back January 9 | 19:02 |
| clarkb | tonyb: safe travels! | 19:02 |
| tonyb | "cheers mate!" | 19:03 |
| clarkb | #topic Server Updates | 19:03 |
| clarkb | Just this morning we put three new mirror nodes into use | 19:04 |
| fungi | thanks tonyb! | 19:04 |
| tonyb | https://review.opendev.org/q/topic:mirror-distro-updates+is:open finishes off the process IIUC | 19:04 |
| tonyb | well the CI side, I need to remove the old servers | 19:05 |
| clarkb | with a manual step of removing the server and volumes | 19:05 |
| clarkb | ya that | 19:05 |
| tonyb | ++ | 19:05 |
| clarkb | I should be able to review those soon. I'll shout if I see anything amiss | 19:05 |
| tonyb | Starting on JVB+meetpad later today | 19:05 |
| tonyb | thanks clarkb | 19:06 |
| clarkb | sounds good. For meetpad I think we can do two appraoches. One is to replace the jvb then the main meetpad server in sequence and have them sync up with each other at each step of the way. The other would be to deploy a new meetpad and a new jvb independent of the old stuff and then swap over dns. I'm not sure if the existing ansible is built for the second option | 19:07 |
| clarkb | it may just try to automatically make everything work together based on ansible groups | 19:07 |
| clarkb | anything else server related? | 19:08 |
| tonyb | Good to know. I'll poke around | 19:08 |
| tonyb | not from me | 19:08 |
| clarkb | #topic DIB Bionic Support | 19:08 |
| clarkb | DIB's python3.6 unittesting is broken | 19:08 |
| clarkb | #link https://review.opendev.org/c/openstack/diskimage-builder/+/901093 proposal to drop py36 testing | 19:08 |
| clarkb | In this change ianw suggested we (opendev) pause/stop building bionic nodes and DIB would then actively remove py36 support entirely | 19:09 |
| ianw | o/ | 19:09 |
| clarkb | I have no problem with removing py36 unitesting in dib but am a bit more cautious around the assumption we don't need to build bionic nodes. That may be necessary due to changes in clouds for example | 19:10 |
| clarkb | My proposal would be that dib remove py36 testing as necessary toget its CI stable again, but not actively take steps to break py36 yet | 19:10 |
| clarkb | the code base is fairly stable so I don't expect big changes that would cause that to happen unless dib explicitly set a python requires or similar | 19:10 |
| clarkb | ianw: feel free to jump in with other info, background or thoughts | 19:11 |
| ianw | yeah, so i don't think anyone has anything 3.6 breaking as you say | 19:11 |
| ianw | but opendev might be in a position of wanting a bionic build from a nodepool host with a dib that can't do that (theoretically) | 19:11 |
| frickler | so we'd just need to stop building new bionic images, we can still use the existing ones, rught? | 19:12 |
| ianw | my thought was to just avoid that possibility up front -- but it's only a possibility | 19:12 |
| fungi | i agree with not dropping bionic image builds while we're still relying on them. we've gotten ourselves into situations in the past where we lost our local copy of an image we were no longer able to build and so couldn't reupload it to providers | 19:12 |
| JayF | You could still use older versions of DIB to build bionic images, correct? | 19:12 |
| clarkb | frickler: we could manually manage bionic images copying from one cloud to another yes as a possibility | 19:12 |
| ianw | someone could, i'm sure, manually build with old versions of dib or manually move images. this would suck for that person :) | 19:13 |
| clarkb | JayF: not easily that would potentially require a dedicated builder just for bionic | 19:13 |
| JayF | ack | 19:13 |
| clarkb | basically my point is bionic buidls work today | 19:13 |
| clarkb | we can drop testing without changing that state and then we don't need to pause builds | 19:13 |
| clarkb | if bionic builds do break in say a yaer or two hopefully we don't need bionic anymore and if we do we can decide if we stop building them at that point or fix dib | 19:14 |
| clarkb | rather than an upfront shift to explicitly say this isn't supported and intentional steps to make it not work | 19:14 |
| clarkb | I do think the suggestion for a tag nowish is a good idea as a fallback point if that becomes necessary | 19:15 |
| ianw | so no intentional steps to make it not work -- maybe i mentioned that but i don't think we've ever done that | 19:15 |
| fungi | at the moment, openstack stable/ussuri and older branches rely on bionic jobs | 19:15 |
| corvus | yeah, i think as long as we have bionic images, we need to keep the builder going. if it breaks, then at that point we should consider that we no longer have bionic images. | 19:15 |
| clarkb | ianw: I may have also misparsed | 19:15 |
| ianw | it sounds like you're ok with the uncertainty window that i was thinking should be closed | 19:15 |
| ianw | which is fine too | 19:16 |
| corvus | keeping an image around without the ability to build a new version is not feasible in the long run -- as soon as we can't build an image, that's when the clock starts on either fixing it or removing it. | 19:16 |
| clarkb | also I've just checked a recent dib change and py36 passed | 19:16 |
| clarkb | so maybe this is a non issue? | 19:16 |
| fungi | under openstack's old extended maintenance model it would normally have forced stable/ussuri to eol around mid-2025 | 19:16 |
| clarkb | https://review.opendev.org/c/openstack/diskimage-builder/+/902497 this change | 19:16 |
| fungi | with the new unmaintained model, maybe we don't have to worry abotu it for quite that long | 19:17 |
| corvus | to build on ianw's nomenclature, i think the uncertainty window is "when does a bionic build finally accidentally break" | 19:17 |
| ianw | huh, maybe stestr had a similar discussion and restored 3.6 support? | 19:18 |
| clarkb | ianw: that would be my hunch | 19:18 |
| ianw | anyway, i think the whole thing is on borrowed time until we have this discussion again :) | 19:18 |
| ianw | so basically, it seems like we're ok with dib not actively breaking 3.6, but also not taking efforts to do 3.6 testing when it fails | 19:19 |
| clarkb | yes I think we should have a general plan. I'll prpose this: Keep py36 testing as long as it is passing without a ton of effort and opendev still needs bionic images. If testing starts to fail and there isn't an easy fix drop the testing but don't explicitly break old pythons. Tag at this point as fallback. if at some point int he future bionic buidls break opendev can either | 19:19 |
| clarkb | fix it in dib with less automated testing or accelerate bionic removal | 19:19 |
| corvus | clarkb: ++ | 19:20 |
| tonyb | I think 3.6 is still unsupported: https://github.com/mtreinish/stestr/commit/cc66736df90968126585540b220fea663b124bbf | 19:20 |
| clarkb | additionally I think it would be trivial to cap stestr for py36 testing | 19:20 |
| fungi | environment marker, i guess | 19:20 |
| clarkb | now that I've had more time to digest what initiated this change I think I'm actually -2 even if stestr is broken | 19:20 |
| clarkb | we just install older stestr for py36 and move on | 19:20 |
| clarkb | fungi: yes exactly | 19:21 |
| clarkb | anyway that is beside teh point considering py36 testing does work | 19:21 |
| ianw | well i would say removing 3.6 in dib has been a topic of interest for a long time, so it's good to have a discussion about it | 19:21 |
| ianw | anyway, i think we've sorted out the major points so can follow-up in the changes | 19:22 |
| clarkb | cool. As a side note I think rhel 8's main python is 3.6 as well so not bionic specific? | 19:22 |
| clarkb | but rhel 8 has packaging for newer pythons so less impactful | 19:23 |
| clarkb | #topic Python Container Updates | 19:23 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/898756 And parent add python3.12 images | 19:23 |
| clarkb | Not aware of any major movement here, but a reminder that I thinki we can go ahead and approve ^ | 19:23 |
| clarkb | #topic Gitea 1.21 | 19:24 |
| tonyb | Yeah no progress, the time I wanted to spend on this got redirected into a downstream project | 19:24 |
| clarkb | Lots of movement on the gitea side of things though its a bit one step forard two back | 19:24 |
| clarkb | the tl;dr is that we added a new larger key to gitea as the first step to rotating the old key out | 19:25 |
| clarkb | the next step is adding the key to gerrit and having it use the new larger key. Unfortunately when we attempted that on firday none of it worked | 19:25 |
| clarkb | And the problem was entirely pebkac | 19:25 |
| fungi | le sigh | 19:25 |
| clarkb | I used review02 host side file paths to ssh keys and not gerrit container filesystem file paths to keys | 19:26 |
| clarkb | so our .ssh/config wasn't working beacuse the file "didn't exist" | 19:26 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/902490 Configure Gerrit to use new SSH key | 19:26 |
| clarkb | I've pushed a new change up which corrects that and has been tested on some held nodes | 19:26 |
| fungi | i'm going to blame it being a long week and we weren't as observant as we might have been were it not the end of the day on a friday | 19:26 |
| clarkb | I don't think I've got time to do a gerrit restart today given how much we've already gotten wrong around that. I want to ensure I can focus on it | 19:27 |
| clarkb | but maybe tomorrow? | 19:27 |
| tonyb | So that ssh config works as expected on the test nodes? | 19:27 |
| clarkb | tonyb: yes | 19:27 |
| tonyb | Oh cool | 19:27 |
| clarkb | tonyb: the key was using the IdentifyFile path using the container's filesystem paths and not the hosts | 19:27 |
| fungi | tomorrow will work for me | 19:27 |
| tonyb | tomorrow is good for me. | 19:28 |
| clarkb | excellent we can sync back up again tomorrow and figure out timing | 19:28 |
| tonyb | ++ | 19:29 |
| clarkb | #topic Upgrading Zuul's DB Server | 19:29 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-zuul-mysql-upgrade info gathering document | 19:30 |
| clarkb | I didn't end up having much time to look into this, but I did briefly pull up postgres' HA documentation (note postgres clustering is completely different so you want to use HA for search terms) | 19:30 |
| clarkb | The impression it gave me was that there are a lot of options and your use case is important to picking the right one and none are super drop in out of the box easy | 19:31 |
| clarkb | So basically if we go down the postgres path we'll need to do a bit of investigating | 19:31 |
| clarkb | Has anyone else done any research to add to the pile? | 19:32 |
| clarkb | I'm guessing that is a no. I'm hopeful that I'll have time to look into this more around the holidays as research tends to be a bit more time flexible for me | 19:34 |
| clarkb | #topic Annual Report Season | 19:34 |
| clarkb | #link https://etherpad.opendev.org/p/2023-opendev-annual-report OpenDev draft report | 19:34 |
| clarkb | There is a draft in this etherpad now | 19:34 |
| tonyb | I did a small amount on Mysql 5.X to 8.x DB upgrades. | 19:35 |
| clarkb | hrm can I undo twice? | 19:35 |
| clarkb | #undo | 19:35 |
| opendevmeet | Removing item from minutes: #link https://etherpad.opendev.org/p/2023-opendev-annual-report | 19:35 |
| clarkb | #undo | 19:35 |
| opendevmeet | Removing item from minutes: #topic Annual Report Season | 19:35 |
| clarkb | I can ! | 19:35 |
| tonyb | As expected it's basically a dump+restore but going through 6 and 7 before 8 so really 3 dump+restores | 19:35 |
| clarkb | tonyb: anything jump out to you as important? | 19:35 |
| corvus | oh if we just want to upgrade i think we would just dump and restore | 19:36 |
| corvus | once | 19:36 |
| tonyb | I picked 8 as it was the latest in trove | 19:36 |
| tonyb | corvus: That's not waht the docs say, but they could be overly cautious | 19:36 |
| corvus | i mean, it's sql | 19:36 |
| corvus | i'm talking mysqldump + mysql | 19:37 |
| fungi | yeah, it would mean an outage, but hopefully not too long of one | 19:37 |
| tonyb | Yeah me too | 19:37 |
| corvus | i've done that exact upgrade something like 20 times over the past month. it's fine. :) | 19:37 |
| fungi | also we could time a dry run to find out approximately how long of an outage we're talking about | 19:37 |
| tonyb | corvus: Okay | 19:37 |
| clarkb | could be a generic suggestion for specific corner cases | 19:37 |
| clarkb | but in this case it sounds like we know this db can make the transition just fine | 19:38 |
| tonyb | Okay. I was worried it would triple the outage window but it sounds like not | 19:38 |
| corvus | fungi: it's basically about the same amount of time our recent schema change took | 19:39 |
| fungi | noted | 19:39 |
| corvus | because our schema change was "rewrite the two biggest tables" | 19:39 |
| corvus | (hopefully the actual cpu time, not the "figure out what went wrong" time) | 19:40 |
| fungi | i guess the workload for updates and inserts are comparable in that case | 19:40 |
| corvus | well, i'm trying to say they're both inserts :) | 19:40 |
| fungi | ah | 19:41 |
| corvus | which is why they're equivalent in this specific case | 19:41 |
| fungi | makes sense, yep | 19:41 |
| clarkb | #topic Annual Report Season | 19:41 |
| clarkb | #link https://etherpad.opendev.org/p/2023-opendev-annual-report OpenDev draft report | 19:41 |
| clarkb | as I mentioned prior to the undos there is a draft here now. I'm considering this draft fairly complete but please do make suggestions or point out missing content if you notice anything like that. I've also got placeholders for stats data that fungi will eb collecting | 19:42 |
| corvus | i read that yestrday and it looks good to me! thanks! | 19:42 |
| clarkb | This document goes into the foundation's annual report and is a good way to us to show off to people that may not be in the know typically about what we do | 19:43 |
| clarkb | so definitely call out anything you feel is important and missing | 19:43 |
| clarkb | #topic Mailing lists and Gmail requirements | 19:44 |
| fungi | this is something frickler spotted and brought up in irc earlier today | 19:44 |
| clarkb | This is a new one. Apparently gmail is throttling list.openstack.org beacuse it lacks spf and dkim. Adding one or the other seems to be sufficient according to the message they respond with to stop getting rate limited | 19:44 |
| clarkb | #link https://review.opendev.org/q/topic:%22argh-gmail%22 Add an SPF record for the listserv | 19:44 |
| fungi | also according to the article i linked in the commit message, it will go from throttling to rejecting in february | 19:45 |
| clarkb | I have no concerns or objections with the proposed dns record update | 19:45 |
| clarkb | As written it should largely be a noop and it would be good to confirm it makes google happy | 19:45 |
| fungi | as little love as i harbor for gmail, i think it will mean less work for our mailman server if it doesn't have to hold all those deferrals, and less work for list owners/moderators dealing with gmail users getting their subscriptions put on hold | 19:46 |
| * frickler likes the gerrit topic | 19:46 | |
| corvus | lgtm | 19:46 |
| fungi | the two changes in that topic cover the list sites we do dns in opendev for | 19:46 |
| fungi | the other 5 domains i'll make similar edits to in their respective hosts (rackspace and cloudflare) | 19:46 |
| fungi | since i'm not hearing objections, i'll un-wip them now | 19:47 |
| tonyb | Seems good to me | 19:47 |
| fungi | and again, thanks to frickler for spotting this and bringing it up | 19:48 |
| clarkb | I'm trying to think how we can test it directly after the dns updates are made but my gmail account isn't subscribed to anything aynmore because years ago I got auto unsubbed for some reason that had to do with gmail | 19:48 |
| clarkb | but it is probably sufficient to make the changes and monitor the exim logs | 19:49 |
| fungi | yes, that's my plan | 19:49 |
| fungi | see if, after a while, it stops logging deferrals | 19:49 |
| clarkb | sounds good | 19:50 |
| clarkb | #topic Open Discussion | 19:50 |
| clarkb | I've got one more "its should be straightfoward review stack because it doesn't actually touch production": https://review.opendev.org/c/opendev/system-config/+/901469/ and parent add gerrit 3.9 images and testing | 19:50 |
| fungi | oh, i'm going to be travelling a big chunk of thursday and friday, so am not expecting to be around much | 19:51 |
| fungi | should be back on for a bit in my friday afternoon. may also be able to check in from the hotel thursday night my time | 19:51 |
| clarkb | tonyb: I also left the python testing with newer python in containers or else item under open discussion as a reminder to ask if you were able to look at the existing job roles for that yet | 19:51 |
| clarkb | fungi: thanks for the heads up | 19:51 |
| tonyb | clarkb: I have not, Later today for sure! | 19:52 |
| clarkb | I'll give it a few more minutes for any other topics but its looking like we might be able to end early and go find a meal | 19:53 |
| clarkb | thank you for your time and effort everyone! | 19:56 |
| fungi | thanks clarkb! | 19:56 |
| clarkb | #endmeeting | 19:56 |
| opendevmeet | Meeting ended Tue Dec 5 19:56:14 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:56 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-12-05-19.00.html | 19:56 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-12-05-19.00.txt | 19:56 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-12-05-19.00.log.html | 19:56 |
| tonyb | Thanks all! | 19:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!