| clarkb | meeting time | 19:00 |
|---|---|---|
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Jan 17 19:01:08 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/MB3XWSVFXEROBXFXU5V3DX3D3QVMYLQR/ Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | I didn't have anything to announce. Did anyone else? | 19:02 |
| fungi | nothing particularly exciting, no | 19:02 |
| clarkb | #topic Topics | 19:03 |
| clarkb | #topic Bastion Host Updates | 19:03 |
| clarkb | I have a small update for ansible installation https://review.opendev.org/c/opendev/system-config/+/843330 | 19:03 |
| clarkb | Old bridge is still up. But I expect we'll shut it down real soon now | 19:04 |
| clarkb | Other than that I think new bridge has been happy. Was there anything else to mention on this topic? Probably drop it from the weekly agenda once old bridge is shutdown | 19:05 |
| ianw | (sorry bouncer issues) | 19:05 |
| clarkb | ianw: anything to add before we move onto the next item? | 19:06 |
| ianw | not really, there's still some things with some backup changes, and possibly some work on the parallel running | 19:06 |
| ianw | i haven't looked at either lately though | 19:06 |
| clarkb | oh right. I've been bad and mean to do reviews of those changes but keep getting distracted | 19:06 |
| * clarkb makes a note on the todo list and maybe that will help | 19:07 | |
| fungi | yeah, i had some down time over the holiday weekend and tried to dig into older system-config changes which have been growing stale | 19:07 |
| clarkb | #topic Mailman 3 | 19:08 |
| fungi | it would be good to get the parallel stuff finished | 19:08 |
| clarkb | ++ | 19:08 |
| fungi | no progress on the mm3 front since last week's meeting | 19:08 |
| clarkb | fungi: lsat week I said feel free to reach out if we can help with the mailman 3 stuff. You've got changes but they are WIP'd. Anything new? | 19:08 |
| clarkb | ah that answers that. | 19:09 |
| clarkb | Anything we can do to help move it along? | 19:09 |
| fungi | but year end/beginning paperwork is mostly wrapped up now, so i should have more time to get back to it | 19:09 |
| clarkb | sound good. | 19:10 |
| ianw | not mailman3, but on mailman2 ... | 19:10 |
| ianw | it's currently in emergency with some custom apache rules | 19:10 |
| fungi | oh, right, that's one thing that sucked up a bit of our collective time last week | 19:10 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/869779 | 19:11 |
| clarkb | I'll review that change after the meeting | 19:11 |
| fungi | thanks! | 19:11 |
| clarkb | but I guess we can remove the host from emergency once that lands? | 19:11 |
| fungi | should be able to, yes | 19:11 |
| ianw | would allow us to codify those rules | 19:11 |
| ianw | we'd have to make sure we define the current rules in the secrets for the host | 19:12 |
| clarkb | excellent, ya I'll get on that as soon as the meeting is done | 19:12 |
| ianw | although, i'm not 100% sure if that issue is ongoing | 19:12 |
| ianw | anyway, can double check and we can discuss outside meeting | 19:13 |
| clarkb | #topic Quo vadis Storyboard | 19:14 |
| clarkb | This is something i Just haven't had time to get to with all of the other paperwork and distractions. I'm hoping to be able to get to it at some point though | 19:14 |
| clarkb | #topic Gerrit 3.6 | 19:15 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/870118 Fix Gerrit 3.6 image build | 19:15 |
| clarkb | Gerrit updated the bazel version they use which is stricter about checking some flogger logging code. This has resulted in one of the plugins we build erroring on the bazel build | 19:15 |
| clarkb | The change above has us consume the latest version of that plugin which fixes the error. Looking at the difference between 3.6.3 and latest of that plugin I expect this to be safe | 19:16 |
| clarkb | Then separately I've got a stack of changes to drop gerrit 3.5 image builds and add 3.7 image builds and test all that | 19:16 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/869763 beginning of stack to remove 3.5 images and add 3.7 and associated testing. | 19:16 |
| ianw | ++ will review | 19:17 |
| clarkb | probably the most interesting bit of all that is the modifications to the upgrade testing to make the upgrade functional | 19:17 |
| clarkb | converts Verified to a submit requirement dropping the function specification (the docs actually say we should set function to noop but they also say function is deprecated so I decided to try just not setting it and that appeas to work) | 19:18 |
| clarkb | but ya its straightforward to start, the real fun will be as we test 3.7 more and figure out what all in our acls needs updating | 19:18 |
| ianw | ++ | 19:19 |
| ianw | ... on that | 19:19 |
| ianw | as we found out late last week, we may have committed some bad ACL's and not noticed | 19:19 |
| ianw | full details are at | 19:19 |
| ianw | #link https://etherpad.opendev.org/p/760YNeM5OEFS1hlr7bE5 | 19:19 |
| ianw | i think maybe if fungi could review that, i can take on doing an audit of where we're at, following the checklist in there, at some quiet time | 19:20 |
| clarkb | ianw: my only other piece of feedback is that afte rremoving the cache file maybe we want to manually run the tool against a specific project or three just to be sure we don't have unexpected behavior with the cache file out of the way before we do it against everything | 19:21 |
| clarkb | otherwise lgtm | 19:21 |
| fungi | i'll take a look, but may not have time until tomorrow | 19:21 |
| ianw | good idea, actually it's probably better to do something like loop the project list and keep each in a separate log file. i can make an update to the procedure | 19:22 |
| ianw | i think manage-projects works by getting the list of all projects, but then filtering it by any command-line parameters, iirc from code inspection last week | 19:23 |
| clarkb | yes that sounds right | 19:23 |
| clarkb | Sounds like that is all for Gerrit | 19:25 |
| clarkb | #topic Linaro Cloud Move | 19:25 |
| clarkb | The new nb04 arm64 builder is up and running in osuosl and I think ianw mentioned the new linaro cloud is running jobs? | 19:25 |
| clarkb | Ed emailed earlier today asking for an update. YOu can update us then copy paste into an email :) | 19:25 |
| ianw | i'm not sure on that -- a node did get allocated, but the wrong flavor :) so there is some communication happening | 19:26 |
| ianw | but yeah, nb04 has been my main focus and that seems to be working. so i think we can get rid of nb03 to have one less thing in linaro | 19:26 |
| ianw | but now i need to debug the node launches in the new cloud | 19:27 |
| clarkb | ianw: one easy thing might be to rename the flavor if we can | 19:27 |
| clarkb | the other idea I had was maybe we can set max-servers in the old linaro cloud to zero at this point? | 19:27 |
| ianw | oh that should be fixed -- by dropping the min-ram in the config, it makes it an exact match on the flavor name | 19:28 |
| clarkb | ah | 19:28 |
| ianw | but something is up with launching the nodes i'm not sure about yet, it seems, there's nothing running | 19:28 |
| clarkb | I see | 19:28 |
| clarkb | probably a good idea to give ed that update. Basically we're close and shutting down resources in the old cloud just have to figure out new boots in the new cloud? | 19:29 |
| ianw | i'll have to dig through the logs. i got a bit sidetracked because there's a ton of extra messages in there from a leaking node issue, that i had a fix for but now is in merge conflict with the new statemachine stuff :) | 19:29 |
| ianw | yep, i will respond, sorry haven't even opened emails yet :) | 19:29 |
| clarkb | thanks! and thank you for pushing this along | 19:29 |
| ianw | no worries, getting closer! always interesting finding the things we didn't expect | 19:30 |
| clarkb | #topic Upgrading Old Servers | 19:30 |
| clarkb | As mentioned last week I didn't have any news and didn't expect to have news today. I don't. That said I'm hopeful if I can burn through some of my backlog this week I might actually get to booting a new server before our next meeting | 19:30 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades Notes | 19:30 |
| clarkb | Please add your name next to any hosts you plan on dealing with | 19:31 |
| clarkb | to avoid stepping on toes as we work through this | 19:31 |
| clarkb | #topic Fips job updates | 19:31 |
| clarkb | ade_lee added this to the agenda after I sent it it out but we've got plenty of time so I think we can dig in | 19:31 |
| clarkb | there are two outstanding changes to make this work. The one ot zuul-jobs is the one I'm concerned about since it makes a backward incompatible change. Does anyone know if that has been communicated like fungi requested? | 19:32 |
| clarkb | https://review.opendev.org/c/zuul/zuul-jobs/+/866881 that change | 19:32 |
| clarkb | then the other question I had is if the second change can land at any time or if it needs to wait for ^ | 19:32 |
| clarkb | https://review.opendev.org/c/openstack/project-config/+/867112 | 19:32 |
| ade_lee | clarkb, it hasn't yet - but I can send it out -- I wanted to get some +2's first | 19:33 |
| ade_lee | in case there were any other changes needed | 19:33 |
| clarkb | ade_lee: gotcha, in that case I can try to review the zuul-jobs change today and then you can send email based on that I Guess | 19:33 |
| ade_lee | clarkb, as far as I know only one team is using these jobs (and that is tripleo which is setting the boolean true in any case) | 19:34 |
| clarkb | ade_lee: is the project-config chang eeffecitvely a noop until the zuul-jobs change lands? | 19:34 |
| fungi | i think the project-config change can probably land at any time since it's just supplying additional context to the zuul-jobs parent? | 19:34 |
| fungi | but i didn't review it with that possibility in mind | 19:34 |
| ade_lee | yeah I think so | 19:34 |
| clarkb | ade_lee: zuul-jobs is a bit of a global zuul library. What this means is within opendev there may be only one team but we have to be mindful of other potential users | 19:34 |
| ade_lee | I already did talk with the tripleo folks | 19:34 |
| ade_lee | ack understood | 19:35 |
| ade_lee | clarkb, if you're good with the change, I'll send it out once I get a +2 | 19:35 |
| clarkb | ok I'll try to take a look today (it won't happen immediately after the meeting as I need to review themailman change then find lunch) | 19:35 |
| clarkb | #topic Gitea Upgrades | 19:36 |
| ade_lee | cool thanks | 19:36 |
| clarkb | Another new topic since we have time. | 19:36 |
| ianw | fungi: i we don't really care if the extant parent job is passed the secret | 19:36 |
| ianw | i.e. if we merge that project config change now? | 19:36 |
| fungi | ianw: right, it doesn't currently do anything with it | 19:36 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/870850 Upgrade Gitea to 1.17.4 and its child upgrades us to 1.18.1 | 19:36 |
| clarkb | Over the holidays there were a couple of gitea updates that I meant to upgrade us to. I think the 1.17.4 upgrade is very straightforward and quite safe. The 1.18.1 upgrade form 1.17.4 is mroe involved. THere are template updates and golang is bumped to 1.19 and there are some config changes | 19:37 |
| clarkb | anyway reviews welcome and I think we can hold a 1.18.1 host for manual interaction too just to be extra sure (I haven't gotten that far yet) | 19:37 |
| clarkb | I wanted zuul to tell me if we got anything catastrophically wrong before I worried about holding stuff | 19:38 |
| ianw | ++ will review | 19:39 |
| clarkb | #topic Open Discussion | 19:39 |
| clarkb | That was all I had on the agenda | 19:39 |
| clarkb | Anything else? | 19:39 |
| fungi | i got nuthin' | 19:39 |
| ianw | apropos nothing, but i was wondering if maybe we should do something like apache redirect opendev.org / to some sort of fancier static page | 19:40 |
| clarkb | we can also have gitea serve it via the template file pretty easily | 19:40 |
| clarkb | the upside to that is it helps people find repos and orgs in gitea more easily | 19:40 |
| clarkb | I want ot say fungi has been noolding on that sort of change | 19:40 |
| ianw | i still feel like that kind of goes from 0 - 100mph on changes via gerrit and gating really quickly | 19:40 |
| ianw | "that" being the current page we have there | 19:41 |
| clarkb | ya I think the content can certainly use editing. But I think having gitea be navigable from the content is a good thing (and using the gitea templates gets us that integration for free ish) | 19:41 |
| fungi | yeah, for now the change i have up to add donor logos takes the current assets directory approach | 19:41 |
| frickler | what became of the afs disk usage topic? do we want to add it to the agenda to at least watch it regularly? | 19:42 |
| ianw | fungi: oh, i feel like i've missed that, is it just one change? | 19:42 |
| fungi | the last time i talked about shadowing the main page in apache, one of the objections was that we need to be careful any sub-paths it incorporates don't shadow git namespaces | 19:42 |
| ianw | frickler: we did purge some openeuler stuff, but it didn't make a big difference | 19:43 |
| clarkb | frickler: sure, can you add it? | 19:43 |
| clarkb | the openeuler cleanup would've been about 100GB iirc | 19:43 |
| fungi | #link https://review.opendev.org/869091 Feature our cloud donors on opendev.org | 19:43 |
| fungi | ianw: ^ | 19:43 |
| ianw | will look, thanks | 19:43 |
| fungi | ianw: and i too am interested in some reworking of the page content at the very least | 19:44 |
| frickler | oh, I missed the openeuler cleanup. with that we're at least well below 90% again | 19:44 |
| fungi | ianw: like some of the faq we have up there is probably no longer necessary now that we've been around for a while | 19:44 |
| clarkb | frickler: I do think more cleanup would be good especially to understand the growth in some of the volumes | 19:45 |
| clarkb | but openeuler cleanup was a quick bandaid since the change was already written | 19:45 |
| ianw | i think i mentioned it last week, but i feel like purging xenial nodes, and everything related to it in various mirror volumes would be a help | 19:45 |
| clarkb | ya that would make a good dent | 19:45 |
| fungi | absolutely | 19:45 |
| clarkb | but also doesn't explain why ubuntu has grown so much over the last 6 monmths | 19:46 |
| clarkb | (which may also expose somethign inefficient we are doing) | 19:46 |
| ianw | we could make the mirror scripts dump some sort of du log | 19:47 |
| ianw | although i'm not sure how long that would take as it stats everything | 19:47 |
| frickler | ok, I'll add the topic to the agenda with those ideas listed | 19:48 |
| clarkb | Sounds like that may be it for today? Thank you for your time and your help running OpenDev | 19:50 |
| clarkb | Oh! I've just remembered one other item | 19:50 |
| clarkb | the service coordinator election. I plan to send email to service-discuss making january 31 - february 14 the official time frame for nominations since no one objected | 19:51 |
| clarkb | I'll try yo get that out after lunch today | 19:51 |
| fungi | thanks! | 19:51 |
| clarkb | #endmeeting | 19:51 |
| opendevmeet | Meeting ended Tue Jan 17 19:51:38 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:51 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-01-17-19.01.html | 19:51 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-01-17-19.01.txt | 19:51 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-01-17-19.01.log.html | 19:51 |
| clarkb | and now we can have a few extra minutes for breakfast/lunch/dinner :) | 19:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!