| *** diablo_rojo_phone_ is now known as diablo_rojo_phone | 16:29 | |
| *** yoctozepto_ is now known as yoctozepto | 16:29 | |
| *** diablo_rojo_phone is now known as Guest674 | 16:30 | |
| clarkb | Meeting time | 19:00 |
|---|---|---|
| clarkb | We'll get started shortly | 19:00 |
| frickler | o/ | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue May 17 19:01:14 2022 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| ianw_ | o/ | 19:01 |
| clarkb | #link https://lists.opendev.org/pipermail/service-discuss/2022-May/000336.html | 19:01 |
| clarkb | Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | A friendly reminder that teh Summit happens June 7-9. I expect some of us will be traveling for that and distracted that week and probably on either end of it as well | 19:02 |
| clarkb | Also I'll be out tomorrow | 19:03 |
| clarkb | #topic Topics | 19:03 |
| clarkb | Time to dive right in | 19:03 |
| clarkb | #topic Improving CD Throughput | 19:03 |
| clarkb | I don't have much new on this topic. But did want to call out something ianw_ noticed yesterday. If our base job fails then we don't periodicallyrun manage-projects | 19:04 |
| clarkb | We should probably monitor https://zuul.opendev.org/t/openstack/builds?project=opendev%2Fsystem-config&pipeline=periodic&skip=0 and check that it is running through its list of jobs | 19:04 |
| frickler | maybe add it as a regular topic to check that? | 19:05 |
| frickler | works fine in the qa meetings | 19:05 |
| clarkb | frickler: to the meeting? We can do that | 19:05 |
| frickler | yes | 19:05 |
| clarkb | sure, I can add that and we can give it a try | 19:06 |
| clarkb | #topic Container Maintenance | 19:06 |
| clarkb | ianw_: mentioned wanting to look at mariadb upgrades when doing the Gerrit 3.5 upgrade. | 19:07 |
| clarkb | ianw_: have you had a chance to look into that yet? Looks like you were working on holding a test node | 19:07 |
| ianw_ | not yet sorry, but yeah, working on it with that held node, and also testing the downgrade there | 19:08 |
| ianw_ | (meant to do it yesterday but got distracted by other things) | 19:08 |
| clarkb | no problem. Just wanted to check if there was new info on that as it was something I identified under the broad container maintenance header | 19:08 |
| clarkb | Will be interesting to see what we learn about it | 19:08 |
| clarkb | #topic Support for Jammy Jellyfish | 19:09 |
| clarkb | we're still waiting on wheel mirrors for Jammy before we consider this done. | 19:09 |
| clarkb | I did want to mention that I think the openafs cleanups that jammy triggered are largely compelte with exception of xenial removal | 19:09 |
| clarkb | however, even without xenial removal we have freed up significant disk space and managed to reset quotas to better reflect current volume consumption | 19:10 |
| ianw_ | i can get onto that soon, but what was the latest on reprepro? it's sort-of-working-but-with-errors-that-we-can-seem-to-ignore-for-now? | 19:10 |
| frickler | yes | 19:10 |
| clarkb | ianw_: for the PPA stuff do we have jammy openafs packages built now and managed by changes to the new repos? | 19:11 |
| *** ianw_ is now known as ianw | 19:11 | |
| ianw | yes we do, there is a test ... | 19:12 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/841525 | 19:12 |
| ianw | be good to get that in | 19:12 |
| clarkb | ++ I'll review that | 19:12 |
| clarkb | Sounds like we mostly just need to push changes to add wheel mirroring and review them etc. No major issues at this point? | 19:13 |
| clarkb | new afs volumes will need to be created too. Note I set all the wheel volume quotas to 20GB today | 19:13 |
| clarkb | Are there any other Jammy spin up concerns at this point? | 19:14 |
| ianw | i still need to fully look into the phased deb bits from the dib side, on the todo | 19:15 |
| clarkb | Sounds like that may be it | 19:18 |
| clarkb | #topic Gerrit 3.5 Upgrade | 19:18 |
| ianw | (oh and tangentially to jammy openafs, i also fixed an issue with the new bionic openafs packages that were built relying on the backports repo being enabled. i've disbled that in the ppa and rebuilt them) | 19:18 |
| ianw | thanks to frickler for noticing that | 19:18 |
| fungi | that was a great find | 19:20 |
| ianw | i put the gerrit 3.5 upgrade on the agenda | 19:21 |
| clarkb | ya sorry being distracted | 19:21 |
| ianw | i have started a checklist at | 19:21 |
| ianw | #link https://etherpad.opendev.org/p/gerrit-upgrade-3.5 | 19:21 |
| clarkb | I left some notes on there which you seem to have noticed | 19:22 |
| clarkb | the main thing that jumped out to me is enabling the conflicts checking. The mergeability checking showed us that people do rely on and use those features so we shoudl toggle them on before upgrading | 19:23 |
| ianw | yeah, i can send changes to turn that on explicitly | 19:23 |
| clarkb | ++ thanks | 19:24 |
| ianw | the other one was that there's a new ssh "set-account" ability to remove external id's | 19:24 |
| ianw | might make it slightly easier for duplicate accounts | 19:24 |
| clarkb | ya I think the migration automatically adds that perm to admins? But then we can possibly take advantage of it to clean up the user db further | 19:25 |
| clarkb | learning what we can about the downgrade and possibility of a mariadb upgrade at the same time are the other major things. Basically we seem to have the known things under control and now its a matter of investigating the unknowns a bit | 19:26 |
| ianw | and out of interest i did dump the username: id's to see which ones conflicted in case-insensitive; that's an interesting list | 19:26 |
| clarkb | yup, a few of them are people getting a second account and needing to set a non conflicting usename | 19:26 |
| fungi | i ran into one for a third-party ci system last week | 19:26 |
| clarkb | I noticed patterns like ^ when doing the db cleanups | 19:26 |
| ianw | assuming i get those other bits marked off, do we want to think about a date? | 19:27 |
| clarkb | Yes, I made a note on the agenda to talk about scheduling | 19:28 |
| clarkb | https://releases.openstack.org/zed/schedule.html is the openstack release schedule which is a big thing to consider when planning this | 19:28 |
| clarkb | that makes note of the summit too which is nice | 19:28 |
| ianw | the options seem to be pre summit or post -- pre meaning we have it done and deployed for that, post being i guess slightly safer | 19:29 |
| clarkb | personally I feel like I have enough distractions pre summit to make doing it between now and then difficult. But maybe soon after the summit? That gives us plenty of time to announce it too | 19:29 |
| clarkb | With previous releases I've done enough java'ing that I'd worry about doing an upgrade and not having some time to do that if necessary again | 19:30 |
| ianw | fair enough, i do like the idea that we go into the summit with it fully updated just as a point about the way we deploy, but i see the argument | 19:31 |
| fungi | sounds good to me | 19:31 |
| fungi | after i mean. just worried about summit prep and maintenance | 19:31 |
| fungi | not that i'm super against before | 19:31 |
| clarkb | ya I think if I didn't have a bunch of commitments for the summit I need to prepare for and then attend while there I'd be ok before | 19:31 |
| clarkb | but I do have those things and I need to be sure I get around to them :) | 19:32 |
| clarkb | The week after a summit tends to be pretty quiet too (not sure if that will hold during a pandemic) so could be a good time for it from a user perspective | 19:32 |
| ianw | something like the 12th? | 19:32 |
| clarkb | I return the 11th. The 12th would work for me assuming I can power through jetlag | 19:33 |
| clarkb | as far as conflicts go thats a fairly easy one to deal with. | 19:34 |
| clarkb | fungi: frickler ^ thoughts? | 19:34 |
| frickler | I won't summit so I'm pretty flexible about it | 19:34 |
| fungi | i'm not going to be around i don't think... checking | 19:34 |
| ianw | hrm 13th june is a public holiday here actually, so i am likely as not to be afk from that weekend | 19:35 |
| fungi | yeah, i'll be on my way home from boston on the 12th | 19:35 |
| clarkb | what about the 19th then? gives us a week to recover from the event then do the upgrade | 19:35 |
| fungi | i can absolutely do the 19th | 19:35 |
| clarkb | or depending on location enjoy a public holiday :) | 19:35 |
| clarkb | and then we can also give everyone a months notice if we announce it in the next day or two | 19:36 |
| ianw | ++ i'll be happy to drive and do it very early on my monday, and hopefully !australian people can just be standyby in case of issues | 19:37 |
| clarkb | sounds like a plan | 19:37 |
| clarkb | and to be clear thats australian 20th, but late 19th UTC ? | 19:38 |
| clarkb | ianw: and is that something you want to send email for or should I work on that bit? | 19:38 |
| ianw | say 20:00 UTC if that's ok. that's 6am which is fine | 19:38 |
| clarkb | wfm | 19:39 |
| ianw | i can send the announcement today | 19:39 |
| clarkb | June 19 starting at 20:00 UTC. Shouldn't take more than a few minutes so blocking out an hour is probably plenty? | 19:39 |
| clarkb | (I really do appreciate that gerrit upgrades are quick now) | 19:39 |
| ianw | ++ (famous last words, haha) | 19:39 |
| clarkb | Anything else on the subject of Gerrit upgrades? | 19:40 |
| ianw | but yeah, we really have a ton of ci/cd that gives us great confidence! | 19:41 |
| clarkb | ++ | 19:41 |
| ianw | nope, sounds good, thanks! | 19:41 |
| fungi | wfm! | 19:41 |
| clarkb | #topic Open Discussion | 19:41 |
| clarkb | Just to reiterate I do plan to get a new annual cert for wiki.o.o next week (one of my pre summit todos that interferes with pre summit gerrit upgrades) | 19:41 |
| clarkb | frickler: ^ fyi since you called that out | 19:42 |
| corvus | i have thing | 19:42 |
| clarkb | corvus: go for it | 19:42 |
| corvus | i'm a little behind, but shortly i'm going to send an email reminder to zuul-discuss about the deprecation of the 'queue' attribute on project-pipeline stanzas | 19:43 |
| corvus | we (opendev) should run this script to find instances of it: https://opendev.org/zuul/zuul/src/branch/master/tools/deprecated-queues.py | 19:43 |
| corvus | or at least our tenants should :) | 19:44 |
| corvus | so my question is, how would we like to handle that? | 19:44 |
| clarkb | I don't mind running it and sending email to our various users. I have done similar in the past for zuul config errors | 19:44 |
| corvus | cool, should be easy to run, and you can do a whole tenant at a time | 19:44 |
| corvus | istr there were many instances in the openstack tenant | 19:45 |
| fungi | and we have only a handful of tenants | 19:45 |
| corvus | there's like 1 or 2 in zuul; i'll take care of that | 19:45 |
| clarkb | corvus: any ballpark on when it will become an error to have in your config? | 19:45 |
| corvus | clarkb: v7 i think; but no idea when that is. probably > 1 mo. | 19:45 |
| clarkb | ok that works. Mostly so that I can warn them with info on when it will convert to an error in the emails | 19:46 |
| corvus | (should have been v6, but i think v6 was too fast/urgent for that) | 19:46 |
| corvus | note that "become an error" in the case of a project-pipeline definition is sort of equivalent to "stop running many/all jobs" | 19:46 |
| clarkb | ah right | 19:46 |
| clarkb | I'll also call this out in the TC meeting thursday (that is likely to be before I send email about it) so that they are aware | 19:47 |
| clarkb | and ya I spot checked with codesearch last week and there are a number with those configs in place | 19:47 |
| clarkb | I'll likely get to running it and sending emails on thursady just not before the first thing in the mroning meeting | 19:48 |
| corvus | that's all from me | 19:48 |
| clarkb | I'll give ti a couple more minutes for any other topics and if not we can end 10 minutes early | 19:49 |
| clarkb | Seems like that was it. Thank you everyone! | 19:51 |
| clarkb | As always feel free to reach out in #opendev or on the mailing list if something comes up before the next meeting | 19:51 |
| clarkb | #endmeeting | 19:51 |
| opendevmeet | Meeting ended Tue May 17 19:51:24 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:51 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2022/infra.2022-05-17-19.01.html | 19:51 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2022/infra.2022-05-17-19.01.txt | 19:51 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2022/infra.2022-05-17-19.01.log.html | 19:51 |
| fungi | thanks clarkb! | 19:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!