| -opendevstatus- NOTICE: all new logins to https://review.opendev.org are currently failing. investigation is ongoing, please be patient | 08:53 | |
| *** mmalchuk_ is now known as mmalchuk | 16:06 | |
| clarkb | Just about meeting time | 18:59 |
|---|---|---|
| clarkb | I'm very quickly trying to eat an apple and put together some meeting agenda notes | 18:59 |
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Jan 23 19:00:21 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/HPFGK4QDZU24FUZFA6BHEAYLQIG224WD/ Our Agenda | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/HPFGK4QDZU24FUZFA6BHEAYLQIG224WD/ Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | Service coordinator nominations open February 6, 2024 - February 20, 2024 | 19:01 |
| clarkb | I made that official in an email to the service-discuss list | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TB2OFBIGWZEYC7L4MCYA46EXIX5T47TY/ | 19:01 |
| clarkb | Happy to answer any questions people have about that | 19:01 |
| clarkb | #topic Server Upgrades | 19:03 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/905510 Upgrading meetpad service to jammy | 19:03 |
| clarkb | I think we're largely waiting on a second reviewer for this stack? tonyb testing was happy after we fixed the websockets issue? | 19:03 |
| tonyb | Yup testing was definitely happier | 19:05 |
| clarkb | cool | 19:05 |
| tonyb | There are still certifuicate issues | 19:05 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades | 19:05 |
| clarkb | ya the hsts stuff is annoying | 19:06 |
| tonyb | but I think having real LE certs will help that | 19:06 |
| clarkb | turns out there are ways to work around it like using incognito mode | 19:06 |
| clarkb | however in this case I'm not sure if it will help since jitsi willsend the headers for strict verification | 19:06 |
| tonyb | I was using ... https://paste.opendev.org/show/blYpdCY39nZbSijUoV2l/ ... which seemed to "do better" | 19:07 |
| clarkb | thats good to know | 19:08 |
| clarkb | in the etherpad I linked above there are notes for wiki replacement | 19:09 |
| clarkb | tonyb: did you have a chance to see the notes funig and I added? any concerns with what we wrote? | 19:09 |
| tonyb | I read them. I don't have any concerns | 19:11 |
| clarkb | anything else on this topic? | 19:11 |
| fungi | quick summary is that we want to make sure the openid and patrolling extensions work, the others are likely less important for now | 19:11 |
| clarkb | for the wiki if there is any confusiion with the ongoing gerrit + openid problems | 19:12 |
| fungi | overall the plan lftm | 19:12 |
| fungi | er, lgtm | 19:12 |
| tonyb | Yup. I'm fairly confident that as the first step will be to deploy the *same* git versions but containerised we'll be okay | 19:12 |
| tonyb | ... upgrades will be more work | 19:12 |
| tonyb | but really I want to decouple the OS and application | 19:12 |
| fungi | thanks! | 19:12 |
| tonyb | yw | 19:13 |
| tonyb | I think we're ready for #next_topic | 19:13 |
| clarkb | #topic Python Container Updates | 19:13 |
| clarkb | Nothing new here that I am aware of | 19:14 |
| clarkb | I think this has largely been on the far end of the priority list due to all the other stuff going on. And thats ok. We've minmimzed the the amount of images we have to build and support etc | 19:14 |
| clarkb | #topic Upgrading Zuul's DB Server | 19:14 |
| clarkb | I kept this on the agenda despite the general agree with a rough plan last week | 19:14 |
| clarkb | mostly because I seem to recall saying I would let people object for the next week | 19:15 |
| clarkb | any objections to the plan of trying to run a mysql/mariadb for zuul that we can eventually cluster later if we get to it? | 19:15 |
| fungi | also worth noting, there were several outages of that trove instance, though zuul seems to have weathered them okay | 19:15 |
| clarkb | if there are no objections now then I think we can move on and remove this topic from the agenda for next week | 19:16 |
| corvus | no objections here | 19:16 |
| fungi | the silent ayes have it | 19:16 |
| clarkb | #topic AFS Quota Issues | 19:16 |
| clarkb | I haven't made any progress on ubuntu ports trimming | 19:17 |
| clarkb | #topic Broken Wheel Builds on CentOS | 19:19 |
| clarkb | I think we have openafs packages that should work now | 19:20 |
| fungi | that's what it sounded like yesterday | 19:20 |
| clarkb | In theory I would expect that means we've got more jobs passing for this now. However, I think the final publication stuff may still have the wrong volume names? | 19:20 |
| clarkb | probably worth doing another pass on checking the job statuses | 19:22 |
| fungi | as in they're publishing with the wrong platform stub in the filenames? | 19:22 |
| clarkb | as I suspect we may have pushed the failure forward to the next broken thing | 19:22 |
| clarkb | fungi: ya the centos8 amd64 stuff afiles because it was trying to publish to a volume openafs claimed didn't exist | 19:22 |
| fungi | that was something that changed in a recent ansible, if memory serves | 19:22 |
| clarkb | I suspect we may have made a stream volume but we're still pushing to the non stream location which was cleaned up? | 19:23 |
| fungi | i think it broke for some rh-like platforms in our last ansible update | 19:23 |
| fungi | where the release var started including the minor number rather than just the major | 19:23 |
| tonyb | There was a similar change for Debian a while back | 19:23 |
| clarkb | ah ya that could be part of the problem | 19:24 |
| fungi | we switched which ansible var we use in some places, but probably missed some too | 19:24 |
| clarkb | in any case we should find time to do another pass on job statuses and failure and take it from there | 19:24 |
| clarkb | #topic OpenDev Pre PTG | 19:25 |
| clarkb | Looking at try to do two days Wednesday and Thursday sometime in February: February 7+8 or February 14+15 or February 21+22 | 19:26 |
| clarkb | Have two blocks of time each day one that works better for EU and another for APAC. Probably 14:00-16:00UTC and 22:00-00:00 UTC. | 19:26 |
| clarkb | I haven't heard any objections to any of these days or times | 19:26 |
| clarkb | I'm kinda leaning towards the 14th and 15th as that gives time to prepare but isn't so far out in the future | 19:26 |
| clarkb | I'm happy to hear feedback though. THis is me mostly trying to accomodate what I perceive to be the issues with various timezones as well as my own meeting schedule (tuesdays are really busy) | 19:27 |
| fungi | i have schedule availability for all of them, but yeah maybe sooner is better than immediately before ptg week | 19:27 |
| corvus | 14/15 slightly better for me | 19:27 |
| frickler | ptg is a full month later? | 19:27 |
| clarkb | frickler: ptg is first week of april I think /me double checks | 19:27 |
| tonyb | 14+15 works best for me | 19:28 |
| clarkb | April 8 - 12 is the PTG | 19:28 |
| fungi | oh, yeah, i guess any of them is more than a month before the ptg | 19:28 |
| frickler | anyway I'm fine with any of these dates | 19:28 |
| clarkb | fungi: ya I mostly want to get a head start on some of this stuff | 19:28 |
| fungi | absolutely | 19:28 |
| clarkb | as far as topics go I'd like to do a group brainstorm/planning/prioritization sort of thing for the various debt we've got hanging around | 19:29 |
| tonyb | I could do the "EU" timeslot but I do worry I wouldn't be a great asset | 19:29 |
| clarkb | think podman / modern docker compose, mariadb upgrades, keycloak id stuff, openmetal/inmotion cloud redeployment, prometheus, deprecated zuul configs (think stdout/stderr split in command tasks) and so on | 19:30 |
| clarkb | I've got a set of notes in my notetaking file that I need to transplant into an etherpad and then others can also add ideas as well as indicate interest on topics so that we can do our best to accomodate split scheduling | 19:30 |
| fungi | also i'd generally throw a "sustainability" discussion item in there | 19:30 |
| clarkb | ++ | 19:31 |
| clarkb | lets also say we'll do it February 14 and 15 | 19:31 |
| fungi | wouldn't hurt to do a "big picture" overview of everything we're still managing and ask ourselves what else might be on the losing side of cost vs benefit | 19:32 |
| clarkb | and then we can use the meetpad corresponding with the planning etherpad as the location | 19:32 |
| clarkb | fungi: thats a great idea. I think that sort of big picture will help with the prioritzation aspect of figuring out where to apply ourselves | 19:32 |
| tonyb | ++ | 19:33 |
| clarkb | once other things settle down I'll send emails making this all official | 19:33 |
| clarkb | and work on getting that agenda etherpad populated | 19:33 |
| fungi | thanks! | 19:33 |
| clarkb | #topic Open Discussion | 19:33 |
| clarkb | speaking of other things settling down the gerrit openid stuff has been fun | 19:34 |
| fungi | "fun" | 19:34 |
| clarkb | maybe we should call the meeting early and get back to that? Is there anything else to bring up now? | 19:34 |
| frickler | inmotion failures? | 19:34 |
| clarkb | oh ya I haven't had time to look at that | 19:35 |
| clarkb | tonyb: maybe after today's school run we can dig into that together? | 19:35 |
| frickler | the mirror host seems to be offline since sunday | 19:35 |
| clarkb | tonyb: would be a good introduction to how things are set up there because it like the linaro cloud are "different" | 19:35 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily for a restart, in order to attempt to restore OpenID login functionality | 19:35 | |
| frickler | but nodepool seems to have been unhappy for much longer | 19:35 |
| tonyb | clarkb: Sounds good to me | 19:36 |
| clarkb | frickler: in the past we've definitely seem things leak in ways that create nodepool failures that slowly get worse over time | 19:36 |
| clarkb | wouldn't surprise me if it finally got bad enough that the mirror had nowhere to run but we'll haev to check logs | 19:36 |
| fungi | i did add the inmotion mirror to the emergency disable list, in order to get the base deploy job working again. it's been failing since sunday, which coincides with the errors ironic reported | 19:36 |
| tonyb | Oh, I have a meeting 2100-2200UTC but apart from that ... | 19:37 |
| frickler | ack | 19:37 |
| clarkb | tonyb: ya I think school pickup is 2200-2300 ish | 19:37 |
| tonyb | Okay perfect | 19:37 |
| clarkb | I have to walk because I'll eb without the car | 19:37 |
| clarkb | sounds like that may be it and we need to get back to debugging gerrit things | 19:40 |
| clarkb | thank you everyone! | 19:40 |
| clarkb | #endmeeting | 19:40 |
| opendevmeet | Meeting ended Tue Jan 23 19:40:20 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:40 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2024/infra.2024-01-23-19.00.html | 19:40 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2024/infra.2024-01-23-19.00.txt | 19:40 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2024/infra.2024-01-23-19.00.log.html | 19:40 |
| -opendevstatus- NOTICE: OpenID logins for the Gerrit WebUI on review.opendev.org should be working normally again since the recent service restart | 20:01 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!