| * fungi waves | 19:00 | |
| clarkb | hello there | 19:00 |
|---|---|---|
| fungi | ohai | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Sep 19 19:01:25 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/ZBZMOM7RXD4AXORIXJO537T3YDOJPFTW/ Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | We are deep into the OpenStack release period and OpenStack elections end tomorrow | 19:01 |
| clarkb | go vote if you haven't already | 19:02 |
| clarkb | #topic Mailman 3 | 19:03 |
| clarkb | fungi: the plan is still for starlingx and openinfra to migrate on thursday at ~15:30 UTC? | 19:03 |
| fungi | #link https://etherpad.opendev.org/p/mm3migration maintenance plan starts at line 198 | 19:03 |
| fungi | yes | 19:03 |
| fungi | tomorrow i'll prime the rsyncs and remind the relevant community managers/liaisons | 19:04 |
| fungi | but we're on track there | 19:04 |
| clarkb | fungi: probably want to update dns records at the same time tomorrwo too? | 19:04 |
| clarkb | (to reduce TTLs?) | 19:04 |
| fungi | i did the dns ttl adjustments early because one of the domains is in cloudflare | 19:04 |
| fungi | already crossed off the list | 19:04 |
| fungi | wanted to make sure i could still get to it | 19:04 |
| clarkb | aha | 19:05 |
| fungi | this time we'll approve the change earlier ahead of the window so we don't end up starting late | 19:05 |
| fungi | #link https://review.opendev.org/895205 Move OpenInfra and StarlingX lists to Mailman 3 | 19:05 |
| fungi | please review | 19:05 |
| fungi | i plan to approve it by 13:30 utc on thursday | 19:06 |
| clarkb | sounds like a plan. Any other prep work besides reviewing that change we can help with? | 19:06 |
| fungi | it can technically be approved as far ahead as we like, but easier not to need to roll that back if we end up postponing for some reason | 19:07 |
| clarkb | and avoids having the two lists of mailing lists diverge | 19:07 |
| clarkb | though at this point I suspect we'd say please wait for the migration to complete before adding a new list | 19:07 |
| fungi | no remaining prep work for this window, though assuming it goes well i'll start planning for the openstack lists (15:30-19:30 utc on thursday 2023-10-12, a week after their release) | 19:07 |
| fungi | i've already jotted some preliminary notes for that one, and plan to split the server/config management deprovisioning to a separate change and put the old server in emergency disable to avoid any accidents | 19:08 |
| fungi | you can find the section for the final maintenance at the very end of the previously mentioned pad | 19:09 |
| clarkb | sounds good, thank you for pushing this along | 19:09 |
| fungi | sure, looking forward to being done with it after what's been about a year of off-and-on effort from several of us | 19:10 |
| clarkb | #topic Server Upgrades | 19:11 |
| clarkb | nothing new here... | 19:11 |
| clarkb | #topic Nodepool image upload situation | 19:11 |
| fungi | the timeout increase merged yeah? | 19:12 |
| clarkb | yup yesterday | 19:12 |
| clarkb | our image builds look good too. The thing we lack in the dashboard is a listing of how old images are in each cloud provider | 19:12 |
| clarkb | but I think in a week we can do a nodepool image-list and check for any that are more than 7 days olver | 19:13 |
| clarkb | *more than 7 days old | 19:13 |
| fungi | in semi-related news, cloudnull is back at rackspace and possibly has leverage/mandate to assign effort for fixing some of the problems we've observed | 19:13 |
| frickler | there are 6 and 13d old ones | 19:13 |
| clarkb | frickler: the 13d images are the fedora ones | 19:14 |
| clarkb | we'll need to clean up that dashboard to remove them I think | 19:14 |
| fungi | 6d and 13d sounds perfect. like we'll be getting 0d and 7d tomorrow | 19:14 |
| clarkb | oh you mean in the cloud sorry | 19:14 |
| clarkb | all that to say early signs are this is working well enough for us again | 19:14 |
| clarkb | but lets check back in again in a week and ensure that the complete set of changes is looking good | 19:15 |
| frickler | ack | 19:15 |
| fungi | we should probably also do another leaked upload cleanup to see if we keep getting more | 19:15 |
| clarkb | fungi: good idea | 19:16 |
| frickler | yes, now would be a good time to see if the 6h timeout helps with that | 19:16 |
| fungi | i can try to find time for that later this week | 19:17 |
| clarkb | thanks. Anything else nodepool related? | 19:18 |
| fungi | nothing i'm aware of | 19:19 |
| clarkb | #topic Zuul PCRE deprecation | 19:19 |
| clarkb | #link https://etherpad.opendev.org/p/3FbfuhNmIWT33fCFK1oK Draft announcement for OpenDev users (particularly OpenStack) | 19:19 |
| clarkb | I think corvus was looking for feedback on this before sending it out. At this point I want to say we have chimed in so corvus you are probably good to send that when ready? | 19:19 |
| frickler | note I added some comments in the etherpad, so it shouldn't be sent as is | 19:20 |
| clarkb | thank you everyone for reviewing that | 19:22 |
| frickler | I also tasked kopecmartin to look at the qa projects and started doing patches for OSC myself, those are the largest batches of warnings I saw | 19:22 |
| clarkb | tripleo-ci repo had a lot of them too last I looked | 19:22 |
| frickler | or some of tha largest, yes | 19:22 |
| frickler | but tripleo was to be retired somehow? need to check the timeline for that | 19:22 |
| fungi | yeah, that's one to bring up with the tc | 19:23 |
| fungi | i wouldn't sink lots of effort digging into tripleo, we can just ask the tc how they want it handled | 19:23 |
| fungi | maybe it can be retired instead | 19:24 |
| clarkb | I think that repo supports the stable branches they've kept open but in that case they should be able to fix it | 19:24 |
| clarkb | either way we can come up with a solution | 19:24 |
| fungi | right. it's a question of whether they're keeping those open in light of the new "unmaintained" and opt-in or automatic eol resolution | 19:25 |
| clarkb | #topic Python image updates | 19:25 |
| clarkb | #link https://review.opendev.org/q/(+topic:bookworm-python3.11+OR+hashtag:bookworm+)status:open | 19:25 |
| clarkb | I think I've decided that we should avoid the Gerrit update until after the openstack release. The two drivers for this are that Irealized there are plenty of other less impactful services that we can work on in the meantime (changes for some pushed up under that link) and Gerrit made a 3.7.5 release that we should also update to at the same time | 19:26 |
| clarkb | that means our gerrit update will be a 3.7.4->3.7.5 upgrade, java 11 -> java 17 upgrade, and bullseye -> bookworm upgrade | 19:26 |
| clarkb | we can split those up or go at once but either way enough changing that I think we should avoid the openstack release | 19:27 |
| frickler | sounds reasonable to me | 19:27 |
| clarkb | In the meantime reviews on the other changes I've pushed are appreciated and I think I can approve and monitor those while we wait | 19:27 |
| clarkb | note I've asked kopecmartin to weigh in on the refstack update too | 19:28 |
| clarkb | I'm hopeful that we'll be able to drop bullseye and python3.9/3.10 image builds soon enoguh. Then we can look at adding 3.12 builds once that release occurs | 19:28 |
| clarkb | but one step at a time :) | 19:28 |
| clarkb | #topic Redeploying the InMotion/OpenMetal cloud | 19:29 |
| clarkb | I sent email to yuriy to gather some initial information on what this involves. I cc'd fungi and frickler | 19:29 |
| clarkb | Yuriy responded, but it wasn't imemdiately clear to me what a redeployment would be based on if we deployed today. Yuriy did talk about how in the new year they would be on 2023.1 or 2023.2 openstack. | 19:30 |
| clarkb | Anyway I asked for further clarification and volunteered to set up time to meet and coordinate if necessary (I think that yuriy in particular finds the more synchronous interaction beneficial) | 19:30 |
| frickler | so currently they would be able to deploy yoga iiuc | 19:30 |
| fungi | in light of that, it might make sense to delay rebuilding for a few more months | 19:30 |
| clarkb | ya I wanted more clarification on the base OS stuff before we decide | 19:31 |
| clarkb | but if the base OS doesn't move forward much by redeploying today then a delay may be worthwhile | 19:31 |
| fungi | right, that's more the deciding factor than openstack version, since we can in-place upgrade openstack | 19:32 |
| clarkb | Other items of note: we should call the new cloud openmetal not inmotion. They seem interested in continuing to help us successfully consume this hardware and service as well. Thank you openmetal for the help | 19:32 |
| frickler | but we wouldn't rename the cloud before the redeploy? other than possibly in grafana? | 19:33 |
| clarkb | frickler: I think we can if it is important to them. It is a bit of a pain to do as we have to shutdown and startup a new provider in nodepool | 19:34 |
| fungi | yeah, i would say anything that's a significant lift in the renaming department should get wrapped into the redeployment | 19:34 |
| clarkb | doable but if we can tie it into a redployment that will be easiest | 19:34 |
| fungi | we did something similar for internap->iweb though | 19:34 |
| frickler | also they want to rework the networking | 19:35 |
| frickler | seems currently we have 6 x /28, they'd provide a single /26 instead | 19:35 |
| fungi | easy things to rename we can do straight away because why not? and right if they express urgency then we can accommodate that | 19:35 |
| clarkb | frickler: yup that should hopefully simplify thigns for us | 19:35 |
| fungi | yes, their platform previously could only allocate networks in /28 prefixes | 19:36 |
| fungi | but they've apparently overcome that design limitation now | 19:36 |
| fungi | (i wonder if they're any closer to ipv6) | 19:36 |
| clarkb | It also sounded like we may need to do the self signed cert thing | 19:36 |
| clarkb | kolla supports that so hopefuilly just a matter of setting the correct cars and having kolla rerun | 19:36 |
| clarkb | overall though an encouraging first email trade. I'll try to keep the thread alive and drive it towards something we can make a plan off of | 19:37 |
| frickler | in 2023.2 kolla might support LE | 19:37 |
| fungi | thanks! | 19:37 |
| clarkb | #topic Open Discussion | 19:38 |
| clarkb | Anything else? | 19:38 |
| fungi | if we served the dns for the api endpoints in our opendev.org zone, we could probably wrangle our own le with our usual ansible | 19:38 |
| corvus | thanks for the suggestions on the email, i'll update the etherpad later | 19:39 |
| fungi | just missing the bit to inject that into kolla | 19:39 |
| corvus | just wanted to make sure we were all on board with the tone and the requested actions | 19:39 |
| fungi | yep, still lgtm | 19:39 |
| frickler | fungi: you'd just place the resulting cert file into the correct location in the kolla config | 19:39 |
| fungi | frickler: right, we're missing however we'd tie that file deployment into our ansible | 19:40 |
| fungi | i guess we'd put something in our inventory for wherever that is | 19:40 |
| clarkb | ianw had started looking into that | 19:41 |
| fungi | seems like the linaro cloud could benefit from the same | 19:41 |
| clarkb | basically a lighter weight base role application for systems where we don't want to manage firewalls and email and so on | 19:41 |
| clarkb | yup it was the linaro cloud where he was proving this out | 19:42 |
| clarkb | basically add our users and potentailly other lightweight stuff. I could see working some of the LE provisioning into that | 19:42 |
| clarkb | speaking of LE I wonder if we can unfork acme.sh yet | 19:42 |
| clarkb | Last call for any other items otherwise we can probably end a bit early today | 19:43 |
| fungi | thanks clarkb! | 19:43 |
| frickler | ack, thx all | 19:44 |
| clarkb | #endmeeting | 19:45 |
| opendevmeet | Meeting ended Tue Sep 19 19:45:06 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:45 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-09-19-19.01.html | 19:45 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-09-19-19.01.txt | 19:45 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-09-19-19.01.log.html | 19:45 |
| clarkb | thank you everyone | 19:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!