| clarkb | Hello, it is weekly meeting time | 19:00 |
|---|---|---|
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Jan 18 19:01:08 2022 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | I expect this meeting to be another quicker one as we're all still sort of rolling into the new year | 19:01 |
| ianw | o/ | 19:01 |
| clarkb | #link https://lists.opendev.org/pipermail/service-discuss/2022-January/000314.html Our Agenda | 19:01 |
| clarkb | But we do have an agenda to run trhough | 19:01 |
| clarkb | ianw: good morning! | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | Service coordinator nominations run January 25, 2022 - February 8, 2022 (that starts next week) | 19:01 |
| clarkb | Another reminder that service coordinator nominations will begin soon | 19:02 |
| clarkb | OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/ | 19:02 |
| clarkb | the OpenInfra Summit CFP is accepting papers and if you'd like to help program the event you can nominate yourself to be on the programming committee | 19:02 |
| clarkb | there are a number of tracks including a CI/CD track that we might find interesting | 19:02 |
| clarkb | Deadlines for both are fast approaching (I think early february for CFP and end of the week ish for programming committee?0 | 19:04 |
| clarkb | #topic Actions from last meeting | 19:04 |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-01-11-19.01.txt minutes from last meeting | 19:04 |
| clarkb | Last meeting was pretty easy going. No actions recorded | 19:05 |
| clarkb | #topic Topics | 19:05 |
| clarkb | #topic Improving OpenDev's CD throughput | 19:05 |
| clarkb | I have on my todo list an item that says I should review this spec but I haven't yet | 19:06 |
| clarkb | #link https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets | 19:06 |
| clarkb | infra-root ^ would be good to look at that in the near future. I'll try to prioritize it myself once I've flushed a bit more of the backlog (getting really close) | 19:06 |
| clarkb | ianw: anything else to add on this topic? I suspect you are in backlog catch up mode too and this may not be near the top | 19:07 |
| ianw | yeah nothing more on this one yet | 19:07 |
| clarkb | #topic Container maintenance | 19:08 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-container-maintenance | 19:08 |
| clarkb | Good news: all buster based images should be updated to bullseye now. Thank you to everyone who helped me push that along | 19:09 |
| fungi | contrats! | 19:09 |
| clarkb | I've got a change that came out of that to stop running a limnoria fork at https://review.opendev.org/c/opendev/system-config/+/821331 which I updated today to consume from a tag instead of tip of master | 19:09 |
| fungi | and thanks for all the work on it | 19:09 |
| clarkb | I think that might want to be landed on a friday when there are fewer meetings | 19:09 |
| ianw | ++ thanks for pushing that through! | 19:10 |
| clarkb | jentoio I'll be reaching out sometime soon to start looking at the dedicated user work with our containers as that is the next thing on the container todo list | 19:11 |
| jentoio | sounds good, thanks | 19:12 |
| fungi | i'd love to find the time to work on deploying the storyboard containers we're already building | 19:12 |
| clarkb | There are also a few other large todos on that etherpad including zookeeper upgrades, mariadb upgrades, and intermediate registry pruning if anyone gets bored and wants to take a look :) | 19:12 |
| fungi | also we'll have container deployment work involved in the upcoming mailman3 effort | 19:13 |
| clarkb | fungi: yup. Re storyboard converting it to a newer mariadb may be a good idea as a first step on the mariadb upgrades planning | 19:13 |
| fungi | oh, in the dockerfile? | 19:14 |
| clarkb | I think the zk upgrade may be as simple as shifting the version ahead and letting ansible do its thing as newer zk is supposedly compat with older | 19:14 |
| clarkb | fungi: in the docker compose file | 19:14 |
| fungi | er, that's what i meant, of course ;) | 19:14 |
| clarkb | for zk we should probably test that assumption a bit before going ahead with it though | 19:14 |
| clarkb | Anyway that was all I had. Happy to help others that dig into this and I plan to pull tasks off the list as I'm able myself | 19:15 |
| clarkb | #topic Nodepool image cleanups | 19:15 |
| clarkb | Tumbleweed is gone. Haven't heard any complains | 19:15 |
| clarkb | CentOS 8 removal has begun. I pushed up a bunch of changes to infra related repos as well as openstack requirements and zuul-jobs to remove the use of centos8 | 19:15 |
| clarkb | We should no longer be updating the wheel mirror for centos 8 as well | 19:16 |
| clarkb | I sent email to service-discuss and openstack-discuss warning people and some specific projects of our plan to remove centos 8 | 19:16 |
| clarkb | Those emails indicated we would remove the label from nodepool/zuul entirely around the end of the month | 19:17 |
| clarkb | Overall it seems people have been receptive and are converting jobs to centos 8 stream or removing centos entirely. Which is basically what we were looking for so yay | 19:17 |
| clarkb | However, `ping` is currently broken for non root users on centos 8 stream. Something to be aware of if conversion happen and don't work. That distro needs to revert its most recent iputils package update or update systemd to systemd-239-55 to correct this | 19:18 |
| clarkb | A couple of projects like devstack and tripleo have worked around this by adding sysctl to mimic the system-239-55 update | 19:18 |
| clarkb | ianw: for fedora 34 frickler indicated that neutron is using that platform for testing. Do you know what is required to move them ahead to fedora 35? | 19:19 |
| clarkb | fedora 34 is the other distro in nodepool that would be good to remove in the near future as that one doesn't boot reliably | 19:20 |
| fungi | yeah, i think we have it booting successfully in only two providers? | 19:20 |
| ianw | i think we need to go through a few bootloader things and cut a dib release | 19:20 |
| clarkb | ianw: to remove fedora 34 or to fix it? | 19:21 |
| ianw | sorry, to fully fix fedora 35 booting and keep 9-stream happy, etc. | 19:22 |
| ianw | just context switching back in, sorry | 19:23 |
| clarkb | Gotcha general rhel system improvements then. But no reason to keep fedora 34 around if we can sort out neutron? and even then fedora has a short shelf life so neutron may need to accept it is going away soon either way | 19:23 |
| ianw | so f35 wasn't booting, https://review.opendev.org/c/openstack/diskimage-builder/+/818851 fixed that and i think is released now | 19:23 |
| ianw | no, it's just merged | 19:24 |
| ecsantos[m] | Oh, sorry to butt in, just a quick note about openSUSE. SUSE dropped OpenStack in 2019: https://www.zdnet.com/article/suse-drops-openstacks/ | 19:25 |
| ecsantos[m] | That's why the mirrors were broken, apparently. I have a change semi-ready to address it | 19:25 |
| ianw | it caused a regression in 9-stream "centos" element -- fixed with https://review.opendev.org/c/openstack/diskimage-builder/+/821772 | 19:25 |
| ianw | there's a couple of other changes in the queue related now | 19:26 |
| ianw | so i think we go through those and try to find a coherent point to make a 3.17.0 release. that should free up the fedora work | 19:27 |
| clarkb | ianw: sounds good, thanks | 19:27 |
| clarkb | #topic Scheduling Gerrit Project Renames | 19:27 |
| clarkb | last week we penciled in January 24 for this as that seemed to fit the openstack release cycle well and shouldn't interfere with zuul much either | 19:28 |
| clarkb | fungi: I think you were thinking later in the day UTC time. Like 22:00 UTC? | 19:28 |
| fungi | i plan to be on hand all day, so am happy to help as long as it's while i'm still awake | 19:28 |
| fungi | very early like in the 00:00-03:00 UTC range would also work for me | 19:29 |
| ianw | should we perhaps schedule the 3.4 upgrade at the same time? | 19:29 |
| ianw | (i need to finish the donwgrade/revert section, but i think it's good to go) | 19:30 |
| clarkb | I'm happy to try the 3.4 upgrade around then too. But is ~6 days enough advancewarning for a gerrit upgrade? | 19:30 |
| clarkb | I guess the delta here is much smaller. The biggest gap is probably zuul working with gerrit | 19:30 |
| fungi | it seems tight, but i'm okay with the idea | 19:30 |
| clarkb | If we do it that way I think we do the renames first | 19:30 |
| clarkb | that way if we rever the 3.4 upgrade we'll have completed the renames | 19:30 |
| ianw | ++ | 19:31 |
| fungi | also we should plan enough time in the window to test and roll back | 19:31 |
| clarkb | fungi: ++ | 19:31 |
| clarkb | ianw: when is a good time to start all that for you? | 19:31 |
| clarkb | is 22:00 UTC too early? | 19:31 |
| ianw | no, that is fine and probably is better for EOD in US time | 19:32 |
| ianw | (than earlier) | 19:33 |
| clarkb | ok I can put together an email today announcing that we'll plan to rename projects starting at 22:00 UTC January 24 as well as upgrade gerrit to 3.4 with an expected end time around 00:00 UTC (I think 2 hours should be enough, an hour for each action) | 19:33 |
| ianw | #link https://etherpad.opendev.org/p/gerrit-upgrade-3.4 | 19:34 |
| clarkb | infra-root ^ does that proposed plan seem reasonable? | 19:34 |
| fungi | wfm | 19:34 |
| ianw | sounds excellent, thanks | 19:34 |
| clarkb | alright I think we can go to the next topic then | 19:34 |
| clarkb | #topic Spring Cleaning for Old Reviews | 19:34 |
| clarkb | frickler: put this on the agenda and poinst out that repos like system-config have >300 open reviews most of which are more than a year old and have merge conflicts | 19:35 |
| clarkb | should we (auto) abandon these? | 19:35 |
| clarkb | My initial thought is that maybe we should all try to spend an hour going through the backlogs and manually prune but that is a lot of effort. Maybe we can do a better breakdown and see how many are puppet era and abandon those at least (since we aren't really puppeting anymore) | 19:36 |
| clarkb | things from a more modern ansible+docker era are likely worth double checking for relevance before abandoning | 19:37 |
| ianw | i'm not opposed | 19:38 |
| ianw | i also see old things as "ideas that may be kind of valid but never quite floated to the top at the time they were proposed, but may one day" and it doesn't bother me so much having them around | 19:39 |
| clarkb | off the top of my head anything >3 years seems very safe to abandon. Anything >2 is probably fine too. But I see at least one change in my backlog from september 2020 that is probably worth another look at | 19:39 |
| clarkb | ianw: yup that is my hesitation whcih is why I'm ok with clearing stuff that we are reasonably sure is related to the old puppet way of doing things | 19:40 |
| clarkb | since they are very unlikely to apply any longer | 19:40 |
| clarkb | I guess think on this a bit and we can propose a plan next week? I do think that having cleaner gerrit dashboards would be nice for getting myself organized | 19:41 |
| ianw | agree | 19:41 |
| clarkb | #topic Open Discussion | 19:42 |
| fungi | i guess if i see a notification from gerrit about anything i've proposed getting abandoned and i think it's still worthwhile, i'll revisit and restore it | 19:42 |
| clarkb | fungi: that seems reasonable too. We can always restore later | 19:42 |
| clarkb | My Gerrit 3.3.9 update change just landed. ianw re your autohold may be worth rerunning anyway to ensure you're on the latest images for the revert testing | 19:43 |
| clarkb | ianw: I do wonder if the autohold hit a scale otu scheduler bug. Might need to go look at zuul debug logs | 19:43 |
| clarkb | But I bring this up because I'd like to do a much smaller gerrit restart today. Probably after the kids get back from school this afternoon and things are quiet | 19:43 |
| fungi | also i'm getting ready to turn the jitsi-meet servers back on once the meeting wraps (new stable images without any log4j in them were finally tagged a few hours ago) | 19:44 |
| clarkb | \o/ | 19:44 |
| clarkb | I'll give this a few more minutes before calling it a meeting. | 19:44 |
| jentoio | a comment on opendev docs, specifically opendev/system-confg | 19:45 |
| clarkb | jentoio: go for it | 19:45 |
| jentoio | as I read them I'm finging lots of outdated info. For example, major systems, logstash | 19:46 |
| jentoio | I guess its fine to update them as I find outdated info, is my question | 19:46 |
| fungi | for logstash, probably better to not spend time updating those docs | 19:46 |
| jentoio | is it worth working on | 19:47 |
| clarkb | jentoio: yup pushing changes to update out of date info is greatly appreciated. Even if you just push a note warning that indicates the info is out of date | 19:47 |
| fungi | we've got an active effort underway which should result in us removing all the logstash docs anyway | 19:47 |
| clarkb | logstash is a special case beacuse we are in the process of removing it entirely. But in general pointing things out with a warning blob or updating out of date info to be correct would be great | 19:47 |
| fungi | though sure, a warning banner on the logstash docs can't hurt | 19:47 |
| jentoio | lots of lionks with http not http(s) | 19:47 |
| fungi | we've added https to a lot of our services/sites recently, so yes any of them that work via https would be great to switch in hyperlinks | 19:48 |
| jentoio | okay, i'm hearing that pushing updates to updated docs is fine | 19:48 |
| clarkb | ++ | 19:48 |
| jentoio | except logstash | 19:48 |
| fungi | ianw's design for broad automation of lets encrypt has allowed us to be far less choosy about what gets a cert | 19:48 |
| clarkb | Sounds like that may be all. Thank you everyone. I'll give you about 10 minutes back for $meal :) | 19:50 |
| clarkb | I myself need some lunch | 19:50 |
| clarkb | #endmeeting | 19:51 |
| opendevmeet | Meeting ended Tue Jan 18 19:51:02 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:51 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2022/infra.2022-01-18-19.01.html | 19:51 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2022/infra.2022-01-18-19.01.txt | 19:51 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2022/infra.2022-01-18-19.01.log.html | 19:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!