| clarkb | anyone else here for the meeting? we will begin shortly | 18:59 |
|---|---|---|
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Jul 20 19:01:09 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2021-July/000269.html Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | I didn't have any announcements | 19:01 |
| clarkb | #topic Actions from last meeting | 19:01 |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-07-13-19.01.txt minutes from last meeting | 19:02 |
| clarkb | #action someone write spec to replace Cacti with Prometheus | 19:02 |
| diablo_rojo_phone | O/ | 19:02 |
| ianw | o/ | 19:02 |
| clarkb | As far as I know that action hasn't been done. But good news is I think we're getting throug some of the stuff that had been distracting us and so that might get done soon :) | 19:02 |
| clarkb | #topic Specs Approval | 19:02 |
| clarkb | #link https://review.opendev.org/796156 Supporting communications on our very own Matrix homeserver | 19:02 |
| clarkb | Last week we said we'd put this up for approval this week. Big part of that was people weren't around last week that might want to review the change | 19:03 |
| * mordred does dance | 19:03 | |
| clarkb | fungi: ianw ^ if you can give that a read before Thursday EOD clarkb time that would be great | 19:03 |
| fungi | yeah | 19:03 |
| mordred | clarkb: you're your own timezone now | 19:03 |
| clarkb | tristanC has a change up to run a matrix gerritbot too which might be interesting to others | 19:04 |
| clarkb | #topic Topics | 19:04 |
| clarkb | #topic Gerrit Server Upgrade | 19:04 |
| clarkb | This happened. Thank you ianw for doing the bulk of the work on this one. | 19:05 |
| clarkb | We've had a few minor hiccups with things like java sql conncetor libraries and getting ssh host keys sorted last minute and making the change to reflect this in config management pass actually pass | 19:05 |
| clarkb | all told those are small issues and this went really well. | 19:06 |
| clarkb | I wanted to call out for followups we now have the ability to do some gerrit cache and memory usage tuning that we didn't really have room for before | 19:06 |
| fungi | the end result is we now have a bigger gerrit with a local db for the reviewed file checkmarks? | 19:06 |
| clarkb | Maybe we let the server run for a week or two then look at data? | 19:06 |
| clarkb | fungi: yup | 19:06 |
| clarkb | The othe rthing that ianw has mentioned is we should all check our homedirs and the gerrit2 homedir for files we want to live on the new server too | 19:07 |
| clarkb | I've already moved the ones I know I wanted to move, but still need to go look at the rest to see if any of it is worth carrying over | 19:07 |
| clarkb | #link https://etherpad.opendev.org/p/gerrit-upgrade-2021 Look here for changes to review in followup to the server move | 19:08 |
| clarkb | the etherpad also has a few changes that ianw has pushed in followup. I think I've reviewed them all, and other reviews would be nice too | 19:08 |
| clarkb | ianw: anything else to say about this? | 19:08 |
| clarkb | Do we want to keep it on the agenda for next week or can we consider this done? | 19:09 |
| ianw | nope, it seems like everything is going well. i'll wait for frickler_pto to become frickler and then i think we can retire the old server | 19:09 |
| ianw | oh and i think i have some log archives i can move across too | 19:09 |
| fungi | how is the mariadb connector fix applied to the image build? sed -i? | 19:09 |
| ianw | fungi: currently we reverted to the mysql connector, which is still compatible | 19:10 |
| clarkb | fungi: we landed it via normal gerrit changes after manually editing the config file | 19:10 |
| clarkb | we have both libs present in the image to support the old and new server db setups so we just had to change the connection line to specify the other lib | 19:10 |
| fungi | oh, got it, so all in configuration | 19:10 |
| clarkb | yup, once mariadb connector works properly we can update the config as well as update our image to remove the old mysql lib | 19:10 |
| fungi | we're not actually running with the mariadb connector patched | 19:10 |
| clarkb | correct | 19:11 |
| fungi | that makes sense, thanks | 19:11 |
| ianw | yep; no rush to get back. i was just using the mariadb connector because, well, we were connecting to mariadb :) | 19:11 |
| fungi | i wonder why they added a special one for that | 19:11 |
| ianw | just the way the subclasses work i think | 19:12 |
| ianw | basically string matching the db uri and then choosing classes | 19:12 |
| ianw | i do wonder why they added it with at least two bugs that made it not work at all, and nobody noticed, however | 19:12 |
| fungi | just seems odd that with as little as gerrit uses an rdbms for now, they'd incur the maintenance overhead of slight variations of a generic sql connector | 19:13 |
| fungi | historical baggage i suppose | 19:13 |
| fungi | anyway, not necessary to dig into now | 19:13 |
| clarkb | it may have been added back in 2.16 or prior for the actual db when that existed | 19:13 |
| clarkb | ya | 19:13 |
| clarkb | #topic Gerrit Account Cleanup | 19:14 |
| clarkb | I've been thinking about scheduling for this and the project renaming that we said we'd all do "week after the upgrade" | 19:14 |
| clarkb | In my head it makes sense to do the external id pruning Say on Monday. That is 3 weeks after I disabled the related accounts. Then do the project renames later in the week, say Friday. | 19:15 |
| fungi | that sounds reasonable to me | 19:15 |
| clarkb | The reason for this is if we need to do account surgery after the external id cleanup happens due to the cleanups we can take advantage of the outage to do it | 19:15 |
| clarkb | we did surgery on two accounts during the move for unrelated reasons and it went well but does require a downtime | 19:15 |
| clarkb | I don't expect we'll end up doing that as part of the project rename downtime, but it is nice to have the option | 19:16 |
| clarkb | Ok I'll plan to run the external id cleanup script on Monday then. | 19:16 |
| fungi | yep | 19:16 |
| clarkb | #topic Gerrit Project Renames | 19:17 |
| clarkb | As just mentioned I'm suggesting Friday July 30 for this and probably early that day. Say 15:00UTC ? | 19:17 |
| clarkb | that gives us time to do more testing if we can find time to do that testing but also the buffer to account cleanups which we can use to fix accounts if necessary | 19:17 |
| fungi | i'm around and happy to do it then | 19:18 |
| clarkb | ok lets do it | 19:18 |
| clarkb | I can send out email about that as well | 19:18 |
| clarkb | #topic Gitea01 backup issues | 19:19 |
| clarkb | I don't expect anything new has come up about this but wanted to double check | 19:19 |
| ianw | no i haven't looked into it sorry | 19:19 |
| clarkb | no worries you were busy :) | 19:20 |
| ianw | i haven't heard any more on the ipv6 issues | 19:20 |
| ianw | i could take it out of the lb and reboot it i guess | 19:20 |
| fungi | and does look like it's still going on | 19:20 |
| ianw | the switch it off and on again method probably has a high likelyhood of fixing things | 19:20 |
| clarkb | hah ok | 19:21 |
| fungi | yeah, neutron likes to get a kick in the interface every now and then | 19:21 |
| ianw | we can sync back after that happens one way or the other | 19:22 |
| clarkb | sounds good | 19:22 |
| clarkb | #topic Gitea 1.14.4 upgrade | 19:22 |
| clarkb | I started working on this when all the other gerrit and zuul work was happening and didn't want to impact those as they seemed more important | 19:22 |
| clarkb | but now that is largely behind us so I'd like to go ahead and land this upgrade when I have time to monitor it. Currently I think I have time tomorrow morning any objections to landing the change to upgrade gitea then? | 19:23 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/800274 gitea upgrade | 19:23 |
| clarkb | Its a fairly large gitea upgrade as far as the change delta goes, but I did hold a node and it looked good | 19:23 |
| clarkb | and our testing caught one issue with a changing api so that seems to be working well for us | 19:23 |
| fungi | i'm good with it, finally reviewed and checked it out today | 19:24 |
| clarkb | Sounds like no objections. If things are quiet when I start my day tomorrow I'll approve it | 19:24 |
| ianw | ++ | 19:25 |
| clarkb | #topic High AFS utilization | 19:25 |
| fungi | i'll be more around tomorrow anyway, more than today | 19:25 |
| clarkb | fungi: cool | 19:25 |
| clarkb | Yesterday it was pointed out that the opensuse mirror was sad. It had a stale lock which I removed and then manually did updates for after and is happy now | 19:25 |
| clarkb | In the process of looking into that I noticed htat our afs vicepa utilization is high | 19:26 |
| clarkb | we have less than 10% remaining disk space | 19:26 |
| clarkb | The debian volume looks like it may need to grow as well as we are right up against quota there | 19:26 |
| fungi | were we able to reclaim much with the centos cleanup? | 19:26 |
| clarkb | fungi: yup, but then we backfilled with openeuler mirroring :) | 19:26 |
| fungi | we can push to try and drop debian-stretch images | 19:26 |
| ianw | yeah, although bullseye is fully frozen now, so we can probably drop some debian soon | 19:27 |
| clarkb | ya I was going to ask about that particularly since debian is near its quota | 19:27 |
| ianw | or yeah, just push to drop it now | 19:27 |
| clarkb | I think that would be a good next step if we think it isn't too disruptive to users for some reason | 19:27 |
| clarkb | The other one that is large and I'm not sure still used is mirror.yum-puppetlabs | 19:27 |
| fungi | but yes, stretch is already "oldstable" (the release before the current stable) and is about to become "oldoldstable" | 19:28 |
| clarkb | I think tripleo and/or puppet-openstack may be the users of the yum puppetlabs mirror. I'm wondering if they continue to use it at all and if so if they only use a small subset | 19:28 |
| fungi | we should probably ask the puppet-openstack team if they're still using that, and if so whether they're using all or just a small part | 19:28 |
| clarkb | currently that mirror seems to have packages for old fedora and sles and so on | 19:29 |
| clarkb | fungi: ++ exactly | 19:29 |
| clarkb | I suspect we can trim that done one way or another. | 19:29 |
| clarkb | I can send email to openstack-discuss asking puppet-openstack and tripleo to weigh in on it | 19:29 |
| clarkb | fungi: ianw: any idea what the first step for debian trimming would be? | 19:29 |
| fungi | probably check for zuul configs referencing debian-stretch, though searching stable branches will be hard | 19:30 |
| fungi | probably better would be to start a discussion proposing we drop it, and then see if we get pushback | 19:30 |
| clarkb | fungi: that seems reasonable. Is that something you might want to send to service-discuss? | 19:31 |
| ianw | yeah, i can have a look at it | 19:31 |
| clarkb | ianw: that would be great thanks | 19:31 |
| ianw | i think both discussion, but also directly proposing changes might help | 19:31 |
| fungi | stretch has been obsolete for ~2 years since buster came out, and most projects aren't supporting their branches past that anyway | 19:31 |
| ianw | anything we can do to get dib's testing more under control is good too :) | 19:31 |
| clarkb | The last note I put related to this on the agenda is I've noticed discussion about adding Alma linux to our node list. | 19:32 |
| fungi | i can bring up dropping stretch on the openstack-discuss ml | 19:32 |
| clarkb | I think starting with clenaup is the right option but we should probably be prepared to add a third 1TB device to our vicepa setups if the linux distros keep multiplying | 19:32 |
| clarkb | There is probably another disucssion there about whether we can delete centos entirely if we do alma or similar but I think it is a bit early for that | 19:33 |
| ianw | i mean we can build images before putting the mirror in | 19:33 |
| fungi | aha, alma linux is in the same vein as rocky linux and (non-stream) centos. i knew the name sounded familiaer | 19:33 |
| clarkb | ianw: yup that is true, though I suspect ne wdistros like that may want us to offloda for them as much as possible | 19:34 |
| clarkb | fungi: yes, alma seems to be the one iwth all the momentum right now | 19:34 |
| fungi | yes, mirrors are more of a general speedup, and matter more for our more heavily-used distros | 19:34 |
| fungi | so we might just consider mirrors to be out of scope for less-used images we offer | 19:34 |
| fungi | if a particular label only sees use in a handful of builds a week, the overhead in maintaining a mirror for the software is clearly overkill | 19:35 |
| fungi | taking that into account, we could always just consider dropping some of our mirrors of significantly under-utilized stuff even if we don't drop the images | 19:36 |
| clarkb | fungi: yup, that is why I also mentioned the possibility of mv centos alma && sed -i -e s/centos/alma/ *zuul.yaml :) | 19:36 |
| ianw | (this was a bit of my concern with openeuler as it seems like that has a long path to being used) | 19:37 |
| clarkb | But all of that currnetly depends on imgaes and people being interested in the work and so | 19:37 |
| clarkb | ianw: thats a good point. I guess we can take a high level view once the afs pruning happens and we know where we stand | 19:38 |
| clarkb | then start thinking a bit more critically about the various distros and whether we need full mirrors for all of them | 19:38 |
| clarkb | Not something we need to solve today, I just saw the demand seems to be creeping up and wanted to call it out | 19:38 |
| clarkb | #topic PTG Participation | 19:39 |
| clarkb | I discovered yesterday that the deadline to sign up for PTG participation is tomorrow | 19:39 |
| clarkb | The PTG is running October 18-22, 2021. If you are curious yes this sign up deadline is earlier than in the past | 19:40 |
| fungi | it feels sudden, but i guess that's because i've been on vacation | 19:40 |
| clarkb | The office hours we did last time around seemed to work well for 2 hours in the EU friendly timezone. | 19:40 |
| clarkb | s/timezone/time block/ | 19:40 |
| clarkb | I'm happy to sign up for similar this time around as a result. | 19:40 |
| clarkb | Does anyone think we should have more time than that? | 19:41 |
| clarkb | Or perhaps think we should skip entirely? | 19:41 |
| fungi | what we did last time worked for me | 19:42 |
| clarkb | fungi: ok, I'm proposing we trim that back to a single 2 hour block during the EU afternoon NA morning since that is where we got participation last time | 19:43 |
| clarkb | ianw: ^ will you feel excluded if we do that ? | 19:43 |
| ianw | no | 19:43 |
| clarkb | alright I'll sign us up for that. If anyone wants mor etime let me know and I'll adjust to try and include that | 19:44 |
| clarkb | #topic Open Discussion | 19:44 |
| clarkb | That was all I had. Except I remember thinking of something I should bring up during the meeting earlier today and I've completely forgotten what it was | 19:45 |
| clarkb | Anything else? | 19:45 |
| fungi | i got nuthin' | 19:45 |
| ianw | nothing more from me | 19:46 |
| clarkb | Alright then, thank you everyone for your time and now you can go find $meal :) | 19:47 |
| fungi | thanks clarkb! | 19:47 |
| clarkb | As always feel free to find us in #opendev or at service-discuss@lists.opendev.org and we can discuss anything there that was missed here | 19:47 |
| clarkb | #endmeeting | 19:47 |
| opendevmeet | Meeting ended Tue Jul 20 19:47:57 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:47 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2021/infra.2021-07-20-19.01.html | 19:47 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2021/infra.2021-07-20-19.01.txt | 19:47 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2021/infra.2021-07-20-19.01.log.html | 19:47 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!