Tuesday, 2022-04-12

clarkbJust about meeting time. We'll get started in a ocuple of minutes18:59
ianwo/19:00
clarkb#startmeeting infra19:01
opendevmeetMeeting started Tue Apr 12 19:01:09 2022 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
opendevmeetThe meeting name has been set to 'infra'19:01
clarkb#link https://lists.opendev.org/pipermail/service-discuss/2022-April/000331.html Our Agenda19:01
clarkb#topic Announcements19:01
clarkbNo announcements. We've got the PTG behind us and a couple of months to the summit. I don't think there are any other major events that are likely to impact us until the summit19:02
clarkb#topic Actions from last meeting19:03
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-04-05-19.01.txt minutes from last meeting19:03
clarkbno actions recorded last meeting. I did think about dropping this part of hte meeting but then ianw took an action (now complete) so I didn't at the time. I think we can keep it for now19:03
clarkb#topic Topics19:03
clarkbLet's dive right in19:03
clarkb#topic Improving OpenDev CD Throughput19:03
clarkbThe zuul changes that remove the patched ansible stuff have begun to arrive in gerrit. I have not yet had a chance to review them but we should probably try to stay on top of that19:04
clarkbAs far as meeting goes to discuss our use of Zuul for CD and the potential impacts ^ has I can do tomorrow around 2000 or 2100 UTC  but then the rest of the week is looking less good for me. Could do Monday next week? I know frickler is on vacation already and not sure if that bumps into ianw's holidays?19:05
fungiit's probably not urgent if we want to push it off a bit19:06
ianwumm that is easter weekend here, unlikely to be around19:06
clarkbya I know. Not sure when you'll be back :)19:06
fungi(i mean, like, push it to closer to the end of april)19:07
clarkbfungi: ah ya that works for me too19:07
clarkbya ok why don't we worry about finding time after the holiday then19:08
ianw++19:08
clarkband I will try to review those changes and keep on top of the incremental updates that I expect opendev to start seeing in the near future (since we deploy from ~master)19:08
fungifrom my perspective, the ansible restrictions in zuul aren't really adding any effective security layer, so when they end up in our deployment doesn't have a lot of bearing on when or if we make changes to our cd approach19:09
clarkbgood point19:10
fungithe ansible restrictions were always wishful thinking from a security perspective19:11
fungithem going away doesn't change that19:11
clarkbya I think where we ended up is that the change prompts a discussion but not necessarily an urgent reworking of anything19:12
clarkbgood to make sure we're still comfortable with things periodically19:12
clarkbSeems like that may be it for this topic. We can pick this up again afte rthe holiday (which ya'll should enjoy)19:13
clarkb#topic Container Maintenance19:13
clarkbDon't see any changes to review on this yet. jentoio feel free to reach out with questions. Though I totally understand if you are busy19:13
jentoioworking on grafana should have something to review soon19:14
clarkbthat is great to hear. THank you!19:14
clarkb#topic Spring Cleaning Reviews19:14
clarkb#link https://review.opendev.org/q/project:opendev/system-config+status:open+topic:system-config-cleanup Resurrected changes19:14
clarkbThese changes have been getting reviews. Thank you everyone19:15
clarkbianw dug into some of the gerrit config related changes and pointed out that we probably don't need to incrase our open file ulimits. I suspect that our more aggressive git packing is to blame for this19:15
clarkbShould we go ahead and abandon https://review.opendev.org/c/opendev/system-config/+/445616 given that?19:16
ianwi think so, i think if we start to see it at this point we should debug again, as it seems like a lot has changed since then19:17
clarkb++19:17
clarkbI'll go ahead and abandon it after the meetingthen19:17
clarkbAnd the other thing ianw noticed was that while we do have git diff timeouts and there is an old change to increase that timeout it seems that only stackalytics generates those errors. So we may be able to work with stackalytics and avoid landing a change like this too19:17
ianw++ i will send an email today about that to relevant parties19:18
clarkbhttps://review.opendev.org/c/opendev/system-config/+/529426 https://review.opendev.org/c/opendev/system-config/+/689858 are a couple of super easy ones if you have time fungi19:19
clarkbthere are others that could use review too but ar elikely a bit more involved19:19
fungisure!19:19
clarkbthanks again everyone for whittling away at this. We aren't down to the ~10 or so changes that is probably ideal, but it is progress :)19:19
ianwthanks for driving it too!19:20
clarkb#topic Scheduling Project Renames19:20
clarkbso just today my friday (and a bit of thursday) schedule has changed due to kids being out of school and wife needing to do things. This means I'm definitely not going to be able to drive this but can possibly be a second set of hands on friday still19:21
clarkbI'm happy to reschedule, but also sounded like fungi didn't htink it was a problem for him to drive it and I'll do my best to support19:21
fungiyeah, i can take care of it. we've still only got the one rename to do, right?19:22
clarkbyup I've only seen the one19:22
clarkbOk sounds like we're still on track then. Definitely say something if we need to reschedule and I can send email19:23
fungisure thing19:23
clarkb#topic Time for a git-review release19:23
clarkbI think we're || that close to being ready for a git-review release? And in particular you were asking for clarifation on whether or not we are comfortable dropping support for older python3 and I guess older git ~1.8 now?19:24
clarkbWhere I've landed on that is I think it is ok for older platforms to use older git-review. They can also manually git push gerrit HEAD:refs/for/$branch as well (which is how I work with upstream gerrit)19:24
fungiyeah, the python 3.5 support removal merged independently of the change which needed it19:24
clarkbPut another way I'm happy to hav ea release note that says python3.6 or newer is necessary as well as anothe release note saying git 2.10.x or whatever the versions is is required19:25
fungiso i've pushed up a revert in case we decide to defer or adjust the other change resulting in us not needing to drop 3.5 after all19:25
clarkbianw: ^ you indicated a concern for centos 7 and its git version (~1.8) do you know of active users on that platform that wouldn't be able to pin git-review? Or was that just a general concern?19:25
fungiyeah, the other change still under discussion includes such a release note about git minimum version19:26
ianwit was just a general concern about a platform with 3.6 and a lower git ootb19:26
clarkbGot it, I think if the relase notes (and maybe the manpage/docs) indicate a minimum git version we're ok there19:27
ianwi did propose https://review.opendev.org/c/opendev/git-review/+/837451 to enforce the minimum version with a sane message19:27
clarkbah ok that works too19:27
fungito be clear, the only reason i pushed the python 3.5 removal change up was because the otehr change needed newer git than we have on xenial, so if we went forward there it would force us to drop the xenial job, removing our python 3.5 testing19:27
clarkbfungi: yup and I think we're trying to remove xenial testing anyway so this is all a step in the right direction19:27
fungiif we're going to the trouble of testing whether there's git 2.10 present, we could simply use that to determine whether to add that command-line flag anyway19:28
clarkbdoesn't sound like anyone is really opposed to the is direction we just need to review the changes and land them?19:28
ianwyeah, i think we have to move on.  centos 7 is right on the edge of things i'd expect maybe someone still uses19:28
clarkbfungi: ya but then we aren't testing really old git either...19:28
fungibut if the goal is to support centos 7, we need jobs which test it19:28
fungiright19:29
clarkbthats the main reason why I'm ok with moving deps up and telling users to pin on older platforms19:29
ianwthat's fair enough.  i would just like it to fail with "your git is too old" rather than raise an exception in the middle with the system() call passing invalid flags, though19:29
clarkbianw: yup that wfm19:29
ianwwe already do a git version check anyway, so it's mostly just pulling it out into common code19:30
fungiokay, so seems like we have consensus to merge the signed commits workaround, require git>=2.10, drop python 3.5 support, and tag a 2.3.0 (or do people think dropping a python version means we need to make it 3.0.0?)19:30
clarkbfungi: 2.3.0 is probably fine here19:30
fungigiven that, i'll drop the revert, and we can finish up merging the rest19:31
ianwyeah, i could go either way19:31
clarkbthis is a command line tool not a library where these changes might have transitive affects19:31
clarkbif it were a lib 3.0 would probably be appropriate19:31
ianwor you can take the linux approach of "this feels like a 3-type-thing" :)19:31
ianwnot sure git-review will ever have a "big" change, it seems like it does what it does 19:32
fungiwell, we made 2.0.0 when we dropped python 2.7 support19:32
clarkbin general I'm in favor of ensuring these tools are easier to use for active up to date users. And I think simplifying testing and bumping those minimums helps us do that without a bunch of effort19:32
ianw++19:33
fungiyeah, totally19:33
clarkbalright sounds like we have a plan then. I'll look to review the followup with helpful error message shortly19:34
clarkb#topic Time for a Bindep release19:35
clarkbBindep has similar received some updates from modern users. In this case a change adding Alma Linux support19:35
clarkbbut before that can land we need a change to address our test suite's trouble with the latest distro 1.7.0 release19:35
clarkbI've just realized we should add a release note for the ALma linux support. I'll get that pushed up after the meeting too19:36
clarkbthen if we can land the changes (fedora mirror trouble currently hitting us) I think we can push a 2.11.0 release?19:36
clarkbdoes that sound reasonable?19:37
clarkboh wait there is a release note in the change already. Even better :)19:37
fungiyeah19:38
fungilooks like we haven't merged any changes since 2.10.2 so these would be the only ones to consider for the release19:38
ianwseems ok to me.  this seems the second time the mirror has been out of sync, i'll try having a look, i know it's been discussed today extensively19:39
fungiand i agree, alma linux support and changing a dep minimum count for a minor rev19:39
clarkbcool I'll try to stay on top of that and ensure it gets done19:39
fungiianw: looks like we pulled a lot of changes from it around 05-06z today and then updates after that were empty19:39
fungiso i think the mirror we're pulling from is stuck mid-update19:40
ianw:/19:41
clarkbya if we can find a more up to date upstream we could swap them out19:41
fungithough there might be a lengthy resync due to slight differences on the new mirror19:42
fungi(variations in timestamp resolution, perms, et cetera)19:43
clarkbright19:44
clarkbAnything else on this topic?19:44
funginot from me19:45
clarkb#topic Removing our ELK systems19:45
clarkbThis is a topic that I'm ninja adding to the agenda19:45
clarkbI think dpawlik's efforts to use opensearch for openstack's needs are ocming along well and recently fungi had to restart the logstash geard anyway since it had a massive queue and had fallen over19:46
clarkbadditionally we told openstack we could keep things running until theend of yoga which just happened19:46
clarkbAll that to say any objections to start unwinding these services and return resources back to the cloud provider?19:46
clarkbI think the first step is to remove the gearman log job submissions from our base jobs19:46
fungiagreed19:46
clarkbthen if no panic occurs we can shutdown the gearman works, the elasticsearches, and the kibana etc19:47
fungii was tempted to do that during the release since it was clearly not actually serving any purpose and only slowing the jobs down19:47
corvusthere may be a grafana dashboard or some graphs to clean up after that too19:47
clarkbcorvus: oh right good thing to note19:47
clarkbthe zuul dashboard tracks a job queue. I'll go ahead and propos ea clneap of that as one of the first things since its impact is small19:48
clarkbfungi: I should be able to push some changes up for that this afternoon19:48
fungii'll be able to review after dinner, thanks19:49
clarkbcool, doesn't sound like there is any concern proceeding with the plan we put in place about a year ago. I just wanted to start making progress on it since the deadline we gave has now passed19:50
clarkb#topic Open Discussion19:51
clarkbAnything else?19:51
fungione thing which came up earlier today so i didn't have time to get added to the agenda19:52
fungiopendev/irc-meetings could use a broader pool of core reviewers19:52
fungichanges for it are relatively time-sensitive, but trivial, usually fine with only a single core reviewer approving19:53
fungiright now its core team is our infra core set plus ttx19:53
fungiand he's indicated to me that he probably can't give it the attention he has in the past19:53
clarkbI'd be willing to add whoever volunteers if they've had some history with code review on opendev19:54
fungifigured it's an opportunity to do something like we do with project-config19:54
clarkb++19:54
fungimaybe even lower-risk, so could be a great place for newer folks to get familiar with the approval workflow19:54
fungibut also i wonder if we have a lot of smaller projects like that which might benefit from a blanket core review group to ease management/membership. i haven't really put much thought into that though19:55
fungigetting more volunteer reviewers is the first step either way19:56
fungiso probably best not to prematurely optimize for them19:56
clarkbyup I think finding interesting people is the hard part :)19:56
fungiwell, interested people anyway19:56
clarkbbut if we can find them then I agree that is a low risk project to add them to and get them involved19:57
clarkboh heh19:57
clarkbwell I'm definitely not interesting :)19:57
fungiwe have plenty of interesting people (but more never hurts!)19:57
fungianyway, sounds like there's not a concern with adding some new reviewers to irc-meetings if we get volunteers19:58
clarkbnot from me19:58
fungiprojects like that and project-config are where i see an opportunity to direct community volunteers to help with some of the review burden19:58
clarkb++19:59
fungiif projects feel like their changes to those aren't merging quickly enough, there's a pretty obvious answer ;)19:59
ianwoh i had a quick one to consider after fixing our status tweeting19:59
ianw#link https://review.opendev.org/c/opendev/statusbot/+/83708219:59
ianwwill also log #log messages.  just to increase engagement with what we're doing.  just a thought20:00
clarkboh we both +2'd at the same time :)20:00
clarkbI'm going tocall the meeting here. Thank you everyone20:00
fungithanks clarkb!20:01
clarkbenjoy the long weekend if it is one for you :)20:01
clarkb#endmeeting20:01
opendevmeetMeeting ended Tue Apr 12 20:01:06 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:01
opendevmeetMinutes:        https://meetings.opendev.org/meetings/infra/2022/infra.2022-04-12-19.01.html20:01
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/infra/2022/infra.2022-04-12-19.01.txt20:01
opendevmeetLog:            https://meetings.opendev.org/meetings/infra/2022/infra.2022-04-12-19.01.log.html20:01

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!