| clarkb | Anyone else here for the meeting? | 19:00 |
|---|---|---|
| clarkb | I'm still getting things together, but we'll start momentarily | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Feb 1 19:01:05 2022 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| ianw | o/ | 19:01 |
| frickler | o/ | 19:01 |
| fungi | ohai | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2022-January/000316.html Our Agenda | 19:02 |
| clarkb | #topic Announcements | 19:02 |
| clarkb | Service coordinator nominations run January 25, 2022 - February 8, 2022 You have another week :) | 19:02 |
| clarkb | As always let me know if you have questions about that and I'd be happy to answer them | 19:02 |
| clarkb | OpenInfra Summit CFP needs your input: https://openinfra.dev/summit/ | 19:03 |
| clarkb | If you'd like to talk at the open infra summit there is a ci/cd track as well as other tracks you may be interested in proposing towards. I think you have until february 9 for that | 19:03 |
| clarkb | And finally Zuul v5 released today! The culmination of much long term planning and effort. Thank you everyone who helped make that possible | 19:03 |
| clarkb | side note our zuul isntall still says v4.12.something but we're running the same commits that were tagged v5 | 19:04 |
| clarkb | #topic Actions from last meeting | 19:04 |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-01-25-19.01.txt minutes from last meeting | 19:05 |
| clarkb | There were no actions recorded | 19:05 |
| clarkb | #topic Topics | 19:05 |
| clarkb | #topic Improving Opendev's CD throughput | 19:05 |
| clarkb | #link https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets | 19:05 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog | 19:05 |
| clarkb | Unfortunately the Gerrit upgrade and server patching had me far more distracted last week than I would've liked | 19:06 |
| clarkb | I haven't had time to look at these yet. They are still on my todo list though... | 19:06 |
| clarkb | maybe I should make an action for everyone to review those :) | 19:06 |
| clarkb | #action infra-root Review OpenDev CD throughput related spec for secrets management: https://review.opendev.org/c/opendev/infra-specs/+/821645 | 19:06 |
| clarkb | ianw: is there anything else to add to this topic? | 19:06 |
| ianw | no, no work has been done on this one | 19:07 |
| clarkb | #topic Container Maintenance | 19:08 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-container-maintenance | 19:08 |
| clarkb | My time for this last week was largely sidelined by server patching | 19:08 |
| clarkb | I don't really have anything new to add to this unfortunately. | 19:08 |
| clarkb | #topic Nodepool Image Cleanup | 19:08 |
| clarkb | Changes to remove CentOS 8 have been pushed as promised by the end of January. However, at least one project (OSA) is still struggling with removing centos 8 so we can hold off until they are ready since they are actively workign to correct this | 19:09 |
| clarkb | Once projects like OSA are ready we can land the chagnes in this order: | 19:09 |
| clarkb | #link https://review.opendev.org/c/opendev/base-jobs/+/827181 | 19:09 |
| clarkb | #link https://review.opendev.org/c/openstack/project-config/+/827184 | 19:09 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/827186 | 19:09 |
| ianw | so we found yesterday that the centos mirror infrastructure stopped returning links | 19:10 |
| clarkb | It looks like centos 8 upstream is starting to archive itself which is causing some issues here and there and people will be motivated to start moving | 19:10 |
| clarkb | ya that | 19:10 |
| ianw | yeah, so things are really only working because we run in a little mirror bubble | 19:11 |
| fungi | bless this bubble | 19:11 |
| ianw | i've stopped the image builds (https://review.opendev.org/c/openstack/project-config/+/827195) because they will fail | 19:12 |
| ianw | if upstream modifies their mirror bits and we rsync that on an run, then we will be totally broken | 19:13 |
| ianw | so tbh i feel like we could probably pull the images now, and if jobs fail people need to switch them to 8-stream and make them non-voting if it doesn't work ootb and fix it | 19:13 |
| fungi | the problem with dropping our images is that if something happens we can't upload them again | 19:14 |
| clarkb | ianw: do we know why the rsyncing hasn't broken yet? | 19:14 |
| clarkb | I think I'm ok with leaving this up until projects migrate and removing sooner if the upstream infrastructure is no longer tenable | 19:15 |
| fungi | oh, i see what you mean, drop the images in our providers and stop providing centos-8 nodes, full stop | 19:15 |
| ianw | clarkb: when i last checked, they hadn't moved the 8/ directories into vault.centos.org | 19:15 |
| ianw | but that may happen at any time i guess, which would make them disappear from the mirror and we'd pull that | 19:16 |
| fungi | not that we should have to handhold anyone, but i know we've been focused mostly on openstack's use of centos-8 nodes... does anyone happen to know if starlingx is also impacted? (or did they never finish moving off centos-7?) | 19:16 |
| frickler | mirror.centos dropped like 90G in size at around 11:30 today | 19:16 |
| fungi | and yeah, this is probably good to make folks on openstack-discuss aware of. centos-8 is going away even if we do nothing. your jobs are breaking today, sorry! | 19:17 |
| clarkb | fungi: ++ maybe the thing to do is respond to my thread that warned people about the removal. Indicate that centos-8 doesn't work if you talk to upstream anymore and as a result we're going to remove things? | 19:18 |
| clarkb | jrosser isn't here but was one who wanted to keep them up | 19:20 |
| ianw | ahh, yeah, http://mirror.iad.rax.opendev.org/centos/8/os isn't there ... | 19:20 |
| ianw | so that might have happened overnight | 19:20 |
| clarkb | ianw: I assume os/ includes important packages :) | 19:20 |
| ianw | (or today, depending on how you look at it :) | 19:20 |
| clarkb | I definitely think we should accelerate the removal given ^ | 19:20 |
| ianw | looks like from the logs it largely cleared itself out @ 2022-02-01T10:43:44,980764868+00:00 | 19:22 |
| clarkb | anyone want to volunteer to respond to the thread? The changes should be ready to go once we're ready | 19:23 |
| ianw | i can chase up on it, reply and merge those things through today | 19:24 |
| clarkb | thank you | 19:24 |
| fungi | i've got a few other deadlines looming so probably can't give it the immediate attention it deserves | 19:24 |
| fungi | thanks ianw! | 19:24 |
| clarkb | #topic Cleaning up old reviews | 19:25 |
| clarkb | #action clarkb to produce a list of repos that can be retired. We can then retire the repos and abandon their open reviews as step 0 | 19:25 |
| clarkb | I'll go ahead and record this now as an explicit action for my todo list. I think we still start with ^ which should take out a chunk of reviews then reevaluate when this is done | 19:25 |
| clarkb | frickler: anything else to add to this topic? | 19:25 |
| frickler | nope, didn't do anything on that yet | 19:25 |
| clarkb | #topic Gerrit mergeability checking | 19:26 |
| clarkb | When we upgraded to gerrit 3.4 we lost mergability checking by default. Gerrit disabled this functionality by default as it can use a disproportionate amount of resources to calculate merge conflicts | 19:26 |
| clarkb | The functionality is still in Gerrit though and we can opt into it via a config switch | 19:26 |
| clarkb | A few users have mentioned that the infromation was useful to them. | 19:27 |
| fungi | particularly folks using it to omit unmergeable changes from their review dashboards, seems like | 19:27 |
| clarkb | I'm not opposed to reenabling the functionality but do have some minor concerns. The biggest is that this will likelymake reindexing projects take longer now. But we were ok with that in the past. | 19:27 |
| clarkb | Other concerns are that gerrit has a tendency to remove functionality entirely after disabling it by default so we may have to accept it going away one day. But no automated tooling relies on this functionality so the cost remains with humans. If they remove it entirely we should be fine other than having some sad users | 19:28 |
| clarkb | There is also the potential that we'd be exposing ourselves to bugs in that functionality since few other users are going to use it. If that happens we can always disable it again | 19:28 |
| frickler | maybe we could send them (gerrit devs) feedback that we would like not to loose that functionality? | 19:29 |
| clarkb | All that to say I think despite my concerns there are good solutions should the concerns become a problem which means I'm ok with reenabling this | 19:29 |
| clarkb | frickler: ya that is another option. Basically "we're toggling to non default here please keep it working" | 19:29 |
| fungi | i'm also fine with bringing it back if someone proposes a patch | 19:29 |
| ianw | ++ i agree with having it on, i always found it useful, and also agree with sending some feedback that we're turning it on | 19:30 |
| frickler | I can look at doing a patch, since I'm one of the users who like to have it | 19:30 |
| ianw | it seems like we could probably create conflicting changes and push them during the testing at least | 19:30 |
| clarkb | I guess there is the msall matter of weather or not we need to offline reindex after enabling it. But I suspect that it will just start adding the info to new changes | 19:30 |
| clarkb | frickler: thank you! | 19:30 |
| clarkb | and ya our testing should alrgely cover major concerns with enabling it | 19:31 |
| ianw | it would be super cool to check that with selenium but probably just seeing in a screenshot is enough | 19:31 |
| frickler | clarkb: can you #action me on that or can I do that myself? | 19:31 |
| frickler | just so I don't forget it | 19:31 |
| clarkb | frickler: you should be able to do it yourself. | 19:31 |
| frickler | #action frickler propose patch to re-enable Gerrit mergeability checking | 19:32 |
| clarkb | (the bot doesn't give a lot of feedback though so I guess we'll find out after the meeting is done) | 19:32 |
| clarkb | #topic Gerrit issues we are tracking | 19:32 |
| clarkb | First up is the regression with gerrit ignoring signed tag acls for pushing tags. My patch to fix this which I tested manually on a held test node landed upstream and we have restarted Gerrit with that code and removed our workaround | 19:33 |
| clarkb | We are just waiting on someone to push a signed tag and confirm it is happy now. Once that is done I'll merge the 3.4 fix into gerrit 3.5 as well | 19:33 |
| clarkb | Next is url text substitution for gitweb links doesn't provide the hash value for ${commit} in all cases and gitea needs that | 19:34 |
| clarkb | #link https://bugs.chromium.org/p/gerrit/issues/detail?id=15589 | 19:34 |
| clarkb | fungi and I are working to test a fix that I pushed upstream. Still no review comments though | 19:34 |
| clarkb | One neat thing we are trying to do with our testing though is depends-on against upstream gerrit and running that code in our test jobs | 19:35 |
| fungi | also i'm somewhat blocked on zuul-client autohold's --change option not working as advertised | 19:35 |
| clarkb | It seems t owork from what I've seen so far | 19:35 |
| fungi | i'm testing a workaround with --ref instead, but we're probably going to need to fix zuul-client to be able to continue doing change-specific autoholds (or go back to the rpc client in the meantime) | 19:35 |
| clarkb | And finally yesterday we noticed that git pulls over ssh can backlog in gerrit where the tcp connection is made and gerrit recognizes there is a pull waiting but the tasks remain in waiting and are not processed by a thread | 19:36 |
| clarkb | If this happens long enough and the backlog grows eventually it leads to Zuul being very backlogged with its mergers | 19:36 |
| clarkb | #link https://bugs.chromium.org/p/gerrit/issues/detail?id=15649 | 19:36 |
| clarkb | Upstream asked for a thread dump which we have. We just need to audit it for any over exposure of sensitive info | 19:37 |
| clarkb | I'll try to work on that I guess since I've been working upstream with Gerrit more and more | 19:37 |
| clarkb | #topic Open Discussion | 19:38 |
| clarkb | Anything else? | 19:38 |
| frickler | not sure where to discuss, but lp is nearing the 2000000 bug count | 19:39 |
| frickler | so overlap with storyboard ids will happen | 19:39 |
| clarkb | fun. I'm not sure I personally grasp the impact of that. fungi would probably know better | 19:39 |
| fungi | that's a good reminder | 19:39 |
| frickler | we will loose the option to migrate existing bugs keeping their ids | 19:40 |
| fungi | it basically means we can no longer migrate projects from lp to sb and expect a 1:1 correlation between imported bug numbers | 19:40 |
| fungi | yeah, exactly | 19:40 |
| clarkb | I see | 19:40 |
| clarkb | solvable but with degraded ease of migration | 19:40 |
| clarkb | (since we would have to map to new numbers) | 19:40 |
| frickler | but then I also don't see a tendency to further do migrations to sb | 19:40 |
| fungi | we can probably do some logic to uprev any imported bugs in the 20k+ range and continue to import earlier reports the way we did in the past | 19:41 |
| clarkb | ya thats a good point | 19:41 |
| clarkb | If anything it seems like projects are looking at github issues more than anything else | 19:41 |
| fungi | and asking us to turn on gitea's issues feature, yeah | 19:41 |
| clarkb | thank you for calling that out | 19:42 |
| fungi | (but that means fixing the clustering problem, account management, and a host of other challenges) | 19:42 |
| clarkb | As a general heads up my availability over the next week may be spotty. I'm going to do my best to be around but not sure what my availability will be like. | 19:42 |
| clarkb | fungi: yup not an easy task. | 19:42 |
| ianw | have upstream fixed the clustering issues? | 19:43 |
| ianw | or not so much fixed, but added? | 19:43 |
| clarkb | yes I think the elasticsearch backend is there. Not sure if it will work with opensearch though | 19:44 |
| clarkb | unfortunate that the gitea effort happened while elasticsearch became less open but at least in theory we could run gitea with an opensearch cluster, a mariadb cluster, and a shared cephfs fs | 19:45 |
| ianw | interesting ... another one for the todo list :) | 19:45 |
| ianw | all that running together ... sounds like kubernetes might fit in ... | 19:46 |
| fungi | that's what we tried to use the first time! | 19:46 |
| clarkb | ya that was the original goal with gitea | 19:46 |
| clarkb | but when we realized the indexes weren't distributed it wasn't tenable until that got fixed (and they addressed that with the elasticsearch backend option) | 19:46 |
| fungi | i think that kubernetes cluster might still exist even, but we'd almost certainly want to rebuild it from scratch if we take it in that direction | 19:47 |
| clarkb | ++ | 19:47 |
| fungi | if memory serves, we also ran into trouble with rook | 19:47 |
| clarkb | figuring out how to manage a k8s cluster is probably step 0 | 19:48 |
| fungi | but that's almost certainly improved in the meantime | 19:48 |
| fungi | yeah, managing ceph within kubernetes was a struggle back then | 19:48 |
| clarkb | since there are many options and every option we tried perviously had its downsides (magnum didn't do upgrades via the api and you couldn't upgrade directly because there wasn't enough disk to grab two copies of the k8s images) | 19:48 |
| clarkb | We don't need to solve that in this meeting though. But if people want to investigate that again now might be a good time to start looking into it | 19:49 |
| clarkb | Sounds like we may be winding down. I'll give it a couple more minutes for any last minute items then call it a meeting | 19:50 |
| clarkb | Thank you everyone! | 19:52 |
| clarkb | #endmeeting | 19:52 |
| opendevmeet | Meeting ended Tue Feb 1 19:52:33 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:52 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2022/infra.2022-02-01-19.01.html | 19:52 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2022/infra.2022-02-01-19.01.txt | 19:52 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2022/infra.2022-02-01-19.01.log.html | 19:52 |
| fungi | thanks clarkb! | 19:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!