Tuesday, 2022-02-15

clarkbAnyone else here for our meeting?19:00
clarkb#startmeeting infra19:01
opendevmeetMeeting started Tue Feb 15 19:01:06 2022 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
opendevmeetThe meeting name has been set to 'infra'19:01
clarkb#link http://lists.opendev.org/pipermail/service-discuss/2022-February/000319.html Our Agenda19:01
clarkb#topic Announcements19:01
ianwo/19:01
clarkbAs discussed last week I sent email to the list explicitly for the service coordinator nomination period. I haven't need any nominations yet. Does that mean I'm "it" again19:01
clarkbI'll make it official after lunch today if no one else indicates interest19:03
clarkb#topic Actions from last meeting19:03
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-02-08-19.01.txt minutes from last meeting19:03
clarkb#action frickler propose mergeability check reenablement change19:03
clarkbI don't see that change yet, but also know frickler isn't around today. Hopefully soon though19:03
clarkb#topic Topics19:04
clarkb#topic Improving OpenDev's CD throughput19:04
clarkbThis item and the next one continue to be deprioritized due to other things for me :/19:05
clarkbianw: not sure there are any updates directly related to this though your gpg encrypted logs setup may be related19:05
clarkbdid you want to talk about that really quickly? Or is it still in refinement mode?19:05
fungiit looked like a really neat idea19:05
fungii think he said it was ready for evaluation?19:06
ianwit's ready for review.  it falls under the general heading of "making bridge not special"19:06
ianw#link https://review.opendev.org/q/topic:bridge-encrypt-logs19:06
clarkbthe tldr is encrypt ansible logs with root gpg keys so that they can be uploaded as zuul artifacts?19:06
ianwyes, but i think the key is not just root gpg keys -- anyone could add a key per prod job19:07
clarkbah neat19:07
ianwbut for root users who can already see the logs, the benefit is that instead of having to dig into bridge.o.o:/var/logs/... when we see a failed job, we can grab the logs for that run directly19:08
clarkbright, its more consistent with the typical zuul experience. My only concern was I had to fiddle with gpg recently to test signed tag pushes with Gerrit and gpg really didn't want to work headlessly. But I imagine most of the time this will be done on my desktop/laptop and I won't notice19:09
clarkb(mostly me just grumping that gpg does't work without a display)19:09
ianwthat is indeed correct, and there's about 8-10 changes in the encrypt-files role where i was figuring out how to get gpg to import a pubilc key and trust it headlessly :)  there's some comments around that section about several things that *don't* work :)19:10
clarkbThis is a neat idea. Thank you for putting it together. Anything else on this topic?19:11
ianwin terms of decrypting -- i repurposed the download logs script we have that pulls things via the manifest19:11
ianwas long as you have gpg-agent setup, it "just works" to decrypt; you run that script and it does it all for you19:11
ianwbut you're welcome to just grab the .gpg file directly too19:12
ianwthat is all, we can discuss in reviews, thanks! :)19:12
clarkb#topic Container Maintenance19:12
clarkbNo real movement here. I keep forgetting to do the limnoria update on a thursday afternoon when meetings don't happen19:12
clarkbAnd then haven't had time to dig into the dedicated user work. But I really do want too :/19:13
clarkb#topic Nodepool image cleanup19:13
clarkbFedora 34 is now completely gone including its mirror content19:13
clarkbI did notice that we still have the centos-8 python wheel mirror, but that content is relatively small. I'm not too worried about it19:14
clarkbI think we can call this topic done for now? frickler looked into xenial cleanup and there is a bit of work to be done19:14
ianwoh, i can clean that up19:14
clarkb#link https://etherpad.opendev.org/p/ubuntu-xenial-jobs Ubuntu Xenial cleanup19:14
clarkbI think we need to push on ^ those items a bit before we are naywhere near removing the image19:15
clarkbI did try to add notes about things there though19:15
clarkbThis went pretty smoothly even with the shorter than expected centos-8 mirror grace period19:15
clarkbThank you to everyone that updated jobs and helped with this cleanup19:15
clarkb#topic New Nodepool Images19:16
clarkbThis is a meeting chair addition to the agenda :)19:16
clarkb#link https://review.opendev.org/c/openstack/project-config/+/828435 Add rocky linux images to nodepool19:16
clarkbI think that change is about ready to go. I half suspect we'll get image build failures and then pause the image while we sort through them19:16
clarkbIf we are ok with that iterative process reviews would be good19:16
clarkband I think we'll run these without mirrors and see how that does19:17
ianwjust on the previous one; i have a couple of low-priority py3 and centos wheel build updates with 19:17
ianw#link https://review.opendev.org/q/topic:8-stream-wheel19:17
clarkbthanks19:18
ianwi plan to do 9-stream, that's why dropping the virtualenv requirement19:18
ianwthe goal for rocky is more kolla and things, rather than full devstack runs?19:19
clarkbianw: yes aiui kolla specifically is asking for it. I think it would be up to the qa team or new volunteers to step up and do devstack work19:19
clarkbianw: the way the TC describes platform support it might make sense for them to say replace centos stream with rocky/euler/alma19:20
fungithouvgh always possible that goal will expand in scope if there are more centos stream regressions19:20
clarkb(the real target is rhel 8 and those three may be more representative?)19:20
clarkbI do think it would be good to avoid having 3 different rhel8 clones (this rocky change gets us to 2. If it becomes an issue we should discuss with interested parties in why we'd need the whole set19:21
fungiright, part of why openstack yoga is being tested with python 3.6 even though centos stream 9 is out with newer python is that there is no rhel 9 yet19:21
fungiopenstack zed may end up in the same situation19:22
clarkbBut ya after centos-8-stream stopped working at the beginning of the year for a bunch of jobs and they didn't revert nor push a fix I can understand why people want something more stable19:22
clarkbso having an option or two like euler and rocky is a good idea imo19:22
fungithey want there to be at least one release overlap where a user can move from rhel 8 to rhel 9 while running the same version of openstack19:22
fungiso we're probably looking at openstack wanting to test zen on a rhel 8 clone as well, the way rhel 9's schedule seems to be progressing19:23
clarkbya I think from our position we're trying to enable interested parties and less prescribing what they should test19:23
clarkbin this case people are interested in replacing stream with a proper clone to get closer to what people are likely to use in production aiui19:23
clarkb(kolla specifically)19:24
fungithough it's worth noting, on the openstack development pain points front, continuing to test with python 3.6 when lots of libs on pypi are dropping support for it is causing a lot of headache19:25
clarkbanyway I just wanted to call out the change and the fact that we might have to pause builds and iterate. If we are ok with that then we can proceed. If we want more upfront vetting then we'll have to sort out local builds19:25
ianwi think it's close enough that submitting and pausing if failing is reasonable19:25
fungithat seems fine to me19:26
clarkbme as well I've already +2'd19:26
clarkband thanks for listening :)19:26
ianw(both close enough to working by itself, and close enough theoretically to things that already work :)19:26
clarkb#topic Cleaning up old reviews19:26
clarkb#link https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please double check19:27
clarkbBased on this list I went ahead and pushed some chagnes to start the retirement process19:27
clarkb#link https://review.opendev.org/c/opendev/system-config/+/829119 Removes unused repos from integration testing.19:27
clarkb#link https://review.opendev.org/c/openstack/project-config/+/829121 Sets noop jobs on unused repos so that they can be retired.19:27
clarkbThe first one has the reviews it needs as does its parent. I guess we approve that when we are able to watch it for any unexpected puppet fallout19:28
clarkbI can do that probably tomorrow (today is a busy one already)19:28
clarkbThen the second change ensures we can push up and land all the retire this repo changes in the repos themselves19:28
clarkbIf you notice something doesn't look right please let me/us know. Cleanup like this is always a little scary particularly since our coverage in testing is less good for puppet19:29
clarkb#topic Gitea 1.16.119:30
clarkb#link https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.1 when we are ready19:30
clarkb#link https://104.130.74.7:3081/opendev/system-config Test site via held node here19:30
clarkbianw has looked this over and called out one weird behavior that seemed to correct itself. I suspect a race in when the information is queried and the state of classification of the repo contents19:30
ianwyeah, it was odd that it changed when i was looking, as the host had been up for a few days.  i guess looking triggers some sort of refresh19:31
clarkbIf others can look at it carefully that would be helpful as the gitea 1.16 changelog is quite large. The thing I'm most concerned about is ssh functionality as they specifically call out breaking changes to that. 828184 attempts to accomodate the ssh changes and updates our testing to push via ssh to mimic gerrit replication19:31
clarkb#topic Gerrit Gitea Weblinks19:32
clarkbI've spent a good chunk of the last day and a half figuring out how to java better. I'm hopeful the fixups for this upstream are now in a mergeable state19:33
clarkb#link https://gerrit-review.googlesource.com/c/gerrit/+/329279 Allow for sha1 to be specified in filelinks19:33
clarkb#link https://gerrit-review.googlesource.com/c/plugins/gitiles/+/330361 related gitiles plugin update19:33
clarkbAssuming those land we can rebuild our gerrit and restart with some gitea link config19:34
clarkbI got a separate bug fix for gerrit ls-members --recursive ssh command landed whcih a rebuild would now pick up if we want to do that (it isn't urgent now that we know the bug exists and can check via http instead)19:34
ianwso it's already in 3.4 branch?19:35
clarkbIt is nice to see that upstream is responsive to these issues and is willing to guide us through fixing them19:35
clarkbianw: ya ls-members was fixed on 3.3 and the merge to 3.4 happened yesterday19:36
clarkbthe other two are proposed to 3.419:36
ianw++; i'm happy to restart later this afternoon if we have a change bumping it19:36
clarkbI havne't pushed a change for that yet. I also don't think it is urgent if we just want to wait for the gitea link work to hopefully land19:37
clarkbuntil then do group membership listings via rest or the web ui :)19:37
clarkb#topic Open Discussion19:38
clarkbAnything else?19:38
clarkb#link https://review.opendev.org/c/opendev/system-config/+/829141 improve haproxy checks for gitea19:38
clarkbfungi called out that our old checks weren't quite right as apache could be up and gitea could be down and haproxy would think the service was still up. This change attempts to address that by doing http checks which should check both apache and gitea are functional as a unit19:38
fungiyesterday i moved production zanata and refstack over to authenticating with id.openinfra.dev instead of openstackid.org, in case anyone hears of problems logging into those19:40
fungii haven't touched the keycloak poc yet, because we might want to redeploy it soonish anyway, i'm guessing, as we put more of the configuration under management19:40
clarkbSounds liek that may be it?19:42
fungii'm still a teensy bit worried i missed something on refstack since i had to resort to using sed on a mysqldump to work around its use of openid fields as foreign key constraints, but so far people seem to be having no trouble with it19:42
clarkbthats good19:42
fungiand no, i didn't have anything else at the moment19:43
clarkbI think we can call it there and we can all go find food :) I need to get ready for my next meeting.19:43
clarkbthank you everyone. We'll see you here next week19:43
fungithanks!19:43
clarkb#endmeeting19:43
opendevmeetMeeting ended Tue Feb 15 19:43:43 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:43
opendevmeetMinutes:        https://meetings.opendev.org/meetings/infra/2022/infra.2022-02-15-19.01.html19:43
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/infra/2022/infra.2022-02-15-19.01.txt19:43
opendevmeetLog:            https://meetings.opendev.org/meetings/infra/2022/infra.2022-02-15-19.01.log.html19:43

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!