Friday, 2025-01-31

jaltman https://www.openafs.org/release/openafs-1.8.13.2.html00:21
fricklerdpawlik: tonyb: can you please stop the SF CI from reviewing (at least) devstack patches until things are passing by default? having a -1 vote on e.g. https://review.opendev.org/c/openstack/devstack/+/939825 has essentially made me not look at this patch for a couple of days and I assume others may be affected similarly12:19
fungijaltman: thanks, and yes that's the version in debian/sid now12:36
fungisince it seems quiet right now and there's no openstack release team meeting this week, i'm going to pop out to get breakfast and run an errand, but should be back in about an hour13:45
dpawlikfrickler: hey, we are busy today, but I will take a focus on Monday on it. If I will not reply, ping me once again. Sorry13:56
fungiokay, i guess that was more like 30 minutes really14:22
dpawlikcc karolinku[m] (please check what frickler wrote)14:37
dpawlikfrickler - https://review.rdoproject.org/r/c/config/+/56132 that should be enough, right ?14:40
karolinku[m]this patch needs to be merge to fix that issue (by adding support for c10) https://review.opendev.org/c/openstack/devstack/+/93725114:50
karolinku[m]cc jcapitao 14:50
karolinku[m]but temporally, before it's merged, we can make it non-voting14:50
fricklerthere were some weird checksum errors in jobs, not sure if pypi is doing strange things or our mirrors? e.g. https://zuul.opendev.org/t/openstack/build/ab1699191fdf491c92c7d4eaf4d57360/console15:24
jcapitao[m]frickler: hi, is it the "-1" from SF CI job that is annoying ? or is it the entire job that you want to go away until devstack CS10 patch is merged15:24
fricklerjcapitao[m]: I'm mainly concerned about the -1, though the job in itself isn't really helpful currently, either15:25
jcapitao[m](asking to make the right change) 15:25
jcapitao[m]yeah I agree, the job is useless until the CS10 patch is merged15:26
fricklerjcapitao[m]: you'll also want to discuss with the QA team how this should work longer term. iiuc the current policy is to only mark distros as supported in devstack that are testing in the upstream CI15:26
jcapitao[m]ok, so as you mentioned in a comment, I think we should bring that topic in next PTG15:30
jcapitao[m]I think we can remove the SF CI job until then15:32
clarkbfrickler: neither of the hashes reported by the job log match the hash at https://pypi.org/project/ara/1.6.0/#ara-1.6.0-py2.py3-none-any.whl and if I download that file the hash matches what pypi reports15:53
clarkbso ya I don't know what is happening there15:54
opendevreviewClark Boylan proposed opendev/system-config master: Switch our haproxy image to quay opendevmirror location  https://review.opendev.org/c/opendev/system-config/+/94053616:02
clarkbinfra-root if we want to do ^ and maybe the refstack db backend update (https://review.opendev.org/c/opendev/system-config/+/940347) those might be two container image updates appropriate for a Friday16:02
clarkbthen we're down to the more difficult ones and can followup later16:03
fungiwhen i download ara-1.6.0-py2.py3-none-any.whl from that same proxy and directly from pypi i get identical contents (same checksums)16:03
fungiwhich also matches the checksum claimed on pypi's interface16:03
clarkbfungi: do they match the one listed at https://pypi.org/project/ara/1.6.0/#ara-1.6.0-py2.py3-none-any.whl ? 16:03
fungiyes16:04
clarkbya which is a third value. I suspect that pypi was serving bogus data for a time16:04
clarkband we cached that which resulted in differing shas over a period of time16:04
clarkbinfra-root I'm going to restart apache on eavesdrop to force it to pick up the new ssl certs on all the backend worker processes16:04
clarkbthats done, the cert checker should stop complaining now16:05
fungithanks, should we do that on ptg.o.o too?16:07
clarkbfungi: its the same host so it got both of them16:09
fungioh, right16:09
clarkbptg.opendev.org CNAME eavesdrop01.opendev.org.16:10
fungiyeah, i can't even blame lack of caffeine this time16:10
fungimaybe my brain has just decided to take off early for the weekend16:10
clarkbI'm there myself if I'm honest16:11
fungiit's been a week16:11
clarkbthe abrupt weather change has my body saying "its time to hibernate"16:11
opendevreviewClark Boylan proposed opendev/system-config master: Switch our haproxy image to quay opendevmirror location  https://review.opendev.org/c/opendev/system-config/+/94053616:25
clarkbthat wasn't testing zuul too just gitea. I updated fiel matchers so that both are covered16:26
opendevreviewBrian Haley proposed zuul/zuul-jobs master: Update ensure-twine role  https://review.opendev.org/c/zuul/zuul-jobs/+/94007417:09
opendevreviewMerged opendev/system-config master: Switch refstack to opendevmirror hosted mariadb image  https://review.opendev.org/c/opendev/system-config/+/94034717:19
opendevreviewBrian Haley proposed zuul/zuul-jobs master: Update ensure-twine role  https://review.opendev.org/c/zuul/zuul-jobs/+/94007417:23
clarkbthe refstack db deployed a minute or two ago. The service appears to remain up17:30
fungiconfirmed17:34
clarkbI'm thinking monday may be a good day to try the grafana upgrade17:40
fungiyeah, i was going to ask17:40
clarkbOur test coverage should be pretty good but considered we know the last time we tried to upgrade it broke being cautious there seems like a good idea17:41
clarkbhttps://review.opendev.org/c/opendev/system-config/+/940536 passes ci including zuul and gitea jobs if we want to swap the haproxy location. This seems a lot safer since it should be one equivalent image for another (just the source changes)17:44
clarkbI guess I can check that the docekr hub hosted image hasn't changed since we mirrored it yseterday17:45
clarkbheh it did update 19 hours ago17:45
clarkbwhich I now need to go graph logs for and cross check17:45
opendevreviewJames E. Blair proposed openstack/project-config master: Add zuul-providers project  https://review.opendev.org/c/openstack/project-config/+/94054017:46
opendevreviewJames E. Blair proposed openstack/project-config master: Add zuul-providers to opendev and zuul tenants  https://review.opendev.org/c/openstack/project-config/+/94054117:46
clarkbhttps://zuul.opendev.org/t/openstack/build/1b2b6d41e0a44d1b805aae2a2c8c269c/log/job-output.txt#554 this digest matches the digest at https://hub.docker.com/layers/library/haproxy/lts/images/sha256-75c6ef628f55d747c3ffe607dd84e0f8fd834f1c9e4a90ab3afdf722ad887273 so I think we are good to proceed. And even then haproxy is stateless so should be fine if we downgrade (whcih might not be17:48
clarkbok for mariadb)17:48
corvusclarkb: fungi ^ i think we're ready to start doing a bit more with zuul-launcher nodes.  those 2 changes will make a new project that we can include in all the tenants (just opendev and zuul for now though) that will hold the image/label/flavor/provider configs.17:50
corvusi think we can keep the image build jobs in the opendev/zuul-jobs repo and keep that only in the opendev tenant17:51
corvuswith that in place, i think we can probably start maybe running some zuul unit test jobs on the new nodes17:51
fungilgtm17:51
clarkband the split is because we don't need every tenant running image build jobs but we do need to expose the images and labels etc to every tenant?17:51
fungialso not sure if you saw my comment on the first one, but do we want it added to our gerritbot config too?17:55
opendevreviewMerged zuul/zuul-jobs master: Update ensure-twine role  https://review.opendev.org/c/zuul/zuul-jobs/+/94007418:10
opendevreviewMerged openstack/project-config master: Add zuul-providers project  https://review.opendev.org/c/openstack/project-config/+/94054018:11
clarkbcorvus: re ^ should we approve the change to add it to zuul as soon as it exists in gerrit or wait for some ocntent? (I guess that would probably require force pushing which we'd like to avoid so adding it to zuul as soon as it is in gerrit makes sense to me)18:12
corvusclarkb: re split: yes exactly (we could filter by config object to exclude jobs, but also, we said that repo was for "opendev-tenant-specific jobs, so i think it makes sense to move this stuff to another repo18:32
corvusclarkb: and yes, i think we can create first then add content18:32
corvusfungi: i added it to the gerritbot config in the second change, just forgot to mention in the commit msg18:33
corvuslooks like it exists in gerrit, so i'll approve the next change18:34
fungioh, yep18:34
fungithanks!18:34
opendevreviewMerged openstack/project-config master: Add zuul-providers to opendev and zuul tenants  https://review.opendev.org/c/openstack/project-config/+/94054118:45
clarkbmakes sense18:51
clarkbany opinions on https://review.opendev.org/c/opendev/system-config/+/940536 to change the haproxy source?18:54
corvus+218:57
corvusi think the blip is fine18:57
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Remove provider configuration  https://review.opendev.org/c/opendev/zuul-jobs/+/94054318:59
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Add provider configuration  https://review.opendev.org/c/opendev/zuul-providers/+/94054418:59
corvusthat is an unexpected configuration error on 54319:00
clarkbcorvus: the inaugust repos are in opendev's tenant19:00
clarkbwhich would explain the relationship at least19:01
corvusyeah, but for it to report means it's a new error....19:01
corvusi'm going to do a tenant reconfig on opendev to see if anything shows up in the static error list afterwords19:01
corvusnope.19:02
corvushowever, after doing that, a recheck worked.19:03
clarkbthat seems even more unexpected?19:03
clarkblike some sort of data inconsistency in the backend?19:04
fungialso reminder that we've got some stuck gate and promote builds in the opendev tenant, indefinitely queued docs and noop jobs19:04
fungii never did find time to dig into those19:04
corvuswell, "no error" is the correct state, so i'm leaning towards "we just fixed some previously incorrect in-memory/zk state"19:04
corvusfungi: yeah, that sort of thing seems to add to the "corrupt state" theory19:05
fungioldest are 90 days19:05
clarkbya I debated trying to dequeue them before but they didn't seem to be causing problems so left them as is19:05
clarkbbut maybe we should try and start over and see if problems crop back up again19:05
clarkbits possible whatever caused that was in old zuul and isn't a problem anymore19:06
corvusyeah my inclination would be to dequeue and observe.19:06
fungiyoungest is a little over a couple weeks, but all of them have other jobs queued up behind them waiting on those to complete19:06
corvusif that's everyone else's feeling, i can go ahead and dequeue them while i'm here19:06
fungiwfm19:06
fungiall gone!19:08
clarkb++19:08
corvus#status log dequeued a handful of stuck changes in opendev tenant19:08
opendevstatuscorvus: finished logging19:08
corvusi dequeued from bottom to top, just to avoid any extra weirdness.19:08
corvusoh heh i forgot to add a pipeline config to zuul-providers19:09
fungid'oh! sorry i missed that when reviewing19:09
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Add provider configuration  https://review.opendev.org/c/opendev/zuul-providers/+/94054419:10
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Add initial zuul pipeline config  https://review.opendev.org/c/opendev/zuul-providers/+/94054519:10
corvushey, we should put that change in the jeepyb initial change creator thingy19:11
clarkbthe bare minimum to merge changes does seem like a good idea19:11
clarkbfungi: you good with updating the haproxy source too? https://review.opendev.org/c/opendev/system-config/+/94053619:13
clarkbI'm around and can ensure thinsg restart properly / debug if necessary19:13
opendevreviewJames E. Blair proposed opendev/jeepyb master: Create a stub .zuul.yaml  https://review.opendev.org/c/opendev/jeepyb/+/94054719:14
opendevreviewMerged opendev/zuul-providers master: Add initial zuul pipeline config  https://review.opendev.org/c/opendev/zuul-providers/+/94054519:15
clarkbI believe that jeepyb change will actually create new projects in a test gerrit installation so it should have decent coverage, but I'll try to check in 20 minutes or however long that takes to confirm19:16
corvusok. i wrote that without mentally paging in all of jeepyb, so i'm kind of just assuming "do it next to where we write .gitreview" is the correct approach.19:17
clarkbyup I believe that is the case19:17
opendevreviewMerged opendev/zuul-jobs master: Remove provider configuration  https://review.opendev.org/c/opendev/zuul-jobs/+/94054319:17
fungihuh, opendev-zuul-jobs-noop actually runs things, i guess19:17
clarkbits in the else block for "no other source of data is present" and we're starting from an empty repo case19:17
fungi(contrary to its name)19:17
claygHowdy; I'd like some "infra" assistance with https://review.opendev.org/c/openstack/swift/+/836755 - which is currently set to "WIP" according to gerrit, but it's actively be worked on (by a new author) and we don't know how to change the status.  I think I would have previously asked this question in #openstack-infra but maybe that's not a channel anymore?  Regardless I see @clarkb here so I'm sure he'll tell me to talk to 😁19:19
fungi#openstack-infra is still an active channel too, yes19:19
clarkbfungi uses the built in wip feature a lot (I don't). I know the original author can toggle it. Do we need to use admin bits to unset it?19:20
claygfungi: sorry, should I cross post there or is ok to discuss here?19:21
clarkbhere is fine19:21
corvusfungi: i think it is a noop on a real node19:21
fungiclayg: either is fine, #openstack-infra is more for openstack-specific talk, but we'll answer anywhere19:21
claygclarkb: we also feel like as an author we can change the setting - but we don't know how to contact the orig author - we have considered ditching the history/comments and starting a new change if we can't change the author19:21
clarkbhttps://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#set-ready-for-review I think we can use this rest api endpoint as an admin19:24
fungiclayg: https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/swift.config is missing a togglewipstate perm like you see in https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/ironic.config#L1419:24
corvusthe "delete provider" workflow needs some work (the zuul launcher is unhappy with having image build artifacts that belong to a provider it doesn't know about).19:24
clarkboh I guess as an alternative we update acls and let swift do it. That might be more future proof19:24
clarkbcorvus: do we think landing the change to readd that config will make it happy again in the short term?19:25
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Revert "Remove provider configuration"  https://review.opendev.org/c/opendev/zuul-jobs/+/94054919:25
corvusno, i think we should try a nodepool-style removal19:26
corvus(because the change to re-add the config will add it with different canonical names, so it won't match)19:26
clarkbgotcha so we need to revert, then delete the resources, then delete the config, then add new config elsewhere.19:27
clarkbclayg: do you want to propose a change to the swift.config file fungi linked that updates it to be like ironic? We can get that landed pretty quickly then you should be able to remove the wip flag yourself19:28
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Remove resources from providers  https://review.opendev.org/c/opendev/zuul-jobs/+/94055019:28
clarkbcorvus: looks like we may need to force merge the revert?19:28
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Remove provider configuration  https://review.opendev.org/c/opendev/zuul-jobs/+/94055119:29
corvusoh yeah, that's interesting.  that seems like an error that should have shown up the first time.  maybe that was being masked by the inaugust error.19:30
opendevreviewClay Gerrard proposed openstack/project-config master: let core fix WIP state  https://review.opendev.org/c/openstack/project-config/+/94055219:30
corvusthere is a way to navigate that without a force-merge, but it would take all day because it involves running image build jobs.  so i think we should just force it.19:31
opendevreviewMerged opendev/zuul-jobs master: Revert "Remove provider configuration"  https://review.opendev.org/c/opendev/zuul-jobs/+/94054919:32
clarkbya force merge makes sense to me too19:32
clarkbits a super isolated area of the config too19:32
fungisounds fine to me19:33
claygclarkb: fungi: oh wow, thanks!  I'll give it a little bit and see if it helps 🤞19:35
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Remove opendev-zuul-jobs-noop  https://review.opendev.org/c/opendev/zuul-jobs/+/94055419:35
corvushttps://review.opendev.org/940550 is ready now19:36
clarkbapproved19:37
clarkbI +2'd the followup changes so you can approve when ready19:38
fungias have i19:38
opendevreviewMerged opendev/zuul-jobs master: Remove resources from providers  https://review.opendev.org/c/opendev/zuul-jobs/+/94055019:39
corvushttps://zuul.opendev.org/t/opendev/image/debian-bullseye looks empty, seems happy, i'll approve now19:40
opendevreviewMerged openstack/project-config master: let core fix WIP state  https://review.opendev.org/c/openstack/project-config/+/94055219:41
clarkbI'm going to make a note about how we might want to toggleWipState across the board. It would still need to be a project by project basis though to get the perms correct (I'll put this on the meetup etherpad)19:42
opendevreviewMerged opendev/zuul-jobs master: Remove provider configuration  https://review.opendev.org/c/opendev/zuul-jobs/+/94055119:42
fungiclarkb: sgtm, we could just script a bulk change19:46
opendevreviewMerged opendev/zuul-providers master: Add provider configuration  https://review.opendev.org/c/opendev/zuul-providers/+/94054419:52
clarkbclayg: I think you should be able to toggle it back to active yourself now19:52
clarkbyou may need to hard refresh on the change page to reload things to show you the button to do that19:53
clarkbI think it says something like "set active" ?19:53
clarkboh looks like you already did19:53
fungiyeah, there was conversation in #openstack-swift19:54
clarkbI was just catching up with the deployment job and was slower than yall19:55
clarkband now it is time to reheat some pizza for lunch19:55
clarkbI'm still happy to keep an eye on haproxy things if we want to approve that. It runs the gitea job which takes about an hour so should be plenty of time to eat19:55
claygclarkb: yes i'm all squared - thank you so much!  5* rating on #openstack-infra support.  Would buy again 😉20:00
opendevreviewClark Boylan proposed opendev/jeepyb master: Create a stub .zuul.yaml  https://review.opendev.org/c/opendev/jeepyb/+/94054720:49
opendevreviewClark Boylan proposed opendev/jeepyb master: Fix E471 linter error  https://review.opendev.org/c/opendev/jeepyb/+/94056020:49
clarkbcorvus: ^ I went ahead and fixed the linter error unrelated to your change and stacked yours on top20:49
corvusclarkb: thanks!20:49
corvusclarkb: fungi i keep forgetting that the image build job *does* need to be in the same repo as the image definition...20:50
corvusi think we should go ahead and pull the image build jobs into zuul-providers since i think it still makes sense to be there.20:50
clarkbok, then we selectively load the necessary bits into each tenant from zuul-providers?20:51
corvusi'd love to have them split but i think there's a permissions situation that's resolved by having them together20:51
corvusyeah20:51
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Remove image build jobs  https://review.opendev.org/c/opendev/zuul-jobs/+/94056120:55
clarkbI think we need to upodate the opendev tenant config for zuul-providers but not zuul tenant?20:56
corvusyeah20:57
corvusi'm working on re-encrypting the secret20:57
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Add image build jobs  https://review.opendev.org/c/opendev/zuul-providers/+/94056221:01
clarkbcorvus: I think you need to copy the playbooks too?21:02
corvusheh yep21:02
clarkband maybe delete them from zuul-jobs side though that is less required21:02
corvusclarkb: the tenant config is correct -- it already has jobs in the opendev tenant21:02
clarkback21:03
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Add image build jobs  https://review.opendev.org/c/opendev/zuul-providers/+/94056221:04
opendevreviewMerged opendev/zuul-jobs master: Remove image build jobs  https://review.opendev.org/c/opendev/zuul-jobs/+/94056121:04
clarkboh wow that is a very large change now21:05
clarkbI think I'm mostly going to trust that you copied things properly :)21:05
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Add image build jobs  https://review.opendev.org/c/opendev/zuul-providers/+/94056221:05
corvuscopied a few more things21:05
corvusbut yeah, all the dib-* and roles/* and playbooks/* are straight copies21:05
corvusi omitted the noop.yaml playbook21:06
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Cleanup image build playbooks, elements, and roles  https://review.opendev.org/c/opendev/zuul-jobs/+/94056321:06
clarkbI still see no_log: true in at least one place I would expect it. But if you've done a straight copy we should have that in all the necesary places21:08
corvusgood idea.  "git grep no_log" says the same things for the old repo and new:21:09
corvusplaybooks/opendev-build-diskimage-base/post-inner.yaml:  no_log: true21:09
corvusroles/image-upload-swift/tasks/main.yaml:  no_log: true21:09
corvusokay that stuff is working its way through gate (which includes an image build), so that'll be a bit21:10
opendevreviewMerged opendev/zuul-jobs master: Cleanup image build playbooks, elements, and roles  https://review.opendev.org/c/opendev/zuul-jobs/+/94056321:13
clarkbcorvus: reviewing testing I found a small issue in the jeepyb change if you want to fix that or let me know if I should edit it21:15
corvusi'll take a look21:15
opendevreviewJames E. Blair proposed opendev/jeepyb master: Create a stub .zuul.yaml  https://review.opendev.org/c/opendev/jeepyb/+/94054721:22
corvusoh oops i think i undid your thing clark 1 sec21:23
clarkboh ya justneeds to be rebased again21:24
opendevreviewJames E. Blair proposed opendev/jeepyb master: Create a stub .zuul.yaml  https://review.opendev.org/c/opendev/jeepyb/+/94054721:24
opendevreviewJames E. Blair proposed opendev/zuul-providers master: Add image build jobs  https://review.opendev.org/c/opendev/zuul-providers/+/94056223:24
corvusclarkb: ^ that required one env var change due to the name of the repo; i think that's trivial enough i'm going to self-reapprove.23:25
clarkbI reviewed it quickly the delta diff was clear23:25
clarkbbut go ahead and approve23:25
corvuskk thx23:25

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!