jaltman | https://www.openafs.org/release/openafs-1.8.13.2.html | 00:21 |
---|---|---|
frickler | dpawlik: tonyb: can you please stop the SF CI from reviewing (at least) devstack patches until things are passing by default? having a -1 vote on e.g. https://review.opendev.org/c/openstack/devstack/+/939825 has essentially made me not look at this patch for a couple of days and I assume others may be affected similarly | 12:19 |
fungi | jaltman: thanks, and yes that's the version in debian/sid now | 12:36 |
fungi | since it seems quiet right now and there's no openstack release team meeting this week, i'm going to pop out to get breakfast and run an errand, but should be back in about an hour | 13:45 |
dpawlik | frickler: hey, we are busy today, but I will take a focus on Monday on it. If I will not reply, ping me once again. Sorry | 13:56 |
fungi | okay, i guess that was more like 30 minutes really | 14:22 |
dpawlik | cc karolinku[m] (please check what frickler wrote) | 14:37 |
dpawlik | frickler - https://review.rdoproject.org/r/c/config/+/56132 that should be enough, right ? | 14:40 |
karolinku[m] | this patch needs to be merge to fix that issue (by adding support for c10) https://review.opendev.org/c/openstack/devstack/+/937251 | 14:50 |
karolinku[m] | cc jcapitao | 14:50 |
karolinku[m] | but temporally, before it's merged, we can make it non-voting | 14:50 |
frickler | there were some weird checksum errors in jobs, not sure if pypi is doing strange things or our mirrors? e.g. https://zuul.opendev.org/t/openstack/build/ab1699191fdf491c92c7d4eaf4d57360/console | 15:24 |
jcapitao[m] | frickler: hi, is it the "-1" from SF CI job that is annoying ? or is it the entire job that you want to go away until devstack CS10 patch is merged | 15:24 |
frickler | jcapitao[m]: I'm mainly concerned about the -1, though the job in itself isn't really helpful currently, either | 15:25 |
jcapitao[m] | (asking to make the right change) | 15:25 |
jcapitao[m] | yeah I agree, the job is useless until the CS10 patch is merged | 15:26 |
frickler | jcapitao[m]: you'll also want to discuss with the QA team how this should work longer term. iiuc the current policy is to only mark distros as supported in devstack that are testing in the upstream CI | 15:26 |
jcapitao[m] | ok, so as you mentioned in a comment, I think we should bring that topic in next PTG | 15:30 |
jcapitao[m] | I think we can remove the SF CI job until then | 15:32 |
clarkb | frickler: neither of the hashes reported by the job log match the hash at https://pypi.org/project/ara/1.6.0/#ara-1.6.0-py2.py3-none-any.whl and if I download that file the hash matches what pypi reports | 15:53 |
clarkb | so ya I don't know what is happening there | 15:54 |
opendevreview | Clark Boylan proposed opendev/system-config master: Switch our haproxy image to quay opendevmirror location https://review.opendev.org/c/opendev/system-config/+/940536 | 16:02 |
clarkb | infra-root if we want to do ^ and maybe the refstack db backend update (https://review.opendev.org/c/opendev/system-config/+/940347) those might be two container image updates appropriate for a Friday | 16:02 |
clarkb | then we're down to the more difficult ones and can followup later | 16:03 |
fungi | when i download ara-1.6.0-py2.py3-none-any.whl from that same proxy and directly from pypi i get identical contents (same checksums) | 16:03 |
fungi | which also matches the checksum claimed on pypi's interface | 16:03 |
clarkb | fungi: do they match the one listed at https://pypi.org/project/ara/1.6.0/#ara-1.6.0-py2.py3-none-any.whl ? | 16:03 |
fungi | yes | 16:04 |
clarkb | ya which is a third value. I suspect that pypi was serving bogus data for a time | 16:04 |
clarkb | and we cached that which resulted in differing shas over a period of time | 16:04 |
clarkb | infra-root I'm going to restart apache on eavesdrop to force it to pick up the new ssl certs on all the backend worker processes | 16:04 |
clarkb | thats done, the cert checker should stop complaining now | 16:05 |
fungi | thanks, should we do that on ptg.o.o too? | 16:07 |
clarkb | fungi: its the same host so it got both of them | 16:09 |
fungi | oh, right | 16:09 |
clarkb | ptg.opendev.org CNAME eavesdrop01.opendev.org. | 16:10 |
fungi | yeah, i can't even blame lack of caffeine this time | 16:10 |
fungi | maybe my brain has just decided to take off early for the weekend | 16:10 |
clarkb | I'm there myself if I'm honest | 16:11 |
fungi | it's been a week | 16:11 |
clarkb | the abrupt weather change has my body saying "its time to hibernate" | 16:11 |
opendevreview | Clark Boylan proposed opendev/system-config master: Switch our haproxy image to quay opendevmirror location https://review.opendev.org/c/opendev/system-config/+/940536 | 16:25 |
clarkb | that wasn't testing zuul too just gitea. I updated fiel matchers so that both are covered | 16:26 |
opendevreview | Brian Haley proposed zuul/zuul-jobs master: Update ensure-twine role https://review.opendev.org/c/zuul/zuul-jobs/+/940074 | 17:09 |
opendevreview | Merged opendev/system-config master: Switch refstack to opendevmirror hosted mariadb image https://review.opendev.org/c/opendev/system-config/+/940347 | 17:19 |
opendevreview | Brian Haley proposed zuul/zuul-jobs master: Update ensure-twine role https://review.opendev.org/c/zuul/zuul-jobs/+/940074 | 17:23 |
clarkb | the refstack db deployed a minute or two ago. The service appears to remain up | 17:30 |
fungi | confirmed | 17:34 |
clarkb | I'm thinking monday may be a good day to try the grafana upgrade | 17:40 |
fungi | yeah, i was going to ask | 17:40 |
clarkb | Our test coverage should be pretty good but considered we know the last time we tried to upgrade it broke being cautious there seems like a good idea | 17:41 |
clarkb | https://review.opendev.org/c/opendev/system-config/+/940536 passes ci including zuul and gitea jobs if we want to swap the haproxy location. This seems a lot safer since it should be one equivalent image for another (just the source changes) | 17:44 |
clarkb | I guess I can check that the docekr hub hosted image hasn't changed since we mirrored it yseterday | 17:45 |
clarkb | heh it did update 19 hours ago | 17:45 |
clarkb | which I now need to go graph logs for and cross check | 17:45 |
opendevreview | James E. Blair proposed openstack/project-config master: Add zuul-providers project https://review.opendev.org/c/openstack/project-config/+/940540 | 17:46 |
opendevreview | James E. Blair proposed openstack/project-config master: Add zuul-providers to opendev and zuul tenants https://review.opendev.org/c/openstack/project-config/+/940541 | 17:46 |
clarkb | https://zuul.opendev.org/t/openstack/build/1b2b6d41e0a44d1b805aae2a2c8c269c/log/job-output.txt#554 this digest matches the digest at https://hub.docker.com/layers/library/haproxy/lts/images/sha256-75c6ef628f55d747c3ffe607dd84e0f8fd834f1c9e4a90ab3afdf722ad887273 so I think we are good to proceed. And even then haproxy is stateless so should be fine if we downgrade (whcih might not be | 17:48 |
clarkb | ok for mariadb) | 17:48 |
corvus | clarkb: fungi ^ i think we're ready to start doing a bit more with zuul-launcher nodes. those 2 changes will make a new project that we can include in all the tenants (just opendev and zuul for now though) that will hold the image/label/flavor/provider configs. | 17:50 |
corvus | i think we can keep the image build jobs in the opendev/zuul-jobs repo and keep that only in the opendev tenant | 17:51 |
corvus | with that in place, i think we can probably start maybe running some zuul unit test jobs on the new nodes | 17:51 |
fungi | lgtm | 17:51 |
clarkb | and the split is because we don't need every tenant running image build jobs but we do need to expose the images and labels etc to every tenant? | 17:51 |
fungi | also not sure if you saw my comment on the first one, but do we want it added to our gerritbot config too? | 17:55 |
opendevreview | Merged zuul/zuul-jobs master: Update ensure-twine role https://review.opendev.org/c/zuul/zuul-jobs/+/940074 | 18:10 |
opendevreview | Merged openstack/project-config master: Add zuul-providers project https://review.opendev.org/c/openstack/project-config/+/940540 | 18:11 |
clarkb | corvus: re ^ should we approve the change to add it to zuul as soon as it exists in gerrit or wait for some ocntent? (I guess that would probably require force pushing which we'd like to avoid so adding it to zuul as soon as it is in gerrit makes sense to me) | 18:12 |
corvus | clarkb: re split: yes exactly (we could filter by config object to exclude jobs, but also, we said that repo was for "opendev-tenant-specific jobs, so i think it makes sense to move this stuff to another repo | 18:32 |
corvus | clarkb: and yes, i think we can create first then add content | 18:32 |
corvus | fungi: i added it to the gerritbot config in the second change, just forgot to mention in the commit msg | 18:33 |
corvus | looks like it exists in gerrit, so i'll approve the next change | 18:34 |
fungi | oh, yep | 18:34 |
fungi | thanks! | 18:34 |
opendevreview | Merged openstack/project-config master: Add zuul-providers to opendev and zuul tenants https://review.opendev.org/c/openstack/project-config/+/940541 | 18:45 |
clarkb | makes sense | 18:51 |
clarkb | any opinions on https://review.opendev.org/c/opendev/system-config/+/940536 to change the haproxy source? | 18:54 |
corvus | +2 | 18:57 |
corvus | i think the blip is fine | 18:57 |
opendevreview | James E. Blair proposed opendev/zuul-jobs master: Remove provider configuration https://review.opendev.org/c/opendev/zuul-jobs/+/940543 | 18:59 |
opendevreview | James E. Blair proposed opendev/zuul-providers master: Add provider configuration https://review.opendev.org/c/opendev/zuul-providers/+/940544 | 18:59 |
corvus | that is an unexpected configuration error on 543 | 19:00 |
clarkb | corvus: the inaugust repos are in opendev's tenant | 19:00 |
clarkb | which would explain the relationship at least | 19:01 |
corvus | yeah, but for it to report means it's a new error.... | 19:01 |
corvus | i'm going to do a tenant reconfig on opendev to see if anything shows up in the static error list afterwords | 19:01 |
corvus | nope. | 19:02 |
corvus | however, after doing that, a recheck worked. | 19:03 |
clarkb | that seems even more unexpected? | 19:03 |
clarkb | like some sort of data inconsistency in the backend? | 19:04 |
fungi | also reminder that we've got some stuck gate and promote builds in the opendev tenant, indefinitely queued docs and noop jobs | 19:04 |
fungi | i never did find time to dig into those | 19:04 |
corvus | well, "no error" is the correct state, so i'm leaning towards "we just fixed some previously incorrect in-memory/zk state" | 19:04 |
corvus | fungi: yeah, that sort of thing seems to add to the "corrupt state" theory | 19:05 |
fungi | oldest are 90 days | 19:05 |
clarkb | ya I debated trying to dequeue them before but they didn't seem to be causing problems so left them as is | 19:05 |
clarkb | but maybe we should try and start over and see if problems crop back up again | 19:05 |
clarkb | its possible whatever caused that was in old zuul and isn't a problem anymore | 19:06 |
corvus | yeah my inclination would be to dequeue and observe. | 19:06 |
fungi | youngest is a little over a couple weeks, but all of them have other jobs queued up behind them waiting on those to complete | 19:06 |
corvus | if that's everyone else's feeling, i can go ahead and dequeue them while i'm here | 19:06 |
fungi | wfm | 19:06 |
fungi | all gone! | 19:08 |
clarkb | ++ | 19:08 |
corvus | #status log dequeued a handful of stuck changes in opendev tenant | 19:08 |
opendevstatus | corvus: finished logging | 19:08 |
corvus | i dequeued from bottom to top, just to avoid any extra weirdness. | 19:08 |
corvus | oh heh i forgot to add a pipeline config to zuul-providers | 19:09 |
fungi | d'oh! sorry i missed that when reviewing | 19:09 |
opendevreview | James E. Blair proposed opendev/zuul-providers master: Add provider configuration https://review.opendev.org/c/opendev/zuul-providers/+/940544 | 19:10 |
opendevreview | James E. Blair proposed opendev/zuul-providers master: Add initial zuul pipeline config https://review.opendev.org/c/opendev/zuul-providers/+/940545 | 19:10 |
corvus | hey, we should put that change in the jeepyb initial change creator thingy | 19:11 |
clarkb | the bare minimum to merge changes does seem like a good idea | 19:11 |
clarkb | fungi: you good with updating the haproxy source too? https://review.opendev.org/c/opendev/system-config/+/940536 | 19:13 |
clarkb | I'm around and can ensure thinsg restart properly / debug if necessary | 19:13 |
opendevreview | James E. Blair proposed opendev/jeepyb master: Create a stub .zuul.yaml https://review.opendev.org/c/opendev/jeepyb/+/940547 | 19:14 |
opendevreview | Merged opendev/zuul-providers master: Add initial zuul pipeline config https://review.opendev.org/c/opendev/zuul-providers/+/940545 | 19:15 |
clarkb | I believe that jeepyb change will actually create new projects in a test gerrit installation so it should have decent coverage, but I'll try to check in 20 minutes or however long that takes to confirm | 19:16 |
corvus | ok. i wrote that without mentally paging in all of jeepyb, so i'm kind of just assuming "do it next to where we write .gitreview" is the correct approach. | 19:17 |
clarkb | yup I believe that is the case | 19:17 |
opendevreview | Merged opendev/zuul-jobs master: Remove provider configuration https://review.opendev.org/c/opendev/zuul-jobs/+/940543 | 19:17 |
fungi | huh, opendev-zuul-jobs-noop actually runs things, i guess | 19:17 |
clarkb | its in the else block for "no other source of data is present" and we're starting from an empty repo case | 19:17 |
fungi | (contrary to its name) | 19:17 |
clayg | Howdy; I'd like some "infra" assistance with https://review.opendev.org/c/openstack/swift/+/836755 - which is currently set to "WIP" according to gerrit, but it's actively be worked on (by a new author) and we don't know how to change the status. I think I would have previously asked this question in #openstack-infra but maybe that's not a channel anymore? Regardless I see @clarkb here so I'm sure he'll tell me to talk to 😁 | 19:19 |
fungi | #openstack-infra is still an active channel too, yes | 19:19 |
clarkb | fungi uses the built in wip feature a lot (I don't). I know the original author can toggle it. Do we need to use admin bits to unset it? | 19:20 |
clayg | fungi: sorry, should I cross post there or is ok to discuss here? | 19:21 |
clarkb | here is fine | 19:21 |
corvus | fungi: i think it is a noop on a real node | 19:21 |
fungi | clayg: either is fine, #openstack-infra is more for openstack-specific talk, but we'll answer anywhere | 19:21 |
clayg | clarkb: we also feel like as an author we can change the setting - but we don't know how to contact the orig author - we have considered ditching the history/comments and starting a new change if we can't change the author | 19:21 |
clarkb | https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#set-ready-for-review I think we can use this rest api endpoint as an admin | 19:24 |
fungi | clayg: https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/swift.config is missing a togglewipstate perm like you see in https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/ironic.config#L14 | 19:24 |
corvus | the "delete provider" workflow needs some work (the zuul launcher is unhappy with having image build artifacts that belong to a provider it doesn't know about). | 19:24 |
clarkb | oh I guess as an alternative we update acls and let swift do it. That might be more future proof | 19:24 |
clarkb | corvus: do we think landing the change to readd that config will make it happy again in the short term? | 19:25 |
opendevreview | James E. Blair proposed opendev/zuul-jobs master: Revert "Remove provider configuration" https://review.opendev.org/c/opendev/zuul-jobs/+/940549 | 19:25 |
corvus | no, i think we should try a nodepool-style removal | 19:26 |
corvus | (because the change to re-add the config will add it with different canonical names, so it won't match) | 19:26 |
clarkb | gotcha so we need to revert, then delete the resources, then delete the config, then add new config elsewhere. | 19:27 |
clarkb | clayg: do you want to propose a change to the swift.config file fungi linked that updates it to be like ironic? We can get that landed pretty quickly then you should be able to remove the wip flag yourself | 19:28 |
opendevreview | James E. Blair proposed opendev/zuul-jobs master: Remove resources from providers https://review.opendev.org/c/opendev/zuul-jobs/+/940550 | 19:28 |
clarkb | corvus: looks like we may need to force merge the revert? | 19:28 |
opendevreview | James E. Blair proposed opendev/zuul-jobs master: Remove provider configuration https://review.opendev.org/c/opendev/zuul-jobs/+/940551 | 19:29 |
corvus | oh yeah, that's interesting. that seems like an error that should have shown up the first time. maybe that was being masked by the inaugust error. | 19:30 |
opendevreview | Clay Gerrard proposed openstack/project-config master: let core fix WIP state https://review.opendev.org/c/openstack/project-config/+/940552 | 19:30 |
corvus | there is a way to navigate that without a force-merge, but it would take all day because it involves running image build jobs. so i think we should just force it. | 19:31 |
opendevreview | Merged opendev/zuul-jobs master: Revert "Remove provider configuration" https://review.opendev.org/c/opendev/zuul-jobs/+/940549 | 19:32 |
clarkb | ya force merge makes sense to me too | 19:32 |
clarkb | its a super isolated area of the config too | 19:32 |
fungi | sounds fine to me | 19:33 |
clayg | clarkb: fungi: oh wow, thanks! I'll give it a little bit and see if it helps 🤞 | 19:35 |
opendevreview | James E. Blair proposed opendev/zuul-jobs master: Remove opendev-zuul-jobs-noop https://review.opendev.org/c/opendev/zuul-jobs/+/940554 | 19:35 |
corvus | https://review.opendev.org/940550 is ready now | 19:36 |
clarkb | approved | 19:37 |
clarkb | I +2'd the followup changes so you can approve when ready | 19:38 |
fungi | as have i | 19:38 |
opendevreview | Merged opendev/zuul-jobs master: Remove resources from providers https://review.opendev.org/c/opendev/zuul-jobs/+/940550 | 19:39 |
corvus | https://zuul.opendev.org/t/opendev/image/debian-bullseye looks empty, seems happy, i'll approve now | 19:40 |
opendevreview | Merged openstack/project-config master: let core fix WIP state https://review.opendev.org/c/openstack/project-config/+/940552 | 19:41 |
clarkb | I'm going to make a note about how we might want to toggleWipState across the board. It would still need to be a project by project basis though to get the perms correct (I'll put this on the meetup etherpad) | 19:42 |
opendevreview | Merged opendev/zuul-jobs master: Remove provider configuration https://review.opendev.org/c/opendev/zuul-jobs/+/940551 | 19:42 |
fungi | clarkb: sgtm, we could just script a bulk change | 19:46 |
opendevreview | Merged opendev/zuul-providers master: Add provider configuration https://review.opendev.org/c/opendev/zuul-providers/+/940544 | 19:52 |
clarkb | clayg: I think you should be able to toggle it back to active yourself now | 19:52 |
clarkb | you may need to hard refresh on the change page to reload things to show you the button to do that | 19:53 |
clarkb | I think it says something like "set active" ? | 19:53 |
clarkb | oh looks like you already did | 19:53 |
fungi | yeah, there was conversation in #openstack-swift | 19:54 |
clarkb | I was just catching up with the deployment job and was slower than yall | 19:55 |
clarkb | and now it is time to reheat some pizza for lunch | 19:55 |
clarkb | I'm still happy to keep an eye on haproxy things if we want to approve that. It runs the gitea job which takes about an hour so should be plenty of time to eat | 19:55 |
clayg | clarkb: yes i'm all squared - thank you so much! 5* rating on #openstack-infra support. Would buy again 😉 | 20:00 |
opendevreview | Clark Boylan proposed opendev/jeepyb master: Create a stub .zuul.yaml https://review.opendev.org/c/opendev/jeepyb/+/940547 | 20:49 |
opendevreview | Clark Boylan proposed opendev/jeepyb master: Fix E471 linter error https://review.opendev.org/c/opendev/jeepyb/+/940560 | 20:49 |
clarkb | corvus: ^ I went ahead and fixed the linter error unrelated to your change and stacked yours on top | 20:49 |
corvus | clarkb: thanks! | 20:49 |
corvus | clarkb: fungi i keep forgetting that the image build job *does* need to be in the same repo as the image definition... | 20:50 |
corvus | i think we should go ahead and pull the image build jobs into zuul-providers since i think it still makes sense to be there. | 20:50 |
clarkb | ok, then we selectively load the necessary bits into each tenant from zuul-providers? | 20:51 |
corvus | i'd love to have them split but i think there's a permissions situation that's resolved by having them together | 20:51 |
corvus | yeah | 20:51 |
opendevreview | James E. Blair proposed opendev/zuul-jobs master: Remove image build jobs https://review.opendev.org/c/opendev/zuul-jobs/+/940561 | 20:55 |
clarkb | I think we need to upodate the opendev tenant config for zuul-providers but not zuul tenant? | 20:56 |
corvus | yeah | 20:57 |
corvus | i'm working on re-encrypting the secret | 20:57 |
opendevreview | James E. Blair proposed opendev/zuul-providers master: Add image build jobs https://review.opendev.org/c/opendev/zuul-providers/+/940562 | 21:01 |
clarkb | corvus: I think you need to copy the playbooks too? | 21:02 |
corvus | heh yep | 21:02 |
clarkb | and maybe delete them from zuul-jobs side though that is less required | 21:02 |
corvus | clarkb: the tenant config is correct -- it already has jobs in the opendev tenant | 21:02 |
clarkb | ack | 21:03 |
opendevreview | James E. Blair proposed opendev/zuul-providers master: Add image build jobs https://review.opendev.org/c/opendev/zuul-providers/+/940562 | 21:04 |
opendevreview | Merged opendev/zuul-jobs master: Remove image build jobs https://review.opendev.org/c/opendev/zuul-jobs/+/940561 | 21:04 |
clarkb | oh wow that is a very large change now | 21:05 |
clarkb | I think I'm mostly going to trust that you copied things properly :) | 21:05 |
opendevreview | James E. Blair proposed opendev/zuul-providers master: Add image build jobs https://review.opendev.org/c/opendev/zuul-providers/+/940562 | 21:05 |
corvus | copied a few more things | 21:05 |
corvus | but yeah, all the dib-* and roles/* and playbooks/* are straight copies | 21:05 |
corvus | i omitted the noop.yaml playbook | 21:06 |
opendevreview | James E. Blair proposed opendev/zuul-jobs master: Cleanup image build playbooks, elements, and roles https://review.opendev.org/c/opendev/zuul-jobs/+/940563 | 21:06 |
clarkb | I still see no_log: true in at least one place I would expect it. But if you've done a straight copy we should have that in all the necesary places | 21:08 |
corvus | good idea. "git grep no_log" says the same things for the old repo and new: | 21:09 |
corvus | playbooks/opendev-build-diskimage-base/post-inner.yaml: no_log: true | 21:09 |
corvus | roles/image-upload-swift/tasks/main.yaml: no_log: true | 21:09 |
corvus | okay that stuff is working its way through gate (which includes an image build), so that'll be a bit | 21:10 |
opendevreview | Merged opendev/zuul-jobs master: Cleanup image build playbooks, elements, and roles https://review.opendev.org/c/opendev/zuul-jobs/+/940563 | 21:13 |
clarkb | corvus: reviewing testing I found a small issue in the jeepyb change if you want to fix that or let me know if I should edit it | 21:15 |
corvus | i'll take a look | 21:15 |
opendevreview | James E. Blair proposed opendev/jeepyb master: Create a stub .zuul.yaml https://review.opendev.org/c/opendev/jeepyb/+/940547 | 21:22 |
corvus | oh oops i think i undid your thing clark 1 sec | 21:23 |
clarkb | oh ya justneeds to be rebased again | 21:24 |
opendevreview | James E. Blair proposed opendev/jeepyb master: Create a stub .zuul.yaml https://review.opendev.org/c/opendev/jeepyb/+/940547 | 21:24 |
opendevreview | James E. Blair proposed opendev/zuul-providers master: Add image build jobs https://review.opendev.org/c/opendev/zuul-providers/+/940562 | 23:24 |
corvus | clarkb: ^ that required one env var change due to the name of the repo; i think that's trivial enough i'm going to self-reapprove. | 23:25 |
clarkb | I reviewed it quickly the delta diff was clear | 23:25 |
clarkb | but go ahead and approve | 23:25 |
corvus | kk thx | 23:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!