| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 04:45 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 04:48 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 04:50 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 04:50 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add newuidmap and podman https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 04:58 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add newuidmap and podman https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 05:00 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 05:01 |
| *** ralonsoh_out is now known as ralonsoh\ | 05:46 | |
| *** ralonsoh\ is now known as ralonsoh | 05:47 | |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism https://review.opendev.org/c/zuul/zuul-jobs/+/795336 | 07:16 |
| mnasiadka | clarkb: we need a new DIB release to switch rocky-container to quay.io (https://opendev.org/openstack/diskimage-builder/commit/cdaf45b9e00af4f4f29f80439abe11e55f18306f) - how do I ,,orchestrate'' this? | 07:34 |
| mnasiadka | IIRC release team does not do DIB releases | 07:34 |
| frickler | mnasiadka: there's a dib IRC channel, not sure who still hangs out there | 07:39 |
| mnasiadka | Well, usually ianw did the releases - I'm happy to help with that, but I guess I would need some additional rights to push tags to DIB repo :) | 07:39 |
| mnasiadka | Or we can ,,onboard'' DIB to openstack/releases repo under cycle independent | 07:40 |
| mnasiadka | but yes, let's move the discussion to DIB channel | 07:41 |
| frickler | I'm not there fwiw and it also doesn't seem to get logged. ping here again if you can make no progress | 07:41 |
| opendevreview | ribaudr proposed openstack/project-config master: Add team IRC ops for #openstack-nova https://review.opendev.org/c/openstack/project-config/+/949707 | 08:36 |
| *** ralonsoh_ is now known as ralonsoh | 09:21 | |
| opendevreview | Merged openstack/diskimage-builder master: Remove qemu-debootstrap from debootstrap element https://review.opendev.org/c/openstack/diskimage-builder/+/946550 | 10:38 |
| opendevreview | Merged openstack/diskimage-builder master: Remove the usage of pkg_resource https://review.opendev.org/c/openstack/diskimage-builder/+/933324 | 12:28 |
| opendevreview | Merged openstack/project-config master: Add team IRC ops for #openstack-nova https://review.opendev.org/c/openstack/project-config/+/949707 | 13:05 |
| opendevreview | Merged openstack/project-config master: to create a new repo for a cfn new launched sub-group heterogeneous distributed training framework https://review.opendev.org/c/openstack/project-config/+/949555 | 13:29 |
| fungi | https://superuser.openinfra.org/articles/opendev-and-rackspace-building-stronger-open-infrastructure-together/ | 13:31 |
| frickler | does ^^ mean that IPv6 is ready now? *scnr* | 13:37 |
| fungi | i read it as "real soon now" ;) | 13:39 |
| fungi | dan_with might know how close they are to dual-stack global networking | 13:39 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky 8/9 builds, labels and provider config https://review.opendev.org/c/opendev/zuul-providers/+/949696 | 13:52 |
| Clark[m] | mnasiadka: we don't run dib releases with Openstack releases because of chicken and egg problems/concerns. But ya someone in the release group needs to push a tag. I can do it if I ever dig my gpg key out of cold storage. Fungi did one once recently too iirc. | 13:52 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 13:52 |
| mnasiadka | Clark: ianw sorted it out on #openstack-dib - thanks :) | 13:53 |
| Clark[m] | Oh cool | 13:53 |
| fungi | yeah, there was a fair bit of discussion over there about the release process | 13:53 |
| mnasiadka | Well I think there was a couple of important patches in DIB since Dec 2024 (previous release) - so user-experience wise it would be good to release things more often :) | 13:54 |
| mnasiadka | at least rocky builds now use quay.io instead of docker hub | 13:54 |
| fungi | sure, also our nodepool/zuul deployments use released dib, so we can't take advantage of any changes until there's a release (which is why infra-core is included in diskimage-builder-release) | 14:00 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 14:02 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: ensure-dib: Add podman and rootlesskit https://review.opendev.org/c/zuul/zuul-jobs/+/949697 | 14:02 |
| clarkb | I'll approve teh gitea 1.23.8 update in a few minutes if there aer no objections | 14:42 |
| clarkb | I don't see anything in scrollback or email that would indicate I should not do this but let me know if I missed something | 14:43 |
| fungi | please do, i'm around all day | 14:47 |
| clarkb | done it is on its way in (which will probably take about an hour or maybe even a little more | 14:48 |
| mnasiadka | Ok then, the niz-rocky builds are on image conversion level so they will be good to go when it finishes - I'm off the hook :) | 14:49 |
| clarkb | mnasiadka: thanks again for getting those images sorted out | 14:56 |
| mnasiadka | no problem, happy to help - nice difference from chasing breakages in Kolla world ;-) | 15:05 |
| mnasiadka | Question to some more Gerrit-knowledgeable people - in Kolla we have Review-Priority and Backport-Candidate labels - is there a way that a vote on this label would override other votes? As in person A votes RP+1, another person wants to change that to -1 - and we end up with one +1 and one -1 - I'd like that to be more of a ,,label'' than a vote with only one value... | 15:18 |
| mnasiadka | well, than a vote with multiple values from multiple people | 15:19 |
| clarkb | I think hashtags are better suited to this problem | 15:19 |
| clarkb | I forget why others have said taht won't work for review priority though. Maybe it is because anyone can set hashtags if we open them up globally (we haevn't yet but that is the idea) | 15:20 |
| clarkb | but no in general individual own their votes. The actions you take on those votes can be scoped to specific groups or users, But I don't think we can generically say this vote goes away if someone else votes something different | 15:20 |
| mnasiadka | I think we have an ACL in Kolla that allows them only for core-reviewers - let me check | 15:20 |
| mnasiadka | well, hashtags sound like a better suited solution for that instead of standard voting mechanism | 15:22 |
| corvus | only other thing i'd say about votes is there are different options for calculating the winner (max with and without blocking, for example). not sure if that's flexible enough to accommodate what you want. but also, hashtags ftw. | 15:50 |
| fungi | right... it's in that category of technical solutions to social challenges: document the hashtags your project intends to use for specific situations, if someone misuses them then have a chat with that person about it, and if they're unreasonable then escalate it to project leadership/platform admins | 15:54 |
| fungi | micro-managing per-project access to set and remove hashtags is almost certainly an overoptimization | 15:55 |
| fungi | if a user starts abusing access by going around randomly setting or removing hashtags on projects, i have no qualms about disabling their account immediately | 15:56 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM Forced fail on Gerrit to test 3.11 upgrade and downgrade https://review.opendev.org/c/opendev/system-config/+/893571 | 16:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit images to 3.10.6 and 3.11.3 https://review.opendev.org/c/opendev/system-config/+/949778 | 16:04 |
| clarkb | there is a newer gerrit release for 3.10 and 3.11. I figure getting those updated is a godo step 0 before we start testing upgrade stuff. Then the second change there has a couple of holds in place to make testing of the upgrade easy | 16:06 |
| fungi | agreed | 16:07 |
| clarkb | gitea change should merge in a minute or two. Its uploading logs | 16:38 |
| opendevreview | Merged opendev/system-config master: Update to gitea 1.23.8 https://review.opendev.org/c/opendev/system-config/+/949544 | 16:39 |
| clarkb | and it is deploying now | 16:41 |
| clarkb | https://gitea09.opendev.org:3081/opendev/system-config/ is up and reports the expected version. The page rendered how I expect it. I'll do a clone test next | 16:44 |
| fungi | lgtm! | 16:44 |
| clarkb | clone works | 16:45 |
| fungi | 09 seems to be working | 16:45 |
| clarkb | ya so far it looks happy. We're through at least gitea11 at this point | 16:48 |
| clarkb | https://zuul.opendev.org/t/openstack/build/5585152355814cc089d9c1fdde0e2138 success and my checks of individual backeds look good | 16:54 |
| clarkb | infra-root checking the giteas I notice that gitea10-gitea14 report no space left on device for /var/log/syslog and the journals on May 10 (from dmesg -T). df -h reports plenty of disk now and we do seem to be writing to syslog and maybe the journal as well. Gitea seems to have up to date content too and / is not mounted ro. Also gitea09 doesn't seem to have been hit by this | 17:02 |
| clarkb | not sure what is going on there but it is weird enough that it may be worth someone else doing a quick check that there isn't anything terribly wrong we need to intervene for | 17:02 |
| clarkb | I'm beginning to suspect some temporary blip in storage for those servers and when storage resumed normal operations so did our servers | 17:03 |
| clarkb | I think if things were persistently sad the upgrade we just did would have failed (due to being unable to fetch and store new docker images) | 17:04 |
| clarkb | infra-root https://104.130.253.194/q/status:open+-is:wip is a held Gerrit 3.11 which we can use to interact with it and see that it works as expected. I also held a 3.10 node and that is the node I'll use to test the upgrade and downgrade process | 17:30 |
| clarkb | basically this 3.11 node should be safe to use at any time as a "what does 3.11 look like" check then the other node iswhere things will go up and down and change versions | 17:30 |
| clarkb | visually this doesn't seem all that different | 17:31 |
| fungi | the free space is enough that i doubt it dipped into root-only overhead during rotation | 17:44 |
| fungi | also 09 and 10 have basically identical utilization at the moment | 17:45 |
| clarkb | ya and it should rotate more regularly than only on the 10th and not since | 17:45 |
| clarkb | this is why I suspect something on the underlying cloud | 17:45 |
| fungi | agreed | 17:45 |
| corvus | clarkb: could check cacti graphs | 17:46 |
| clarkb | ah yup | 17:46 |
| clarkb | corvus: that was a great idea. Gitea10 does seem to have run out of disk on the 10th | 17:47 |
| clarkb | it was very sawtooth | 17:47 |
| clarkb | suddenly I'm reminded of the tarball generation problem and i think that must be what happened hwere | 17:47 |
| fungi | okay, so maybe rotation related after all | 17:47 |
| clarkb | we run a cron to prune those daily and that wasn't keeping up | 17:47 |
| fungi | oh! yes, tarballs | 17:48 |
| clarkb | fungi: ya not log rotation but rotation of the tarball artifacts | 17:48 |
| fungi | agreed, that would definitely explain it | 17:48 |
| fungi | and 09 just got lucky i guess | 17:48 |
| corvus | while /bin/true; rm tarballs; end | 17:48 |
| fungi | you forgot the "do" ;) | 17:48 |
| clarkb | ya I think we can probably just update the cron to run hourly or twice a day or similar | 17:48 |
| clarkb | I'll prep a change for that so we have it if it becomes useful again | 17:48 |
| clarkb | (right now things seems stable for the last few days) | 17:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run gitea tarball cleanup every 6 hours https://review.opendev.org/c/opendev/system-config/+/949790 | 17:56 |
| clarkb | in related news github announced anonymous request rate limit changes as even they are being crushed by the bots on the internet vying for AI supremacy | 17:57 |
| clarkb | looks like prior to the 10th it would get close to the limit but not exceed it | 18:01 |
| clarkb | then on the 10th we got "lucky" | 18:01 |
| clarkb | so ya I think 949790 should help mitigate for the future if we don't have better ideas (pretty sure I checked and we cannot disable this feature entirely otherwise I would) | 18:02 |
| opendevreview | Merged zuul/zuul-jobs master: Add a meta log upload role with a failover mechanism https://review.opendev.org/c/zuul/zuul-jobs/+/795336 | 18:16 |
| corvus | i think we could adapt ^ for use in our environment... but then we might not notice cloud storage failures.... | 18:18 |
| corvus | something to think about | 18:18 |
| clarkb | I guess we could generate a random order for the swift backends then pass that entire list to this new role? | 18:20 |
| clarkb | then as long as any one of them succeeds we would avoid job failures. With the current backends we use its likely that 3 fail or 2 fail given that 3 belong to one cloud and 2 to another. So we probably need at least 4 options and at that point you may as well use all 5 | 18:21 |
| fungi | reminiscent of (mike pondsmith/r. talsorian games) cyberpunk lore where where the old net was overrun by rogue ai systems so they had to build the blackwall to keep them from leaking into the new reconstructed net after the datakrash of 2022. even the timeline isn't too far off | 18:22 |
| mnasiadka | fungi: if hashtags are limited to core reviewers we should be fine ;) | 18:28 |
| mnasiadka | Ok - both niz-rocky patches have passed Zuul and are good to go https://review.opendev.org/q/topic:%22niz-rocky%22 | 18:29 |
| corvus | hashtags are very useful for non-core reviewers, i would encourage not limiting them. if it's important enough to restrict, then it should probably be a label (and then look into the submit rules) | 18:34 |
| fungi | mnasiadka: yeah, maybe you misread me. i said limiting hashtags to core reviewers is a wasteful overoptimization at best. if people misuse them (core or otherwise) then talk to them. if they won't listen, talk to us | 18:38 |
| fungi | people problems require people solutions | 18:38 |
| ildikov | Hi y'all, I'm reaching out about an error I got on my patch to update a StarlingX doc page: https://review.opendev.org/c/starlingx/governance/+/949746 | 19:02 |
| ildikov | If I read the logs correctly, it seems like a library issue and not an error in my patch. But I might've missed smth. Can someone please help me confirm one way or the other? | 19:03 |
| fungi | looking | 19:04 |
| Clark[m] | I think that is the problem that occurs when you run old flake8 on modern python (likely due to the node set being updated to noble) | 19:06 |
| Clark[m] | If you update flake8 it should fix the problem | 19:07 |
| ildikov | @Clark[m] great, that confirms my suspicion | 19:12 |
| ildikov | @Clark[m] sooo, how can I update flake8? :) | 19:13 |
| fungi | yeah, current flake8 per https://pypi.org/project/flake8/ is 7.2.0, that job seems to have installed 3.8.4 (from 2020) according to the log, it's probably using an old constraints file but i'm digging into that now | 19:15 |
| fungi | the log says it ran `pip install -U -r /home/zuul/src/opendev.org/starlingx/governance/test-requirements.txt` | 19:17 |
| fungi | so it's coming in as an indirect dependency via https://opendev.org/starlingx/governance/src/branch/master/test-requirements.txt#L1 | 19:17 |
| ildikov | ah, ok | 19:17 |
| ildikov | I checked that file, among other possible ones, and then realized I might need help figuring the dependency out | 19:18 |
| fungi | the current hacking version is 7.0.0 | 19:18 |
| fungi | just for reference | 19:18 |
| fungi | but i'd probably try to match to whatever version of hacking other starlingx projects are using | 19:19 |
| ildikov | ok, cool, I'll check | 19:19 |
| slampton | Hi, I'm an e-mail admin, troubleshooting the issue of not being able to receive e-mail from Gerrit (162.253.55.233). The thing is, we're receiving e-mail from openstack.org from that IP, but not from opendev.org. That suggests to me that you have a difference in either reverse DNS, SPF. DKIM, or DMARC records between those domains. | 19:28 |
| slampton | Proofpoint says they have removed the IP block, even though the lookup tool know gives an error. Can you provide smtp logs from your end, when sending from opendev.org to windriver.com? | 19:28 |
| fungi | slampton: i'll get those now, just a sec | 19:30 |
| slampton | thx | 19:30 |
| fungi | for the record, i requested removal of that address from proofpoint weeks ago and received no response, they seem to be completely unstaffed or otherwise defunct | 19:30 |
| slampton | that's ridiculous, they seem to be staffed just fine, if you are a paying customer | 19:33 |
| fungi | yeah, maybe they just ignore anything put into their removal request form. anyway i do see messages getting accepted right now in the log... (redacted) 2025-05-14 19:29:51 1uFHna-00000006wlT-0tv7 -> REDACTED@windriver.com R=dnslookup T=remote_smtp H=mxb-0064b401.gslb.pphosted.com [205.220.178.238] X=TLS1.2:ECDHE_SECP256R1__RSA_SHA256__AES_256_GCM:256 CV=yes C="250 2.0.0 46mbc8sm08-1 | 19:36 |
| fungi | Message accepted for delivery" | 19:36 |
| fungi | i'm looking to see when the last refusal was | 19:37 |
| mnasiadka | fungi: makes sense, after giving some thoughts we’re probably going to switch backport-candidate from review labels to use hashtags, but for review priority I would probably still stick with review labels (but think about giving the PTL/cores option to delete RP votes - because we sometimes end up with that semi-active person setting review priority - which needs to be lowered) | 19:37 |
| fungi | slampton: most recent rejection i see from pphosted.com in our logs was 2025-05-08 08:22:35 utc, almost a week ago now | 19:39 |
| fungi | there was an incident around 2025-05-12 02:38-04:00 with problems looking up the domain in dns, but that only resulted in messages being temporarily deferred | 19:41 |
| fungi | slampton: if messages aren't getting through for the past week, then my only guess is proofpoint has started 250/2.0.0 accepting the messages but then dumping them in the bitbucket | 19:42 |
| ildikov | I was finally able to find a hacking version that worked, thanks fungi for the help in figuring that out | 19:52 |
| fungi | ildikov: Clark[m] too, but you're welcome! | 19:59 |
| ildikov | yes, thanks Clark[m] too! | 19:59 |
| slampton | fungi: thanks, that correlates with the removal of the IP block. They still seem to be silently dropped, because I don't see anything in our logs from @opendev.org. I've asked Proofpoint to investigate further | 19:59 |
| clarkb | fungi: slampton: looking at my personal inbox I think the deliveries may be coming from review@openstack.org | 20:02 |
| fungi | yes, we don't have e-mail set up for the opendev.org domain for $reasons, so still use @openstack.org for the addresses | 20:02 |
| fungi | we want messages to be deliverable, therefore we only send from addresses at a domain that actually accepts smtp deliveries itself | 20:03 |
| fungi | confirmed, the sender (smtp "mail from") on those messages should be review@openstack.org | 20:06 |
| fungi | the rfc 822 "from:" header will also be review@openstack.org | 20:07 |
| fungi | though with varying names | 20:08 |
| mnaser | OSError: [Errno 39] Directory not empty: '/opt/stack/requirements/.eggs/pbr-6.1.1-py3.10.egg/pbr-6.1.1.dist-info' -> '/opt/stack/requirements/.eggs/pbr-6.1.1-py3.10.egg/EGG-INFO' | 20:28 |
| mnaser | Running setup.py install for openstack_requirements: finished with status 'error' | 20:28 |
| mnaser | waaah, do we just keep catching issues lol | 20:28 |
| mnaser | https://github.com/vexxhost/magnum-cluster-api/actions/runs/15030117460/job/42240444170#step:8:6705 | 20:30 |
| mnaser | https://github.com/pypa/setuptools/releases | 20:31 |
| mnaser | an hour ago a job passed | 20:31 |
| mnaser | trying to look at https://github.com/pypa/setuptools/compare/v80.5.0...v80.7.0 | 20:32 |
| slampton | fungi: aha, so we may be good then | 20:33 |
| mnaser | job passed with setuptools 80.4.0 | 20:34 |
| mnaser | so a bunch happened here https://github.com/pypa/setuptools/compare/v80.4.0...v80.7.0 | 20:35 |
| clarkb | mnaser: the code that raised the exception is 7 years old | 20:35 |
| fungi | but maybe not a code path we used to hit | 20:35 |
| clarkb | fungi: ya or something was previously ensuring the directory on the dst side was empty and now it isn't | 20:36 |
| clarkb | os.rename says you can't rename foo/ into bar/ if bar/ is not empty | 20:36 |
| fungi | also possible | 20:36 |
| mnaser | https://github.com/vexxhost/magnum-cluster-api/actions/runs/15029256236/job/42237574746#step:8:6576 installed just fine an hour ago | 20:37 |
| mnaser | where setuptools was 80.4.0 | 20:37 |
| clarkb | which is what I think happened here. Something had already written to bar/ (/opt/stack/requirements/.eggs/pbr-6.1.1-py3.10.egg/) and kaboom because it already had an EGG-INFO in it | 20:37 |
| fungi | i wonder if that's pbr's metadata writer for the pbr.json file | 20:38 |
| mnaser | is setuptools pinned in opendev? | 20:39 |
| fungi | depends on the project (whether it uss a pyproject.toml), phase (build vs install) and invoking tool (e.g. tox may install specific setuptools versions) | 20:40 |
| fungi | so the answer is "sometimes"? | 20:41 |
| mnaser | oh.. | 20:41 |
| mnaser | i think the flood gates of failure are about to come | 20:41 |
| mnaser | https://zuul.opendev.org/t/openstack/status see 949800,1 -- failed with same reason | 20:41 |
| mnaser | https://zuul.opendev.org/t/openstack/build/13528f8c1f99421391600a49673e66df | 20:41 |
| clarkb | fungi: I think that goes into a different file not EGG-INFO looks like it goes to pbr.json. However, it is defined as a egg_info.writers entrypoint | 20:42 |
| mnaser | im running a quick pip install in a docker container here to see whats in the folder | 20:44 |
| mnaser | https://paste.openstack.org/show/bvXJuXBeCevAgxY48IgH/ | 20:46 |
| mnaser | it looks like everything is identical except for a requires.txt in the EGG-INFO folder | 20:47 |
| mnaser | which contains `setuptools` | 20:47 |
| mnaser | so it feels like something is already generating everything in EGG-INFO and that copies is useless (and problematic) | 20:47 |
| clarkb | mnaser: thinking out loud here about how to debug this I think you want to figure out what creates the EGG-INFO dir with that content and see if that can be moved to later or if we can stop doing the rename as it may be redundant | 20:48 |
| mnaser | i'm trying to probe at this but it is very much out of my knowledge scope | 20:49 |
| mnaser | oh interesting | 20:50 |
| mnaser | i noticed `× python setup.py egg_info did not run successfully.` and when i ran it inside the repo, it fails | 20:50 |
| mnaser | if i `rm -rfv /opt/stack/requirements/.eggs` and run it, it works, but the second run around, it fails | 20:50 |
| mnaser | so its almost like maybe that function is meant to be idempotent (or maybe not meant to run more than once?) -- one or the other | 20:51 |
| clarkb | seems like it would be a bug to not be idempotent as you can't control whether or not people prebuild random things | 20:51 |
| mnaser | i can actually reproduce this same thing with zuul/zuul repo.. now they both use pbr, im trying to find a non-pbr project to see if its maybe pbr related | 20:52 |
| clarkb | mnaser: note you need a setuptools but not pbr package | 20:53 |
| mnaser | ah right yes | 20:53 |
| clarkb | (its possible to not use setuptools at all these days) | 20:53 |
| clarkb | I think doc8 may fit the bill but I haven't double checked to see if it is still using setuptools) | 20:54 |
| mnaser | they got rid of setup.py so i cant replicate that | 20:54 |
| mnaser | found https://github.com/fabric/fabric i can try on | 20:55 |
| mnaser | yea.. issue is not there | 20:55 |
| clarkb | its possible that the egg_info.writers entrypoint is side effecting things in ways we don't want | 20:56 |
| clarkb | mnaser: what you can do is clone pbr and remove the egg_info.writers entry from setup.cfg then build a wheel from that using itself. Then install that wheel and try to install something to see if it works | 20:56 |
| mnaser | lets try | 20:57 |
| clarkb | EGG-INFO doesn't show up in pbr at all though so my hunch is it is related to that entrypoint somehow | 20:57 |
| mnaser | yeah i searched the code for that too and indeed, silent | 20:57 |
| clarkb | oh wait | 20:57 |
| clarkb | we have LocalEggInfo too | 20:57 |
| mnaser | its still happening but i wonder if pip is doing an isolated build | 20:58 |
| mnaser | sorry nvm i am not using pip | 20:58 |
| clarkb | ok looking at your traceback it is specifically failing to install pbr as a setup requires | 21:02 |
| mnaser | oh, you're right | 21:02 |
| clarkb | bindep usese pbr via pyproject.toml and pep517. Its possible that would work while installing setup requires doesn't | 21:02 |
| clarkb | mnaser: if installing setup requires is the issue you can try to preinstall pbr first | 21:02 |
| clarkb | something like venv/bin/pip install pbr && venv/bin/pip install requirements or whatever you want to install | 21:03 |
| clarkb | I wonder if the problem here is that specific setup_requires path and not really pbr at fault as much as some issue with that system (it is still entirely possible that pbr is tickling some bug thourhg) | 21:03 |
| mnaser | with pbr preinstalled it also blows up | 21:04 |
| clarkb | is the traceback different? | 21:04 |
| mnaser | https://www.irccloud.com/pastebin/kbRG91J1/ | 21:04 |
| mnaser | same ol | 21:04 |
| clarkb | huh why is it trying to install setup_requires if they are already present | 21:05 |
| mnaser | https://github.com/pypa/setuptools/commits/main/setuptools/installer.py ther was some changes | 21:06 |
| mnaser | i wonder if its the avoidance of pkg_resources or soemthing | 21:06 |
| clarkb | I was able to install bindep in a venv with setuptools 8.7.0 without preinstalling anything else | 21:06 |
| mnaser | what about requirements? | 21:06 |
| clarkb | *setuptools 80.7.0 | 21:06 |
| clarkb | https://github.com/pypa/setuptools/commit/7379eaa957aaf4f2a01438066afb1674a64545f4 this does look suspcious | 21:07 |
| fungi | pbr did add setuptools as an explicit install_requires in a recent release, though that was months ago still | 21:08 |
| mnaser | actually as an exercise let me try the different setuptools versions and see which one fails | 21:08 |
| clarkb | mnaser: ^ that commit isn't in 80.6.0 | 21:08 |
| mnaser | might help narrow it down | 21:08 |
| clarkb | mnaser: ++ I was just going to suggest trying 8.6.0 | 21:08 |
| mnaser | ok 80.6.0 i can run the egg_info twice with no issues | 21:09 |
| mnaser | (or more than twice) | 21:09 |
| mnaser | 80.7.0 breaks | 21:10 |
| mnaser | https://github.com/pypa/setuptools/compare/v80.6.0...v80.7.0 so 13 commits to look through | 21:10 |
| clarkb | mnaser: the one I linked touches functions in your traceback which is why I called it suspicious | 21:11 |
| mnaser | https://github.com/pypa/setuptools/commit/7cb4c76468735ae69450a3693bed56217afe902c | 21:11 |
| clarkb | ya there is also that one | 21:12 |
| clarkb | unfortunately with 30 character commit messages it is hard to evaluate what the intent was... | 21:12 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Build images daily https://review.opendev.org/c/opendev/zuul-providers/+/949806 | 21:14 |
| clarkb | mnaser: theory: if 7379eaa it changes lookups for existing egg infos. That changed lookup would short circuit if found and return that existing info rather than writing new ones | 21:15 |
| mnaser | well i guess one way to find out | 21:16 |
| clarkb | mnaser: I suspect that the code that updated to find the egg info is buggy and it isn't finding the existing egg anymore so it ends up writing a new one later when it shouldn't | 21:16 |
| clarkb | and then ti goes kaboom | 21:16 |
| mnaser | checkout setuptools locally, revert that patch and install | 21:16 |
| mnaser | lemme try | 21:16 |
| clarkb | mnaser: ya you may also be able to just edit the setuptools code and see if ti fall through around like 90ish here https://github.com/pypa/setuptools/commit/7379eaa957aaf4f2a01438066afb1674a64545f4 | 21:16 |
| clarkb | spefically linke 93 on the new side return egg_dist. I don't think that is happening anymore and thus it falls through to line 121 on the new side which is what ultimately explodes | 21:17 |
| clarkb | My hunch is the old code was returning on line 93 because it found the egg and not falling through | 21:17 |
| mnaser | https://github.com/pypa/setuptools/commit/7379eaa957aaf4f2a01438066afb1674a64545f4 conflicts to revert, so i reverted https://github.com/pypa/setuptools/commit/7cb4c76468735ae69450a3693bed56217afe902c first and no go | 21:18 |
| mnaser | so donig the second one now | 21:18 |
| mnaser | https://github.com/pypa/setuptools/commit/7cb4c76468735ae69450a3693bed56217afe902c fixed it | 21:19 |
| mnaser | gonna try to revert that one _alone_ since i had to do both in order to avoid conflict | 21:19 |
| clarkb | mnaser: your second revert is the same hash as the first. DId you mean 7379eaa? | 21:19 |
| mnaser | sorry, yes, thats right, the second revert fixed it | 21:20 |
| clarkb | fwiw I strongly suspect that code is simply buggy allowing it fall through then explode and I don't think that would be a pbr issue | 21:20 |
| clarkb | its just that pbr is tripping over it | 21:20 |
| clarkb | (because it is a setup_requires?) | 21:20 |
| mnaser | yeah reverting 7379eaa957aaf4f2a01438066afb1674a64545f4 alnoe fixes it | 21:24 |
| mnaser | so.. now what :) | 21:24 |
| mnaser | clarkb: lol https://github.com/pypa/setuptools/issues/4998 | 21:25 |
| fungi | patch or revert as a setuptools pr | 21:25 |
| fungi | or at least that yep | 21:25 |
| fungi | parallel discovery and research is also great confirmation | 21:25 |
| clarkb | if that issue wasn't already up I'd suggest that we edit the function to confirm it is falling through on the second pass | 21:25 |
| clarkb | then file an issue. But since there is already an issue. I think its time for a drink and doing something not python related? | 21:26 |
| mnaser | i dont know if we want to give some sort of heads up so people dont recheck and eat up endless ci resources? | 21:28 |
| clarkb | we could do something like #status notice Setuptools issue 4998 is affecting python CI jobs. Hold off on rechecks until setuptools 80.7.0 is pulled or an 80.8.0 includes a fix. | 21:28 |
| fungi | i'm about to sit down to winner, but sgtm yep | 21:29 |
| clarkb | fungi: that notice specifically is good? | 21:29 |
| fungi | yes, though also now the setuptools release has been yanked | 21:29 |
| mnaser | oh nice | 21:29 |
| mnaser | yeah i was just gonna say that | 21:29 |
| fungi | breaking news! | 21:29 |
| fungi | ;) | 21:29 |
| fungi | (pun intended) | 21:30 |
| clarkb | in that case maybe #status notice Setuptools 80.7.0 broke python package installs for many affecting CI jobs. That release has been yanked and it should be safe to recheck failed changes. | 21:31 |
| fungi | wfm | 21:51 |
| clarkb | #status notice Setuptools 80.7.0 broke python package installs for many affecting CI jobs. That release has been yanked and it should be safe to recheck failed changes. | 21:59 |
| opendevstatus | clarkb: sending notice | 21:59 |
| -opendevstatus- NOTICE: Setuptools 80.7.0 broke python package installs for many affecting CI jobs. That release has been yanked and it should be safe to recheck failed changes. | 21:59 | |
| opendevstatus | clarkb: finished sending notice | 22:02 |
| JayF | https://bugs.launchpad.net/pbr/+bug/2107732 was never triaged and I saw it referenced in a setuptools issue | 22:45 |
| JayF | probably worth a look | 22:45 |
| JayF | it's nice that they are reaching out | 22:45 |
| clarkb | JayF: oslo is responsible for pbr fwiw | 22:46 |
| JayF | Ah. Makes sense. I'll link it over there. | 22:46 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!