| @jim:acmegating.com | Clark: questions on your questions on https://review.opendev.org/848234 | 00:12 |
|---|---|---|
| @clarkb:matrix.org | > <@jim:acmegating.com> Clark: questions on your questions on https://review.opendev.org/848234 | 00:25 |
| Responded, does that make more sense? | ||
| @clarkb:matrix.org | oh wait I see an error ya I think my simplification over simplified | 00:26 |
| @clarkb:matrix.org | let me update | 00:26 |
| @clarkb:matrix.org | and updated | 00:29 |
| @jim:acmegating.com | Clark: that seems plausible :) | 00:45 |
| -@gerrit:opendev.org- Zuul merged on behalf of Rajesh Tailor: [zuul/zuul] 847946: Fix typo from 'execption' to 'exception' https://review.opendev.org/c/zuul/zuul/+/847946 | 01:07 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 848137: Fix exception in blob store cleanup https://review.opendev.org/c/zuul/zuul/+/848137 | 01:35 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Artem Goncharov: [zuul/zuul] 817393: Fix gitlab squash merge https://review.opendev.org/c/zuul/zuul/+/817393 | 02:44 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/nodepool] 848341: Fix race in test_aws_resource_cleanup https://review.opendev.org/c/zuul/nodepool/+/848341 | 03:05 | |
| @chgans:matrix.org | does nodepool support autoscaling out of the box? eg. i want to have up to 10 builders (CPU/IO intensive) and up to 10 "generic" nodes (cheap), and i want these nodes to be kept around and deleted after a grace period. AFAIU, it seems to be supported, eg using labels in config (https://zuul-ci.org/docs/nodepool/latest/configuration.html#attr-labels) and matching them in (AWS) driver https://zuul-ci.org/docs/nodepool/latest/aws.html#attr-providers.[aws].pools.labels. Is this correct? | 04:58 |
| -@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/nodepool] 847698: dib: Bump to 3.22.0 https://review.opendev.org/c/zuul/nodepool/+/847698 | 05:08 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 848234: Fix read-only branches error in zuul-web https://review.opendev.org/c/zuul/zuul/+/848234 | 07:37 | |
| -@gerrit:opendev.org- Zuul merged on behalf of lotorev vitaly: [zuul/zuul] 844742: Replace 'web' section with 'webclient' https://review.opendev.org/c/zuul/zuul/+/844742 | 08:11 | |
| @q:fricklercloud.de | What does the community think about doing regular meetings? Maybe not necessarily weekly, but bi-weekly or monthly? I have the impression that a lot of topics just float by in the chat here and get ignored and lost, so having an occasion where more concentrated focus could be given to them might be useful. | 08:29 |
| Slightly related, how about planning a virtual PTG session loosely coupled to the OpenInfra PTG? | ||
| @mhuin:matrix.org | > <@q:fricklercloud.de> What does the community think about doing regular meetings? Maybe not necessarily weekly, but bi-weekly or monthly? I have the impression that a lot of topics just float by in the chat here and get ignored and lost, so having an occasion where more concentrated focus could be given to them might be useful. | 09:14 |
| > Slightly related, how about planning a virtual PTG session loosely coupled to the OpenInfra PTG? | ||
| That, or moving some discussions tu zuul-discuss | ||
| @q:fricklercloud.de | Hmm, that doesn't seem to be very active either, but I'll give it a try | 09:53 |
| -@gerrit:opendev.org- Zuul merged on behalf of Guillaume Chauvel: [zuul/zuul] 735475: Fix zuul_run_command ret when command is not found https://review.opendev.org/c/zuul/zuul/+/735475 | 10:08 | |
| @westphahl:matrix.org | Clark, corvus : question on https://review.opendev.org/848234 re. the update logic in general + addressed the comments | 13:02 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 846055: Validate dependent job names https://review.opendev.org/c/zuul/zuul/+/846055 | 13:09 | |
| @jim:acmegating.com | Christian Gagneraud: yep, that's the idea :) | 14:00 |
| @jim:acmegating.com | frickler: meetings can be useful when there's ongoing development coordination needed for specific topics. i don't think zuul is in that place, and the mailing list or chat is sufficient for where we are now. | 14:01 |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 848132: zuul-web: return errors as JSON https://review.opendev.org/c/zuul/zuul/+/848132 | 14:08 | |
| @fungicide:matrix.org | > <@chgans:matrix.org> does nodepool support autoscaling out of the box? eg. i want to have up to 10 builders (CPU/IO intensive) and up to 10 "generic" nodes (cheap), and i want these nodes to be kept around and deleted after a grace period. AFAIU, it seems to be supported, eg using labels in config (https://zuul-ci.org/docs/nodepool/latest/configuration.html#attr-labels) and matching them in (AWS) driver https://zuul-ci.org/docs/nodepool/latest/aws.html#attr-providers.[aws].pools.labels. Is this correct? | 14:26 |
| nodepool is primarily designed to elastically provide resources as they're needed, so scales the pool up and down to fit the workload (within the maximum you set in configuration and any quotas in the providers). it's not really meant to "keep things around" after they get allocated to a job and it completes. the idea is that you can keep only the resources your jobs need in the provider and then tear them down as soon as possible in order to minimize things like billable time | ||
| @jpew:matrix.org | Our gate pipeline has a lot of expensive "build" jobs, and then a few longer "test" jobs that are dependencies of the "build" jobs, and we've noticed that behavior Zuul/nodepool uses to satisfy node requests is suboptimal for maximizing throughput. The issue is that many build jobs queue up and request nodes (mode than nodepool can provide). nodepool seems to satisfy requests in more-or-less FIFO order. Thus when the "test" jobs for the change at the top of the gate queue are finally able to run, they have to wait for all of the currently queued "build" node requests before they can run. When this is combined with the gate queue "window", this means we aren't optimally using cluster resources because we hit the window limit while the top change is still waiting for it's test jobs to get assigned a node. It ends up with a "sawtooth" node usage pattern; once the top change gates, the window extendes to the next change and more "builds" start, but the problem starts with the next buildsets "test" jobs.... it would be nice if jobs that had a dependency could "cut" in line to get a build node before jobs that don't have any dependencies | 14:27 |
| @jpew:matrix.org | Or even just be able to statically assign the node allocation priority from the job (NodeSet?) | 14:31 |
| @jim:acmegating.com | jpew: i think the first thing makes the most sense. since that's not implemented, you might consider tweaking the window size. also, there are some tricks you could do with job.hold-following-changes, but that's likely to slow things down more unless you have just the right job timing. | 14:39 |
| @jpew:matrix.org | Ya, I started looking at the code.... apparently there is an undocumented option called `use_relative_priority` which sets the node allocation priority based on the position of the change in the gate queue | 14:41 |
| @jim:acmegating.com | that's not what you want | 14:41 |
| @jim:acmegating.com | jpew: actually, the thing you want may already be implemented -- child jobs do get a priority bump | 14:43 |
| @jim:acmegating.com | the interactions between all these things, priority, window, launcher queues, may be more complex | 14:44 |
| @jim:acmegating.com | jpew: oh, it's only for paused parents | 14:44 |
| @jim:acmegating.com | not complete parents | 14:44 |
| @jim:acmegating.com | so yeah, my original response stands :) | 14:44 |
| @jpew:matrix.org | So the naive approach is to set the priority to `len(parents)` (or `-len(parents)`... not sure which way the priority works). | 14:46 |
| @jim:acmegating.com | lower better | 14:46 |
| @jpew:matrix.org | Is that approach sufficient, and does it need a flag to control it? | 14:47 |
| @jpew:matrix.org | (as an aside, what is use_relative_priority for) | 14:48 |
| @jim:acmegating.com | jpew: honestly i don't know right now, i don't have a lot of time to dig into the nuance. i know that someone is going to need to do a lot of thinking about that before we merge anything though, because that could radically change behavior. i would not think we would want a flag though. | 14:50 |
| @jpew:matrix.org | corvus: Fair enough | 14:51 |
| @jim:acmegating.com | jpew: use_relative_priority adjusts priority across tenants | 14:51 |
| @jim:acmegating.com | it prevents one tenant from monopolizing resources compared to others | 14:51 |
| @jim:acmegating.com | jpew: having said that.... there is a possibility that it may unintentionally help your situation | 14:52 |
| @jim:acmegating.com | (just because of the way it's implemented) | 14:53 |
| @jim:acmegating.com | jpew: ah no, the quantization will neuter it for that use case | 14:53 |
| @jim:acmegating.com | yeah, actually it might help | 14:54 |
| @jim:acmegating.com | maybe give it a try and see :) | 14:54 |
| @jpew:matrix.org | corvus: Ah, ya. We probably want that anyway because we have a lot of tenants on the same cluster | 14:54 |
| @jim:acmegating.com | the quantization means it's only effective for the top 10 changes | 14:55 |
| @jim:acmegating.com | i think we may want to make it the default | 14:55 |
| @jpew:matrix.org | That's fine, as long as the top few changes keep moving it will help us greatly. If/we the behavior breaks on us (since it isn't the intended purpose) we can revisit something based on number of parents | 14:56 |
| @jim:acmegating.com | jpew: cool, please report back your observations :) | 14:56 |
| @jpew:matrix.org | * That's fine, as long as the top few changes keep moving it will help us greatly. If/when the behavior breaks on us (since it isn't the intended purpose) we can revisit something based on number of parents | 14:56 |
| @jpew:matrix.org | corvus: Ya, I'll try to remember to do that :) | 14:57 |
| @jim:acmegating.com | jpew: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/zuul/templates/zuul.conf.j2#L5 | 14:57 |
| @jpew:matrix.org | AH, cool. It's nice to know that's already used in production | 14:58 |
| @jpew:matrix.org | Perhaps I'll add some documentation | 14:58 |
| @jim:acmegating.com | jpew: not necessary :) https://zuul-ci.org/docs/zuul/latest/configuration.html#attr-scheduler.relative_priority | 14:58 |
| @jpew:matrix.org | Oh, hah | 14:59 |
| @jim:acmegating.com | jpew: the behavior is also within a tenant... so it'll even out requests among different projects within the same tenant. sorry my description here was incomplete, but the docs are better ;) | 15:01 |
| @jim:acmegating.com | the way i look at it is: it slows down the second and following changes for each project, so that every project's first change gets nodes first. so if your project uploads one change at a time periodically, you will see fast results. if you upload hundreds of changes to your project all at once, you will be delayed. it's idea of "fair" is that every project gets to make progress on at least one change at the same time. | 15:04 |
| @jpew:matrix.org | Yep | 15:04 |
| @jim:acmegating.com | swest: Clark responded on https://review.opendev.org/848234 (sorry, it took a bit to collect my thoughts on that) | 15:30 |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 848132: zuul-web: return errors as JSON https://review.opendev.org/c/zuul/zuul/+/848132 | 15:41 | |
| @clarkb:matrix.org | > <@jim:acmegating.com> swest: Clark responded on https://review.opendev.org/848234 (sorry, it took a bit to collect my thoughts on that) | 16:05 |
| I'm trying to understand if your comment about updateProjectBranches() implies a need for change. Specifically around improving updateProjectBranches() to handle the None case for all branches if we don't actually need all_branches to be fetched and set. It seems if we have an error and return None once. Then from that point forward we'd continue to track all branches even if possibly unnecessary? | ||
| @jim:acmegating.com | Clark: i was trying to say that no change is necessary, and expand on the lifecycle of that since swest asked about it | 16:06 |
| @clarkb:matrix.org | got it | 16:08 |
| @clarkb:matrix.org | and I guess no vote yet to further discuss the LookupError | 16:08 |
| @jim:acmegating.com | Clark: and to put that another way: updateProjectBranches only needs to run a query if there is an existing cache entry. if there is not an existing cache entry (for whatever reason) it should not perform a query. | 16:09 |
| @jim:acmegating.com | (for further clarity: a cache entry of the empty list *is* a cache entry) | 16:10 |
| @clarkb:matrix.org | oh got it ya the check is not None | 16:10 |
| @clarkb:matrix.org | so an empty list of branches still counts | 16:10 |
| @jim:acmegating.com | (it's just that "no entry because it's unused, and no entry because it failed are equivalent to the logic in that method, and that is correct and should not be changed) | 16:10 |
| @clarkb:matrix.org | right | 16:10 |
| @jim:acmegating.com | and yeah, if you're on board, i think i'm at +1 on the change, pending swest's thoughts on the exception handling. also, i think i want to throw some more tests at it over the weekend. | 16:11 |
| @clarkb:matrix.org | Sorry if I'm catching up slowly. I've had a very slow start to the day. If I was canadian I'd have a good excuse; unfortunately, I am not | 16:11 |
| @jim:acmegating.com | Clark: but at least you apologized? :) | 16:12 |
| @clarkb:matrix.org | yes I think the change makes sense to me now that set(None) and the confusing return logic have been handled | 16:12 |
| @clarkb:matrix.org | Happy Canada Day! :) | 16:12 |
| @jim:acmegating.com | oh. i also think maybe some more comments would be good. i'll leave a note. | 16:12 |
| @clarkb:matrix.org | frickler: I feel like most of the communication in here tends to be focused on code review discussion, debugging, or general information gathering (for example the question about nodepool autoscaling). Of those I feel the first two do aid from synchronous discussion, but also don't need to be scheduled. Maybe we could be better about keeping track of the issues people are having with deployments to try and ease deployment friction? For general information gathering I think that can be asynchronous on eg the mailing list. Then we try to send email for things that need broader consensus (see my email about dropping platform version defaults in zuul job roles for an example). But those aren't served by synchronous meetings. The one thing I could see synchronous meetings helping with is checkups on progress towards major goals, but I'm not sure Zuul is in a spot where that is required right now? Even zuulv5 was done without that. | 16:38 |
| @clarkb:matrix.org | I guess I'm wondering if there is a specific need that isn't being addressed that you ahve in mind that I am missing that would be aided by meetings | 16:38 |
| @clarkb:matrix.org | > <@fungicide:matrix.org> nodepool is primarily designed to elastically provide resources as they're needed, so scales the pool up and down to fit the workload (within the maximum you set in configuration and any quotas in the providers). it's not really meant to "keep things around" after they get allocated to a job and it completes. the idea is that you can keep only the resources your jobs need in the provider and then tear them down as soon as possible in order to minimize things like billable time | 16:43 |
| There is a min-ready setting (https://zuul-ci.org/docs/nodepool/latest/configuration.html#attr-labels.min-ready) to instruct Nodepool to keep a reasonable number of nodes ready to go at any time (note this will be backfilled as demand uses nodes so if max-servers is > min-ready you'll have in use + min-ready present). There is also a max-ready-age setting (https://zuul-ci.org/docs/nodepool/latest/configuration.html#attr-labels.max-ready-age) which will delete old min-ready servers if they aren't used but they will then be backfilled too so this may or may not be what was intended for the original question. | ||
| @avass:vassast.org | > <@clarkb:matrix.org> frickler: I feel like most of the communication in here tends to be focused on code review discussion, debugging, or general information gathering (for example the question about nodepool autoscaling). Of those I feel the first two do aid from synchronous discussion, but also don't need to be scheduled. Maybe we could be better about keeping track of the issues people are having with deployments to try and ease deployment friction? For general information gathering I think that can be asynchronous on eg the mailing list. Then we try to send email for things that need broader consensus (see my email about dropping platform version defaults in zuul job roles for an example). But those aren't served by synchronous meetings. The one thing I could see synchronous meetings helping with is checkups on progress towards major goals, but I'm not sure Zuul is in a spot where that is required right now? Even zuulv5 was done without that. | 16:44 |
| If I compare to GitHub projects i feel like opendev (or maybe just zuul) is missing GitHubs equivalent of an "issues" page. I know storyboard exists, but it's a bit disconnected to the repository and the discussions and doesn't seem to be used a lot (or maybe I'm just bad at checking it?), so it feels underutilized. | ||
| @clarkb:matrix.org | > <@avass:vassast.org> If I compare to GitHub projects i feel like opendev (or maybe just zuul) is missing GitHubs equivalent of an "issues" page. I know storyboard exists, but it's a bit disconnected to the repository and the discussions and doesn't seem to be used a lot (or maybe I'm just bad at checking it?), so it feels underutilized. | 16:46 |
| Ya I think that the story around bug tracking could be better. That said I personally really dislike how project issue trackers get used like a mailing list or forum. It pollutes the list of things that are known as defects that need trackign with discussions about other stuff and it is hard as a casual user to determine if something I've experiencing is already known. | ||
| @clarkb:matrix.org | It also leads to an expectation that more casual contributors need to subscribe to the issue tracker for all the things. And I personally find that very overwhelming and just turn it off instead | 16:47 |
| @fungicide:matrix.org | the opendev sysadmins are definitely looking for help with storyboard, right now the version of the software we're hosting is outdated and missing useful features because we need to find time to rewrite our deployment automation for it | 16:47 |
| @avass:vassast.org | I took a quick look at zuuls docs and i couldn't see storyboard mentioned unless i checked "vulnerability reporting": | 16:50 |
| https://zuul-ci.org/docs/zuul/latest/vulnerabilities.html | ||
| So maybe it's a good idea to mention storyboard on the frontpage of the docs at least to encourage it a bit? | ||
| @avass:vassast.org | * I took a quick look at zuuls docs and i couldn't see storyboard mentioned unless i checked "vulnerability reporting": | 16:51 |
| https://zuul-ci.org/docs/zuul/latest/vulnerabilities.html | ||
| So maybe it's a good idea to mention storyboard on the frontpage of the docs at least to promote it a bit? | ||
| @clarkb:matrix.org | There is a bugs link in the footer of the website which goes there | 16:51 |
| @clarkb:matrix.org | But ya maybe we need to make it more prominent? | 16:51 |
| @jim:acmegating.com | it's on every page, and has a section in the community page | 16:52 |
| @jim:acmegating.com | i'm not sure we should try to funnel more people to storyboard | 16:52 |
| @avass:vassast.org | I think that footer is a bit too out of sight :) | 16:52 |
| @jim:acmegating.com | it's an issue tracking system, which is a place to organize information around issues. it's not a support forum | 16:52 |
| @jim:acmegating.com | so like, if you want to report a bug, i'm pretty sure you'll be able to find the place to do it | 16:52 |
| @jim:acmegating.com | (i can confirm that plenty of people are able to find it) | 16:52 |
| @jim:acmegating.com | i don't think we should encourage folks to start filing issues every time they have a question about zuul | 16:53 |
| @jim:acmegating.com | that will not improve their experience | 16:53 |
| @avass:vassast.org | But i meant the sphinx docs specifically, it's easier to find on the main site. | 16:53 |
| @jim:acmegating.com | > <@avass:vassast.org> But i meant the sphinx docs specifically, it's easier to find on the main site. | 16:54 |
| er, you're saying the docs aren't prominent enough? sorry i thought you meant the bug tracker link | ||
| @avass:vassast.org | I think that maybe it's a good idea for this page to mention storyboard: | 16:54 |
| https://zuul-ci.org/docs/zuul/latest/ | ||
| @jim:acmegating.com | ah, okay. wfm. | 16:55 |
| @avass:vassast.org | I think I've mentally filed https://zuul-ci.org/ as an ad for the product and https://zuul-ci.org/docs/zuul/latest/ as technically information that i actually want, so i personally escape the main site as fast as possible if i ever end up there :) | 16:58 |
| @jim:acmegating.com | aww :( | 16:58 |
| @jim:acmegating.com | and here i treat it as an index for all things zuul related | 16:59 |
| @avass:vassast.org | I think i use the documentation page as an index for the component i want to read about though | 17:02 |
| @fungicide:matrix.org | note that the zuul docs link doesn't provide an index of the client, or nodepool, or other parts of zuul | 17:13 |
| @fungicide:matrix.org | (though they can be found in the drop-down, or at the grandparent level url) | 17:14 |
| @fungicide:matrix.org | also i wholeheartedly agree that the defect and task tracker should not be turned into a support request/trouble ticket system. not only is that software not designed for such a use case, organized software support is something for companies to provide and not an "assignment" of the project's contributors | 17:16 |
| -@gerrit:opendev.org- Zuul merged on behalf of Kenny Ho: [zuul/zuul] 837063: Add build set URL to reporter https://review.opendev.org/c/zuul/zuul/+/837063 | 17:35 | |
| @avass:vassast.org | > <@fungicide:matrix.org> note that the zuul docs link doesn't provide an index of the client, or nodepool, or other parts of zuul | 17:42 |
| Uh yeah i meant this "Documentation" page: https://zuul-ci.org/docs/ | ||
| I realize that was unclear from my side | ||
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 848331: Update to python3.10 https://review.opendev.org/c/zuul/nodepool/+/848331 | 17:46 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 848331: Update to python3.10 https://review.opendev.org/c/zuul/nodepool/+/848331 | 17:49 | |
| @clarkb:matrix.org | So I actually think https://review.opendev.org/q/topic:python3.10 might be reviewable at this point. The dependencies for nodepool that gave us compilation struggles on arm64 now appear to all publish wheels to pypi | 17:50 |
| @clarkb:matrix.org | That means we can drop the reliance on the opendev openstack wheel mirror entirely and fetch wheels from pypi. | 17:50 |
| @clarkb:matrix.org | Selfishly I'd like to get us all the way up to python3.11, but that doesn't happen until october. Python3.10 seems like a good first step on that path | 17:51 |
| @fungicide:matrix.org | we might be able to install https://packages.debian.org/python3.11-full (currently in sid) | 18:01 |
| @clarkb:matrix.org | I'm not sure we should ship production zuul images based on beta python | 18:01 |
| @clarkb:matrix.org | I'm happy to wait for the full 3.11 release | 18:01 |
| @fungicide:matrix.org | oh, no i just meant if you want to test with it | 18:01 |
| @clarkb:matrix.org | oh i see | 18:02 |
| @clarkb:matrix.org | There are docker images for 3.11 that opendev could add at this point | 18:02 |
| @clarkb:matrix.org | its not availability that is the issue, more that I'm wary of pushing it into production too soon | 18:02 |
| @fungicide:matrix.org | oh, absolutely | 18:02 |
| @clarkb:matrix.org | but ya maybe running unittests against it and seeing if it improves runtimes is worthwhile | 18:02 |
| @clarkb:matrix.org | Once the current runs complete (for more data gathering) I've got some updates locally I'll push which convert more jobs to python3.10 | 18:25 |
| @clarkb:matrix.org | that shoudl give us a good amount of coverage | 18:25 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 840356: Bump up to python3.10 https://review.opendev.org/c/zuul/zuul/+/840356 | 18:58 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 848331: Update to python3.10 https://review.opendev.org/c/zuul/nodepool/+/848331 | 19:32 | |
| @clarkb:matrix.org | This is interesting running a pip installed `virtualenv` on ubuntu-jammy results in a venv with no venv/bin/ instead it is venv/local/bin | 19:59 |
| @clarkb:matrix.org | fungi: ^ do you know if that is a new virtualenv behavior affecting older debuntu or if that is new to jammy? | 19:59 |
| @clarkb:matrix.org | I'm about to try and reproduce on focal | 19:59 |
| @clarkb:matrix.org | Ok it does not reproduce with the same virtualenv bootstrapping on focal so something to do with python on jammy | 20:04 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 840356: Bump up to python3.10 https://review.opendev.org/c/zuul/zuul/+/840356 | 20:17 | |
| @clarkb:matrix.org | Well now I am extremely confused. I spun up a new container to try and reproduce the original failures and now it won't even reproduce the old behavior of stashing things in venv/local/bin | 20:58 |
| @clarkb:matrix.org | but it seems that when things are stashed in venv/local/bin the venv acts as if system-site-packages is true even though it is explcitly set to false in it sconfig file | 20:58 |
| @clarkb:matrix.org | that issue is leading to the current retry failures | 20:58 |
| @clarkb:matrix.org | and now I've managed to reproduce in a new container not changing any process that I know of | 21:02 |
| @clarkb:matrix.org | now to see if I can reproduce without zuul installations | 21:03 |
| @clarkb:matrix.org | https://paste.opendev.org/show/blwHnssUggeWQ8CEvJrj/ thats a more minimal reproducer. I have to look at my container that doesn't reproduce more closely to see what might be different | 21:07 |
| @clarkb:matrix.org | oh wait I already deleted that container arg | 21:08 |
| @clarkb:matrix.org | I think that I maybe missed the `pip install -U pip setuptools wheel` step on the one that was happy. So I was using the distro pip the entire time. Something about updating distro pip maybe? | 21:09 |
| @clarkb:matrix.org | Ok using distro pip to install virtualenv produces working virtualenvs | 21:12 |
| @clarkb:matrix.org | Pretty sure the issue is related to `pip install -U pip` | 21:12 |
| @clarkb:matrix.org | ok no it is setuptools | 21:17 |
| @clarkb:matrix.org | everything works after I `pip install -U pip` but the instant I `pip install -U setuptools` the next virtualenv creation breaks | 21:18 |
| @fungicide:matrix.org | i wonder if it's using the debian-patched distutils which is how they override `sudo pip install ...` to install into /usr/local instead of just /usr | 21:24 |
| @fungicide:matrix.org | setuptools semi-recently embedded a copy of distutils in preparation for it being removed from the stdlib, right? | 21:24 |
| @clarkb:matrix.org | Any setuptools >=60 seems to cause this to happen | 21:26 |
| @clarkb:matrix.org | oh ya you think maybe set the flag to undo that? You know I wonder if I hit this with devstack jammy python3.10 work. | 21:27 |
| @fungicide:matrix.org | https://setuptools.pypa.io/en/latest/history.html#v60-0-0 | 21:27 |
| @fungicide:matrix.org | that's the release where distutils absorbing happened | 21:28 |
| @clarkb:matrix.org | Installing setuptools==62.6.0, setting that env var, then running virtualenv produces a working venv | 21:29 |
| @clarkb:matrix.org | So ya it must be that | 21:29 |
| @clarkb:matrix.org | I can't imagine that virtualenv not actually producing a virtualenv is behavior pypa wants, but I alsos suepct that if I file this as a bug I'll get yelled at for using pip and setuptools wrong | 21:30 |
| @fungicide:matrix.org | well, if you're using distro patched toolchains, then yes the distro-patched distutils is required to work with them. if you use the distutils embedded in new versions of setuptools, you need all upstream toolchain not distro patched anything, from what i've gathered | 21:33 |
| @clarkb:matrix.org | sure, the problem is there is no indication that any of this might be happening and when you make a virtualenv it does the oppposite it just aliases the global pyhton | 21:34 |
| @clarkb:matrix.org | at the very least virtualenv shoudl fail | 21:34 |
| @clarkb:matrix.org | but I know that if I try to argue that upstream is unlikely to listen | 21:34 |
| @fungicide:matrix.org | yeah, i agree it's a tangled mess | 21:34 |
| @clarkb:matrix.org | if virtualenv cannot make a virtualenv then it should exit with an error. Not make a virtualenv like thing that pollutes your global instllation | 21:35 |
| @fungicide:matrix.org | and yes, if you want upstream to agree about anything you have to basically eschew distro everything these days | 21:35 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 840356: Bump up to python3.10 https://review.opendev.org/c/zuul/zuul/+/840356 | 21:37 | |
| @clarkb:matrix.org | I expect that might actually pass now with the env var set | 21:37 |
| @clarkb:matrix.org | But I'm not sure we want to merge it in that state? Its probably fine for testing but I dunno it feels wrong | 21:37 |
| @clarkb:matrix.org | > <@fungicide:matrix.org> and yes, if you want upstream to agree about anything you have to basically eschew distro everything these days | 21:38 |
| Ya I'm mostly ok with that with a couple of exceptions. One, I'm still using the distro to bootstrap typically. This specific issue is that. But also regardless of distro vs not the tool should do what it says on the tin or fail. I feel like thats basic user expectations and doesn't matter where you sourced it. | ||
| @fungicide:matrix.org | yeah, you might be able to argue the bug angle with the virtualenv maintainer | 21:39 |
| @fungicide:matrix.org | also, if we don't setenv `VIRTUALENV_CREATOR = venv` then that might be worth a try | 21:40 |
| @fungicide:matrix.org | that's the option that tells virtualenv to just be a passthrough to venv when available | 21:40 |
| @clarkb:matrix.org | we don't because venv doesn't work for other reasons | 21:41 |
| @clarkb:matrix.org | there are comments about this in the code | 21:41 |
| @clarkb:matrix.org | has to do with running the manage-ansible command itself out of a virtualenv | 21:41 |
| @fungicide:matrix.org | ahh, okay | 21:42 |
| @clarkb:matrix.org | that problem may be solveable though I'm not sure | 21:43 |
| @clarkb:matrix.org | maybe that is the better long term option we should be looking at though. | 21:43 |
| @clarkb:matrix.org | since that will always be using the distro provided venv module that comes with the python install if you are using distro python | 21:43 |
| @clarkb:matrix.org | hrm the resulting installation is still broken on that change but it gets past the install step at least | 21:46 |
| @clarkb:matrix.org | Anyway i need to take a break after melting my brain against python and pip and setuptools all afternoon so far | 21:46 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 840356: Bump up to python3.10 https://review.opendev.org/c/zuul/zuul/+/840356 | 23:00 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!