| @iwienand:matrix.org | https://zuul.opendev.org/t/zuul/build/cf3d04e3029741d48251464a0daad7fd failed to build the nodepool image with a 429 rate limit from dockerhub. probably an ephemeral error, but not sure we've seen much of that. perhaps the host was previously doing some other job that was pulling a lot. | 00:37 |
|---|---|---|
| @iwienand:matrix.org | > <@clarkb:matrix.org> are there kubernetes users that might want to weigh in on https://review.opendev.org/c/zuul/zuul/+/867136 and https://review.opendev.org/c/zuul/zuul/+/867137/ ? I'm interested in the image size savings but I think updating the client is potentially a win too? | 00:47 |
| i'm not technically a user, but i've fiddled with the jobs and the osc before, particularly 853914 and it lgtm. along with k8s jobs, i think we need to have a discussion about our openshift testing, because it's stuck on centos-7 without a way off it, afaics | ||
| @iwienand:matrix.org | Clark: if you have a sec for https://review.opendev.org/c/zuul/nodepool/+/867590/ and https://review.opendev.org/867579 that would be good to get the dib gate moving again | 01:36 |
| -@gerrit:opendev.org- Anders Hanson marked as active: [zuul/nodepool] 867744: Add support for Kubernetes 1.24+ https://review.opendev.org/c/zuul/nodepool/+/867744 | 05:46 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 867175: Reuse queue items after reconfiguration https://review.opendev.org/c/zuul/zuul/+/867175 | 07:20 | |
| -@gerrit:opendev.org- Fabien Boucher proposed: [zuul/zuul] 867804: [gitlab driver] Handle missing diff_refs attribute https://review.opendev.org/c/zuul/zuul/+/867804 | 13:36 | |
| -@gerrit:opendev.org- Fabien Boucher proposed: [zuul/zuul] 867805: [gitlab driver] fix "'EnqueueEvent' object has no attribute 'change_url'" https://review.opendev.org/c/zuul/zuul/+/867805 | 14:47 | |
| -@gerrit:opendev.org- Fabien Boucher proposed: [zuul/zuul] 851008: fix "'EnqueueEvent' object has no attribute 'change_url'" https://review.opendev.org/c/zuul/zuul/+/851008 | 14:56 | |
| -@gerrit:opendev.org- Fabien Boucher proposed: [zuul/zuul] 851008: [pagure driver] fix "'EnqueueEvent' object has no attribute 'change_url'" https://review.opendev.org/c/zuul/zuul/+/851008 | 14:57 | |
| @jim:acmegating.com | ianw: fungi left an important comment on 867590 that would be good to address in a followup | 15:46 |
| @clarkb:matrix.org | ya I was just looking at that. Looks like podman and the containernetworking package are synced between the two distro versions currently | 15:50 |
| @fungicide:matrix.org | right, if we pull from future-stable then we shield ourselves from accidentally being impacted by or growing dependent on newer versions after the next release | 16:26 |
| -@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/nodepool] 867590: Dockerfile: use containernetworking-plugins https://review.opendev.org/c/zuul/nodepool/+/867590 | 16:35 | |
| @clarkb:matrix.org | I'm looking at tox v4 compat for zuul and friends. The main thing I'm running into trying understand is zuul/zuul/tox.ini's use of skipsdist and usedevelop | 17:20 |
| @clarkb:matrix.org | tox v4 seems to treat skipsdist as more of a "skip install of repo as package" directive which causes usedevelop to be ignored | 17:20 |
| @clarkb:matrix.org | I also think the use of the install command in zuul (which runs a script to build web things) may cause use develop to be functionally ignored? | 17:21 |
| @clarkb:matrix.org | I'm going to remove skipsdist which will cause usedevelop to be used maybe? But reviewers should look at that aspect of the change closely I guess | 17:21 |
| @jim:acmegating.com | Clark: i haven't put much effort into nox, but i think it's viable. do you want to try that, or stick with tox? | 17:24 |
| @clarkb:matrix.org | I think nox is good for a longer term migration? I'm just trying to avoid zuul having trouble on the 22nd when we land that zuul-jobs change | 17:25 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 867829: Add tox v4 compatibility https://review.opendev.org/c/zuul/zuul/+/867829 | 17:25 | |
| @clarkb:matrix.org | and in that context the sdist behavior probbaly doesn't matter much | 17:25 |
| @jim:acmegating.com | we could always pin it and then nox. but either way works for me. :) | 17:25 |
| @clarkb:matrix.org | my main concern with pinning is that a new developer trying to run zuul tests might not be aware that is necessary | 17:26 |
| @clarkb:matrix.org | but ya that would keep CI happy | 17:26 |
| @jim:acmegating.com | Clark: why the depends-on? | 17:28 |
| @clarkb:matrix.org | corvus: to be sure that it works in CI with newer tox. I think we can drop that if it checks clean then merge today | 17:28 |
| @jim:acmegating.com | gotcha | 17:29 |
| @clarkb:matrix.org | I think nodepool might be ok actually. It does set skipsdist, but none of the other known incompatibilities seem to be there | 17:29 |
| @jim:acmegating.com | Clark: why would skipsdisk cause usedevelop to be ignored in zuul but not nodepool? | 17:30 |
| @jim:acmegating.com | oh because of install_command? | 17:30 |
| @clarkb:matrix.org | I think it is ignored in both. The problem I have is I'm not sure what is more correct here. SKipping the sdist and install altogether or doing an install. In the zuul case I think it doesn't actually develp beacuse it uses the install script. In nodepool it would do a pip install -e if we drop skipsdist | 17:32 |
| @clarkb:matrix.org | corvus: when skipsdist is set I think what happens is we don't install the project at all. The zuul change I just pushed should help clarify some of that behavior as I think the zuul install needs to happen for web testing | 17:35 |
| @clarkb:matrix.org | hrm even with skipsdist unset we aren't installing zuul which is causing things to break | 17:38 |
| @clarkb:matrix.org | oh! because install_command is overriding things maybe? | 17:39 |
| @clarkb:matrix.org | I wonder if you use install_command now if you need to explicitly list the current repo as a requirement? | 17:39 |
| @clarkb:matrix.org | this is a failure mode I didn't anticipate and now I'm extraconfused | 17:41 |
| @clarkb:matrix.org | fungi: ^ fyi | 17:41 |
| @clarkb:matrix.org | hrm I think I get it. I think it is using install_command to install the package too and that is breaking due to how zuu has defined it maybe | 17:42 |
| @clarkb:matrix.org | whereas in the past tox ignored install_command for installing the actual package | 17:42 |
| @jim:acmegating.com | Clark: the old tox3 behavior is that it would run 2 commands: | 17:43 |
| `/usr/bin/bash tools/pip.sh -r/tmp/zuul/requirements.txt -r/tmp/zuul/test-requirements.txt` | ||
| then | ||
| `/usr/bin/bash tools/pip.sh --exists-action w -e .` | ||
| @jim:acmegating.com | those are at the head of py310-1.log and py310-2.log respectively | 17:44 |
| @clarkb:matrix.org | https://zuul.opendev.org/t/zuul/build/3c3e7e0c8e1b413a825b800b6fa303f5/log/job-output.txt#1712 appears to be the new thing that is breaking tools/pip.sh | 17:45 |
| @clarkb:matrix.org | earlier in the log it does the equivalent of `/usr/bin/bash tools/pip.sh -r/tmp/zuul/requirements.txt -r/tmp/zuul/test-requirements.txt` but then it does an install breaking out all the deps which trips our check to install ansible but we haven't installed zuul yet | 17:45 |
| @clarkb:matrix.org | I think this is due to the new isolated build env. | 17:46 |
| @clarkb:matrix.org | Its fetching and building all the deps using the command we expect in an isolated build env. Then it is listing everything that produced and installing them against but via explicit listing | 17:46 |
| @jim:acmegating.com | Clark: want to add a "set -x" to pip.sh to see what's failing? | 17:47 |
| @jim:acmegating.com | (i agree, zuul-manage-ansible seems the most likely, but maybe it could be `pip check`?) | 17:47 |
| @clarkb:matrix.org | corvus: it is this: https://zuul.opendev.org/t/zuul/build/3c3e7e0c8e1b413a825b800b6fa303f5/log/job-output.txt#1803 because we haven't installed zuul yet | 17:47 |
| @jim:acmegating.com | ah missed that | 17:48 |
| @clarkb:matrix.org | but the guard around that command isn't detecting that properly. I think a better check is simply checking if the command is in our path yet | 17:48 |
| @clarkb:matrix.org | Once it is in our path we should be good to run it | 17:48 |
| @jim:acmegating.com | Clark: presumably there will be a 3rd command, and that's what we need to check for with tox4? | 17:48 |
| @clarkb:matrix.org | corvus: yup after the deps are installed it should get around to installing zuul itself and thats the bit I'm not sure about (I think it will pass the -e and maybe it will just work) | 17:49 |
| @jim:acmegating.com | (if we're lucky, we could check for something like "--exists-action" or "-e") | 17:49 |
| @jim:acmegating.com | Clark: maybe easiest thing is to just comment out the zuul-manage-ansible and see what the logs say | 17:50 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 867829: Add tox v4 compatibility https://review.opendev.org/c/zuul/zuul/+/867829 | 17:52 | |
| @clarkb:matrix.org | I think ^ should get us further | 17:52 |
| @clarkb:matrix.org | Unrelated: pypi tweeted out some graphs showing their python3.10 -> python3.11 switch. In particular their cpu utilization dropped. But it made me realize Zuul (at least in OpenDev) switched to python3.11 in prod as quickly as pypi did :) | 17:57 |
| @fungicide:matrix.org | trying to catch up, but even in tox v3 i think people were often abusing usedevelop to mean "install my project in this testenv" rather than for its intended purpose (making sure that the project is installed in "editable mode"). if the intent is simply to make sure the project is installed into a testenv then an explicit {toxinidir} in the testenv's deps list is the clearest way to achieve that. skipsdist continues to be a useful way to say "by default don't build an sdist of this project" but seems to also be mutually exclusive of usedevelop | 18:02 |
| @clarkb:matrix.org | fungi: I don't think adding your project to the deps list works the way you want with usedevelop either | 18:05 |
| @fungicide:matrix.org | for personal projects where i over-optimize testenv specific deps lists, i set skipsdist=true in the global [tox] section and then deps={toxinidir} in [testenv] so that i can later override it with a different deps= instead of inheriting that, e.g. deps=bandit in my [testenv:bandit] section | 18:05 |
| @clarkb:matrix.org | fungi: that will cause the project to be installed normally and then you'll do an editable install over the top of it | 18:06 |
| @clarkb:matrix.org | It looks like my latest patchset managed to get through the installation phase. Now we just need tomake sense of what all it did | 18:06 |
| @fungicide:matrix.org | yes, well to be clear i also find develop/editable installs inherently broken, and the pip/setuptools devs keep trying to kill them because they're too hard to support reliably | 18:06 |
| @fungicide:matrix.org | pretty much every discussion about overhauling packaging winds up with "oh but we have to support editable installs so that's not going to work" | 18:07 |
| @fungicide:matrix.org | pretty sure the writing's on the wall | 18:08 |
| @clarkb:matrix.org | `py311: 213340 W install_package> bash tools/pip.sh --force-reinstall --no-deps /home/zuul/src/opendev.org/zuul/zuul/.tox/.tmp/package/1/zuul-8.0.2.dev54-0.editable-py3-none-any.whl` is what it ended up doing to install zuul. And that ran zuul-manage-ansible again but that only took a couple seconds as it finds the existing install from the previous step | 18:08 |
| @clarkb:matrix.org | I think the patchset I've written most closely approximates the old behavior in that it doesn't seem to sdist and does seem to do an editable install | 18:08 |
| @clarkb:matrix.org | I can remove the depends on if that passes and we can land it early if others agree. | 18:09 |
| @clarkb:matrix.org | Then I guess we should drop skipsdist from the nodepool tox.ini too | 18:09 |
| @fungicide:matrix.org | i think continuing to try to make things work the way they did is fine, i just would strongly advise new projects not to rely on editable installs because they're increasingly treated like a corner case/second class | 18:10 |
| @clarkb:matrix.org | ya I think changing that in zuul is a good next step. But it may not be immediately trivial due to the way the web build works? | 18:11 |
| @clarkb:matrix.org | basically that is orthogonal to tox v3 and v4 compat and should happen separately to enure we are happy with the results | 18:11 |
| @clarkb:matrix.org | and can keep track of all the moving parts | 18:11 |
| @fungicide:matrix.org | for example, pep 517 support in setuptools lagged behind for a year because they couldn't figure out how to support editable mode, and it seemed like they were about to throw in the towel at that point and just declare there would be no future support for it | 18:12 |
| @fungicide:matrix.org | but somebody who really felt strongly about continuing to keep it supported stepped up at the last moment to implement that | 18:12 |
| @clarkb:matrix.org | `/home/zuul/src/opendev.org/zuul/zuul/.tox/py311/lib/python3.10/site-packages/kazoo/handlers/utils.py:225: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated` ok that isn't right | 18:13 |
| @clarkb:matrix.org | .tox/py311/lib/python3.10 wut | 18:13 |
| @clarkb:matrix.org | that doesn't appear to happen under tox 3 looking at some older job runs | 18:14 |
| @clarkb:matrix.org | I need to step out for a bit, but I guess I should try to come up with some trivial reproductions of that | 18:14 |
| @clarkb:matrix.org | But it almost looks like tox doesn't know how to select the correct python version anymore :/ | 18:15 |
| @fungicide:matrix.org | https://docs.python.org/3/library/ssl.html#ssl.PROTOCOL_TLS "Deprecated since version 3.10: TLS clients and servers require different default settings for secure communication. The generic TLS protocol constant is deprecated in favor of PROTOCOL_TLS_CLIENT and PROTOCOL_TLS_SERVER." | 18:16 |
| @fungicide:matrix.org | oh, i see, using python 3.10 to run the py311 build | 18:17 |
| @clarkb:matrix.org | fungi: ya the problem isn't the deprecation. THe problem is its a pyhton 3.11 tox target running under python3.10 | 18:17 |
| @clarkb:matrix.org | kazoo will need an update separately | 18:17 |
| @fungicide:matrix.org | agreed, there were changes around how python versions are selected, i need to look back at the changelog | 18:17 |
| @fungicide:matrix.org | Clark: see "Rework how we handle Python packaging environments" at https://tox.wiki/en/latest/changelog.html#bugfixes-4-0-0a3 | 18:24 |
| @fungicide:matrix.org | i think that's what we're seeing | 18:24 |
| @fungicide:matrix.org | the interpreter version selected for the testenv is not necessarily the one used for package build backends | 18:24 |
| @hanson76:matrix.org | Clark: ianw Looks I managed to get https://review.opendev.org/c/zuul/nodepool/+/867744 to pass all the tests, both in the change itself and in https://review.opendev.org/c/zuul/nodepool/+/867745 | 18:24 |
| Had been nice if someone could review it, I tried to change as little as possible, had to add one new method in FakeCoreClient for the tests to pass, except the actual fix in the kubernetes driver itself. | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 847387: Revert "Pin minikube to 1.25.2" https://review.opendev.org/c/zuul/nodepool/+/847387 | 18:47 | |
| @jim:acmegating.com | Hanson: i rebased that on your change ^ | 18:47 |
| @jim:acmegating.com | > <@fungicide:matrix.org> the interpreter version selected for the testenv is not necessarily the one used for package build backends | 18:49 |
| that sounds less than ideal | ||
| @fungicide:matrix.org | > <@jim:acmegating.com> that sounds less than ideal | 19:05 |
| my best guess at the reasoning is that since the results are cached and reused outside of the context of that particular testenv, they decided having deps randomly built by an arbitrary interpreter version was wrong and that they should instead be consistently built by the interpreter version tox itself is installed on | ||
| @clarkb:matrix.org | fungi: the traceback I pasted is from the test not the build | 19:05 |
| @fungicide:matrix.org | Clark: a local run or is there a build log you can link? | 19:10 |
| @fungicide:matrix.org | and ignore_basepython_conflict is on, right? | 19:11 |
| @fungicide:matrix.org | assuming basepython is set at all | 19:12 |
| @clarkb:matrix.org | fungi: https://zuul.opendev.org/t/zuul/build/ba117e2372a8480e9ca0c0fc26431344/log/job-output.txt#2255 this is all with zuul in zuul | 19:16 |
| @clarkb:matrix.org | ignore_basepython_conflict is set but python 3.11 is installed so that should cause this to happen. It may explain why it doesn't error though | 19:17 |
| @clarkb:matrix.org | Is the issue that basepython = python3 is no longer determining which python to execute in the venv and it is informing the python to install to the venv? | 19:18 |
| @clarkb:matrix.org | If so this may be our first tox v3 vs v4 difference that we can't easily reconcile in the general case | 19:19 |
| @clarkb:matrix.org | Reading the docs that may be it. I'm not sure I like this change for projects that need python2 and python3 support concurrently but should be fine for zuul. I'll get a new patchsets up shortly to have it check. Need to figure out why linters timed out first | 19:21 |
| @fungicide:matrix.org | Clark: looks like line 815 is where this maybe starts going wrong | 19:22 |
| @fungicide:matrix.org | `.pkg: 231 D discover exe for PythonInfo(spec=CPython3.10.6.final.0-64, exe=/home/zuul/.local/tox/bin/python3, platform=linux, `version='3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]', encoding_fs_io=utf-8-utf-8) in /usr [virtualenv/discovery/py_info.py:437]`` | 19:23 |
| @clarkb:matrix.org | Ya so it might be finding `python3` there and calling it good. With old tox it would only fall back to that if the env specific version isn't found | 19:24 |
| @clarkb:matrix.org | But now it seems to prefer base python first. I don't like this change but also know it is unlikely upstream will care so we can just drop basepython | 19:25 |
| @clarkb:matrix.org | It is mostly there to assist the python2 to 3 transition but we are past that | 19:25 |
| @clarkb:matrix.org | Looking at the linters job it complains about multiple packages in a flat layout. That is likely due to removing skipsdist and setting usedevelop false on the linters | 19:26 |
| @clarkb:matrix.org | I think we can continue to skip the sdist there which would preserve old behavior | 19:26 |
| @fungicide:matrix.org | yes, setuptools got picky about subdirectories in your package which don't have an __init__.py basically | 19:27 |
| @fungicide:matrix.org | * yes, setuptools got picky about subdirectories in your package which don't have an ``__init__.py`` basically | 19:27 |
| @clarkb:matrix.org | Ya but it looks like a warning that tox is the treating as an error? And for some reason doesn't actually exit properly and the job times out | 19:28 |
| @clarkb:matrix.org | The docker image installs should check that a proper install works though | 19:29 |
| @clarkb:matrix.org | I have had to look at fixing this issue in other contexts in the past and always find it confusing the docs are not clear at all. | 19:29 |
| @clarkb:matrix.org | It isn't clear to me why it is breaking suddenly now since setup tools hasn't changed versions? Maybe new tox forces newer setup tools? | 19:30 |
| @fungicide:matrix.org | so yeah, i'm starting to thing this is a regression in the basepython behavior. it used to be that basepython=python3 meant "use whatever the python3 installed in the venv is" but now it looks like that may be reinterpreted as "use whatever python3 outside your venv is" | 19:30 |
| @fungicide:matrix.org | * so yeah, i'm starting to think this is a regression in the basepython behavior. it used to be that basepython=python3 meant "use whatever the python3 installed in the venv is" but now it looks like that may be reinterpreted as "use whatever python3 outside your venv is" | 19:30 |
| @clarkb:matrix.org | Yup | 19:30 |
| @clarkb:matrix.org | And didn't influence venv build selection | 19:31 |
| @clarkb:matrix.org | Only process execution | 19:31 |
| @fungicide:matrix.org | https://tox.wiki/en/latest/config.html#basepython seems to suggest that dropping use of basepython is preferable, but also requires that your project and your version of tox both support your default python version | 19:34 |
| @fungicide:matrix.org | in openstack we added that originally to force py3k for tests so that tox wouldn't incorrectly run unversioned testenvs with python 2.7 | 19:36 |
| @fungicide:matrix.org | maybe not so important today? | 19:36 |
| @clarkb:matrix.org | Ya I think for zuul at least we can drop it | 19:36 |
| @clarkb:matrix.org | We don't python2 | 19:36 |
| @clarkb:matrix.org | And it is likely anyone doing python2 has tox installed with python3 and goes the other direction these days | 19:37 |
| @fungicide:matrix.org | > <@clarkb:matrix.org> It isn't clear to me why it is breaking suddenly now since setup tools hasn't changed versions? Maybe new tox forces newer setup tools? | 19:41 |
| for that, i think it's that tox switched to using setuptools as a pep 517 build backend when you ask for an editable install, and setuptools only tuens on most of that package discovery breakage when you use it as a pep 517 build backend | ||
| @clarkb:matrix.org | I see | 19:41 |
| @fungicide:matrix.org | at least i recall having to fix it up in one of my projects to make pyproject.toml stuff work | 19:41 |
| @clarkb:matrix.org | do you understand what setuptools is asking for/ this isn't the first time I've read the docs on this and felt like they don't actually explain the functionality | 19:42 |
| @fungicide:matrix.org | telling tox to do editable-legacy should avoid it | 19:43 |
| @fungicide:matrix.org | but otherwise i think setuptools wants you to explicitly declare those directories/their contents as "data" | 19:44 |
| @fungicide:matrix.org | otherwise its discovery assumes you want them to be importable python packages | 19:44 |
| @clarkb:matrix.org | this isn't editable, it is doing an sdist | 19:44 |
| @clarkb:matrix.org | the other jobs works beacue tehy don't go down the editable path which seems fine | 19:45 |
| @fungicide:matrix.org | ahh, then that may simply be newer setuptools behavior | 19:45 |
| @clarkb:matrix.org | This is probably the 20th time I've read https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#finding-simple-packages and it still makes no sense to me | 19:46 |
| @clarkb:matrix.org | do you know if editing MANIFEST.in is stillvalid? | 19:47 |
| @clarkb:matrix.org | but also we build zuul with what I believe is pretty up to date setuptools in our docker image builds and that isn't editable and its fine | 19:49 |
| @fungicide:matrix.org | i think supplying a manifest is supposed to override the discovery, but doesn't prevent it from running | 19:49 |
| @clarkb:matrix.org | I wonder why it is breaking here | 19:49 |
| @clarkb:matrix.org | I guess what we needto know is how to exress that web/ releasenotes/ etc/ and tool/s should be included like docs/ | 19:51 |
| @clarkb:matrix.org | but I have absolutely no idea how to do that after reading the setuptools docs because while they say you can do this they do absolutely nothing to show you how | 19:51 |
| @clarkb:matrix.org | fungi: I've just created a new virtualenv and updated `pip wheel setuptools build` to latest. Then ran `python -m build /path/to/zuul/repo` and that does not error | 19:56 |
| @clarkb:matrix.org | it doesn't even emit a warning | 19:56 |
| @fungicide:matrix.org | "When using tool.setuptools.packages.find in pyproject.toml, setuptools will consider implicit namespaces by default when scanning your project directory. To avoid pkg.namespace from being added to your package list you can set namespaces = false. This will prevent any folder without an __init__.py file from being scanned." | 19:56 |
| @fungicide:matrix.org | if memory serves, that's the relevant bit | 19:57 |
| @fungicide:matrix.org | as far as what setuptools changes about discovery when used as a pep 517 build backend | 19:58 |
| @clarkb:matrix.org | but zuul isn't a pyproject.toml project and running the canonical pep 517 tool `build` produces no errors. I feel like I'm missing something | 19:59 |
| @fungicide:matrix.org | build doesn't assume you have a pep 517 build backend, it just supports having one (as opposed to calling `setup.py bdist_wheel sdist` or whatever) | 20:00 |
| @fungicide:matrix.org | i could have sworn i saw something in the tox v4 changelog about directly calling pep 517 methods under specific circumstances, thought it was related to the editable-legacy toggle but maybe not | 20:01 |
| @clarkb:matrix.org | https://tox.wiki/en/latest/faq.html#tox-4-packaging-changes is all I see | 20:03 |
| @clarkb:matrix.org | and build does the same thing. It uses an isolated env. That is fine as build works | 20:03 |
| @clarkb:matrix.org | the more I stare at this package discovery doc the more I'm confused too... | 20:03 |
| @fungicide:matrix.org | the tox changelog for v4 includes "the base packaging environment is used to get the metadata of the project (via PEP-517) and to build sdist and dev packages," | 20:06 |
| @clarkb:matrix.org | Are the setuptools docs telling me that if I just add find: then it will work? | 20:06 |
| @clarkb:matrix.org | huh build doesn't write to the local dir? | 20:11 |
| @clarkb:matrix.org | wow it writes into the source dir of what you are building | 20:11 |
| @clarkb:matrix.org | ugh | 20:11 |
| @fungicide:matrix.org | build uses a throwaway temporary location | 20:12 |
| @fungicide:matrix.org | this might help: https://github.com/pypa/setuptools/blob/main/CHANGES.rst#v6110 | 20:12 |
| @fungicide:matrix.org | seems to confirm your interpretation of the custom discovery section in the docs | 20:14 |
| @clarkb:matrix.org | what is odd is I've added the options to include specific dirs and it is still including more than that. Namely playbooks/ are included even if I don't list them | 20:16 |
| @clarkb:matrix.org | part of me is beginning to wonder if python somehow made packaging even more complicated than it was before. | 20:17 |
| @clarkb:matrix.org | or if this is actually easier and since I've got ancient python packagign knowledge I'm finding the change to be difficult but the system isn't actually | 20:18 |
| @fungicide:matrix.org | oh, they keep making it more complicated. this isn't particularly sudden | 20:18 |
| @fungicide:matrix.org | rather, they made it different, in order to try to develop standards rather than the packaging standards being "whatever setuptools/pip do" | 20:19 |
| @clarkb:matrix.org | right but it feels like those standard are somehow even more complicated | 20:19 |
| @clarkb:matrix.org | which is impressive considering what we are comparing it to | 20:19 |
| @fungicide:matrix.org | i think it's proving a fundamental law of information theory. for a standard to support what setuptools and pip already do it has to be at least as complex as they are | 20:20 |
| @clarkb:matrix.org | ha yes | 20:22 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 867829: Add tox v4 compatibility https://review.opendev.org/c/zuul/zuul/+/867829 | 20:24 | |
| @clarkb:matrix.org | I am able to run `tox -elinters` locally with that change and it doesn't seem to produce a different sdist content | 20:25 |
| @hanson76:matrix.org | > <@jim:acmegating.com> Hanson: i rebased that on your change ^ | 20:33 |
| Looks like it worked. | ||
| @clarkb:matrix.org | Hanson: I'll review the changes after lunch | 20:34 |
| @clarkb:matrix.org | fungi: new errors: First is we don't seem to run bindep as part of the linters job but since we do an sdist and that installs all of zuul's deps we need libre2-dev (at least). I'm now wondering if under tox v4 we weren't installing the package at all either due to the skipsdist? Both skipsdist and usedevelop=false were set which might've led to not installing the code at all? The second issue is that it is not getting version info for some reason and reporting the version as 0.0.0. Possibly because tox is making the build env with a copy of the repo without the git content? | 20:37 |
| @clarkb:matrix.org | Thats going to be a widespread issue with pbr users possibly | 20:38 |
| @clarkb:matrix.org | ok ya tox v3 doesn't seem to ever install zuul into the env in that situation. So I think I can change this to do that too. The problem is going to be that masks the isolated build env + pbr + versioning problems | 20:39 |
| @fungicide:matrix.org | possibly. i normally pushed for skipsdist=true + usedevelop=false in order to avoid unnecessarily building and installing the project everywhere tox was run even for things like linters which don't need the project and its deps installed, since that can be a massive and redundant waste of resources | 20:40 |
| @clarkb:matrix.org | yes, though I thought we had a rough convention of using sdists in linters as a sanity check they could be built. But ya I'm just going to preserve the old behavior here | 20:40 |
| @jim:acmegating.com | Clark: we have a build-python-release job | 20:41 |
| @fungicide:matrix.org | my personal preference is to test dist building completely separately, but that never really caught on with our projects | 20:41 |
| @clarkb:matrix.org | The versioning problem in an isolated build env is something we should look at generally because that has the possibility ofbreaking all pbr users | 20:41 |
| @fungicide:matrix.org | for mine, i have a [testenv:dist] which runs build and twine --check and so on | 20:41 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 867829: Add tox v4 compatibility https://review.opendev.org/c/zuul/zuul/+/867829 | 20:44 | |
| @clarkb:matrix.org | fungi: the bit that results in a 0.0.0 version is https://zuul.opendev.org/t/zuul/build/4a45beacba444e988bbaee7f2efd2940/log/job-output.txt#1673 I think it is directly invoking the things you do to have setuptools invoked via pyproject.toml | 20:48 |
| @clarkb:matrix.org | that could explain the difference in behavior with pcakage finding I guess since they are using the new interface and not the compatible one | 20:48 |
| @clarkb:matrix.org | What I'm confused about is why that produces a 0.0.0 versions since pbr should work in that setup I thought | 20:49 |
| @fungicide:matrix.org | yes, pep 517 build backends assume all metadata is static unless you explicitly mark things like version as dynamic | 20:49 |
| @clarkb:matrix.org | fungi: but pbr + pyroject.toml + setuptools uses the same mechanism | 20:50 |
| @clarkb:matrix.org | and it produces correct versions as far as I can recall | 20:50 |
| @clarkb:matrix.org | oh wait it might be pbr that invokes that method though not a direct invocation | 20:50 |
| @clarkb:matrix.org | arg so ya this is completely broken for pbr | 20:50 |
| @fungicide:matrix.org | for my projects i have to set project.dynamic=['version'] in pyproject.toml with pbr | 20:51 |
| @clarkb:matrix.org | build-backend = "pbr.build" | 20:52 |
| @clarkb:matrix.org | talking out loud here, I'm not sure how this is valid in the first place | 20:52 |
| @clarkb:matrix.org | zuul is not a pyproject.toml project. It uses setup.py | 20:52 |
| @clarkb:matrix.org | tox should not assume that it can invoke setuptools backends for pyproject.toml and have everything come out fine. It needs to invoke the setup.py | 20:52 |
| @fungicide:matrix.org | is this in a usedevelop context? | 20:53 |
| @clarkb:matrix.org | fungi: no this is `tox -elinters` on the previous patchset that created an sdist then failed to install it because libre2-dev wasn't around | 20:53 |
| @clarkb:matrix.org | I feel like this is a fundamental flaw in tox. They need to use `build` instead | 20:53 |
| @clarkb:matrix.org | because you cannot assume that you can bypass setup.py on a non pyproject.toml project | 20:54 |
| @clarkb:matrix.org | corvus: this has me thinking nox | 20:55 |
| @clarkb:matrix.org | I thought maybe we could shim in tox v4 but that is a very fundamental flaw that is going to affect any project using pbr I think | 20:55 |
| @jim:acmegating.com | Clark: ack -- if we want to divide up nox tasks, i can think of these: 1) zuul job/role to run nox; 2) finish boilerplate noxfile that passes linters, has copyright header, etc; 3) jobs for multiple python versions; 4..X) each session. | 20:59 |
| @fungicide:matrix.org | Clark: hah, looks like someone just opened this today... https://github.com/tox-dev/tox/issues/2730 "tox4: The usedevelop ineffective with skipsdist / editable package not present in virtual environment" | 20:59 |
| @jim:acmegating.com | oh and finish translating pip.sh into python | 21:00 |
| @clarkb:matrix.org | corvus: ok, I'm not sure I'll have time immediately for that beacuse now I need to try and express my concern on the openstack side of things | 21:01 |
| @clarkb:matrix.org | fungi: ^ for the openstack side of things it would probably be good if you could look at the tox sdist behavior with pbr and see if you have the same concern | 21:01 |
| @fungicide:matrix.org | yes, i need to upgrade my local tox and start playing around with some small projects to see if i can reproduce that behavior | 21:02 |
| @jim:acmegating.com | Clark: i will prepare a tox pin change | 21:03 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 21:08 | |
| - [zuul/zuul] 867057: WIP: play with noxfile https://review.opendev.org/c/zuul/zuul/+/867057 | ||
| - [zuul/zuul] 867900: Pin tox to 3 https://review.opendev.org/c/zuul/zuul/+/867900 | ||
| @hanson76:matrix.org | > <@clarkb:matrix.org> Hanson: I'll review the changes after lunch | 21:08 |
| Not sure if I should do "Worflow +1" when it's reviewed or if you (or someone else) does it when it's time. | ||
| Every "team" has it's own way of working ;) | ||
| @jim:acmegating.com | Hanson: your part is done unless someone comes back with a comment, thanks! one of the zuul maintainers will workflow +1 when it's ready to submit | 21:09 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 867901: Pin tox to 3 https://review.opendev.org/c/zuul/nodepool/+/867901 | 21:14 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 867057: WIP: play with noxfile https://review.opendev.org/c/zuul/zuul/+/867057 | 21:16 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 867902: Add ensure-nox role https://review.opendev.org/c/zuul/zuul-jobs/+/867902 | 21:23 | |
| @jim:acmegating.com | Clark: oh i think the big task is going to be siblings | 21:24 |
| @jim:acmegating.com | i'm pausing my nox work here for the moment with those changes ^ anyone should be able to build on them | 21:25 |
| @clarkb:matrix.org | ack | 21:37 |
| @clarkb:matrix.org | corvus: we'd probably also need zuul-jobs stuff to drive nox? | 22:54 |
| @jim:acmegating.com | Clark: yeah, and the 'tox' role in zuul-jobs is where siblings happens | 23:02 |
| @jim:acmegating.com | so i think that combo is the next step | 23:03 |
| -@gerrit:opendev.org- Zuul merged on behalf of Anders Hanson: [zuul/nodepool] 867744: Add support for Kubernetes 1.24+ https://review.opendev.org/c/zuul/nodepool/+/867744 | 23:41 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!