| *** slaweq_ is now known as slaweq | 14:09 | |
| slaweq | gmann: hi, can You open LP bug for me with that? | 14:12 |
|---|---|---|
| slaweq | I'm in the internal meeting this week and don't have time to look at it but next week I can definitely work on that | 14:13 |
| elodilles | gmann kopecmartin : hi, just to inform you, most of the repositories' wallaby branches went to Extended Maintenance (hopefully the last patches will be merged in the coming days). do you plan to tag tempest and tempest plugins with wallaby-last? or do you tag *-last usually later after? | 14:38 |
| gmann | elodilles: sure, will do this week. thanks | 15:55 |
| dansmith | frickler: gmann: kopecmartin: I forgot the gate is broken and +Wd something.. what's the status of making the debian job nv? | 18:06 |
| dansmith | should I do that real quick and use that to kick this out of the gate? | 18:06 |
| opendevreview | Dan Smith proposed openstack/devstack master: Make debian-bullseye job non-voting https://review.opendev.org/c/openstack/devstack/+/864135 | 18:13 |
| dansmith | well, here it is if that's the decision ^ | 18:13 |
| gmann | dansmith: ack, I have not seen its fix yet so I am fine for making n-v for now | 18:24 |
| gmann | kopecmartin: frickler: would you like to +W on it if around? https://review.opendev.org/c/openstack/devstack/+/864135 | 18:25 |
| gmann | if not then i will do | 18:25 |
| gmann | if not around :) | 18:26 |
| clarkb | dansmith: gmann: the debian bullseye devstack error I'm looking at appears to be due to pyopenssl using a new argument in cryptography that I guess isn't present on the installed cryptogrpahy version: https://zuul.openstack.org/build/7fd3993f306041a198d6e01b57630a24/log/job-output.txt#12018-12060 | 18:42 |
| clarkb | https://github.com/pyca/cryptography/commit/727422868510ef1d5eb554fe49669224e8dc50c0#diff-c9ed41cf466f7643671839013ae11d8315b34fcacde992483edca84d6b2dfe04R146 is the commit that added the name parameter | 18:42 |
| clarkb | my first hunch was that somehow the debian package of cryptography was being used as it is much older than the 38.0.2 that we install from pypi. But ubuntu's version is also quite a bit older | 18:43 |
| clarkb | you might want to hold a node and examine the python paths and similar to see why it is finding old cryptography without that name parameter. | 18:45 |
| clarkb | 38.0.2 does include the parameter https://github.com/pyca/cryptography/blob/38.0.2/src/cryptography/utils.py#L124 | 18:45 |
| clarkb | this might be the pip env isolation problem I helped debug earlier this year. /me looks for logs on that | 18:52 |
| clarkb | https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2022-05-19.log.html#t2022-05-19T23:06:27 is where I ended up with that | 18:56 |
| clarkb | I don't think we're tripping that specific bug anymore since it appears devtsack removed those settings. But maybe pip changes? | 18:57 |
| clarkb | yes, looking at that traceback more closely we are using distro installed pip which appears to use distro installed urllib3 which then imports pip installed pyopenssl which must find distro installed cryptography? | 19:00 |
| clarkb | fungi: do you understand how debian debundles deps for pip? I'm beginning to suspect the issue is in that system. In particular, I don't see them debundling cryptography which pyopenssl seems to depend on and I'm wondering if that is why this sometimes works and sometimes doesn't | 19:19 |
| fungi | clarkb: yeah, just a sec i can get you a link | 19:24 |
| fungi | is this debian or ubuntu? | 19:24 |
| fungi | ubuntu i'm guessing? | 19:24 |
| clarkb | debian | 19:25 |
| fungi | ahh, okay. and venv or non-venv? | 19:26 |
| clarkb | non venv | 19:27 |
| clarkb | fwiw I managed to reproduce the error in a docker container so nothing too special about our test images | 19:27 |
| clarkb | the problem occurs after we install pyOpenSSL from pypi but not before. | 19:28 |
| fungi | clarkb: so this is the package in question: https://packages.debian.org/bullseye/python-pip-whl | 19:28 |
| clarkb | I believe that something about the debundling is broken and is allowing pyOpenSSL to escape which uses newer stuff than what is bundled (because things like cryptography are not bundled) | 19:28 |
| fungi | at least i expect it's that copy you're seeing | 19:28 |
| clarkb | yup those paths look right | 19:29 |
| fungi | rather than fully de-vendor those libs, because pip gets weird about it, they basically build a new copy of them and re-vendor them to where pip wants | 19:29 |
| clarkb | fungi: https://zuul.openstack.org/build/7fd3993f306041a198d6e01b57630a24/log/job-output.txt#12018-12060 is the exception. | 19:29 |
| fungi | so they're technically built from the distro's source packages, but a separate copy bundled into wheel form for venv to install | 19:30 |
| clarkb | fungi: in a local debian bullseye container pip installing the dbcounter package in devstack works fine using python3.9 -m pip using pip from python3-pip until I install pyOpenSSL from pypi | 19:30 |
| clarkb | once you install pyOpenSSL from pypi it breaks. What I'm confused about is how it works at all without pyOpenSSL in the prior state. But I suspect that it is actually using the vendored libs in urllib3. | 19:31 |
| clarkb | But once you install a system level urllib3 it can jump out. Now on Ubuntu they must use a different debundling that avoids this | 19:31 |
| fungi | i think ubuntu doesn't care as much about purity/provenance there and just distributes what python upstream embeds | 19:36 |
| clarkb | I'm beginning to suspect that urllib3 conditionally uses pyopenssl based on whether or not pyopenssl is installed. | 19:36 |
| fungi | i don't see any pyopenssl in https://packages.debian.org/bullseye/all/python-pip-whl/filelist though | 19:37 |
| fungi | yes, that seems far more likely | 19:37 |
| clarkb | so this works fine until you have pyopenssl installed then it jumps out | 19:37 |
| fungi | https://github.com/urllib3/urllib3/blob/main/src/urllib3/contrib/pyopenssl.py | 19:37 |
| fungi | the comments there are... interesting | 19:38 |
| clarkb | ya https://github.com/urllib3/urllib3/blob/main/src/urllib3/contrib/pyopenssl.py#L41 is the line that is exploding | 19:38 |
| clarkb | because it is finding pyopenssl that is apparently too new for the cryptography that it is finding | 19:38 |
| clarkb | if I run from OpenSSL import SSL, crypto in an interpreter outside of the pip process it imports fine | 19:38 |
| fungi | though where is cryptography coming from? the python-pip-whl package doesn't seem to bring it | 19:39 |
| clarkb | fungi: pyopenssl has it listed as a dep. This pulls in the version of python cryptography that works (new enough with the new interface). I believe devstack is also installing the debian package and that breaks | 19:42 |
| clarkb | I installed both to mimic devstack. Let me edit the crptography file(s) to confirm | 19:42 |
| fungi | ahh, okay, so there are two versions of cryptography being installed, one via devstack, another transitively from pip install pyopenssl? | 19:43 |
| clarkb | fungi: on my test setup yes. I think when devstack runs it directly installs both | 19:44 |
| fungi | devstack's is a debian python3-cryptography package i guess, which is not conflicting with the file locations for a pip install of cryptography? | 19:45 |
| clarkb | and I've just confirmed that the pip install on bullseye is finding pip installed pyopenssl but then pyopenssl is finding distro cryptography which is not compat | 19:45 |
| clarkb | fungi: correct /usr/lib vs /usr/local/lib | 19:45 |
| fungi | yeah, so mixing distro packaged python libs and python libs from pypi. i guess we shouldn't be surprised that bursts into flames at the first opportunity | 19:46 |
| clarkb | I've manually installed python3-openssl and that fixes it | 19:46 |
| fungi | makes sense | 19:47 |
| clarkb | I think what is happening is that urllib3 conditionally uses pyopenssl. Since we have installed pyopenssl from pypi it tries to use it. Due to the debundling system it attempts to use debian pacakge installs for everything it can, but pyopenssl isn't there from a debian package so it jumps out to pypi | 19:47 |
| clarkb | cryptography is there and then we break | 19:47 |
| clarkb | with this info I should be able to write a change for devstack that works | 19:47 |
| clarkb | oh wow and to make it even more confusing novnc is what installs python3-cryptography | 19:49 |
| clarkb | that means the two simple solutions are 1) don't install novnc 2) install python3-openssl | 19:50 |
| fungi | novnc->websockify->python3-jwcrypto->python3-cryptography | 19:53 |
| fungi | so yes, at least indirectly it's novnc | 19:53 |
| fungi | looks like upstream novnc recommends installing it as a snap ;) | 19:55 |
| fungi | i suppose we could install novnc from source and then pip install websockify | 19:56 |
| fungi | ahh, novnc is available on npmjs.org, so we could npm install it | 19:57 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Fix dbcounter install on Debian Bullseye https://review.opendev.org/c/openstack/devstack/+/864142 | 20:02 |
| clarkb | thats a "just install python3-openssl" change | 20:03 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Fix dbcounter install on Debian Bullseye https://review.opendev.org/c/openstack/devstack/+/864142 | 21:54 |
| opendevreview | Merged openstack/devstack master: Make debian-bullseye job non-voting https://review.opendev.org/c/openstack/devstack/+/864135 | 22:43 |
| clarkb | ha so adding python3-openssl breaks the ubuntu-focal openstacksdk-functional-devstack job in a very similar way to how debian bullseye was broken before this change | 23:45 |
| clarkb | I'm guessing whatever ubuntu does is impacted in reverse :/ | 23:46 |
| clarkb | I'm not sure I've got the ability to look into that today. | 23:46 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!