| opendevreview | Cyril Roelandt proposed openstack/devstack master: Glance: Properly copy rootwrap.d/* when USE_VENV is True https://review.opendev.org/c/openstack/devstack/+/866951 | 02:04 |
|---|---|---|
| *** yadnesh|away is now known as yadnesh | 03:41 | |
| slaweq | frickler hi, can You maybe hold for me one node for debugging? I need to understand why neutron-tempest-plugin jobs are timinig out on patch https://review.opendev.org/c/openstack/neutron/+/865822 - I'm checking logs since today morning and I have no idea what's wrong there. Locally with same devstack patch all seems to be working fine for me :/ | 10:15 |
| slaweq | I can recheck that patch if You would be able to hold for me one of those jobs | 10:15 |
| frickler | slaweq: sure, which job do you want, neutron-tempest-plugin-openvswitch? or lb too? | 10:41 |
| frickler | slaweq: started with that job, you have time until the timeout to let me know if you also want others. also I'll need your ssh key once again, I still didn't manage to keep track of it | 10:44 |
| slaweq | frickler neutron-tempest-plugin-openvswitch will be good | 10:45 |
| slaweq | frickler and my ssh key https://paste.opendev.org/show/bl52j3D1ESDuw3oJXye4/ | 10:46 |
| slaweq | thank You very much for help | 10:46 |
| frickler | hmm, I have no idea why zuul doesn't like the dependency on that patch now. will need to debug later | 10:53 |
| opendevreview | Slawek Kaplonski proposed openstack/devstack master: [neutron] Don't configure firewall_driver for core ML2 plugin https://review.opendev.org/c/openstack/devstack/+/865544 | 10:57 |
| opendevreview | Slawek Kaplonski proposed openstack/devstack master: Drop lib/neutron module https://review.opendev.org/c/openstack/devstack/+/865014 | 10:58 |
| opendevreview | Slawek Kaplonski proposed openstack/devstack master: Rename lib/neutron-legacy to lib/neutron https://review.opendev.org/c/openstack/devstack/+/865015 | 10:58 |
| slaweq | frickler I rebased devstack patches - lets see if it will work now | 10:58 |
| slaweq | frickler it's running now, You can hold node for me :) | 11:07 |
| frickler | slaweq: weird, there were no merge conflicts, I still wonder why zuul was so picky. | 11:28 |
| slaweq | frickler did You maybe add my ssh key to that node already? | 11:55 |
| slaweq | and put it on hold? | 11:55 |
| frickler | slaweq: root@199.19.213.11, the hold will only apply when (if) it fails/timeouts | 11:58 |
| frickler | but you can look at it while the job is in progress now anyway | 11:58 |
| *** pojadhav is now known as pojadhav|afk | 12:00 | |
| slaweq | frickler and did You put my ssh key there? | 12:12 |
| slaweq | frickler it's strange but it seems that now all jobs are green | 12:19 |
| slaweq | :) | 12:19 |
| frickler | quantum timeout then, disappears when being watched. we can recheck a couple of times to see how stable it is now | 12:57 |
| *** pojadhav|afk is now known as pojadhav | 13:23 | |
| opendevreview | Slawek Kaplonski proposed openstack/grenade master: Use default neutron backend in the base grenade job https://review.opendev.org/c/openstack/grenade/+/862475 | 13:54 |
| ralonsoh | frickler, hi, I'm seeing this error in some neutron jobs | 14:07 |
| ralonsoh | https://zuul.opendev.org/t/openstack/build/f7ad05d06e79486e98e77805c6f6373c | 14:07 |
| ralonsoh | venv: failed with tempest is not allowed, use allowlist_externals to allow it | 14:07 |
| ralonsoh | should we add "tempest" to the allowlist_externals in its own tox.ini? | 14:07 |
| slaweq | ralonsoh it may be related to the tox 4 | 14:07 |
| ralonsoh | it is, yes | 14:07 |
| slaweq | but I didn't check it so I'm not sure about it | 14:07 |
| slaweq | ++ | 14:08 |
| slaweq | ralonsoh shouldn't You be off today? :D | 14:08 |
| ralonsoh | slaweq, maybe... | 14:08 |
| slaweq | hehe | 14:08 |
| frickler | ralonsoh: slaweq: the nova people just found the same bug, sean-k-mooney is doing a patch | 14:10 |
| ralonsoh | perfect! | 14:10 |
| frickler | or gibi actually | 14:11 |
| gibi | :) | 14:11 |
| ralonsoh | gibi, thanks! | 14:12 |
| sean-k-mooney | ya i have to leave for an hour or so so if gibi for frickler can take care of the patch that would be best | 14:13 |
| opendevreview | Balazs Gibizer proposed openstack/devstack master: [tempest]Use the proper venv to run verify-config https://review.opendev.org/c/openstack/devstack/+/866997 | 14:13 |
| gibi | ^^ it is up | 14:13 |
| ralonsoh | frickler, we have the same problem with this patch | 14:53 |
| ralonsoh | https://zuul.opendev.org/t/openstack/build/7585d1de46914130b3348efecfd5fa33 | 14:53 |
| frickler | gibi: kopecmartin: gmann: ^^ then we need to revisit the allowlist_external change in tempest | 14:58 |
| gibi | ralonsoh: I think my devstack patch will fix that too | 14:58 |
| ralonsoh | gibi, no, this build if from your patch | 14:59 |
| gibi | hm | 14:59 |
| gibi | then I don't know what is next | 15:00 |
| gibi | should we do a `tox -evenv-tempest -- pip install .` to get tempest in the venv-tempest? | 15:02 |
| ralonsoh | is done before | 15:02 |
| frickler | actually that tempest change is two weeks old, but the failures only started yesterday | 15:03 |
| gibi | ralonsoh: where? I only see that we install the requirements for tempest | 15:03 |
| frickler | so probably the issue is rather related to the tox4 issue yesterday and what infra did to pin tox | 15:03 |
| ralonsoh | gibi, | 15:03 |
| ralonsoh | 2022-12-08 14:47:20.294442 | controller | ++ lib/tempest:configure_tempest:709 : tox -evenv-tempest -- pip install -c /tmp/tempest_u_c_m.UhBZYwzaK1 -r requirements.txt | 15:03 |
| ralonsoh | 2022-12-08 14:47:21.681455 | controller | venv-tempest: commands[0]> pip install -c /tmp/tempest_u_c_m.UhBZYwzaK1 -r requirements.txt | 15:03 |
| frickler | clarkb: fungi: ^^ I didn't look into what you did yesterday in detail yet, maybe you have a better grip on that | 15:04 |
| gibi | ralonsoh: I don't think we have tempest itself in the requirements file of tempest | 15:04 |
| ralonsoh | no, we don't | 15:04 |
| ralonsoh | we have * | 15:04 |
| ralonsoh | sorry, in the allowed list | 15:04 |
| ralonsoh | but the problem is not this binary, if that is not allowed | 15:05 |
| ralonsoh | is* | 15:05 |
| *** pojadhav is now known as pojadhav|dinner | 15:06 | |
| opendevreview | Balazs Gibizer proposed openstack/devstack master: [tempest]Use the proper venv to run verify-config https://review.opendev.org/c/openstack/devstack/+/866997 | 15:06 |
| frickler | oh, actually those jobs seem to be using tox4, which got stricter about allowlist handling | 15:06 |
| frickler | failing has tox-4.0.2 https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_170/866489/5/check/neutron-ovn-tempest-ovs-release-ubuntu-old/17007bf/controller/logs/pip3-freeze.txt | 15:06 |
| frickler | earlier jobs tox-3.27.1 https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5fa/865857/5/gate/neutron-ovn-tempest-ovs-release-ubuntu-old/5fa7b43/controller/logs/pip3-freeze.txt | 15:07 |
| fungi | frickler: what we did was pin to tox<4 in the ensure-tox role in zuul-jobs, but anything which installs tox some other way is probably still installing latest | 15:07 |
| frickler | yes, we need to cap tox somewhere in there. or amend the allowlist after all | 15:07 |
| ralonsoh | aren't we using allowlist_externals=* in this env? | 15:08 |
| fungi | you may run into other problems after you get it updated with allowlist_externals, for example tox v4 changes how and where it logs things, changes some of its outputs (including colorization by default), and probably more i'm forgetting | 15:08 |
| ralonsoh | this is still allowed in 4.0.2 https://tox.wiki/en/4.0.2/config.html#base-options | 15:09 |
| frickler | seems neutron/lib/ovn_agent is pip_installing tox | 15:09 |
| frickler | and causing some conflicts https://paste.opendev.org/show/bvmYWdmR7jLQH9IwPQ96/ | 15:09 |
| frickler | because u-c is too old | 15:10 |
| ralonsoh | frickler, what job? | 15:10 |
| ralonsoh | just to have an example | 15:10 |
| frickler | ralonsoh: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_170/866489/5/check/neutron-ovn-tempest-ovs-release-ubuntu-old/17007bf/job-output.txt | 15:10 |
| ralonsoh | frickler, is is ok if I make this pip installation conditional? | 15:14 |
| ralonsoh | really? | 15:20 |
| rosmaita | anyone else seeing failures during setup on tempest jobs in stable/yoga? I am seeing this: | 15:21 |
| rosmaita | venv: install_deps> python -I -m pip install -r /opt/stack/tempest/requirements.txt -r /opt/stack/tempest/doc/requirements.txt -c https://releases.openstack.org/constraints/upper/master | 15:21 |
| rosmaita | venv: commands[0]> tempest verify-config -uro /tmp/tmp.hnzw9vd7Gn | 15:21 |
| rosmaita | venv: exit 2 (0.00 seconds) /opt/stack/tempest> tempest verify-config -uro /tmp/tmp.hnzw9vd7Gn | 15:21 |
| rosmaita | happening on tempest-full-py3 on https://review.opendev.org/c/openstack/python-cinderclient/+/865399 | 15:21 |
| ralonsoh | rosmaita, yes ^^ conversation ahead | 15:21 |
| fungi | i think rosmaita joined after most of the discussion | 15:21 |
| rosmaita | ralonsoh: ty, i will have to read the log | 15:21 |
| fungi | but check the log on meetings.o.o | 15:22 |
| fungi | also folks should probably follow up to the tox v4 thread i started on openstack-discuss with any insights they arrive at | 15:22 |
| ralonsoh | fungi, thanks | 15:22 |
| rosmaita | looks like allowlist_externals checking is much more strict in 4 | 15:23 |
| fungi | in addition to now completely ignoring the old (previously deprecated) whitelist_externals directive | 15:23 |
| fungi | and a codesearch turns up a crazy amount of that still in our master branches alone which is likely to need addressing | 15:24 |
| rosmaita | my issue is slightly different, exit code 2 instead of 1, and no mention of allowlist_externals | 15:25 |
| ralonsoh | fungi, but https://review.opendev.org/c/openstack/devstack/+/866997 is now using an env with allowlist_externals=* | 15:25 |
| fungi | i guess that's an attempt to turn off tox's checking of external commands completely? | 15:26 |
| ralonsoh | I think so, yes | 15:26 |
| ralonsoh | but doesn't work | 15:26 |
| fungi | probably an indication we should be using something other than tox if we can't even identify the commands we're going to have it run outside venvs | 15:26 |
| fungi | https://tox.wiki/en/latest/config.html#allowlist_externals does at least imply that allowlist_externals=* is still valid (though not recommended) | 15:29 |
| ralonsoh | yes, that's valid | 15:30 |
| ralonsoh | frickler, should I make this pip installation conditional in the ovn_agent script? | 15:32 |
| fungi | rosmaita: 866997 drops use of the constraints file when installing tempest, so maybe that's related? | 15:33 |
| fungi | what i mean is, maybe a backport of 866997 would address the error you're seeing | 15:34 |
| rosmaita | thanks, will take a look | 15:34 |
| clarkb | I hesitate to approve that change without a better understanding of what is happening (and for that I need more time to look at job logs etc). My main concern is that I believe we already install tempest three separate times in a devstack run and this adds a fourth? | 15:37 |
| fungi | clarkb: you mean installing it into venv-tempest rather than just installing its requirements.txt? | 15:40 |
| fungi | oh, right, it's not changing that line, it's adding another | 15:41 |
| clarkb | fungi: basically if you look at the lines above it we just keep repeating ourselves in different ways and its confusing and probably broken | 15:41 |
| clarkb | I'm working on a patch | 15:41 |
| clarkb | er not a patch a comment | 15:41 |
| frickler | ralonsoh: actually I think the better solution is to make the tempest setup work properly with tox4, but I also don't grok all those twisted venvs. maybe kopecmartin or gmann need to take this over | 15:42 |
| fungi | and yeah, that configure_tempest function is so massive that it's easy to lose track of what it's already done. so while "installing tempest yet again" seems to work around the problem, it would be good to figure out why that is | 15:43 |
| clarkb | I posted a comment | 15:44 |
| clarkb | right I called this behaviour out as very likely buggy a couple of weeks ago. I think that the tox stuff is merely making it a problem | 15:46 |
| clarkb | and adding more of the underlying problem isn't a proper way out | 15:46 |
| *** ykarel_ is now known as ykarel|away | 15:47 | |
| frickler | setting up something like /opt/tempest-venv without tox might be a good idea, yes | 15:47 |
| fungi | i agree, deciding how you want the tempest venv to look and then taking the minimum steps necessary to achieve that rather than the tempest venv being an emergent behavior of the random workarounds people have piled onto that function over the years | 15:49 |
| *** pojadhav|dinner is now known as pojadhav | 16:10 | |
| kopecmartin | ahh, this whole thing is very weird, i'm trying to reproduce that locally, but can't , it works o.O however if i use the newest tox , tox itself fails before it starts creating an env .. ah | 16:14 |
| kopecmartin | what if we commented the verify-config part out? it's just a check, not required for a functional tempest.conf | 16:15 |
| kopecmartin | i mean temporarily until we fix and ideally refactor that confusing part | 16:16 |
| *** yadnesh is now known as yadnesh|away | 17:16 | |
| gmann | we need verify config but let me check why evenv-tempest and we need tempest as command there | 20:05 |
| gmann | also we should not install it multiple times. I think existing installation are needed in config section separately but need to check it | 20:06 |
| gmann | as yoga and zed jobs are also failing, is tox 4 used there also? | 20:19 |
| clarkb | gmann: the change that I made yesterday only impacted jobs that use ensure-tox. Devstack (master at least) pip intalls tox itself in lib/tempest and lib/neutron_plugins/ovn_agent and both of those appear to install latest tox | 20:39 |
| clarkb | so yes I think this may be trying to use tox 4 | 20:39 |
| gmann | clarkb: I am confused where tempest-full job on master using old tox and in stable job using new https://lists.openstack.org/pipermail/openstack-discuss/2022-December/031442.html | 20:45 |
| gmann | it is same type of job | 20:45 |
| clarkb | I'd have to look at the logs to see where they install tox | 20:47 |
| clarkb | maybe tox is in contraints and that capped it? | 20:47 |
| gmann | I do not see it in constraints but it seems we should have this at least for stable branches | 20:48 |
| gmann | that is why grenade jobs failing and not tempest full on master. something fishy going with tox installation in devstack | 20:53 |
| gmann | master vs stable | 20:53 |
| gmann | and focal job on mater devstack use tox>4.0 https://zuul.opendev.org/t/openstack/build/aaad7312f68246c0979e1a43cfe96c3b/log/controller/logs/pip3-freeze.txt#229 | 20:55 |
| opendevreview | Ghanshyam proposed openstack/devstack master: [tempest]Use the proper venv to run verify-config https://review.opendev.org/c/openstack/devstack/+/866997 | 21:06 |
| opendevreview | Ghanshyam proposed openstack/devstack stable/zed: [tempest]Use the proper venv to run verify-config https://review.opendev.org/c/openstack/devstack/+/867055 | 21:08 |
| opendevreview | Ghanshyam proposed openstack/tempest master: DNM:Testing 866997 https://review.opendev.org/c/openstack/tempest/+/867056 | 21:10 |
| fungi | tox doesn't have to be *in* constraints to end up at an old version when constraints are applied. pip will consider tox's dependencies and only install versions of tox which will work with the constrained dependencies (if there are any) | 21:41 |
| opendevreview | Ghanshyam proposed openstack/devstack stable/zed: Testing tox version https://review.opendev.org/c/openstack/devstack/+/867059 | 21:50 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Testing tempest pre-provision account job https://review.opendev.org/c/openstack/tempest/+/867060 | 22:02 |
| opendevreview | Ghanshyam proposed openstack/devstack stable/zed: Testing tox version https://review.opendev.org/c/openstack/devstack/+/867059 | 22:34 |
| opendevreview | Ghanshyam proposed openstack/tempest master: DNM:Testing tox issues https://review.opendev.org/c/openstack/tempest/+/867056 | 22:35 |
| rosmaita | gmann: i think you have a typo in https://review.opendev.org/c/openstack/devstack/+/867059 | 22:38 |
| opendevreview | Ghanshyam proposed openstack/devstack stable/zed: Testing tox version https://review.opendev.org/c/openstack/devstack/+/867059 | 22:42 |
| gmann | rosmaita: yeah, my bad | 22:42 |
| gmann | fixed ^^ | 22:42 |
| rosmaita | thanks! | 22:43 |
| gmann | rosmaita: if this work fine and tox new verson is not coming form anywhere else then this is fix for stable jobs and workaround to master job. and for master jobs we can continue debugging and migrate to tox 4 accordingly | 22:43 |
| rosmaita | i sure hope it works! | 22:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!