| opendevreview | melanie witt proposed openstack/nova stable/victoria: Add functional regression test for bug 1853009 https://review.opendev.org/c/openstack/nova/+/811810 | 00:29 |
|---|---|---|
| opendevreview | melanie witt proposed openstack/nova stable/victoria: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/c/openstack/nova/+/811811 | 00:29 |
| opendevreview | melanie witt proposed openstack/nova stable/victoria: Invalidate provider tree when compute node disappears https://review.opendev.org/c/openstack/nova/+/811812 | 00:29 |
| opendevreview | melanie witt proposed openstack/nova stable/victoria: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/811813 | 00:29 |
| opendevreview | melanie witt proposed openstack/nova stable/victoria: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811814 | 00:29 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Add functional regression test for bug 1853009 https://review.opendev.org/c/openstack/nova/+/811815 | 00:35 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/c/openstack/nova/+/811816 | 00:35 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Invalidate provider tree when compute node disappears https://review.opendev.org/c/openstack/nova/+/811817 | 00:35 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/811818 | 00:35 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811819 | 00:35 |
| opendevreview | melanie witt proposed openstack/nova stable/wallaby: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/811808 | 01:10 |
| opendevreview | melanie witt proposed openstack/nova stable/wallaby: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811809 | 01:10 |
| opendevreview | melanie witt proposed openstack/nova stable/wallaby: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811809 | 01:38 |
| opendevreview | melanie witt proposed openstack/nova stable/victoria: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/811813 | 01:44 |
| opendevreview | melanie witt proposed openstack/nova stable/victoria: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811814 | 01:44 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/811818 | 01:46 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811819 | 01:46 |
| opendevreview | melanie witt proposed openstack/nova stable/train: Add functional regression test for bug 1853009 https://review.opendev.org/c/openstack/nova/+/811821 | 01:58 |
| opendevreview | melanie witt proposed openstack/nova stable/train: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/c/openstack/nova/+/811822 | 01:58 |
| opendevreview | melanie witt proposed openstack/nova stable/train: Invalidate provider tree when compute node disappears https://review.opendev.org/c/openstack/nova/+/811823 | 01:58 |
| opendevreview | melanie witt proposed openstack/nova stable/train: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/811824 | 01:58 |
| opendevreview | melanie witt proposed openstack/nova stable/train: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811825 | 01:58 |
| opendevreview | melanie witt proposed openstack/nova stable/victoria: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/811813 | 03:53 |
| opendevreview | melanie witt proposed openstack/nova stable/victoria: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811814 | 03:53 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/811818 | 04:05 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811819 | 04:05 |
| opendevreview | melanie witt proposed openstack/nova stable/train: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/811824 | 04:25 |
| opendevreview | melanie witt proposed openstack/nova stable/train: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811825 | 04:25 |
| *** elodilles_pto is now known as elodilles | 07:12 | |
| elodilles | bauzas: nova + placement RC2s are now +2+W'd, hopefully will be released soon | 07:18 |
| bauzas | ++ | 07:19 |
| bauzas | elodilles: we have a relnote with https://review.opendev.org/c/openstack/nova/+/811447 | 07:19 |
| bauzas | elodilles: I guess we would need to create a RC3 if we backport it to stable/xena or no ? | 07:19 |
| elodilles | bauzas: yes, i think so | 07:24 |
| bauzas | elodilles: because release team is tagging 24.0.0 on top of the last RC, right? | 07:29 |
| bauzas | and not tagging on the last patch ? | 07:29 |
| bauzas | if so, we need to plan it :) | 07:29 |
| lyarwood | FWIW I'm against the releasenote in Nova | 07:29 |
| lyarwood | it's an os-brick regression | 07:29 |
| lyarwood | it should be fixed or documented there | 07:29 |
| bauzas | I don't disagree | 07:29 |
| bauzas | ah | 07:30 |
| bauzas | good point | 07:30 |
| lyarwood | otherwise we'd end up documenting all known issues ;) | 07:30 |
| bauzas | that's a very valid point | 07:30 |
| bauzas | lyarwood: the problem itself is the dependencies we have | 07:31 |
| elodilles | if we don't need the releasenote, then it is not relevant, but still: yes, last RC is tagged, not the last patch | 07:32 |
| bauzas | lyarwood: we would need to mark this osbrick release not supported by nova, right? (I mean, once we have the new osbrick release that fixes the problem) | 07:32 |
| bauzas | my concern goes to upper constraints and requirements that we have in our tree, if you prefer | 07:33 |
| lyarwood | Yeah once we have the fix | 07:33 |
| lyarwood | otherwise we wouldn't support Xena os-brick which could be awkward | 07:33 |
| lyarwood | and this isn't a critical backend IMHO | 07:33 |
| bauzas | ok, -2d the relnote | 07:58 |
| bauzas | lyarwood: what I'm afraid is that I don't see efforts yet on fixing the os-brick root issue | 07:59 |
| lyarwood | That's on them | 07:59 |
| lyarwood | Brian has said they want to fix it | 07:59 |
| gibi | if Xena nova does not work with xena os-brick but work with an older os-brick then we should tell this info to our deployers | 08:13 |
| opendevreview | zhen proposed openstack/nova stable/wallaby: Add missing __init__.py in nova/db/api https://review.opendev.org/c/openstack/nova/+/811786 | 08:16 |
| bauzas | gibi: it's a requirements question | 08:18 |
| lyarwood | gibi: if it was a generic issue then sure but it's a single backend and a single usecase within that backend | 08:18 |
| lyarwood | gibi: otherwise as I said before we would end up documenting all issues in our libs with each release | 08:19 |
| gibi | bauzas: sure, can we ping reqs now for Xena? | 08:19 |
| gibi | lyarwood: OK, so this is not a wide issue | 08:19 |
| gibi | then I'm OK to let this slip | 08:19 |
| bauzas | gibi: not sure, the problem is that for the moment, I don't see any work on os-brick | 08:19 |
| bauzas | lyarwood convinced me it's a small issue, so we can leave it tracked as only a bug | 08:20 |
| bauzas | gibi: if really we see larger problems, we could document something in our relnotes later on | 08:20 |
| lyarwood | yeah lets just let them fix it and release a new version and we can then blacklist the original | 08:20 |
| lyarwood | tbh we don't even need to blacklist it | 08:21 |
| gibi | Ok | 08:21 |
| lyarwood | as again we'd have to blacklist every previous version that contained issues later fixed lol | 08:21 |
| gibi | you are right, I probably overreacted without having the context | 08:23 |
| gibi | elodilles: it seems the release script is broken https://zuul.opendev.org/t/openstack/build/ff6be46d117e4c229b9d45e36addbbae/log/job-output.txt#849 | 08:57 |
| elodilles | gibi: indeed. it seems there was a new jsonschema release yesterday :) | 09:02 |
| gibi | murphy does not like xena | 09:02 |
| opendevreview | Lee Yarwood proposed openstack/nova master: nova-manage: Always get BDMs using get_by_volume_and_instance https://review.opendev.org/c/openstack/nova/+/811716 | 10:39 |
| lyarwood | if anyone has time today reviews on ^ and the change below it would be appreiciated | 10:39 |
| lyarwood | https://review.opendev.org/q/topic:bug/1943431 also | 10:39 |
| gibi | lyarwood: I'm +2 on https://review.opendev.org/c/openstack/nova/+/811716 (and the parent). Can I trade this with https://review.opendev.org/c/openstack/nova/+/810911 ? :) | 11:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Ensure MAC addresses are in lower case https://review.opendev.org/c/openstack/nova/+/811947 | 11:09 |
| lyarwood | gibi: ack will do | 11:15 |
| lyarwood | and thanks! | 11:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/811947 | 11:28 |
| gibi | lyarwood: I finished with https://review.opendev.org/q/topic:bug/1943431 too. If you have still time then I could trade that with https://review.opendev.org/c/openstack/nova/+/811396 :) | 11:47 |
| lyarwood | gibi: ack need to grab lunch and then sit on a call for a while but I'll review both series after that this afternoon | 11:50 |
| *** tobberydberg_ is now known as tobberydberg | 11:51 | |
| gibi | lyarwood: thank you, enjoy your lunch | 11:51 |
| bauzas | gibi: https://review.opendev.org/c/openstack/nova/+/810909/2 gets a -1 | 12:08 |
| bauzas | have you rechecked ? | 12:09 |
| bauzas | hem no | 12:09 |
| gibi | let me check the result | 12:09 |
| gibi | I recheked it in the morning | 12:09 |
| bauzas | again, grenade failing and the likes | 12:11 |
| bauzas | but I'll review anyway | 12:12 |
| bauzas | you're one of my r-p tagged | 12:12 |
| bauzas | oh, strangely, you aren't | 12:12 |
| bauzas | wtf | 12:12 |
| gibi | the multicell job as two build timeout | 12:12 |
| gibi | the grenade failed on a live migration test I need to look deeper | 12:13 |
| gibi | probably unrelated but I don't have a matcher for that failure yet | 12:13 |
| gibi | teh multicell build timeout is due to vif plug timeout | 12:13 |
| gibi | that is unrelated for sure | 12:13 |
| gibi | the grenade live migration failure is due to https://bugs.launchpad.net/nova/+bug/1912310 | 12:16 |
| gibi | I hit recheck | 12:16 |
| *** bhagyashris is now known as bhagyashris|rover | 12:43 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/811947 | 12:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/811947 | 13:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/811947 | 13:06 |
| bauzas | gibi: https://review.opendev.org/c/openstack/nova/+/810909/2/nova/compute/manager.py#5642 shouldn't we also revert to the original flavor in revert_resize ? | 13:35 |
| gibi | bauzas: is that missing? if yes then that is a separate bug | 13:35 |
| bauzas | I dunno | 13:35 |
| bauzas | I'm afraid you set a field on the instance before confirming | 13:36 |
| bauzas | which means a revert needs to get back | 13:36 |
| gibi | I set it earlier than before, but before it was still set during resize not resize confirm | 13:36 |
| bauzas | gibi: I can't see it in the left side | 13:37 |
| gibi | I'm not moving this field change from resize_confirm to resize. I move it from resize_finish (which is part of resize) to resize | 13:37 |
| bauzas | oh fuck | 13:37 |
| bauzas | you're right | 13:37 |
| bauzas | stupid me | 13:37 |
| bauzas | I confused myself with the resize methods naming | 13:37 |
| gibi | bauzas: sorry, yeah, first I moved, but that breaks upgrade, now I just double set | 13:37 |
| gibi | check the first PS | 13:37 |
| gibi | that has the move implemented | 13:38 |
| bauzas | resize_confirm != resize_finish | 13:38 |
| gibi | bauzas: yepp that naming is hard :D | 13:38 |
| bauzas | yeah you finish a resize before you confirm it :p | 13:38 |
| gibi | you finish the resize_instance RPC call on the destination :D | 13:38 |
| gibi | there can be finsh_confirm_resize too | 13:39 |
| gibi | but I think we dont call it like that | 13:39 |
| opendevreview | Merged openstack/nova master: Reproduce bug 1944759 https://review.opendev.org/c/openstack/nova/+/810763 | 13:47 |
| bauzas | gibi: yeah I know | 13:47 |
| bauzas | it was my brain which fscked | 13:48 |
| opendevreview | Lee Yarwood proposed openstack/nova master: Add regression test for bug #1943431 https://review.opendev.org/c/openstack/nova/+/810755 | 13:50 |
| opendevreview | Lee Yarwood proposed openstack/nova master: compute: Update volume_id within connection_info during swap_volume https://review.opendev.org/c/openstack/nova/+/807025 | 13:51 |
| opendevreview | Lee Yarwood proposed openstack/nova master: fup: Move _wait_for_volume_{attach,detach} to os-volume_attachments https://review.opendev.org/c/openstack/nova/+/810775 | 13:51 |
| opendevreview | Lee Yarwood proposed openstack/nova master: fup: Refactor and simplify Cinder fixture GET volume mock https://review.opendev.org/c/openstack/nova/+/810776 | 13:51 |
| lyarwood | sorry are we holding +Ws until we are out of rc? | 13:52 |
| gibi | lyarwood: I'm hesitant. bauzas? | 13:52 |
| bauzas | holding stable patches ? | 13:52 |
| bauzas | or master ones ? | 13:53 |
| gibi | I think lyarwood is asking for master | 13:53 |
| bauzas | former, yes | 13:53 |
| bauzas | latter, nope | 13:53 |
| gibi | ack, then I'm addin +A | 13:53 |
| bauzas | like, I litterally sent gibi's patch to the gate | 13:53 |
| gibi | to https://review.opendev.org/c/openstack/nova/+/810755 | 13:53 |
| lyarwood | cool yeah I just did for the regression test that landed above | 13:53 |
| bauzas | gibi: we branched Xena on RC1, right? | 13:53 |
| gibi | bauzas: right | 13:54 |
| bauzas | so we're officially working on Yoga with master | 13:54 |
| gibi | in my head RC period is when we want to keep master and stable close for easy backport of last minute issues | 13:54 |
| bauzas | that said, any backported change to stable/xena can't be approved until we deliver GA or it's a regression bugfix | 13:54 |
| bauzas | gibi: today is the last RC day | 13:55 |
| gibi | yepp I know | 13:55 |
| bauzas | I don't think we're taking an absolute risk | 13:55 |
| gibi | OK | 13:55 |
| bauzas | this is tho an interesting thought | 13:55 |
| bauzas | honestly, I think we should make more room for bugfixing during our release cadence | 13:56 |
| bauzas | but let's discuss this at the PTG | 13:56 |
| lyarwood | bug fixing never stops ;) | 14:02 |
| gibi | bauzas: you mean feature freezing at F2? ;) | 14:03 |
| bauzas | I don't know | 14:04 |
| bauzas | I'm not opiniated about the solution | 14:04 |
| bauzas | if we could somehow prioritize bugs over features during a period of time that would be explicitely and properly communicated, this would get my interest | 14:04 |
| bauzas | but that also means that people thinking about new features need to understand in advance our release cadence (ideally drafting before the PTG) | 14:05 |
| opendevreview | Merged openstack/nova master: Add section for 'nova-manage placement audit' tool https://review.opendev.org/c/openstack/nova/+/809479 | 14:10 |
| *** haleyb is now known as haleyb|out | 14:11 | |
| *** bhagyashris_ is now known as bhagyashris|rover | 14:19 | |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748 | 15:19 |
| bauzas | gibi: sorry was unclear with my comment on your fix, as I really didn't wanted you to invest time into functesting different compute versions, but this is possible https://github.com/openstack/nova/blob/d64edd3da2336a5c7c8f69cced45272cbaf638a9/nova/tests/functional/test_cold_migrate.py#L74 | 15:49 |
| bauzas | this is just a fyk | 15:50 |
| gibi | bauzas: that probably works if the old behavior is tight to an old service version. In my case there is no service version bump | 15:51 |
| bauzas | then I can't remember where but I'm also sure I wrote some upgrade checks | 15:52 |
| bauzas | anyway | 15:52 |
| bauzas | iirc, the idea is to pin rpc versions to make the compute appear "old" | 15:53 |
| gibi | yepp, if there is such old version | 15:57 |
| gibi | but in my case there is no version difference between the unpatches and the patches compute | 15:57 |
| gibi | unpatched and patched | 15:58 |
| melwitt | elodilles: hi, what is the other patch you think might be better for fixing l-c on stable/train? re: https://review.opendev.org/c/openstack/nova/+/811762/1#message-d28da6263de9241f21a437daf6a0b7273eec47e4 | 15:59 |
| melwitt | I'm not opinionated how to fix it, so if there is another way, I'm happy to rebase the patches onto that | 16:00 |
| opendevreview | Merged openstack/nova master: Update contributor guide for Yoga https://review.opendev.org/c/openstack/nova/+/809936 | 16:01 |
| opendevreview | Merged openstack/nova master: Add regression test for bug #1943431 https://review.opendev.org/c/openstack/nova/+/810755 | 16:12 |
| elodilles | melwitt: I though about this one: https://review.opendev.org/c/openstack/nova/+/810461 | 16:40 |
| elodilles | I missed that it is not yet merged o:) | 16:41 |
| elodilles | so anyway, both ways work for me | 16:41 |
| melwitt | elodilles: ah ok, thanks for the link. does it need an update based on the latest comments about min version? or would that be a separate patch? | 17:01 |
| melwitt | elodilles: also meant to tell you I saw grenade jobs on stable/train also failing with "AttributeError: module 'jsonschema' has no attribute 'compat'" https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#28564 is this also related to the setuptools issue? | 17:14 |
| elodilles | melwitt: oh, I haven't seen that yet :( that most probably the very same jsonschema 4.0.0 release from yesterday that we faced in openstack-releases' validator :( | 17:34 |
| elodilles | though jsonschema should be upper constrained in stable O.o | 17:37 |
| melwitt | hm yeah | 17:39 |
| melwitt | elodilles: oh but the error was raised from tempest, which would be coming from master | 17:41 |
| elodilles | actually tempest should be used from train-last tag and still, should be installed with upper constraints. or am I wrong? | 17:43 |
| elodilles | seemingly I am wrong at some point otherwise it wouldn't fail :) | 17:44 |
| melwitt | well.. not sure. because if this fails here it seems like it would fail in the tempest repo itself but it's not | 17:46 |
| melwitt | I see that the job began collecting jsonschema==2.6.0 but ended up installing 4.0.0 | 17:48 |
| melwitt | but why... | 17:48 |
| melwitt | or how. it's pinned to 3.0.2 as you said earlier https://github.com/openstack/requirements/blob/stable/train/upper-constraints.txt#L634 | 17:49 |
| melwitt | collecting: https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#38834 | 17:49 |
| melwitt | ok but then it uninstalls 4.0.0 and installs 2.6.0 here https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#38886-38895 | 17:51 |
| melwitt | and then tempest installs 4.0.0 https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#55512 | 17:52 |
| melwitt | gmann: does tempest running on stable branch not use upper-constraints from stable? ^ | 17:52 |
| elodilles | actually, since this is a grenade job, it should be stein | 17:55 |
| elodilles | where jsonschema is constrained 2.6.0 | 17:55 |
| melwitt | oh right | 17:56 |
| elodilles | but tempest @ stein-last requires: jsonschema>=3.2.0 # MIT | 17:56 |
| elodilles | so it is strange how it works | 17:57 |
| elodilles | (just for the record, currently jsonschema===3.2.0 is in the upper-constraints.txt @ master) | 18:00 |
| melwitt | yeah, I don't understand it | 18:12 |
| opendevreview | Lee Yarwood proposed openstack/nova master: trivial: Rename LibvirtConfigGuestPCIeRootPortController https://review.opendev.org/c/openstack/nova/+/812002 | 18:23 |
| opendevreview | Merged openstack/nova master: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810909 | 20:07 |
| gmann | melwitt:elodilles: in train, we still using tempest master which I need to move to train-last soon | 21:32 |
| melwitt | gmann: lmk if I can help. stable/train currently broken due to new jsonschema 4.0.0 from master | 21:32 |
| gmann | in stein yes tempest 26.0.0 is used | 21:32 |
| gmann | melwitt: ok. I need to go out to pick up my wife. I will check after coming back | 21:33 |
| melwitt | gmann: np thanks, I can work on it, just I would need a hint where to make a change | 21:35 |
| gmann | melwitt: at first glance, it seems it is using stein branch constraint only https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#12290 | 21:39 |
| melwitt | hm ok | 21:41 |
| gmann | melwitt: I think tempest job on stable/stein also should fail on this. | 22:16 |
| melwitt | gmann: yeah.. I've been looking at how it says it's using the stein upper-constraints file, but then after it does "tox -r -notest -efull" the 4.0.0 jsonschema gets installed. I don't understand that | 22:17 |
| gmann | melwitt: it is pinned to used stable/stein u-c in stable/stein with tempest 26.0.0 | 22:17 |
| melwitt | so I was looking at https://github.com/openstack/tempest/blob/5d7e46f5689040ddaff798662f7a44fc71758d2e/tox.ini#L14 and I didn't know what the -c does but it looks like this is supposed to be use the env var upper-constraints file if present else use master? | 22:18 |
| gmann | but it should not install latest jsonschema | 22:18 |
| melwitt | here it says "full installed" ... jsonschema==4.0.0 https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#12300 | 22:19 |
| melwitt | I took that to mean the tox -efull env installed 4.0.0 | 22:19 |
| gmann | but later on it uninstall 4.0.0 and install 2.6.0 https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#12423-12427 | 22:22 |
| melwitt | oh, again? oh sorry | 22:23 |
| melwitt | ok here's the last time it installs 4.0.0 before the error https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#28518 | 22:24 |
| gmann | but in final installed version in venv I always see 4.0.0 so not sure who & when it is installed/upgraded | 22:24 |
| melwitt | I get confused by how it installs and uninstalls 4.0.0 repeatedly | 22:24 |
| melwitt | venv installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt, -r/opt/stack/old/tempest/doc/requirements.txt | 22:25 |
| gmann | as it fallback to latest bcz u-c not matched, we should use the compatible constraint while we cut the tempest 26.0.0 instead of stable/stein by default. | 22:32 |
| gmann | at least that is tested combination tempest we released tempest 26.0.0 | 22:37 |
| melwitt | what does it mean u-c didn't match? | 22:38 |
| gmann | tempest 26.0.0 requirement has jsonschema>=3.2.0 and u-c jsonschema== 2.6.0 - https://github.com/openstack/tempest/blob/26.0.0/requirements.txt#L6 | 22:40 |
| melwitt | ohhh I see. ok | 22:40 |
| gmann | that is why it try to go for latest jsonschema | 22:40 |
| gmann | complex thing here is to make tempest venv work on single (non-master) constraints, we have to take care in many place otherwise tox will recreate the tempest venv. many place I mean in devstack, run-tempest role, tox.ini etc | 22:41 |
| melwitt | I see now. I didn't know it would ignore u-c if the requirement was higher than it ... I thought it would have failed like "can't install the tempest that you want" | 22:41 |
| clarkb | melwitt: it should always use the constraint | 22:42 |
| clarkb | or fail I guess, but ya if it doesn't do that I would consider that a bug in pip | 22:42 |
| gmann | it actuall try to lowerdown the version as per -c but we have verify-config script from tempest running tempest in between so it fail | 22:42 |
| gmann | otherwise the final version could be as per u-c only | 22:43 |
| melwitt | hm O.o | 22:43 |
| gmann | like it happen in master testing, it install 3.2.0 as per u-c not latest | 22:43 |
| melwitt | sorry I'm getting even more confused. it's the verify-config script from tempest that is installing > u-c? | 22:44 |
| gmann | melwitt: no in between somehow latest jsonschema is installed and verify-config run fail. why it install latest jsonschema and not failing on version conflict might be bug as clarkb mentioned | 22:45 |
| gmann | basically in both case either failed with conflict or latest jsonschema, our constraint mismatch is issue | 22:46 |
| melwitt | yeah. ok | 22:46 |
| clarkb | Looking at the log pip says "tempest 26.0.0 has requirement jsonschema>=3.2.0, but you'll have jsonschema 2.6.0 which is incompatible." then it installs jsonschema 2.6.0 | 22:47 |
| clarkb | I think constraitns are working properly as a result | 22:47 |
| clarkb | (that was my understanding of how they work, constraints always wins) | 22:47 |
| gmann | we have two way to solve it 1. use tempest tag (<26.0.0) which is compatible with stable/stein constraints 2. use compatible constraints (higher than of stable/stein) with tempest 26.0.0 | 22:47 |
| clarkb | your requirements and constraints are at odds with each other | 22:47 |
| gmann | yeah | 22:47 |
| melwitt | yeah, that's why I wondered if https://github.com/openstack/tempest/blob/5d7e46f5689040ddaff798662f7a44fc71758d2e/tox.ini#L14 is doing something wrong, the log says that "tempest full" is what installed 4.0.0 | 22:48 |
| gmann | requirement is from tempest 26.0.0 which is cut at the time of stable/vicrotia | 22:48 |
| gmann | clarkb: but it does not say who installed 4.0.0 but we see that version as final version in installed version in venv | 22:49 |
| clarkb | are you sure 4.0.0 is the final version? | 22:50 |
| clarkb | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_08e/791807/5/check/neutron-grenade-multinode/08e3749/controller/logs/old/devstacklog.txt shows it is failing because jsonschema.compat isn't present which I assume is in >=3.2.0 | 22:50 |
| gmann | clarkb: https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#28518 | 22:50 |
| gmann | they seems removed in 4.0.0 that is why installing 4.0.0. end up started failing due to this constraints and requirement mismatch | 22:51 |
| gmann | I checked before 4.0.0 release we were istalling latest and it was juts working because of no incompatible change there | 22:52 |
| clarkb | ok I think I see it | 22:52 |
| clarkb | there are two different venvs | 22:53 |
| clarkb | 'venv' and 'venv-tempest | 22:53 |
| clarkb | they both install jsonschema 4.0.0 using venv-tempest installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt and venv installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt, -r/opt/stack/old/tempest/doc/requirements.txt | 22:53 |
| clarkb | But venv-tempest has an extra step where it does venv-tempest run-test: commands[0] | pip install -c /tmp/tempest_u_c_m.03mOFK3yxm -r requirements.txt | 22:54 |
| clarkb | It is exceptionally odd to me to run pip in the run-test portion of a tox target | 22:54 |
| clarkb | but there you have it | 22:54 |
| clarkb | looks like venv-tempest does command = {posargs} so I'm not sure what calls it with that to downgrade | 22:55 |
| clarkb | but that modifies .tox/tempest and not .tox/venv | 22:56 |
| clarkb | then you try ot run something in .tox/venv and it breaks due to the newer version of jsonschema | 22:56 |
| melwitt | my head is spinning 😆 | 22:57 |
| melwitt | but I'm very glad to know what is going on. I'll need time to process it | 22:57 |
| clarkb | https://opendev.org/openstack/devstack/src/branch/master/lib/tempest#L627 that is where the pip install that puts jsonschema 2.6.0 into .tox/tempest via the venv-tempest target comes from | 22:58 |
| clarkb | melwitt: my head is spinning too :) there is a lot of back and forth in here and I'm not really sure I understand why as much as now I at least understand how that exception arises | 22:58 |
| melwitt | yeah... I hope there's a way to fix the root cause, having it be able to do this seems too confusing and error prone | 22:59 |
| clarkb | then we run https://opendev.org/openstack/devstack/src/branch/master/lib/tempest#L649 and it explodes because the version is too new | 23:00 |
| clarkb | gmann: I think that means you need a newer version of tempest that works against newer jsonschema as constrained there? | 23:00 |
| clarkb | rather than older | 23:01 |
| gmann | older | 23:01 |
| clarkb | hrm wait -c/opt/stack/old/requirements/upper-constraints.txt that should be using stable/stein requirements in old/ right? | 23:02 |
| clarkb | because this job ran against train | 23:02 |
| clarkb | stable/stein requirements u-c says jsonschema should be 2.6.0 | 23:02 |
| melwitt | right | 23:03 |
| clarkb | the job-output.txt for that build has made my browser very sad | 23:04 |
| melwitt | my browser has a sad as well | 23:04 |
| gmann | heh, i opened three and it stuck | 23:06 |
| gmann | clarkb: melwitt https://review.opendev.org/c/openstack/devstack/+/812092 | 23:06 |
| gmann | let' see | 23:06 |
| gmann | I am going with 1st solution of lowering the tempest as per stable/stein constraints | 23:06 |
| clarkb | gmann: if that fixes it it doesn't explain why old/requirements somehow became newer requirements | 23:07 |
| clarkb | I do see that old/requirements is initially checked out to 3a3ed8a7 which is correct | 23:07 |
| gmann | clarkb: did not get? | 23:07 |
| clarkb | gmann: requirements u-c for stable/stein has jsonschema===2.6.0 in it | 23:07 |
| clarkb | gmann: that means we should never have installed 4.0.0 | 23:07 |
| gmann | tempest 26.0.0 requirement has jsonschema>=3.2.0 and u-c jsonschema== 2.6.0 - https://github.com/openstack/tempest/blob/26.0.0/requirements.txt#L6 | 23:07 |
| clarkb | that implies to me that the u-c file we are using is not for stable/stein | 23:08 |
| clarkb | gmann: yes but we install 4.0.0 | 23:08 |
| clarkb | which is from a newer u-c | 23:08 |
| melwitt | yeah.. I don't know that there's a way to see the file | 23:08 |
| melwitt | bc I was wondering the same thing earlier | 23:08 |
| gmann | it try to check 2.6.0 as per u-c | 23:08 |
| gmann | 4.0.0 is coming from somewhere, it is not yet in master u-c | 23:09 |
| gmann | master u-c is also 3.2.0 | 23:09 |
| clarkb | hrm | 23:09 |
| gmann | how 4.0.0 is installed and u-c complain did not fail is not known to me | 23:10 |
| clarkb | earlier in the job we see Requirement already satisfied: jsonschema===2.6.0 in /usr/local/lib/python3.6/dist-packages (from -c /opt/stack/old/requirements/upper-constraints.txt (line 232)) | 23:11 |
| clarkb | that shows us that /opt/stack/old/requirements/upper-constraints.txt (line 232) is correct until that point | 23:11 |
| melwitt | line 12300 is the first time I see jsonschema==4.0.0 | 23:13 |
| melwitt | it does "lib/tempest:install_tempest:731: set_tempest_venv_constraints /tmp/tempest_u_c_m.IxpNU7ab69" before that but afaict TEMPEST_VENV_UPPER_CONSTRAINTS=/opt/stack/old/requirements/upper-constraints.txt always | 23:16 |
| clarkb | melwitt: ya and a little later it uses /tmp/tempest_u_c_m.IxpNU7ab69 in a pip install in that venv which downgrades jsonschema | 23:16 |
| clarkb | I wonder if this issue was known and that is why there is the explicit step after the fact of reinstalling with constraints | 23:17 |
| melwitt | oh.... seems likely | 23:17 |
| clarkb | basically for $some reason the initial tox setup doesn't do the right thing. So someone hacked in the extra pip install to devstack to fix things | 23:17 |
| clarkb | except we don't do the extra pip install for the .tox/venv venv target later and it breaks | 23:17 |
| melwitt | there's this note that I don't understand https://github.com/openstack/devstack/blob/57a868dd874922a0caed8ace0dc0426f29129277/lib/tempest#L712 | 23:19 |
| clarkb | ok I think I've got it | 23:19 |
| clarkb | the tox venv setup happens in two passes. The first pass installs all requirements as listed in the requirements file with the constraints | 23:20 |
| clarkb | then it pip install tempest | 23:20 |
| clarkb | The expectation is that all of tempest's deps be previously installed properly but jsonschema isn't because tempest says it wants jsonschema>=3.2.0 | 23:20 |
| clarkb | gmann: ^ maybe this is what you were saying. | 23:20 |
| clarkb | I guess normally you don't have this mismatch because because tempest doesn't branch it gets weird | 23:21 |
| melwitt | ok so when it installs tempest it doesn't use u-c? every command I saw seemed to be passing a u-c file though, that's what I don't get | 23:21 |
| clarkb | melwitt: right tox does a two step install. It installs everything listed in the deps directive. Then it installs the actual repo it is in | 23:22 |
| clarkb | The second step install isn't suepr configurable iirc | 23:22 |
| clarkb | gmann: https://opendev.org/openstack/tempest/src/tag/24.0.0/requirements.txt#L6 is the newest tempest with jsonschema 2.6.0 | 23:22 |
| gmann | clarkb: yes, that is last tag with 2.6.0 | 23:24 |
| gmann | and on the tox step: yes and before u-c things setup the right version verify-config run make it fail | 23:25 |
| gmann | which is run directly with venv not 'going inside the venv and then run ' | 23:25 |
| gmann | 'directly with vnev' i mean it create venv first and then run before u-c things happen | 23:26 |
| gmann | tox -r --notest -efull is also very first creation of vnev but as it does not run anything so it just does not complain on 4.0.0 | 23:27 |
| clarkb | gmann: well full also fixes the constraints after the fact | 23:27 |
| gmann | yes after it fix but in verify-config case it run tempest things before constraints fixes. that is what i am suspecting. 'installing with req and then later at some step tox handle the constraints' | 23:28 |
| gmann | like, tox installing all as per req -> running verify-config -> will take care of constraints(but it fail before verify-config itself) | 23:29 |
| clarkb | https://tox.wiki/en/latest/config.html#conf-install_command I think that can be used to make this better | 23:30 |
| gmann | that is sequence I am suspecting which causing failure. thoguh we had bad compatiblity on req and constraint | 23:30 |
| clarkb | basically create an install command that has -c /path/to/constraints/ then the same constraints application will be used on deps and the actual package | 23:30 |
| gmann | clarkb: yeah, we should do that way | 23:30 |
| gmann | clarkb: and it was like that before jsonschema 4.0.0 which could have been caught if installtion fail on req-constraints mismatch | 23:33 |
| clarkb | well constraints never cause a failure at install time. The constraint just wins always | 23:33 |
| clarkb | but you get the warning about it at least | 23:33 |
| gmann | ah right. | 23:35 |
| gmann | clarkb: melwitt let's see if this work (it should) https://review.opendev.org/c/openstack/devstack/+/812092 | 23:35 |
| gmann | and I will say we always use first compatible stable version of tempest insetad of last and figure out the compatible newer constraints | 23:36 |
| gmann | for EM stable | 23:36 |
| clarkb | gmann: you might try to use 24.0.0 instead | 23:36 |
| clarkb | and then treat that as the last compatible version | 23:36 |
| gmann | clarkb: 24.0.0 might create issue in future if anything else newer comes as it is not tested on stable/stein constraints | 23:36 |
| gmann | for jsonschema might work | 23:37 |
| gmann | I was too optimistic on using the last compatible(newer) tempest with old constraints and assumed it will work :) | 23:38 |
| clarkb | gmann: it will work if you use the tox install command thing. But I guess the ship has sailed for old tags | 23:39 |
| clarkb | but if you update the tox.ini before the next tag it would be corrected for the future | 23:39 |
| gmann | clarkb: yes that I will do for next tag, agree on that | 23:40 |
| gmann | clarkb: and another things with constraints is, as we touch tempest venv using tox command in many place like running test in grenade/run-tempest role ect if constraints mismatch at any place it will re-create the tox which is another issue | 23:40 |
| gmann | so simple things for EM stable is use tested compatible version and constraints and do not assume the things :). | 23:41 |
| clarkb | ya if 20.0.0 is the most aligned with stein's constraints that seems like the one to use | 23:41 |
| gmann | yes, that is version released while openstack stein release. | 23:42 |
| gmann | for tox recreation things I was searching for any flag where we can stop tox to recreate and force-live with what is installed but there is no flag seems | 23:43 |
| gmann | that could have solved our problem with tempest venv testing case | 23:43 |
| gmann | clarkb: melwitt 20.0.0 seems working. https://zuul.openstack.org/stream/9234f33acf1247cdb995416ca9389240?logfile=console.log | 23:44 |
| gmann | let me put nova train DNM patch also as grenade also toush tox env while running test | 23:45 |
| melwitt | thank you gmann | 23:47 |
| opendevreview | Ghanshyam proposed openstack/nova stable/train: DNM: testing nova grenade job https://review.opendev.org/c/openstack/nova/+/812095 | 23:47 |
| gmann | ^^ | 23:47 |
| melwitt | and clarkb++ | 23:47 |
| gmann | I will add note in devstack/tempest about it while pinning tempest and its constraints for future EM stable | 23:48 |
| gmann | and install-command thing | 23:48 |
| melwitt | yes please \o/ | 23:49 |
| gmann | melwitt: sorry about issue, its tempest constraints things on EM stable always cause new issue over time. and honestly saying I am not 100% confident if that happen in future too :) but at least testing with version released duing that stable branch will make things simple | 23:51 |
| gmann | * I am not 100% confident that it will not happen in future too : | 23:52 |
| melwitt | gmann: I'm not complaining as I barely understand all this :) just happy you'll be able to do something to fix or make it less likely to happen in the future | 23:53 |
| gmann | sure, will do those doc things tomorrow if it works fine in grenade too. | 23:54 |
| gmann | melwitt: ah lower constraints seem broken on stable/train. is it new ? https://zuul.opendev.org/t/openstack/build/2cdc193e00974abcb67b58ffe566f371 | 23:56 |
| melwitt | gmann: no it's not. sec | 23:56 |
| gmann | melwitt: it seems failing in other old patches too https://review.opendev.org/c/openstack/nova/+/811824 | 23:56 |
| melwitt | gmann: yeah. there's two options, one is make l-c non-voting https://review.opendev.org/c/openstack/nova/+/811762 and the other is pin setuptools https://review.opendev.org/c/openstack/nova/+/810461 | 23:58 |
| gmann | ah that one. i got it | 23:58 |
| gmann | I voted for first even remove testing l-c | 23:58 |
| melwitt | it sounded like consensus favors the latter. I was just not sure if it needs an update before approving, based on discussion on the patch | 23:59 |
| melwitt | ah, heh. yeah. I am neutral about it | 23:59 |
| melwitt | since we have a fix, we can fix this time ... until next break :) | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!