| tonyb | prometheanfire: I went ahead and did a thing: https://review.opendev.org/c/openstack/governance/+/854685 | 01:27 |
|---|---|---|
| tonyb | My thinking is that if we do this now, I can be an election offical and help with that? | 01:28 |
| prometheanfire | lol | 01:37 |
| prometheanfire | huh, tox failed | 01:38 |
| *** yadnesh|away is now known as yadnesh | 04:20 | |
| sean-k-mooney1 | tonyb: well for me this is less of an issue as nova does not have any voting jobs that are impacted by it efoley_ is more impacted as they are trying to improve the testing on centos based jobs and currently the telemetry project have several based on censtos 8 stream that she is trying to get working. | 10:47 |
| *** sean-k-mooney1 is now known as sean-k-mooney | 11:47 | |
| gmann | sean-k-mooney: tonyb for tempest to support 3.6, we have two solution for that, 1. use old tempest 30.0.0 2. use virtual env with latest tempest and test the py3.6 env/job/cloud | 13:54 |
| *** yadnesh is now known as yadnesh|away | 13:56 | |
| sean-k-mooney | gmann: latest tempest does not work | 13:56 |
| sean-k-mooney | not for centso 8 stream | 13:57 |
| sean-k-mooney | where they python verison is 3.6 | 13:57 |
| sean-k-mooney | and we can meet clifs requirement for 3.7+ | 13:57 |
| sean-k-mooney | we dont have the option of using a newer 3.8 virtual env currently i think in that configurtion | 13:57 |
| sean-k-mooney | we can use a vemvn but that will jsut use the default python3 version | 13:57 |
| gmann | for that yes we need to use old tempest | 13:58 |
| sean-k-mooney | yep efoley has a patch to try and to that for the telemerty jobs | 13:58 |
| sean-k-mooney | by setting the devstack varibles | 13:58 |
| sean-k-mooney | but that need the upperconstirts to be pinned on yoga and xena for py 3.6 | 13:59 |
| sean-k-mooney | for jsonschma and cliff | 13:59 |
| sean-k-mooney | unless we do that in devstack | 13:59 |
| sean-k-mooney | the way we do for EM | 13:59 |
| sean-k-mooney | excpt make that change depend on the python version | 13:59 |
| gmann | yes, make sure tempest constraints also pinned via TEMPEST_BRANCH and TEMPEST_VENV_UPPER_CONSTRAINTS var | 14:00 |
| sean-k-mooney | yep so that on its own is not enough | 14:00 |
| sean-k-mooney | without the patch to the requiremetn repo | 14:00 |
| sean-k-mooney | but that is why efoley is currently trying to do | 14:00 |
| sean-k-mooney | https://review.opendev.org/c/openstack/ceilometer/+/853668/12/.zuul.yaml | 14:01 |
| sean-k-mooney | that depends on the requirementns change https://review.opendev.org/c/openstack/requirements/+/854130 | 14:01 |
| gmann | sean-k-mooney: yeah I agree. and would not be surprise if more such requirement update is needed | 14:02 |
| sean-k-mooney | i have been helping her explore differnt ways to adress this over the last week or two but we basicaly got to the point that either we need to clamp them in the requirement repo + pin tempest or change devstack to use a differnt interperty for those jobs | 14:04 |
| gmann | ok, former one is what we usually do and later one might need bigger changes | 14:05 |
| sean-k-mooney | gmann: would you perfer ot do this in devstack i.e. detech that its python 3.6 on stable yoga/xena and automatically pin tempets there | 14:05 |
| gmann | sean-k-mooney: until they are not EM, we need to use tempest master which is working currently via virtual env for other distro like ubuntu etc | 14:06 |
| sean-k-mooney | yep | 14:06 |
| sean-k-mooney | hench only do it for py36 | 14:06 |
| gmann | sean-k-mooney: and for this c8s, pinning from job side and do change in requirement is good way to proceed | 14:06 |
| sean-k-mooney | ack | 14:07 |
| sean-k-mooney | ya thats why i was suggeign to do it in the job definiton orginally | 14:07 |
| gmann | +1 | 14:07 |
| sean-k-mooney | so if we are agreed on the general direction we just need to figure out what both changes should actully contian | 14:08 |
| gmann | for py2.7 drop case we did the same for few jobs in ironic and neutron stable jobs | 14:08 |
| sean-k-mooney | efoley is pretty expirnce with openstack but this part of the job config is new to her and im kind of reaching my knoladw of this too | 14:11 |
| sean-k-mooney | so if you can point her in the right direction that would help | 14:11 |
| gmann | commented on gerrit | 14:31 |
| opendevreview | Emma Foley proposed openstack/requirements stable/yoga: Update requirements for python36 https://review.opendev.org/c/openstack/requirements/+/854130 | 14:55 |
| opendevreview | OpenStack Proposal Bot proposed openstack/requirements master: update constraint for os-traits to new release 2.9.0 https://review.opendev.org/c/openstack/requirements/+/854821 | 17:13 |
| opendevreview | OpenStack Proposal Bot proposed openstack/requirements master: update constraint for metalsmith to new release 1.8.0 https://review.opendev.org/c/openstack/requirements/+/854825 | 18:10 |
| opendevreview | OpenStack Proposal Bot proposed openstack/requirements master: update constraint for sushy to new release 4.3.0 https://review.opendev.org/c/openstack/requirements/+/854826 | 18:14 |
| opendevreview | OpenStack Proposal Bot proposed openstack/requirements master: update constraint for os-brick to new release 6.1.0 https://review.opendev.org/c/openstack/requirements/+/854828 | 18:19 |
| opendevreview | OpenStack Proposal Bot proposed openstack/requirements master: update constraint for kuryr-lib to new release 2.6.0 https://review.opendev.org/c/openstack/requirements/+/854829 | 18:22 |
| opendevreview | OpenStack Proposal Bot proposed openstack/requirements master: update constraint for monasca-statsd to new release 2.5.0 https://review.opendev.org/c/openstack/requirements/+/854830 | 18:25 |
| opendevreview | OpenStack Proposal Bot proposed openstack/requirements master: update constraint for keystonemiddleware to new release 10.0.1 https://review.opendev.org/c/openstack/requirements/+/854831 | 18:26 |
| opendevreview | OpenStack Proposal Bot proposed openstack/requirements master: update constraint for mistral-lib to new release 2.7.0 https://review.opendev.org/c/openstack/requirements/+/854832 | 18:29 |
| opendevreview | OpenStack Proposal Bot proposed openstack/requirements master: update constraint for ironic-lib to new release 5.3.0 https://review.opendev.org/c/openstack/requirements/+/854835 | 18:38 |
| tonyb | gmann: The second one isn't an option *unless* we are also okay with running tempest, inside the venv with a python interpreter *not* supplied from the vendor | 18:46 |
| gmann | yeah, pining tempest is best approach here | 18:58 |
| tonyb | I don't like that as it imples that Yoga (and active supported branch) will *never* need to be patched (on centos 8) | 19:00 |
| gmann | remember py3.6 support in yoga was best possible effort and it was re-added for centos stream point of view | 19:01 |
| gmann | using tempest last compatible version with 3.6 for 3.6 only distro is right approach | 19:01 |
| tonyb | gmann: I don't see that in https://governance.openstack.org/tc/reference/runtimes/yoga.html its says c8 and py36 are supported and required for testing on yoga | 19:03 |
| tonyb | gmann: but pinning tempest has the potential to block the gate on yoga for all projects that have voting c8 jobs | 19:03 |
| gmann | centos stream 8 and py3.6 is there | 19:04 |
| tonyb | gmann: Yes but nothing about "best effort" is says "all Python-based projects must target and test against, at a minimum" | 19:04 |
| gmann | tonyb: https://review.opendev.org/c/openstack/governance/+/820195, commit message has all the details and ML link why we re-added py3.6 support in yoga | 19:06 |
| gmann | and not all projects has centos stream 8 job | 19:07 |
| tonyb | gmann: I still see nothing about "best effort" I see "c8 and u18.04 are supported and needed" on yoga | 19:07 |
| tonyb | gmann: I know not all projects test on c8, but those that do have the potental or block the yoga gate | 19:08 |
| gmann | I mean due to their request for migration from centos stream 8 to centos stream 9 we added both version as supported an dpy3.6 | 19:08 |
| gmann | py3.6 | 19:08 |
| tonyb | (if we pin tempest) | 19:08 |
| gmann | it does not block if we pin tempest | 19:08 |
| gmann | that is how we tested py27 and py3 on train. pin the tempest for py27 job and use master in py3 jobs | 19:09 |
| tonyb | Okay lest take a different approach | 19:09 |
| tonyb | If we pin tempest on c8 on yoga we pin to an immutable tag correct? | 19:09 |
| gmann | yes | 19:10 |
| tonyb | So assume $project has at least one voting c8 on yoga, along with the normal ubuntu testing | 19:12 |
| tonyb | now a bug in $project is discovered on yoga that Depends-On a tempest (master because branchless) patch to test | 19:13 |
| tonyb | How does the voting c8 job get that tempest chnage? | 19:13 |
| tonyb | *change | 19:13 |
| tonyb | gmann: ? | 19:20 |
| gmann | tonyb: yes for those new kind of tests they do not get not get benefit. that is tradeoff they have todo for having old python version | 19:20 |
| gmann | and any new test in tempest is mostly for the new features on master. it is rare to have new tests for stable branch changes | 19:21 |
| tonyb | It's more than no benefit, it's a blocked gate. | 19:21 |
| gmann | how it will block gate? it will just not letting them to use the tempest master new tests which are mostly added for master changes | 19:22 |
| tonyb | gmann: rare but not impossible, when I was nova-stable we hit a number of issues that needed to be backported and needed tempest test chnages because it alters $projct behaviour | 19:22 |
| gmann | it will benefit them to test the existing thing with existing test so that no backward incompatible backport is done | 19:22 |
| gmann | tonyb: can you link me what all nova changes needed tempest change? | 19:23 |
| tonyb | You end up with a test that works on ubuntu (because it gets the fix) and not on c8 and as c8 can't get a new chnage it will never pass | 19:23 |
| gmann | if any backport need tempest change then it is issue in backport | 19:23 |
| tonyb | gmann: No this was years ago | 19:23 |
| gmann | at least I have not noticed such example since pike | 19:23 |
| gmann | if any backport need tempest change then it is a good question on backport as it will break your user also being tempest as a good user of OpenStack | 19:24 |
| tonyb | gmann: Yes I am aware, I was pretty involved in the stable team | 19:24 |
| gmann | I am not saying none of backport need tempst change it may need because of test bug but it is very rare | 19:25 |
| tonyb | you're getting distracted | 19:25 |
| tonyb | Can you see now how there is *potential* albeit rare for pinning tempest to block the yoga gate | 19:25 |
| tonyb | ? | 19:25 |
| gmann | I am saying using tempest 30.0.0 in stable yoga does not loose test coverage and not create any backport issue | 19:25 |
| gmann | tonyb: only for the c8s jobs | 19:26 |
| tonyb | gmann: please elaborate on that last statement | 19:26 |
| gmann | and if that happen we will see what to do for c8s testing. we do take decision on such testing on stable branches based on case by case and best effort to test every case in upstream | 19:26 |
| gmann | if any backport needed tempest change (very rare) and block c8s job then we can take decision if we want to make c8s job non voting until we have new python interpreter option | 19:28 |
| gmann | going with py3.6 constraints in requriement and use of old tempest is best possible way to forward | 19:28 |
| tonyb | so you can see there is a route for there to be a problem? | 19:28 |
| gmann | tonyb: very very rare and i will not block things for that rare cases | 19:29 |
| tonyb | we're still getting on the same page for the problem discussions of solutions can wait | 19:29 |
| gmann | when that happens we will see | 19:29 |
| tonyb | it will block things for any project that has c8 *and* ubuntu jobs | 19:29 |
| gmann | this is samea for EM branches also, should we block them too? | 19:30 |
| gmann | they are blocked now also | 19:30 |
| tonyb | and if that project is on the integrated gate it will block everything | 19:30 |
| gmann | which c8s job with tempest master is passing? | 19:30 |
| tonyb | *but* the EM period is explictly "best effort" | 19:30 |
| gmann | these distro with such old python version are also same | 19:31 |
| gmann | testing runtime is all about best effort not guaranteed testing things | 19:31 |
| tonyb | Yes it isn't *just* c8 but that's easy to type | 19:31 |
| gmann | which other distro? | 19:31 |
| tonyb | u 18.xx | 19:31 |
| gmann | all other distro supported in yoga has >py3.6 | 19:32 |
| gmann | we do not support u 18.xx in yoga | 19:32 |
| sean-k-mooney | yep excption c8s | 19:32 |
| gmann | only case here broken is c8s | 19:32 |
| sean-k-mooney | we extended 3.6 support to yoga because fo rdo/ooo and c8s | 19:32 |
| gmann | which was added in supported in yoga after lot of discussion as mentioned in the commit msg of patch i mentioned | 19:33 |
| gmann | sean-k-mooney: exactly | 19:33 |
| tonyb | gmann: Okay I as wrong about u18.xx | 19:33 |
| gmann | if any rare case happen and c8s job are blocked then we make them non voting | 19:33 |
| gmann | until anyone want to solve/fix them with new version interpreter | 19:34 |
| tonyb | That is a terrible outcome | 19:34 |
| gmann | one important thing to note: from 2023.1 cycle we are making centos stream as 'bets effort' testing distro as all jobs are periodic not in gate queue | 19:34 |
| tonyb | we're have a rare but definable potential gate blocker and we're just going to "#shurg future peoples problem yolo"? | 19:35 |
| gmann | so they know how difficult is their testing in OpenStack CI due to nature of their distro or so | 19:35 |
| gmann | we discussed that last month only | 19:35 |
| tonyb | gmann: the 2023.1 point is orthogonal | 19:35 |
| sean-k-mooney | well right now emma is tryign to fix the c8s situration | 19:35 |
| sean-k-mooney | by fixing the requirements for py3.6 | 19:36 |
| gmann | yeah, not sure why we are blocking her fix whihc is at least making c8s job working | 19:36 |
| tonyb | sean-k-mooney: Yup I just want to make sure we all understand the problem so we can also understand the potential solutions | 19:36 |
| tonyb | gmann: because the fix *wont* fix anything | 19:36 |
| sean-k-mooney | basically emma want the upstream jobs to catch things before tehy break use downstream in rdo/osp | 19:36 |
| gmann | tonyb: we understand the problem since py2.7 drop time and it is tested like that since than. any distro having old pythn and does not match with tempest master then we need to pit it like for py3.5 | 19:37 |
| tonyb | I showed how to trivually reproduce the issue outside of the gate and the stable/yoga requitremnts fix doesn't / can't help | 19:37 |
| sean-k-mooney | she is also lookign to update some of them to a more recent centos in some cases | 19:37 |
| gmann | most of the project gate using c9s in gate | 19:38 |
| sean-k-mooney | tonyb: did you see the patch where she also pinns the version fo tempest | 19:38 |
| gmann | in yoga gate | 19:38 |
| sean-k-mooney | its a two part fix | 19:38 |
| sean-k-mooney | 1 make sure the requiremetns are co installable in 3.6 and 2 pin tempest to a compatible version in the c8s jobs on stable branches | 19:38 |
| gmann | https://review.opendev.org/c/openstack/ceilometer/+/853668 | 19:39 |
| gmann | and that require requirement change at first | 19:39 |
| tonyb | gmann: but it still fails with the requirements change | 19:42 |
| sean-k-mooney | one option might be to move wallaby and aove to c9s | 19:42 |
| sean-k-mooney | to alight with what we will ship them on downstream | 19:42 |
| sean-k-mooney | i.e. rhel 9 | 19:42 |
| sean-k-mooney | but the issue were discoved when tryitn to backprot change and we still need to supprot train | 19:43 |
| sean-k-mooney | for a long time | 19:43 |
| tonyb | sean-k-mooney: we could but that's a different thing | 19:43 |
| sean-k-mooney | right so initally emma is trying to fix the existing jobs | 19:43 |
| sean-k-mooney | anyway we have to take our dog for a walk and i need to eat | 19:43 |
| sean-k-mooney | so enjoy yoru weekend o/ | 19:44 |
| tonyb | okay so the bottom line is that I disagree with pinning tempest but I'm willing to go down that path. So lets generate the chnage(s) that do that. verify that we get the correct temptest ; *then* look at what requiremnets change(s) are also needed | 19:45 |
| tonyb | gmann: Can you point me at where we do the pinning in the job configs for EM? | 19:46 |
| gmann | tonyb: for this patch, this is wherer pin is dine https://review.opendev.org/c/openstack/ceilometer/+/853668/13/.zuul.yaml | 19:47 |
| gmann | I am checking what it is failing now | 19:47 |
| tonyb | okay me also | 19:49 |
| tonyb | So we're still using the *master* constraints rather than TEMPEST_VENV_UPPER_CONSTRAINTS | 19:50 |
| gmann | commented about the failure in patch | 19:51 |
| gmann | we need to merge this in devstack and backport to yoga https://review.opendev.org/c/openstack/devstack/+/848482 | 19:52 |
| tonyb | Okay in the meantime we can test the backport via depends-on | 19:56 |
| tonyb | I'll do the cherry-pick | 19:56 |
| opendevreview | Merged openstack/requirements master: update constraint for os-traits to new release 2.9.0 https://review.opendev.org/c/openstack/requirements/+/854821 | 21:51 |
| opendevreview | Merged openstack/requirements master: update constraint for metalsmith to new release 1.8.0 https://review.opendev.org/c/openstack/requirements/+/854825 | 22:29 |
| opendevreview | Merged openstack/requirements master: update constraint for sushy to new release 4.3.0 https://review.opendev.org/c/openstack/requirements/+/854826 | 22:43 |
| opendevreview | Merged openstack/requirements master: update constraint for os-brick to new release 6.1.0 https://review.opendev.org/c/openstack/requirements/+/854828 | 22:56 |
| opendevreview | Merged openstack/requirements master: update constraint for kuryr-lib to new release 2.6.0 https://review.opendev.org/c/openstack/requirements/+/854829 | 22:56 |
| opendevreview | Merged openstack/requirements master: update constraint for monasca-statsd to new release 2.5.0 https://review.opendev.org/c/openstack/requirements/+/854830 | 22:56 |
| opendevreview | Merged openstack/requirements master: update constraint for mistral-lib to new release 2.7.0 https://review.opendev.org/c/openstack/requirements/+/854832 | 22:56 |
| opendevreview | Merged openstack/requirements master: update constraint for ironic-lib to new release 5.3.0 https://review.opendev.org/c/openstack/requirements/+/854835 | 22:56 |
| opendevreview | Merged openstack/requirements master: update constraint for keystonemiddleware to new release 10.0.1 https://review.opendev.org/c/openstack/requirements/+/854831 | 23:11 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!