| opendevreview | Merged openstack/project-config master: Retire OpenStack-Chef: remove project from infra https://review.opendev.org/c/openstack/project-config/+/910100 | 07:53 |
|---|---|---|
| *** tosky_ is now known as tosky | 11:36 | |
| frickler | who is our resident pbr specialist? seems the CI has some issues, I'd just drop tempest-full in favor of the -py3 version, but not so sure about the other failures https://review.opendev.org/c/openstack/pbr/+/910690 | 13:33 |
| fungi | it's mostly been stephenfin, clarkb and me these days | 13:35 |
| fungi | i think tempest-full is in there mainly to keep pbr changes from breaking tempest jobs on stable branches that haven't dropped python 2.7 support yet | 13:36 |
| clarkb | fungi: correct | 15:26 |
| clarkb | because you can't easily control the version of pbr used due to how install requires work | 15:26 |
| clarkb | the fix for that previuosly has been to override projects from stable branches to tags as the transition happens | 15:26 |
| clarkb | I think that we're approaching the point where swift may be one of the only things caring about python2.7? One option may be to replace that tempest full job with a swift install job or something along those lines | 15:27 |
| frickler | timburke: ^^ is there a standalone swift job without devstack? | 15:31 |
| fungi | fwiw, looks like pbr is the only project running the tempest-full job since a while, and it was passing up until something changed between 2024-02-06 02:34:20 utc (last success) and 2024-02-07 02:39:57 utc (first failure) | 15:35 |
| fungi | last non-pbr run of that job was for an openstack/keystone stable/train branch change in december | 15:36 |
| clarkb | the job runs against ussuri in pbr I think as that was the last python2.7 release? | 15:37 |
| clarkb | oh nope its train | 15:37 |
| clarkb | I was off by one | 15:37 |
| fungi | openstack/python-openstackclient (stable/train) and openstack/requirements (stable/stein) also ran it circa september | 15:38 |
| clarkb | fwiw it is good that we got a -1 on https://review.opendev.org/c/openstack/pbr/+/909581 that is appropriate | 15:38 |
| clarkb | I'm going to -2 that change | 15:38 |
| frickler | train was eoled some weeks ago, so things would have to be replaced by tags indeed | 15:38 |
| clarkb | frickler: ya I'm guessing the set of overrides is simply incompelte at this point and needs to be updated. That is probably a good intermediate step | 15:39 |
| fungi | yeah, my guess, without checking, is that the bulk of the stable/train eol happened around the time that job began failing | 15:40 |
| frickler | yes, some followup to https://review.opendev.org/c/openstack/pbr/+/906072/2/.zuul.yaml will be needed for all other required repos | 15:40 |
| frickler | then next I'm wondering why this periodic job seems to have stopped running on Feb 29 https://zuul.opendev.org/t/openstack/builds?job_name=pbr-installation-openstack&project=openstack/pbr | 15:44 |
| clarkb | ok I posted a -2 to https://review.opendev.org/c/openstack/pbr/+/909581 | 15:44 |
| clarkb | it is unfortunate that PBR has become an attractive nuisance around python2 cleanup | 15:44 |
| clarkb | frickler: the pbr-installation-openstack job may try and use tripleo repos that were removed? | 15:45 |
| clarkb | oh yup thats what https://review.opendev.org/c/openstack/pbr/+/910690/2/.zuul.yaml aims to fix | 15:47 |
| clarkb | we might need to squash these two changes together? | 15:47 |
| frickler | no, the failure appeared on Feb 22. seems pip freeze output changed: testtools.matchers._impl.MismatchError: 'pkg-b' not in 'build==1.0.3\npackaging==23.2\npbr @ file:///home/zuul/src/opendev.org/openstack/pbr\npkg_b==0.0\npyproject_hooks==1.0.0\ntest_markers==0.0\ntomli==2.0.1' | 15:48 |
| frickler | pkg_b vs. pkg-b | 15:48 |
| fungi | pip's project name normalization bites us again | 15:48 |
| fungi | https://virtualenv.pypa.io/en/latest/changelog.html | 15:51 |
| fungi | that likely coincides | 15:51 |
| clarkb | so we have three things: train-eol overrides on tempest-full, fixing package install checks with normalized names and the tripleo project removals. | 15:52 |
| clarkb | I've got a change in progress for the first, gmann pushed one for the third thing, just need a fix for the normalization | 15:53 |
| fungi | v20.25.1 (2024-02-21) was uploaded to pypi 2024-02-22 01:45:30 utc | 15:53 |
| fungi | Upgrade embedded wheels: setuptools 69.0.3 to 69.1.0, pip 23.3.2 to 24.0 | 15:53 |
| fungi | pip 24.0 release notes mention https://github.com/pypa/pip/pull/12477 which talk about setuptools 69 preserving ) in name metadata rather than converting to - | 15:55 |
| frickler | ah, I've been trying to compare good and bad job logs, but plain "pip freeze" doesn't report the pip version itself | 15:56 |
| fungi | er, preserving _ i mean | 15:56 |
| opendevreview | Clark Boylan proposed openstack/pbr master: Add more train-eol overrides for python2 tempest-full https://review.opendev.org/c/openstack/pbr/+/910961 | 15:56 |
| clarkb | we will likely need to squash all these fixes together but there is a split out train-eol override change | 15:56 |
| frickler | fungi: that sounds very much like the issue we see | 15:57 |
| fungi | frickler: yeah, its probably some change in package name normalization by pip 24.0 and/or setuptools 69 which got quietly updated in a patch release of virtualenv around the time the failures started | 15:57 |
| clarkb | ya immediately zuul returns a -1. I'll squash into gmann's change and then yall can squash into that | 15:58 |
| opendevreview | Clark Boylan proposed openstack/pbr master: Omnibus PBR CI fixups https://review.opendev.org/c/openstack/pbr/+/910690 | 16:01 |
| clarkb | that test installs pkg_b and expects that pip freeze will list it as pkg-b. pip/setuptools/etc updated to preserve pkg_b instead of normalizing to pkg-b? | 16:05 |
| clarkb | if so I can quickly update 910690 to s/-/_/ if we think that is appropriate | 16:05 |
| clarkb | the test case is checking that python version markers work so the actual python package we isntall isn't super critical I don't think | 16:06 |
| fungi | checking git blame might be good to see if there was a normalization-related reason for that check, but yeah seems like it was just "match the expected output" and now the expected output has changed due to external forces | 16:06 |
| clarkb | ya if this test was testing normalization I would be more concerned, But it just wants to check that we can use python version markers | 16:07 |
| clarkb | ya code was always written that way I think ebacuse it normalized that way | 16:08 |
| frickler | maybe make the check work with both variants? like some distro may still be building+testing with older pip? | 16:09 |
| opendevreview | Clark Boylan proposed openstack/pbr master: Omnibus PBR CI fixups https://review.opendev.org/c/openstack/pbr/+/910690 | 16:09 |
| clarkb | hrm ya I guess we acn do that too | 16:10 |
| opendevreview | Clark Boylan proposed openstack/pbr master: Omnibus PBR CI fixups https://review.opendev.org/c/openstack/pbr/+/910690 | 16:15 |
| clarkb | the tempest full job was/is running on jammy? Overriding devstack to train-eol caused the job to fail beacuse the test node wasn't supported. We should've previously been checking out stable/train of devstack though so I'm really confused how this was passing previously | 16:28 |
| clarkb | when it passed it ran on bionic | 16:29 |
| clarkb | oh I know when stable/train was a branch it got the stable/train devsatck configs | 16:30 |
| clarkb | but now that branch is goen that info is gone so we have to override it ourselves I guess | 16:30 |
| clarkb | and this creates a problem beacuse devstack has special nodesets /me will figure this out | 16:30 |
| fungi | we should be able to copy the nodeset definition from the train-eol tag? | 16:37 |
| clarkb | yup. The nodeset is still alive on master actually | 16:37 |
| clarkb | this is one of thsoe things that I would like to see projects cleaning up around release time, so I've gone ahead and copied it over with a new name to make it more stable | 16:38 |
| clarkb | just waiting on test results before pushing a new patchset to see if anything else needs fixing | 16:38 |
| opendevreview | Clark Boylan proposed openstack/pbr master: Omnibus PBR CI fixups https://review.opendev.org/c/openstack/pbr/+/910690 | 16:50 |
| clarkb | ok this still fails and it points out a problem | 17:00 |
| clarkb | we're running the newer job definition even with devstack told to checkout train-eol | 17:00 |
| clarkb | current issue is we appear to be using global virtualenvs but we shouldn't for that old job. I'm going to set the flag to disable that, but if that doesn't work then maybe we should consider dropping the job and adding a swift install check job | 17:01 |
| clarkb | or just relying on python2 unittests to ensure compatibility | 17:01 |
| clarkb | oh no the issue is even more subtle than that. We ran ensure-virtualenv previously but no logner do that beacuse we assume we can use python3 -m virtualenv? | 17:05 |
| clarkb | so ya the tempest-full job while existing in newer devstack does not appear to work anymore/ | 17:05 |
| clarkb | ah ya beacuse tempest defines the job but devstack managed the setup and with stable/train tempest gone that setup specific to python2 is gone | 17:06 |
| clarkb | I'm going to propose we just remove the job, keep the unittests running python2 and if swift wants extra coverage can help us add a swift job to pbr cc timburke | 17:06 |
| opendevreview | Clark Boylan proposed openstack/pbr master: Omnibus PBR CI fixups https://review.opendev.org/c/openstack/pbr/+/910690 | 17:13 |
| clarkb | gmann: ^ fyi it might be time to clean that job out of tempest now | 17:14 |
| opendevreview | Clark Boylan proposed openstack/pbr master: Omnibus PBR CI fixups https://review.opendev.org/c/openstack/pbr/+/910690 | 17:23 |
| opendevreview | Clark Boylan proposed openstack/project-config master: CentOS 7 removal prep changes https://review.opendev.org/c/openstack/project-config/+/910978 | 18:18 |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Cleanup legacy CentOS 7 jobs and nodesets https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/910979 | 18:19 |
| opendevreview | Clark Boylan proposed openstack/project-config master: CentOS 7 removal prep changes https://review.opendev.org/c/openstack/project-config/+/910978 | 18:24 |
| opendevreview | Clark Boylan proposed openstack/project-config master: CentOS 7 removal prep changes https://review.opendev.org/c/openstack/project-config/+/910978 | 18:26 |
| gmann | clarkb: thakns for updating. will check if any more usage otherwise remove | 19:06 |
| fungi | i asked a related question in #openstack-qa | 19:12 |
| opendevreview | Merged openstack/project-config master: CentOS 7 removal prep changes https://review.opendev.org/c/openstack/project-config/+/910978 | 21:49 |
| clarkb | fungi: frickler https://review.opendev.org/c/openstack/pbr/+/910690 did end up passing. If we're happy with the reduction in python2 test coverage (I think the unittests are probably sufficient) we can proceed with that change | 21:53 |
| fungi | lgtm | 21:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!