| *** diablo_rojo is now known as Guest1941 | 00:58 | |
| *** ykarel|away is now known as ykarel | 04:18 | |
| *** ysandeep|out is now known as ysandeep | 05:27 | |
| opendevreview | Hervé Beraud proposed openstack/releases master: add missing release notes https://review.opendev.org/c/openstack/releases/+/812636 | 07:30 |
|---|---|---|
| hberaud | ttx, elodilles o/ How are you? This cycle I would suggest to start by merging the missing release notes patch ^ . To avoid to be late on our annoucement as during wallaby. | 07:32 |
| elodilles | hi hberaud , will review shortly | 07:34 |
| hberaud | ty | 07:34 |
| hberaud | elodilles, ttx also please don't forget to hold all our stable branch release validations while coordinated release is underway #today. | 07:40 |
| *** amoralej|off is now known as amoralej | 07:42 | |
| elodilles | hberaud: oh, right, thanks for the reminder. actually just wanted to ask for reviews for nova stable release patches (wallaby, victoria, ussuri) from ttx o:) it is long waiting there, but maybe one more day is not a big deal | 07:43 |
| hberaud | :) | 07:44 |
| ttx | ok | 07:45 |
| ttx | I hope it will not break your final release patch | 07:46 |
| hberaud | ttx: I think we will have some conflict once this one will be merged | 07:48 |
| elodilles | hberaud: among the release notes there are empty pages (like https://docs.openstack.org/releasenotes/automaton/xena.html ) is it still OK & required to add? | 07:48 |
| hberaud | elodilles: I think it's ok | 07:49 |
| hberaud | however, the files modified by the relnote patch aren't really the same than those modified by the final rc patch | 07:50 |
| hberaud | so, I would argue that the conflicts will be minor conflicts | 07:51 |
| elodilles | do you mean that there won't be big conflicts? | 07:54 |
| hberaud | exact | 07:54 |
| elodilles | ok, +2+W'd | 07:54 |
| ttx | we'll soon see | 07:55 |
| hberaud | a big part of the missing relnote pages are with python-* or oslo projects | 07:56 |
| hberaud | or with cwi projects | 07:56 |
| opendevreview | Merged openstack/releases master: add missing release notes https://review.opendev.org/c/openstack/releases/+/812636 | 08:06 |
| hberaud | \o/ no conflict | 08:07 |
| elodilles | ~o~ | 08:13 |
| hberaud | ttx: Do you know what's the exact time for the xena press release? | 08:14 |
| *** ykarel is now known as ykarel|lunch | 08:15 | |
| *** ysandeep is now known as ysandeep|lunch | 08:28 | |
| ttx | hberaud: yes 1500UTC | 08:52 |
| hberaud | ack | 08:52 |
| ttx | so we need to finish and announce it just before that | 08:53 |
| ttx | between 1400 and 1500utc ideally | 08:53 |
| hberaud | ack | 08:54 |
| hberaud | lgtm | 08:54 |
| *** ykarel|lunch is now known as ykarel | 09:45 | |
| *** ysandeep|lunch is now known as ysandeep | 09:46 | |
| fungi | what's button-pushing time for the main releases change? | 10:05 |
| fungi | i guess that's change 812251 | 10:07 |
| hberaud | fungi: I think we will start pushing it in ~30 minutes | 10:07 |
| fungi | cool, i'm around in case anything breaks | 10:08 |
| hberaud | thanks | 10:08 |
| hberaud | until now the infras (pypi/warehouse, and openstack infra (AFAIK)) seems fully functional | 10:13 |
| hberaud | fungi: Before we start pushing the button please can you confirm that our openstack infra is OK? | 10:25 |
| fungi | yeah, i've been looking over everything | 10:27 |
| fungi | there's a bit more of a test backlog than is common this time of day, but tripleo and osa have approved quite a bit of stuff | 10:28 |
| hberaud | ok | 10:29 |
| hberaud | Then, let's go! | 10:30 |
| ttx | yay! | 10:30 |
| hberaud | elodilles, ttx FYI ^ I start merging the final patch | 10:30 |
| elodilles | hberaud: \o/ | 10:31 |
| hberaud | here we go | 10:31 |
| fungi | seems like check-release-approval takes a long time to run for changes which create lots of tags | 10:32 |
| fungi | but that's not a problem as far as i can tell | 10:32 |
| hberaud | indeed | 10:32 |
| fungi | we consider the final release pre-approved by the ptls et cetera | 10:32 |
| hberaud | https://zuul.opendev.org/t/openstack/builds?job_name=openstack-upload-github-mirror | 10:34 |
| hberaud | https://zuul.opendev.org/t/openstack/builds?job_name=release-openstack-python | 10:34 |
| hberaud | https://zuul.opendev.org/t/openstack/builds?job_name=announce-release | 10:35 |
| hberaud | https://zuul.opendev.org/t/openstack/builds?job_name=propose-update-constraints | 10:35 |
| fungi | we're not backlogged on executors any longer, as of a few minutes ago, but still have a bit of a node request backlog | 10:38 |
| fungi | and the releases change has all of its node assignments now | 10:39 |
| fungi | zuul estimates 10 minutes until the show starts | 10:39 |
| hberaud | ack | 10:39 |
| elodilles | i guess it estimates based on the 'general' job run time and for this release patch we can expect some extra time, right? | 10:47 |
| fungi | possibly | 10:48 |
| fungi | it does base its time estimates on the average completion time for a particluar job | 10:48 |
| fungi | it's already pretty far along though assuming it goes in alpha order (jus finished checking neutron and on to nova) | 10:52 |
| hberaud | sahara seems underway | 10:53 |
| hberaud | zaqar | 10:55 |
| hberaud | done | 10:55 |
| elodilles | 812251 | 10:55 |
| elodilles | i mean: 0 errors found | 10:56 |
| hberaud | \o/ | 10:56 |
| elodilles | now the post tasks | 10:56 |
| opendevreview | Merged openstack/releases master: Xena final releases for cycle-with-rc projects https://review.opendev.org/c/openstack/releases/+/812251 | 10:57 |
| fungi | and now the fun begins | 10:58 |
| * hberaud grab popconr | 10:58 | |
| hberaud | corn | 10:58 |
| elodilles | :] | 10:58 |
| fungi | it's enqueued in the release-post pipeline now | 10:58 |
| *** ysandeep is now known as ysandeep|away | 11:00 | |
| fungi | taking a while for a node to come available for tag-releases, looks like | 11:03 |
| fungi | but it finally got one | 11:03 |
| fungi | and tags are getting enqueued! | 11:08 |
| fungi | barbican's got a release-openstack-python build underway now | 11:10 |
| fungi | and i see them starting for other tags as well | 11:11 |
| fungi | so far, so good | 11:11 |
| fungi | https://pypi.org/project/barbican/ shows 13.0.0 as current since a few minutes ago | 11:18 |
| fungi | nova tag is being created now | 11:18 |
| hberaud | excellent | 11:20 |
| fungi | instead of cloning nova fresh, the script could clone from the copy already cached on disk and then just fetch available refs, would speed things up and put less load on our git servers | 11:21 |
| fungi | all tags have been pushed, now for the long tail of waiting for all their builds complete | 11:25 |
| hberaud | hm we get a docker image failure with monasca-notification | 11:27 |
| hberaud | (publish failure) | 11:27 |
| fungi | release pipeline length peaked at 82 refs, though some had already completed their builds by the time the last ones were being enqueued | 11:27 |
| ttx | hberaud: link? | 11:27 |
| hberaud | https://zuul.opendev.org/t/openstack/build/2b6170b63871409a8ef296bca3883896/log/job-output.txt | 11:28 |
| hberaud | apparently it seems related to a non existing local image | 11:29 |
| fungi | "error: can't find Rust compiler" | 11:29 |
| hberaud | and I seen failure related to rust compiler | 11:30 |
| hberaud | yes this one | 11:30 |
| hberaud | I suppose the both are linked | 11:30 |
| fungi | it's trying to build pyca/cryptography from source, yeah | 11:30 |
| ttx | that job seems to have a complicated history https://zuul.opendev.org/t/openstack/builds?job_name=publish-monasca-notification-docker-image&project=openstack/monasca-notification | 11:31 |
| fungi | log says "/usr/local/lib/python3.6/..." | 11:32 |
| hberaud | only with xena at least | 11:32 |
| fungi | did monasca not move their jobs to focal? | 11:33 |
| hberaud | U, V, W looks ok | 11:33 |
| hberaud | I don't know | 11:33 |
| fungi | though it's a focal node where that was running | 11:34 |
| ttx | comparing logs from successful run vs. failed one | 11:34 |
| opendevreview | Brian Rosmaita proposed openstack/releases master: Add Cinder-specific dates for Yoga schedule https://review.opendev.org/c/openstack/releases/+/812666 | 11:35 |
| ttx | Yeah it seems to be where the fail occurs: Building wheel for cryptography (PEP 517): finished with status 'done' | 11:36 |
| ttx | vs. Building wheel for cryptography (PEP 517): finished with status 'error' | 11:36 |
| fungi | i'm struggling to figure out how they ended up with python 3.6 on an ubuntu-focal node, its default python3 is 3.8 | 11:36 |
| fungi | i suspect the underlying problem is that there's no cp36 wheel for the version of cryptography in upper-constraints.txt | 11:37 |
| ttx | (comparing with https://zuul.opendev.org/t/openstack/build/d44b9b892e744917b9c76f62ab6996e5 which was a successful run) | 11:37 |
| ttx | fungi: how do you see it's running 3.6? | 11:38 |
| ttx | hah /usr/local/lib/python3.6/ | 11:38 |
| fungi | i bet that's inside a container | 11:39 |
| ttx | fungi: the successful run also has that 3.6 line | 11:40 |
| fungi | yep, it's a docker build, so their base image is probably 3.6 | 11:40 |
| fungi | ttx: different version of cryptography? | 11:40 |
| fungi | yep | 11:42 |
| fungi | failure: Collecting cryptography===35.0.0 (from -c /app/upper-constraints.txt (line 189)) | 11:42 |
| fungi | success: Collecting cryptography===3.4.8 (from -c /app/upper-constraints.txt (line 189)) | 11:42 |
| ttx | hah | 11:42 |
| hberaud | weird | 11:42 |
| ttx | hmm some change in the underlying Docker image probably | 11:43 |
| hberaud | I don't think that reenqueuing it will help us | 11:45 |
| fungi | yeah, likely they need to update something about their monasca:base image | 11:47 |
| hberaud | yes I think they need to fix the root cause first | 11:48 |
| ttx | looks like one of the "fs layer" is different, yielding a different sha256 digest in the end | 11:48 |
| ttx | this one is in both, but below that it looks different: 7cfa6613f80e: Pulling fs layer | 11:48 |
| hberaud | Apparently it didn't seems to disturb them previously because this error seems present since a while now | 11:49 |
| fungi | and we're now at the long tail of releasenotes jobs | 11:53 |
| fungi | also seem to be basically caught up on the node backlog | 11:55 |
| *** amoralej is now known as amoralej|lunch | 11:55 | |
| ttx | so on the monasca thing, my understanding is that it ends up installing Monasca inside some docker image that resolves cryptography to two different versions, one of which can't be built under python 3.6 | 11:58 |
| hberaud | +1 | 11:59 |
| fungi | i'm curious why it's trying to build cryptography from sdist anyway, there are cp36-abi3 manylinux wheels for it | 12:00 |
| ttx | actually | 12:01 |
| ttx | https://opendev.org/openstack/requirements/commit/1fa22ce584ef8a5f5ec0c0e606e5e0daf38de148 | 12:01 |
| fungi | but maybe the pip in those docker images is too old to look for abi3 wheels | 12:01 |
| ttx | 18 hours ago we went to 35.0.0 | 12:01 |
| hberaud | ah good catch | 12:01 |
| fungi | but on master, not stable/xena | 12:02 |
| ttx | that sounds like a bad idea | 12:02 |
| ttx | hah | 12:02 |
| fungi | so monasca's using master constraints list in their stable/xena image build | 12:03 |
| ttx | "Cloning monasca-common in version: master" | 12:05 |
| hberaud | If I take an old build (https://zuul.opendev.org/t/openstack/build/6f647ab90b3040b08683c3f640154131/log/job-output.txt) the error are differents, they don't match this current case. So the both error don't seems related. But even by fixing it I'm not sure this job will be fixed | 12:05 |
| ttx | at least for that ref refs/tags/5.0.0 it does consider it's master | 12:05 |
| hberaud | this build ^ is older than the req update | 12:06 |
| ttx | hberaud: I would not consider that publication to be part of "the release", so i would defer the fix | 12:06 |
| hberaud | +1 | 12:06 |
| ttx | we don;t do docker images | 12:06 |
| ttx | that is an additional release job, not one of ours :) | 12:07 |
| hberaud | I'll simply send an email to the team to inform them | 12:07 |
| ttx | (it's clearly way too flaky for one of ours) | 12:07 |
| ttx | yep | 12:07 |
| hberaud | I'm not fan to spend time to repair a light during a landing step :) ( https://en.wikipedia.org/wiki/Eastern_Air_Lines_Flight_401 ) | 12:10 |
| hberaud | Else for now all jobs seems green | 12:11 |
| hberaud | I think we are ready to run the check to find the missing tarballs | 12:18 |
| hberaud | ttx, fungi did you noticed other issues until now? | 12:19 |
| ttx | checking | 12:21 |
| ttx | all clear for me | 12:22 |
| hberaud | +1 | 12:22 |
| hberaud | elodilles: I think you can start your verification | 12:24 |
| fungi | i haven't seen any problems | 12:26 |
| hberaud | thanks | 12:26 |
| elodilles | hberaud: actually i've started 'tox -e venv -- missing-releases --series xena' more than half an hour ago and it succeded: did not find any errors | 12:26 |
| fungi | the release notes jobs will still be going for a while | 12:26 |
| fungi | but otherwise i think it's all clear | 12:26 |
| elodilles | the only interesting print outs were these: https://paste.opendev.org/ | 12:27 |
| elodilles | i mean these: https://paste.opendev.org/show/809811/ | 12:28 |
| elodilles | (sorry, captcha tricked me :P | 12:28 |
| hberaud | elodilles: if I'm right the latest job finished around 11:56UTC so I think your run is ok | 12:28 |
| elodilles | hberaud: ack | 12:31 |
| elodilles | the missing-releases command did not consider those 'missing python 3 tarballs' as errors | 12:33 |
| elodilles | armstrong: fyi ^^^ | 12:35 |
| hberaud | yes | 12:36 |
| hberaud | I think that we could ignore them | 12:36 |
| armstrong | Hello elodilles | 12:37 |
| elodilles | hi | 12:37 |
| armstrong | elodilles: are you working on the task yet? | 12:38 |
| elodilles | armstrong: i've run already: 'tox -e venv -- missing-releases --series xena' | 12:38 |
| armstrong | ok | 12:39 |
| elodilles | and fortunately it did not show any missing things | 12:39 |
| armstrong | great! | 12:39 |
| elodilles | (actually I did a test run prior the final release patch merge, just to have a comparison what should I look for) | 12:40 |
| elodilles | but the command gives clear output anyway | 12:41 |
| armstrong | elodilles: After the release merge shall we run another test? | 12:41 |
| elodilles | armstrong: I ran it after the release merge and when I saw that relevant post jobs have finished | 12:42 |
| elodilles | armstrong: but feel free to run it by yourself as well as a doublecheck | 12:42 |
| armstrong | Oh I did know that the final release is merged :) | 12:43 |
| *** ysandeep|away is now known as ysandeep | 12:44 | |
| elodilles | if you read back the channel you'll see what happened as we were discussing the things here :) | 12:45 |
| hberaud | elodilles, ttx: please can you review the drafts or our next announcements https://etherpad.opendev.org/p/relmgmt-weekly-emails | 13:00 |
| hberaud | https://www.openstack.org/software/xena/ LGTM | 13:13 |
| armstrong | hberaud: the draft LGTM | 13:16 |
| elodilles | hberaud: both mail looks good \o/ | 13:16 |
| hberaud | ty! | 13:17 |
| *** amoralej|lunch is now known as amoralej | 13:17 | |
| ttx | all tarballs checked out? | 13:23 |
| ttx | Looking into emails | 13:23 |
| ttx | LGTM, I just made it clear you need to adjust URL by slpashing color | 13:25 |
| clarkb | I've done some napkin math and I think the tag pipeline will need about 5 hours to finish clearing its queued entries. These entries appear to be publishing release notes | 13:26 |
| clarkb | They are running one by one due to a semaphore and take about 5 minutes each | 13:26 |
| hberaud | ttx: yeah all the tarballs seems there | 13:26 |
| hberaud | ttx: thanks for the splashing color | 13:27 |
| ttx | That shall not block the release but we may think about improvements there | 13:27 |
| ttx | (the tag pipeline backlog) | 13:27 |
| hberaud | I agree | 13:28 |
| hberaud | I propose to mark xena as released in ~ 45 minutes (by merging https://review.opendev.org/c/openstack/releases/+/812454 ), it will be 2:15pm UTC, is it ok for you? | 13:30 |
| hberaud | and we could send the announcements emails around 2:30pm UTC | 13:31 |
| ttx | hberaud: works for me | 13:31 |
| ttx | https://www.openstack.org/software/xena/ is ready | 13:31 |
| hberaud | yepp | 13:32 |
| hberaud | -p | 13:32 |
| fungi | the reason we ended up doing a semaphore on the release notes job is that if it gets enqueued twice for different branches of the same project, the publication rsync can collide | 13:34 |
| fungi | clarkb: ^ worth pondering alternative solutions to that problem i suppose | 13:34 |
| fungi | release notes builds for different projects can run fine in parallel, just not for different branches of the same project | 13:35 |
| clarkb | fungi: my first thought on that is maybe zuul needs a semaphore that only mutexes the same project | 13:35 |
| clarkb | or maybe we need to configure variants that use a different semaphore per project | 13:35 |
| hberaud | clarkb: here is the original discussion https://meetings.opendev.org/irclogs/%23openstack-release/%23openstack-release.2020-10-07.log.html#t2020-10-07T14:42:49 | 13:38 |
| clarkb | ya I remember the issue. It isn't actually fatal but causes the jobs to fail which we don't want from a user sanity perspective | 13:42 |
| opendevreview | Hervé Beraud proposed openstack/releases master: Mark Xena as released https://review.opendev.org/c/openstack/releases/+/812454 | 13:46 |
| stephenfin | Can I merge the docs changes to add Xena index pages so? | 14:03 |
| hberaud | stephenfin: which change? | 14:03 |
| hberaud | I guess yes | 14:03 |
| stephenfin | https://review.opendev.org/c/openstack/openstack-manuals/+/812121 | 14:04 |
| stephenfin | Also, is Train in extended maintenance now? What about Ussuri? | 14:04 |
| hberaud | concerning xena will we officialize it by merging https://review.opendev.org/c/openstack/releases/+/812454 in ~10 minutes | 14:05 |
| elodilles | stephenfin: ussuri will transition to EM in a month | 14:05 |
| elodilles | stephenfin: see https://releases.openstack.org/ | 14:05 |
| stephenfin | aight, I'll merge the two patches to mark Train as EM and release Xena so | 14:06 |
| hberaud | and train is already EM | 14:06 |
| stephenfin | but will hold off one for Ussuri | 14:06 |
| hberaud | LGTM | 14:06 |
| ykarel | Reviews please https://review.opendev.org/c/openstack/releases/+/812275 | 14:13 |
| ykarel | elodilles, ttx ^ | 14:13 |
| hberaud | ykarel: Today is the final release for Xena. We scheduled it for 3pm UTC. We will defer this patch after the official announce. | 14:15 |
| hberaud | puppet is a trailing project, it won't hurt if we merge it in awhile | 14:16 |
| hberaud | It's time to go with https://review.opendev.org/c/openstack/releases/+/812454 | 14:17 |
| elodilles | stephenfin: have you seen Andreas' comment on https://review.opendev.org/c/openstack/openstack-manuals/+/812121 ? | 14:17 |
| stephenfin | I have. I pulled the patch and am trying to fix the previous one now | 14:17 |
| elodilles | stephenfin: unfortunately it's still on gate queue | 14:17 |
| stephenfin | It won't merge without the second +W | 14:17 |
| stephenfin | *without the +W | 14:18 |
| elodilles | stephenfin: ok, sorry, then never mind :X | 14:18 |
| stephenfin | Nope, all good :) I should have it fixed shortly | 14:18 |
| stephenfin | As an aside, these are the disadvantages of not having anyone else to look over patches now /o\ | 14:19 |
| elodilles | :/ | 14:19 |
| hberaud | fungi: FYI release announce sent at openstack-announce@list.openstack.org | 14:27 |
| fungi | thanks! | 14:27 |
| fungi | aprice: ^ | 14:27 |
| aprice | yay!! | 14:28 |
| ttx | woohoo! | 14:28 |
| ykarel | hberaud, okk in past it used to be merged before | 14:28 |
| ykarel | but ok after announcement too | 14:28 |
| fungi | approving it now | 14:28 |
| hberaud | ykarel: not a big deal | 14:28 |
| aprice | congrats everyone! | 14:29 |
| hberaud | Congrats everyone! | 14:29 |
| fungi | http://lists.openstack.org/pipermail/openstack-announce/2021-October/002056.html | 14:29 |
| hberaud | Well, now it's time to send it to openstack-discuss | 14:31 |
| opendevreview | Merged openstack/releases master: Mark Xena as released https://review.opendev.org/c/openstack/releases/+/812454 | 14:32 |
| hberaud | twitted https://twitter.com/4383hberaud/status/1445759007046594569 | 14:33 |
| hberaud | and sent http://lists.openstack.org/pipermail/openstack-discuss/2021-October/025214.html | 14:33 |
| hberaud | https://releases.openstack.org/ will be updated in few minutes | 14:35 |
| hberaud | job is running yet | 14:36 |
| *** diablo_rojo__ is now known as diablo_rojo | 14:39 | |
| hberaud | publish-tox-docs-releases is queued | 14:39 |
| diablo_rojo | whoop whoop! | 14:39 |
| ttx | Not for future releases, we should probably wait until releases.o.o shows the release as "released" before sending announcement | 14:44 |
| ttx | Note* | 14:44 |
| hberaud | note | 14:44 |
| diablo_rojo | lol | 14:44 |
| hberaud | yes that what I was thinking too | 14:45 |
| hberaud | The mail could be queued at least but we should wait before approving it. I forgot to check this point before :\ | 14:47 |
| fungi | looks like it's done now | 14:48 |
| hberaud | Hm the page isn't updated | 14:48 |
| fungi | that volume gets released every 5 minutes | 14:49 |
| hberaud | ok | 14:49 |
| fungi | so should show up in a minute assuming nothing else delays that | 14:49 |
| fungi | it's going now | 14:50 |
| * hberaud hberaud restrein him to frantically reload the page | 14:50 | |
| fungi | and done! | 14:50 |
| hberaud | drum roll => https://releases.openstack.org/ | 14:51 |
| hberaud | And yes! | 14:51 |
| diablo_rojo | Well done everyone! | 14:52 |
| hberaud | Thank you all for all your helps during this cycle and congrats for your works | 14:52 |
| diablo_rojo | Thank you for leading us! | 14:52 |
| hberaud | My pleasure :) | 14:52 |
| hberaud | Then now that announces are made and that xena is out, I'm honored to pass the PTL baton to elodilles :) | 14:53 |
| hberaud | Hail to elodilles our fearless leader! | 14:53 |
| ttx | yay! | 14:53 |
| johnsom | Thank you release team for making the Xena release nice and smooth for the Designate/Octavia teams! We appreciate the work. | 14:58 |
| *** ykarel is now known as ykarel|away | 15:38 | |
| *** ysandeep is now known as ysandeep|out | 16:15 | |
| *** amoralej is now known as amoralej|off | 17:01 | |
| *** ysandeep|out is now known as ysandeep | 18:03 | |
| *** ysandeep is now known as ysandeep|out | 19:24 | |
| clarkb | my tag pipelien estimates appear to be a bit off. Still 20 queued entries to go (from about 58 when I first brought it up) | 19:37 |
| clarkb | but it continues to make steady progress | 19:38 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!