| *** marios is now known as marios|ruck | 05:06 | |
| ttx | woohoo, tests passed, and PTLs +1s are piling up | 06:19 |
|---|---|---|
| elodilles | yepp, so far so good \o/ | 06:42 |
| elodilles | zuul load is moderate, though we would count with some load (right now i see multiple tripleo patches on the gate) | 06:46 |
| elodilles | * we should | 06:46 |
| elodilles | yesterday the gate passed for yoga-final around ~1.5 hrs. adding the post release jobs and some final tasks... i think 11:30 UTC (the original starting time we aimed for) could work: we'll have 3.5 hrs to finish the tasks | 06:50 |
| elodilles | maybe if we are not that patient we can start a bit earlier :] we will see | 06:52 |
| ttx | yeah I would not mind pushing the button early if we have good confidence the gate is in good shape | 07:07 |
| elodilles | ++ | 07:17 |
| *** amoralej|off is now known as amoralej | 07:19 | |
| opendevreview | Alfredo Moralejo proposed openstack/releases master: Create new bug release in openstacksdk for wallaby https://review.opendev.org/c/openstack/releases/+/835788 | 08:07 |
| *** marios|ruck is now known as marios|ruck|lunch | 10:01 | |
| *** marios|ruck|lunch is now known as marios|ruck | 10:30 | |
| fungi | i'm not quite around yet, but can be if the button-pushing is starting earlier than discussed | 10:33 |
| elodilles | fungi: ack, thx | 10:35 |
| elodilles | since we passed 10:30 UTC we have less than an hour for starting the machinery :] | 10:38 |
| elodilles | if we start somewhat sooner, around 11:00 UTC then we have even less time :-o | 10:38 |
| ttx | I +2ed it | 10:41 |
| fungi | zuul looks reasonably quiet and i don't see any obvious problems indicated on the status graphs | 10:42 |
| fungi | yeah, stuff is still passing jobs and merging | 10:47 |
| fungi | and there's no real backup | 10:47 |
| elodilles | ttx fungi : roger! | 10:49 |
| elodilles | this also looks good so far: https://status.python.org/ | 10:50 |
| fungi | you can always try to merge a release-test release as a canary to make sure, but i wouldn't worry about it since we were tagging release candidates as late as yesterday | 10:55 |
| elodilles | yepp, unfortunately (& fortunately) that was not so long ago :S | 10:56 |
| ttx | let's go! | 11:02 |
| elodilles | so we have passed 11:00 UTC & zuul & python infrastructure looks good | 11:03 |
| elodilles | let's start then! | 11:03 |
| ttx | yay! | 11:03 |
| fungi | sounds good | 11:03 |
| fungi | i'm caffeinating now | 11:03 |
| elodilles | +2+W'd | 11:04 |
| elodilles | zuul has picked it up! it's on the gate! | 11:05 |
| fungi | yep, lookin' good so far | 11:05 |
| elodilles | jobs are in the queue: https://zuul.opendev.org/t/openstack/status#835322 | 11:05 |
| fungi | and nodes are getting assigned now | 11:10 |
| ttx | I have a meeting at 12:30utc that should last about an hour | 11:10 |
| elodilles | ttx: ack | 11:11 |
| fungi | i guess you'll have to turn down the release soundtrack for a little while | 11:11 |
| elodilles | the job is expected to finish around that time and post-release jobs will start afterwards | 11:12 |
| *** dviroel|out is now known as dviroel | 11:17 | |
| opendevreview | Merged openstack/releases master: Yoga final releases for cycle-with-rc projects https://review.opendev.org/c/openstack/releases/+/835322 | 11:36 |
| fungi | w00t | 11:36 |
| fungi | and now, the tags | 11:37 |
| fungi | tag-releases is already running | 11:38 |
| elodilles | oh, it was quite fast! | 11:40 |
| elodilles | i guess because we don't need the list-changes job here | 11:40 |
| elodilles | let's see how the tagging goes | 11:40 |
| fungi | it's pushed a bunch of them already | 11:41 |
| elodilles | yepp, patches are gathering in 'release' queue | 11:43 |
| fungi | https://pypi.org/project/adjutant-ui/ "4.0.0 Released: 1 minute ago" | 11:44 |
| ttx | it's happening | 11:51 |
| elodilles | :] | 11:57 |
| elodilles | tag & release queues are quite busy | 11:57 |
| fungi | per discussion in #opendev the iweb nodes (~20% of our quota) are impacted by api-related issues at the moment so i've emergency approved a change to stop booting new nodes there while they work on fixing it | 11:58 |
| fungi | but it looks like we have a lot of available capacity right now anyway so it shouldn't impact things | 11:58 |
| fungi | and the problem in the provider is dns-related leading to inaccessibility of the openstack apis, so shouldn't impact running builds | 11:58 |
| fungi | just wanted to let everyone know | 11:59 |
| fungi | unfortunate timing, but it is what it is | 11:59 |
| fungi | in good news. zuul doesn't even seem to be breaking a sweat processing all these tags | 12:02 |
| elodilles | fungi: ack, thanks for the heads up | 12:04 |
| elodilles | tag queue is still highly loaded with 'publish-openstack-releasenotes-python3' jobs for releases (67 at the moment) | 12:06 |
| elodilles | but release queue size is slowly decreasing (26 at the moment) | 12:07 |
| elodilles | soon i should start the missing-releases command if i'm not mistaken | 12:08 |
| fungi | i think the releasenotes jobs ended up using a semaphore to serialize them so they wouldn't step on one another, right? i'll double-check that's why they're piling up | 12:08 |
| elodilles | no, they are not interfere with each other, they are just added in the queue :) | 12:12 |
| elodilles | everything seem OK so far. fingers crossed :X | 12:13 |
| fungi | the disruption in iweb is over, according to mgagne | 12:13 |
| elodilles | \o/ | 12:14 |
| elodilles | release queue is estimated to be empty in a minute | 12:14 |
| fungi | confirmed, https://zuul.opendev.org/t/openstack/job/publish-openstack-releasenotes-python3 shows the builds all share a semaphore called publish-releasenotes | 12:15 |
| fungi | that's why it's only running one of them at a time | 12:15 |
| *** amoralej is now known as amoralej|lunch | 12:16 | |
| fungi | that semaphore was added ~1.5 years ago in https://review.opendev.org/756553 | 12:16 |
| elodilles | we are not in a hurry, so that's OK :) | 12:17 |
| fungi | commit message confirms it was added to address publication races | 12:17 |
| elodilles | one last 'announce-release' job is running (~3 mins left) | 12:18 |
| elodilles | started to run the 'missing-releases' command locally | 12:20 |
| fungi | seems the releasenotes jobs spend a lot of time at "TASK [submit-logstash-jobs : Submit logstash processing jobs to log processors]" which we're due to rip out when we take the old logstash system offline | 12:20 |
| fungi | should have been gone already but there were delays with the replacement | 12:21 |
| elodilles | no worries we are still in time i guess :) | 12:23 |
| elodilles | hmmm, what I see so far, 2 missing wheels: | 12:23 |
| elodilles | did not find python 3 wheel https://tarballs.openstack.org/ansible-role-atos-hsm/ansible_role_atos_hsm-3.0.0-py3-none-any.whl | 12:23 |
| elodilles | did not find python 3 wheel https://tarballs.openstack.org/ansible-role-thales-hsm/ansible_role_thales_hsm-3.0.0-py3-none-any.whl | 12:23 |
| fungi | those are from this release? | 12:23 |
| elodilles | this might be related to the setuptools issue? :/ | 12:23 |
| elodilles | yepp | 12:24 |
| elodilles | and one more: | 12:24 |
| elodilles | did not find python 3 wheel https://tarballs.openstack.org/heat-tempest-plugin/heat_tempest_plugin-1.5.0-py3-none-any.whl | 12:24 |
| fungi | i don't see them listed at https://lists.openstack.org/pipermail/release-job-failures/2022-March/thread.html | 12:25 |
| elodilles | however 'found python 2/3 wheel' is there for them | 12:25 |
| fungi | yeah, that's probably a project that forgot to drop universal from their setup | 12:25 |
| elodilles | hopefully they're just false negative signals | 12:25 |
| elodilles | fungi: hmm, that's quite possible | 12:26 |
| fungi | ansible-role-atos-hsm 3.0.0 was tagged today at 11:40z | 12:26 |
| fungi | https://pypi.org/project/ansible-role-atos-hsm/#files shows a ansible_role_atos_hsm-3.0.0-py2.py3-none-any.whl | 12:27 |
| fungi | so same situation | 12:27 |
| elodilles | yepp, probably all the same | 12:28 |
| fungi | yeah, just checked, same for all | 12:28 |
| fungi | so those projects need to fix their setup.cfg | 12:28 |
| fungi | nothing brok in the release jobs | 12:28 |
| fungi | er, broke | 12:28 |
| elodilles | ++ | 12:31 |
| fungi | i assume we don't need to wait for all the releasenotes builds to complete i order to announce the release, if it comes down to that? | 12:35 |
| fungi | er, in order | 12:36 |
| fungi | at the current run rate we're probably looking at half a day for those to complete | 12:40 |
| elodilles | yes, i think so, though we are in time, aren't we? | 12:40 |
| elodilles | fungi: oh, i see | 12:40 |
| fungi | they're taking around 5-10 minutes each, plus some delay between, and we have >60 still to go | 12:41 |
| elodilles | i thought the jobs take 1-2 minutes each | 12:41 |
| fungi | see the duration column at https://zuul.opendev.org/t/openstack/builds?job_name=publish-openstack-releasenotes-python3 | 12:42 |
| fungi | they may each be taking a couple minutes extra due to whatever's going on with the old logstash systems | 12:42 |
| fungi | what's especially unfortunate though is that the semaphore was added to that job in order to protect against publication races when two tags are pushed at the same time for different branches of the same project. there's really no need to serialize these across different projects | 12:43 |
| elodilles | the missing-release command is finished, we have 3 issues (the above mentioned ones doesn't seem to appear there, so the script knew they are false negatives): https://paste.opendev.org/show/btkRzLl3C1h9SstBLQdc/ | 12:43 |
| elodilles | the problem is with tap-as-a-service | 12:44 |
| elodilles | if i'm not mistaken it was manually released in the past | 12:44 |
| elodilles | fungi: any idea what we could do regarding tap-as-a-service | 12:46 |
| fungi | i'm looking | 12:46 |
| fungi | this was the python release build for it: https://zuul.opendev.org/t/openstack/build/bb62e96739f54f869a55e46553a5edc4 | 12:47 |
| elodilles | i don't see failing job in the log :-o | 12:48 |
| fungi | oh, https://tarballs.openstack.org/tap-as-a-service/ is redirecting to https://tarballs.opendev.org/x/tap-as-a-service/ because it was moved out of openstack originally | 12:50 |
| elodilles | yes, it was moved ~half a year ago if i'm not mistaken :-o | 12:51 |
| fungi | right, and the old redirect for it never got undone | 12:52 |
| fungi | you can see the new releases here: https://static.opendev.org/project/tarballs.opendev.org/openstack/tap-as-a-service/ | 12:52 |
| fungi | i'll push up a change to take out the redirect, and i'll separately copy the old releases from x/tap-as-a-service to openstack/tap-as-a-service | 12:53 |
| elodilles | fungi: thanks \o/ | 12:53 |
| fungi | https://review.opendev.org/835829 should solve it | 13:00 |
| fungi | and thinking about it more, i probably shouldn't copy the old tarballs into the new (or new again) location, since those weren't made when the project was an official part of openstack anyway | 13:01 |
| *** amoralej|lunch is now known as amoralej | 13:03 | |
| elodilles | hmm | 13:03 |
| fungi | the releases made as part of openstack will be reachable once that redirect removal merges and deploys though | 13:06 |
| elodilles | awesome, thanks \o/ | 13:07 |
| elodilles | should we continue with the next tasks? | 13:08 |
| fungi | i expect so, unless you just want to wait for as many release notes to be updated as possible before the deadline | 13:09 |
| elodilles | the www update & mark series as released ones | 13:10 |
| elodilles | this: https://review.opendev.org/c/openstack/openstack-manuals/+/835367 | 13:10 |
| elodilles | and this: https://review.opendev.org/c/openstack/releases/+/835323/ | 13:10 |
| fungi | but i doubt anyone will be confused if some of them aren't updated until after the announcement (i doubt anyone will even notice) | 13:10 |
| fungi | yeah, seems like those should be safe to merge now | 13:10 |
| elodilles | for the www update patch i wrote: "(short) before 14:00 UTC update the doc pages with: [www] Set yoga as released (the pages needs ~1 hrs to refresh, so needs to be done well in time)" | 13:11 |
| fungi | do you have approval permissions for openstack-manuals, or do we need to find someone? | 13:11 |
| elodilles | fungi: i guess if i remove my -W then Andreas' +W will start the job | 13:12 |
| elodilles | if i'm not mistaken :) | 13:12 |
| fungi | oh, yep that works. you may need to add a recheck since i'm not sure the workflow -1 to 0 switch will enqueue it into the gate | 13:13 |
| elodilles | let's find it out | 13:13 |
| fungi | though if rechecking is necessary it has to be in a separate comment anyway (recheck comment when adding or changing a vote gets ignored because it fails to match the comment regex for the pipeline trigger) | 13:13 |
| elodilles | i'll wait 1 or 2 mins and recheck if zuul doesn't pick it up | 13:15 |
| elodilles | -W removed | 13:15 |
| fungi | looks like it's in the gate, so that worked | 13:19 |
| elodilles | well, Andreas -W & +W'd it :) | 13:19 |
| fungi | yep, just saw, so that's what actually did it | 13:19 |
| fungi | better than a recheck since we don't have to wait for it to go back through check | 13:20 |
| elodilles | yepp :) | 13:22 |
| elodilles | ttx: in case the meeting is over, then please review this: https://review.opendev.org/c/openstack/releases/+/835323/ | 13:22 |
| elodilles | we are in time with this i think | 13:23 |
| elodilles | so far :X fingers crossed | 13:23 |
| fungi | i'm going to step away momentarily to get a shower, since things seem to be on track | 13:35 |
| elodilles | fungi: ack | 13:35 |
| elodilles | www patch has merged. we just need to wait until page is refreshing and we'll see it on docs.o.o | 13:36 |
| ttx | I'm back | 13:37 |
| ttx | +2 on the "released" patch | 13:39 |
| ttx | elodilles: did you run missing-releases? | 13:40 |
| ttx | quickly running a link checker to be extra sure | 13:42 |
| elodilles | ttx: yes, i ran it and found tap-as-a-service missing (for which fungi tracked down the root cause and started to apply fixes, see the discussion ^^^) | 13:49 |
| elodilles | ttx: i'm about to +W the 'released' patch now then | 13:51 |
| ttx | +1 | 13:51 |
| elodilles | https://docs.openstack.org has been refreshed! it shows now Yoga as the current supported release \o/ | 13:54 |
| fungi | awesome | 13:54 |
| ttx | fungi: my link checker only found broken links in the header/footer. Quite a few though | 14:03 |
| opendevreview | Merged openstack/releases master: Mark Yoga as released https://review.opendev.org/c/openstack/releases/+/835323 | 14:05 |
| elodilles | ttx: which ones? tap-as-a-service or the ansible-role-*-hsm wheels? | 14:05 |
| ttx | https://etherpad.opendev.org/p/xJvP9GdInAl69JMw7mf0 | 14:05 |
| ttx | elodilles: no outside of the release content, in the footer and header | 14:05 |
| ttx | I think those are in some theme we need to fix | 14:06 |
| elodilles | hmm, i see. | 14:07 |
| ttx | only 3 are really broken | 14:09 |
| ttx | I'll push a fixfor them | 14:11 |
| elodilles | (meanwhile, releases patch has merged, but releases.o.o is not yet updated) | 14:14 |
| elodilles | i don't even find the missing pages. security i guess is https://security.openstack.org but don't know the others :S | 14:22 |
| opendevreview | Thierry Carrez proposed openstack/releases master: Fix links in releases.o.o footer https://review.opendev.org/c/openstack/releases/+/835834 | 14:23 |
| ttx | This syncs it with the one used in openstack.org | 14:23 |
| elodilles | releases.o.o has been refreshed as well: https://releases.openstack.org/ | 14:23 |
| ttx | Let's wait for the docs build to approve it | 14:23 |
| ttx | I think we are all set | 14:24 |
| ttx | Feel free to send emails, I'm preparing a tweet :) | 14:24 |
| elodilles | ttx: your patch won't fix the links in docs.o.o, right? | 14:25 |
| elodilles | as i see it has the same wrong links :S https://docs.openstack.org | 14:26 |
| elodilles | in the footer | 14:26 |
| ttx | I only care about releases.o.o :) Yes it will only fix the customized copy of the theme we use on releases-side | 14:27 |
| elodilles | ack :) | 14:27 |
| elodilles | so i'm preparing the announcement mail \o/ | 14:28 |
| elodilles | ttx: "Remind Foundation staff to switch openstack.org Yoga messaging on" do we need this? or you've already done that? o:) | 14:29 |
| elodilles | ttx: i see that you added 'confirmed ok' to 'Coordinate the timing of the email with the press release from the Foundation staff' | 14:29 |
| ttx | Checking | 14:30 |
| elodilles | (i'm ready to send the announcement mail meanwhile) | 14:31 |
| fungi | and i'm available to approve it through the moderation queue | 14:37 |
| elodilles | fungi: ack | 14:41 |
| elodilles | hmmm, the 'stay in touch' part of the generated docs in the footer looks weird: https://309a0a1a1f63db90b41e-6e6a98e42a02be15877850d4946fa43f.ssl.cf2.rackcdn.com/835834/1/check/openstack-tox-docs/679b370/docs/index.html | 14:42 |
| elodilles | though i don't know whether it is just wrong there or will be the same once we merge the 'fix links' patch | 14:42 |
| ttx | You can announce elodilles | 14:44 |
| ttx | Let me know when the mail is sent so I can push my tweet :) | 14:44 |
| gmann | \o/. thanks release team for all your effort. | 14:45 |
| elodilles | fungi: mail is sent to openstack-announce | 14:46 |
| elodilles | gmann: \o/ thanks too \o/ | 14:47 |
| fungi | waiting for it to arrive at the listserv | 14:47 |
| fungi | seems like it may be taking a minute | 14:47 |
| fungi | elodilles: you're subscribed to openstack-announce, right? | 14:48 |
| elodilles | fungi: nope :S | 14:48 |
| fungi | oh, then your post may be automatically rejected | 14:48 |
| ttx | elodilles: do you have a twitter handle I can tag? | 14:48 |
| elodilles | fungi: let me subscribe quickly | 14:49 |
| fungi | elodilles: yeah, looks like mailman sent you a rejection notice | 14:49 |
| fungi | :q | 14:50 |
| fungi | hah, you're not my vi process | 14:50 |
| elodilles | i've subscribed now | 14:51 |
| elodilles | i mean i guess i need some verification first :) | 14:51 |
| fungi | yeah, it should send you a confirmation token by e-mail | 14:52 |
| fungi | check your junk mail if it's not in your inbox | 14:52 |
| elodilles | ok, verified, resending the announcement | 14:52 |
| fungi | awesome, standing by to approve | 14:52 |
| elodilles | sent | 14:54 |
| fungi | yep, got it. double-checking the sample really quickly | 14:54 |
| elodilles | ttx: well, i don't have twitter so i'll rely on you about twitter announcement o:) | 14:54 |
| elodilles | fungi: ack | 14:55 |
| ttx | alright :) | 14:55 |
| fungi | approved it | 14:55 |
| elodilles | fungi: \o/ | 14:55 |
| elodilles | here we have it: http://lists.openstack.org/pipermail/openstack-announce/2022-March/002059.html | 14:55 |
| fungi | yep. looks good | 14:56 |
| fungi | right on schedule | 14:56 |
| fungi | ho hum, yet another on-time release | 14:56 |
| elodilles | just. in. time. :D | 14:56 |
| ttx | yay! | 14:57 |
| gmann | /o\ | 14:58 |
| elodilles | thanks everyone! \o/ | 14:59 |
| elodilles | ttx: thanks for the tweet :] | 15:00 |
| *** marios|ruck is now known as marios | 15:00 | |
| *** dviroel is now known as dviroel_ | 15:04 | |
| *** dviroel_ is now known as dviroel | 15:04 | |
| fungi | maybe it should have been "ho hum, a 25th on-time release" | 15:06 |
| elodilles | :] | 15:06 |
| fungi | the narrow terminal for my mail client truncates the announcement subject at "OpenStack Yoga is off" | 15:08 |
| fungi | i thought that was fun | 15:09 |
| *** marios is now known as marios|ruck | 15:10 | |
| elodilles | :S | 15:11 |
| elodilles | :D | 15:11 |
| fungi | i'm headed out to my appointment in a few minutes, but looks like all the excitement here is done | 15:39 |
| fungi | following clarkb's suggestion i restarted the geard process on the logstash server which has eliminated the several-minute delay for that task during each releasenotes job, which has sped things up some, but we're probably still looking at a few hours until the remaining releasenotes builds complete | 15:40 |
| elodilles | fungi: ack, thanks for the help! | 15:42 |
| *** dviroel is now known as dviroel|lunch | 15:44 | |
| clarkb | feel free to ping me if something comes up in the next hour while fungi is out too | 15:45 |
| *** amoralej is now known as amoralej|off | 15:50 | |
| elodilles | clarkb: ack, thanks in advance! | 15:52 |
| *** marios|ruck is now known as marios|out | 16:20 | |
| fungi | okay, i'm back | 16:54 |
| fungi | 27 releasenotes jobs in the tag pipeline still | 16:59 |
| *** dviroel|lunch is now known as dviroel | 17:00 | |
| elodilles | it's progressing slowly but steady | 17:20 |
| opendevreview | Elod Illes proposed openstack/releases master: Fix links in releases.o.o footer https://review.opendev.org/c/openstack/releases/+/835834 | 18:26 |
| fungi | the very last releasenotes build just started. ~8.75 hours start to finish for all of those to run serially | 20:39 |
| elodilles | it really took a while :) | 20:41 |
| elodilles | but we are almost there :D | 20:41 |
| fungi | and done! | 20:42 |
| elodilles | \o/ | 20:42 |
| fungi | we should really figure out if there's a better way to not have rsync writes collide so those can safely be run in parallel instead | 21:04 |
| *** dviroel is now known as dviroel|out | 21:16 | |
| fungi | looking at the rsync manpage, it's possible we could use --temp-dir and make sure to avoid --delay.* and --partial.* options | 21:59 |
| clarkb | fungi: aiui it was fine before it just raised a failure and that made people concerned. Maybe have a check of the content after the rsync to see if it is complete and not fail if so? | 21:59 |
| fungi | clarkb: the underlying problem is that rsync creates tempdirs in the same directories as the files being replaced, and --delete wants to remove another concurrent processes tempdirs in the same tree | 22:00 |
| fungi | if we told it not to create tempdirs in the same place as where files are being deleted, we could probably work around it (except in the rare case where two releasenotes builds try to update the same index.html at the exact same moment) | 22:01 |
| fungi | --inplace might be another option, as i think that may avoid creating tempfiles/tempdirs entirely | 22:02 |
| fungi | most of these features are trying to solve problems we don't have since we write to a read-write afs volume and then separately release that to the read-only replicas | 22:03 |
| fungi | so things like atomic file replacement are less critical as we have atomic file*system* replacement | 22:04 |
| clarkb | ah yup since we do that on the whole tree | 22:04 |
| clarkb | that makes sense | 22:04 |
| fungi | another approach might be to have rsync wait if it sees evidence another rsync is already updating the same tree (might be toctou racy tho) or have it retry on error (might lead to hysteresis/livelock between processes?) | 22:07 |
| fungi | but probably asking rsync to do less is the simplest solution | 22:07 |
| fungi | it's smart about some things we want, but its attempts at being smart about other things we don't need are getting in the way | 22:08 |
| *** ianw_pto is now known as ianw | 22:24 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!