Wednesday, 2025-04-02

*** bauzas9 is now known as bauzas01:40
opendevreviewElod Illes proposed openstack/releases master: Add missing release note links for 2025.1 Epoxy #2  https://review.opendev.org/c/openstack/releases/+/94611206:33
elodillesrelease-team: when zuul verifies this then this can be reviewed & merged as soon as possible ^^^06:38
fricklerelodilles: done06:42
fricklerelodilles: seems the release highlights for kolla, kolla-ansible and kayobe would all get rendered into a single block, is that expected? https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7e9/openstack/7e9db142d16a463faacf116a4536d895/docs/epoxy/highlights.html#kolla06:43
opendevreviewMerged openstack/releases master: Add missing release note links for 2025.1 Epoxy #2  https://review.opendev.org/c/openstack/releases/+/94611206:44
elodillesfrickler: thanks!06:55
elodillesfrickler: you mean the cycle highlights?06:55
elodillesfrickler: but yes, cycle highlights are rendered with grouped by teams06:56
elodillesit always worked like that06:59
elodillesat least, as far as i know07:03
fricklerelodilles: ok, so then we'll need to adopt that kolla change07:05
elodilleswhich change? o:)07:10
fricklerthe one from which I posted the docs build https://review.opendev.org/c/openstack/releases/+/94604107:11
elodillesah, i see07:14
zigoRelease day is today, isn't it ?08:10
elodillesit is, indeed :]08:16
zigo:)08:18
zigoI have all of Epoxy already in Unstable, just waiting for new tags. This should hopefully end up all in Debian 13, aka Trixie.08:19
zigoAlso, I was able to tempest validate everything already ... :)08:19
zigo(in Bookworm backports though)08:20
fricklerseems there's some trouble still lurking between eventlet and py3.13 :-/08:23
elodilleszigo: thanks for the info, good to hear that! o/08:38
elodillesfrickler: what do you mean? :-o08:38
zigofrickler: Like what? Do you have more details?08:38
opendevreviewRiccardo Pittau proposed openstack/releases master: Relase python-ironicclient 5.10.1 for epoxy  https://review.opendev.org/c/openstack/releases/+/94612008:44
opendevreviewRiccardo Pittau proposed openstack/releases master: Release ironic-inspector-client 5.3.1 for epoxy  https://review.opendev.org/c/openstack/releases/+/94612108:46
rpittauzigo, elodilles, fixed releases for cliff 4.9.1  compatibility ^08:46
elodillesrpittau: as i see those contains only non-functional (test code) changes (i do understand that those are gate fixes), i kind of think they should not be released at all :/08:57
elodillesbut we can think about that after the coordinated release today...08:58
zigoGreat ! :)09:03
zigorpittau: Will there be a release for inspector-client too ?09:03
rpittauelodilles: yep, it's just to fix the unit tests, unfortunately cliff 4.9.1 broke them in both clients and the patches were not merged in time 09:03
rpittauzigo: yeah it's tehre :)09:03
zigoOh, sorry, didn't read.09:04
rpittaunp!09:04
fricklerzigo: elodilles: that was what I picked up in the eventlet removal channel https://github.com/eventlet/eventlet/issues/103009:12
zigoDie eventlet, die ... :/09:14
zigofrickler: Do you know if the related patches https://github.com/eventlet/eventlet/pull/1031 and https://github.com/eventlet/eventlet/issues/1032 are good enough?09:15
zigoAh, the 2nd one isn't a patch...09:15
fricklerzigo: I didn't follow this in detail yet, best ask in #openstack-eventlet-removal09:16
fungieverything going smoothly so far?09:53
elodillesT-509:55
* fungi grabs some much needed coffee09:55
elodillesfungi: yepp, though we haven't started the main part of the process yet09:56
elodillesfungi: enjoy your early morning coffee :)09:56
elodillesfrickler: are you around to approve the patch @ 10:00 UTC?09:56
elodilleswaiting for +W: https://review.opendev.org/c/openstack/releases/+/94590409:59
elodillesthis looks green: https://status.python.org/10:01
elodilleszuul also looks OK to me10:02
fungiyeah, still catching up but i don't see any reports of known problems10:03
elodillesfungi: ACK, then we are good to go i think10:03
elodillesOkay, let's not wait any more, i'm pushing the red button then10:05
elodilles+W'd10:05
elodillesmerge took 39 mins for Dalmatian release. let's see how this will compare with that10:11
frickleroops sorry, got distracted, I had somehow in my mind that we'd start later10:17
fungiwe shoot for doing the marketing press release around 15:00 utc, so the hope is that 5 hours will be enough for doc updates to reach websites and so on10:20
elodillesyepp, we have started the things around 10:00 UTC in the past couple of cycles and it proved to be a good timing10:22
elodillesbtw i had a related process update patch, which extends the release duties exactly with these details, reviews are welcome after the release o:) https://review.opendev.org/c/openstack/releases/+/94586110:28
opendevreviewGrzegorz Koper proposed openstack/releases master: Add Kolla/Kolla-Ansible/Kayobe Epoxy cycle highlights  https://review.opendev.org/c/openstack/releases/+/94604110:31
opendevreviewMerged openstack/releases master: 2025.1 Epoxy final releases for cycle-with-rc projects  https://review.opendev.org/c/openstack/releases/+/94590410:38
elodillesmerged10:38
elodillesthat was quick :-o10:38
elodilles33 mins this time10:38
elodillesbetter than half a year ago10:39
fricklermight depend on whether it lands on old rax nodes or ovh like in this case10:42
elodillesyepp10:42
elodillesthat might count i guess10:42
elodillesthe release pipeline is getting populated with deliverables10:44
elodilles9 at the moment10:44
elodilles1110:44
elodilles13, slowly piling up10:45
fungihttps://pypi.org/project/adjutant-ui/ has 10.0.0 "Released: 5 minutes ago"10:53
fungiso things seem to be generally working10:53
elodillesgood good10:54
fungirelease notes jobs are also running in parallel, so the temporary semaphore removal does seem to have worked again10:55
elodilles+110:55
elodilleswe are at letter "t" in the alphabet and 56 tasks in release pipeline10:56
elodilles67 and this should be the max if i'm not mistaken10:59
elodilleswith zun as the last deliverable10:59
fricklerI guess I should have reviewed https://review.opendev.org/c/openstack/releases/+/945861 earlier, that makes the start time explicit ;)11:01
elodillesyepp o:)11:02
elodillesrelease queue is decreasing slowly11:03
fricklerwe should wait with approving https://review.opendev.org/c/openstack/releases/+/945948 until that's done, do we?11:08
fungizuul utilization shot up right at 11:00 utc and now we seem to be maxing out quota and running a bit of a node request backlog11:09
fungiprobably the constraints update proposals triggered out of the release jobs11:11
fungioh, there were some kolla-ansible changes pushed that want to run a screenful of jobs11:11
fricklerhmm, bad timing by kevko, but then we (kolla) are also struggling to get our stuff done for the release, so ...11:14
fungiouch, each change runs 68 jobs (many of which are multi-node), but what's most surprising is that 70% of those jobs are non-voting... why?11:14
elodillesyeah i see that check queue is also growing :S11:14
fungiseems like if >2/3 of your jobs are non-voting, you're better off using the experimental pipeline for those11:14
elodillesfrickler: we have to check whether things got released properly first before we mark series as released11:15
fungianyway, this hopefully shouldn't impact release throughput much, check is lower priority for getting nodes assigned11:15
fricklerit depends on the change, most of the time much less jobs are running, but if some basic stuff is touched that affects all scenarios, things blow up a bit. we can discuss more in the kolla channel if needed11:16
fungiyeah, almost every change in that series they pushed ran the full set of their jobs, looks like11:16
fungianyway, in positive news, release notes updates are working: https://docs.openstack.org/releasenotes/cinder/2025.1.html11:17
fungiwe're just waiting for nova to finish and then they'll all be done11:17
fungiand the release pipeline is burning down at a reasonably fast clip too11:18
frickler(also yes, experimental pipeline is an option, but triggering it and watching for results requires a lot of reviewer discipline)11:18
elodillesa couple of deliverables' jobs are still in 'queued' or 'waiting' state, otherwise yes, the queue is shrinking11:21
fricklerhmm, why does this have branch: master instead of stable/2025.1? I vaguely remember we had the same thing last time, maybe I should check old logs https://review.opendev.org/c/openstack/requirements/+/94613111:21
elodillesoh, that's odd11:22
frickleror maybe I misremember, this still looks broken https://opendev.org/openstack/requirements/src/branch/stable/2024.2/upper-constraints.txt#L54611:24
fricklerI mean, the master branch of reqs should also be updated, but would also happen via the automated bot updates11:24
fricklerguess I'll just create backports, same thing should happen for taas in a bit11:25
fungigood catch11:27
fungiokay, release notes jobs have all finished11:27
elodilles7 items in the release pipeline still, jobs are getting slowly their nodes11:29
fricklerand gerrit's cherry-pick menu reminds me that there is still a stable/newton branch to clean up :-/ 11:29
fungii guess the constraints update jobs are so slow because of the 10-minute pause in https://opendev.org/openstack/project-config/src/branch/master/playbooks/proposal/propose-update-constraints.yaml11:31
fungii wonder if that could be converted to a polling loop or something11:31
elodillesthe problem is more that still release-openstack-python jobs are not getting their nodes11:32
fricklerah, yes, that is just my hack to avoid things not being updated in pypi when the reqs checks run.11:32
elodillesafter all those finished i could run already the missing-releases check11:33
fricklersince this is mostly not relevant relevant for the release, we could temporarily revert it like the semaphore thing for the next release11:33
fungilooks like there's one about to finish and one that hasn't gotten a node assignment yet11:33
fungiit just started running, finally11:33
elodilles\o/11:34
elodillesyepp. vitrage-dashboard was waiting as last11:34
ttxBad weather for the ski this afternoon, so I'm around to assist!11:34
fungiso once that one gets as far as uploading to pypi and the tarballs site, you should be able to run the check and have it come through clean11:34
ttxAny issue so far?11:34
elodillesi've started the missing-releases check (it's also slow so hopefully vitrage-dashboard gets relesed when the script reaches it)11:35
fungittx: sorry to hear that!11:35
fungino, smooth sailing11:35
fungiokay, vitrage-dashboard is done11:36
fungiso that's everything uploaded11:37
fungiand no errors still according to https://lists.openstack.org/archives/list/release-job-failures@lists.openstack.org/11:37
ttxfungi: what time is the PR drop? 15utc?11:37
fungittx: correct11:37
fungiso we have about 3.5 hours still11:37
fungishould be plenty of time11:38
ttxPerfect!11:38
elodilles(i'm running the missing-releases check with this fix: https://review.opendev.org/c/openstack/releases/+/945630 this results some false negative results for releases prior mid-january as we said... and i still see some py2.py3 false errors too :S)11:40
elodilles'did not find python 3 wheel https://tarballs.openstack.org/ansible-role-atos-hsm/ansible_role_atos_hsm-9.0.0-py3-none-any.whl'11:41
elodilleswe have 'ansible_role_atos_hsm-9.0.0-py2.py3-none-any.whl' instead11:42
elodillesmaybe this one is the only one with this kind of error ^^^11:43
fungithey need to drop the universal flag from their package config unless they're still intending to support python 2.711:46
elodillesyes11:48
zigottx: Where do you ski usually ? :)11:50
fricklerelodilles: is the check still running or is all well now?11:50
elodillesOK, so this is the list of errors: https://paste.opendev.org/show/bQ7g1ftJoYMpIr2emBzc/11:51
elodilleslet me quickly check if there are any valid error amongst them11:51
elodillesfrickler: just finished11:51
elodillesOK, all false negative11:54
fungilooks like we're still waiting for two constraints proposal jobs to get nodes assigned, but otherwise everything's finished in the release pipeline11:54
elodilleswe are good11:55
fungiought to be able to proceed11:55
elodillesall deliverables are available \o/11:55
ttxzigo: I'm in Tignes right now11:55
zigoOh, lucky you !11:56
elodillesnext task should be: "Before 13:00 UTC update the doc pages with: [www] Set 2025.1 Epoxy as released (the pages need ~1hr to refresh, so needs to be done well in time)"11:57
elodillesplease review the patch: https://review.opendev.org/c/openstack/openstack-manuals/+/94531911:58
fricklerelodilles: reviewed already, please remove your W-1 and I'll W+111:58
elodilleswe have ~1hr until 13:00 UTC, do you want to wait a bit or simply +W it now?11:58
fungiyeah, landing that ahead of schedule doesn't hurt, we've encountered problems in the past we needed to fix11:59
fricklerIMHO no need to delay it11:59
elodillesACK, removing -W11:59
elodillesfrickler: please +W it :)12:01
fricklerand done12:01
elodilles~o~12:02
fungiokay, the remaining constraints update jobs have finished now12:11
fricklerthe previous manuals update should hopefully be promoted just in time when the new one merges (more or less, in hindsight I likely shouldn't have approved that earlier)12:14
fungidoesn't matter, promote is supercedent12:17
elodilles'[www] Set 2025.1 Epoxy as released' has merged 12:17
elodillesso let's just wait until docs.o.o gets refreshed12:18
fungiit'll take time, the promote job for that won't start running for probably another 10 minutes, and takes around 35-40 minutes to complete12:19
fungiand then there's an afs vos release cron that needs to fire12:19
fungiso best case we're looking at that being live on docs.o.o around 13:15 utc i think12:20
elodillesthat sounds OK to me. and if everything is good after the refresh i guess we can merge this patch as well: https://review.opendev.org/c/openstack/releases/+/94594812:21
elodilles'Mark 2025.1 Epoxy as released'12:22
elodillesand then we have to remind Foundation staff to switch openstack.org 2025.1 Epoxy messaging on12:22
elodillesfungi: maybe you could handle this ^^^12:23
fungiand send the announcement to the openstack-announce ml12:23
elodillesyepp, after releases.o.o got refreshed12:23
fungiyeah i've been giving the staff updates throughout the process12:23
zigoIt looks like I'll be finished at the same time as you guys ! :)12:23
elodilleszigo: nice work!12:24
zigo(that's a personal record ...)12:24
fungipromote for the final docs site update is running now12:38
fungidepending on how slow it is syncing remaining content into afs, it might be closer to 13:30 before it's live12:39
elodillesthat's still good i think12:39
fungisince we've got a bit of a wait at this stage, i'm taking the opportunity to grab a shower12:42
fungibbiab12:42
fricklerlooks like these are failing since a month :-/ https://zuul.opendev.org/t/openstack/builds?job_name=upstream-translation-update&project=openstack/openstack-manuals12:42
zigoI'm done... :)12:44
fricklerissue seems to be trying to use new flake8 on old python. probably this job cannot use anything newer than bionic, though?12:44
zigo\o/12:44
elodilles:S12:44
fricklerzigo: congrats, all tests passed? or "only" done with repackaging?12:44
zigoAll tests already passing on Bookworm, not on Unstable/Trixie yet, though all done with packaging Epoxy final in Unstable + bookworm backports.12:45
fungifrickler: we've already had to make some adjustments related to that, but at some point our workarounds are likely to cease being viable12:45
fungishort story is that the java needed to run the abandoned zanata client is contemporary with ubuntu 18.04 lts12:46
fungiwe can't really solve this properly until the i18n sig finishes their migration to weblate12:46
frickleryes, I know. let's hope there's a bit more progress this cycle12:47
frickleroh, found it, guess we should revert https://review.opendev.org/c/openstack/openstack-manuals/+/943251 for now. sorry takashi12:51
fricklerhmm, gerrit's revert/cherry-pick actions also seem even slower than usual. can't say they were really snappy before, though, either12:53
fungiopenstack-manuals promote finished just in time for the 13:15 vos release pulse13:14
fungiwatching the server-side log for that now13:15
fungi2025-04-02 13:15:02,085 release INFO     Releasing docs13:15
elodilles\o/13:15
fungi2025-04-02 13:15:47,900 release INFO     Release of docs successful13:16
fungishould be live now13:16
elodillesi guess then we can merge this as well, no need to wait with this either: https://review.opendev.org/c/openstack/releases/+/94594813:16
fungisounds good13:16
elodillesACK, +2+W'd13:17
fungiprobably the only step i'd wait on is sending the openstack-announce ml post, maybe aim to send it by 14:40 so i can approve it through moderation at 14:45 and that way it'll get delivered to subscribers just before the press release embargo is lifted13:17
elodillesfungi: ACK, will do that13:18
elodillesi've prepared the mail content fwiw: https://etherpad.opendev.org/p/relmgmt-weekly-emails13:19
elodillesmails. one towards openstack-announce and one towards openstack-discuss13:19
fricklerhmm, https://docs.openstack.org/2025.1/deploy/ still says " This is documentation in progress for the next release. "? also that page is still empty, because all deployment projects are trailing13:22
elodillesfrickler: but that is true for trailing projects13:23
elodillesit's a bit unfortunate, but that is the case13:24
fricklerhmm, but maybe the wording could be better then? also this has been forgotten before https://review.opendev.org/c/openstack/openstack-manuals/+/93434813:24
elodilleswe can maybe push a patch with some different wording there since we have time?13:26
elodillesdo you have any better idea what to write there?13:27
fricklerI'll have to think about it. maybe add some reference to the cycle-trailing deadline? 13:27
elodillesyepp, that is better than 'in progress'13:28
elodillesor some general statement like 'deployment tools releases are scheduled approximately 2 months after the coordinated official release of OpenStack components' or something like that13:30
elodillesbut that's also not true as deployment tools CAN release earlier if they are ready13:30
elodillesor just skip the 'approximately 2 months' from the above13:33
opendevreviewMerged openstack/releases master: Mark 2025.1 Epoxy as released  https://review.opendev.org/c/openstack/releases/+/94594813:33
elodillesthis has merged too ^^^ \o/13:33
fungistill have to wait for publish-tox-docs-releases to run in release-post and then the next vos release should publish it from afs13:37
elodillespublish-tox-docs-releases job just started13:42
fungivos release should fire in a few seconds13:49
fungiand done13:50
elodilleshttps://releases.openstack.org/13:51
elodilles\o/13:51
fungihttps://releases.openstack.org/ is updated now13:51
ttx\o/13:51
elodillesfrom our side we are mostly ready, i'll send the mail to announce in ~45 mins (prepared here, feel free to edit if you want to rephrase anything: https://etherpad.opendev.org/p/relmgmt-weekly-emails )13:55
ttxlgtm13:57
gouthamrw00t13:59
fungii'm standing by for openstack-announce moderator duty when that happens14:00
ttxAll set on the foundation marketing side14:27
ttxLet us know when you post the email and we'll follow up with social media activity14:27
elodillesACK14:28
elodillesI'll prepare now the mail to announce list14:28
fungi"latest release" and "source code" links at https://www.openstack.org/software have been updated to point at the epoxy urls now14:29
elodillescool \o/14:31
elodillesmail sent14:32
elodillesto announce14:32
elodilles(will send to openstack-discuss as soon as I can point to the announce mail o:))14:32
fungii'll take a look over it in just a sec, thanks!14:36
fungialso https://review.opendev.org/945885 has now merged to reinstate the job semaphores14:36
elodilles+114:36
fungii mostly didn't want to forget, so approved that revert while it was fresh in my mind14:38
fungiopenstack-announce post lgtm, i'll click approve in about 5 minutes14:40
elodilles++14:40
fungiand approved14:45
elodillesbtw, the main page still shows dalmatian as latest to me14:45
elodillesfungi: thanks \o/14:45
fungiyes, the main page is going to get updated closer to the top of the hour14:46
fungielodilles: actually, they updated it right after you said that, refresh the page and it should be correct now14:47
elodillesah, i was impatient o:)14:48
opendevreviewDan Smith proposed openstack/releases master: Release 3.4.0 for os-traits  https://review.opendev.org/c/openstack/releases/+/94616214:51
fungii have received both copis of the announcement now14:52
fungicopies14:53
fricklerso when can we approve 2025.1.X and 2025.2 releases? right away for the latter?14:53
clarkbI too have received both emails14:53
fungifrickler: if you're really concerned about tags getting signed by the flamingo cycle key, then you could wait until tuesday (as we're scheduled to rotate the signing key on monday)14:54
fungibut i don't expect that's strictly necessary14:54
fungiwe've not been that particular about it in the past anyway14:55
opendevreviewElod Illes proposed openstack/releases master: Add 2025.2 Flamingo deliverable files  https://review.opendev.org/c/openstack/releases/+/94616314:55
fricklerfungi: good point, I'm mainly asking about https://review.opendev.org/c/openstack/releases/+/945183 . not sure if someone like zigo cares about the gpg signing at all?14:57
elodillesfrickler: i've prepared the flamingo init patch ^^^ maybe that should be the first14:58
fricklerright, that will merge-conflict anyway. let me do some diffing on that14:59
elodillesfor stable/2025.1 epoxy releases maybe we can wait a couple of days with any release just to let people consume the current official packages. at least i remember us doing like this in the past15:00
frickleryes, I remember that, too, but wasn't sure whether it is still customary today15:01
elodillesnoonedeadpunk: this is the time to double check OSA repos under deliverables/flamingo :) https://review.opendev.org/c/openstack/releases/+/94616315:01
elodillesnoonedeadpunk: simply push an update patch on top of that ^^^ if needed15:01
noonedeadpunkelodilles: ok, that is much better, but have an edit :)15:03
fungiokay, seems like all the criticial activities are completed, so i'm going to go grab lunch, but will check back in soon15:03
elodillesfungi: thanks fungi for your support! enjoy your lunch! o/15:04
opendevreviewDmitriy Rabotyagov proposed openstack/releases master: Add 2025.2 Flamingo deliverable files  https://review.opendev.org/c/openstack/releases/+/94616315:06
noonedeadpunkthanks for the ping!15:06
noonedeadpunkdoh, on top, not in place...15:06
elodillesnoonedeadpunk: no worries, this is good like this as well :)15:07
noonedeadpunksorry.. I can revert....15:07
elodillesno no, this is okay15:07
noonedeadpunkJust in a midset that things in releases are not very editable once merged15:07
elodillesyepp, most cases that's true :)15:08
elodilleswe can wait until the validator job finishes o:)15:10
clarkbcongrats! I woke up and checked in on things and everything seemed to be going well so I walked the kids to school like any other day. I like when releases are "like any other day"15:12
elodillesme too :) thanks too clarkb o/15:13
ttxThanks for leading the process! I feel like I should go skiing every release week15:27
elodilles:D15:36
elodillesfortunately this release was without any glitch, apparently :)15:37
elodillesttx: enjoy your time off!15:38
elodillesand have a nicer weather for skiing for the coming days15:38
fungittx: you *should* go skiing every week16:33
dansmithcan someone approve this ptl-approved trivial release to unblock other work in nova? https://review.opendev.org/c/openstack/releases/+/94616216:49
elodilles+2'd it17:03
dansmithelodilles: thanks!17:05
fricklerand +317:10
fricklerelodilles: do you want to self approve https://review.opendev.org/c/openstack/releases/+/946163 then or is ttx still around?17:10
dansmithfrickler: thanks also17:12
opendevreviewMerged openstack/releases master: Release 3.4.0 for os-traits  https://review.opendev.org/c/openstack/releases/+/94616217:18
elodilles+2+W'd, let's bootstrap 2025.2 Flamingo 🦩19:23
opendevreviewMerged openstack/releases master: Add 2025.2 Flamingo deliverable files  https://review.opendev.org/c/openstack/releases/+/94616319:47
clarkbwe need more utf8 flamingos19:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!