| opendevreview | Tony Breeds proposed openstack/governance master: [docs] Add links to release based runtimes https://review.opendev.org/c/openstack/governance/+/899301 | 00:27 |
|---|---|---|
| spotz[m] | Thanks Jayf! | 10:22 |
| frickler | tc-members: the details of how to do "Unmaintained" are still in flux (https://review.opendev.org/c/openstack/project-team-guide/+/897505) but the deadline where Yoga should end being maintained is in two days, are there any ideas as to how this should be handled? | 12:33 |
| knikolla | frickler: I'll update the patch today to reflect the last series of comments. Do you think there are other gaps in that doc that I should make sure to touch on today? | 14:24 |
| frickler | knikolla: I think the most important gap is in telling the release team how they should actually act on this | 15:11 |
| frickler | like, iiuc they should create patches to eol yoga branches at some time, likely this week? how long should be the "opt-in" period then be for candidate un-maintainers to -1 those patches? | 15:13 |
| frickler | also should we change the yoga Next Phase from "Extended Maintenance estimated 2023-11-02" to something else on https://releases.openstack.org/? or does that need the p-t-g patch first? | 15:15 |
| JayF | I wish we would've had a specific TC agenda item for this | 15:36 |
| JayF | but now that I've mailed out the agenda it seems too late to add one | 15:36 |
| opendevreview | Kristi Nikolla proposed openstack/project-team-guide master: Update docs for Unmaintained https://review.opendev.org/c/openstack/project-team-guide/+/897505 | 15:49 |
| fungi | it probably warrants its own ml discussion anyway, since the release managers don't necessarily attend the tc meetings | 15:50 |
| knikolla | I'll attend the release team meeting this Friday. I'm not aware of the usual wait times that the release team gives team for the various deliverables and processes. I will ask them about reasonable timeframes. | 15:54 |
| fungi | knikolla: they're skipping their meeting this week according to https://etherpad.opendev.org/p/caracal-relmgt-tracking | 15:57 |
| knikolla | yep, just saw that. | 15:57 |
| fungi | cool, just didn't want you to show up and feel lonely ;) | 15:58 |
| fungi | so anyway, async discussion is probably better unless you want to wait nearly 2 weeks | 15:59 |
| fungi | (and also ttx won't be around next week) | 15:59 |
| knikolla | Posted on -release channel. | 16:04 |
| gmann | as per resolution, we need to keep last three EM branch as unmaintained so if yoga moves to EM before we implement the process then we should keep last three from Yoga which will be stable/yoga, stable/xena, and stable/wallaby | 16:10 |
| frickler | in my understanding (and elodilles seemed to share that) the resolution went into effect once it was approved. so yoga moving to EM would no longer be an option? | 16:28 |
| fungi | but also elodilles is out this week, i think | 16:33 |
| frickler | well their response was from earlier today, so there seem to be different shades of "out" ;) not that I wouldn't often be guilty of the same :D | 16:43 |
| gmann | yoga is still in scope as they are pre-SLURP releases and when SLURP release things start (from stable/2023.1) then we keep it for SLURP releases only and non-SLURP release does not fall into this process "Only SLURP releases are eligible for having an Unmaintained branch." | 17:00 |
| gmann | "Until the first SLURP release ends its maintained phase, all current branches are eligible for Unmaintained." | 17:02 |
| gmann | https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branches.html#transition | 17:02 |
| elodilles | well, i wasn't on PTO until today, but in a minute I'll be on PTO (Wednesday, Thursday and Friday) o:) | 17:02 |
| gmann | so as per my reading here. I think yoga, zed need to move to 'Unmaintained' state as default and after that we will keep only SLURP releases in 'Unmaintained' state | 17:03 |
| gmann | so 2023.2 (non SLURP) is not eligible for 'Unmaintained' instead it will directly move to EOL | 17:03 |
| elodilles | something like that was my understanding as well ^^^ (though i was not sure whether 2023.2 can be kept as unmaintained if any project signals their opt-in) | 17:05 |
| JayF | tc-members: 53 minutes until weekly meeting | 17:07 |
| frickler | as mentioned in the p-t-g review, with SLURP not guaranteeing hitless rolling upgrades being possible, they aren't useful for all deployments, so it is questionable if e.g. kolla will support it at all. as a consequence, there may be an interest in also keeping non-slurp branches active | 17:08 |
| dansmith | what does "not guaranteeing hitless rolling upgrades" mean? | 17:09 |
| dansmith | meaning not guaranteeing live/online upgrade behavior? | 17:09 |
| frickler | yes, kind of the usual thing operators have come to expect from cycle-to-cycle upgrades | 17:11 |
| dansmith | AFAIK few projects even provide that for upgrades across once cycle | 17:12 |
| dansmith | and I'm not aware of any that test it other than nova | 17:12 |
| frickler | I think at least all the integrated projects do | 17:13 |
| dansmith | cinder has a job there cinder-volume is left un-upgraded on one machine while the rest is upgraded? | 17:13 |
| dansmith | glance most certainly does not | 17:13 |
| frickler | at least in terms of claiming support for that and fixing bugs if it doesn't work. not sure about actual testing | 17:13 |
| dansmith | we used to have a tag for rolling upgrades and I didn't think anyone but nova, swift, and maybe neutron claimed it | 17:14 |
| dansmith | rosmaita: does cinder allow for a cinder-volume running on the previous release to co-exist (i.e. speak RPC) to any/all of the stack running on the newer release? | 17:16 |
| dansmith | RPC and database of course, but I assume RPC is more the concern | 17:16 |
| dansmith | I guess my point is that we didn't require rolling upgrades in slurp because it was asserted (or perhaps assumed) that not everyone _could_ do that without undue hardship | 17:19 |
| dansmith | but if they can, we should roll that into the definition of slurp | 17:19 |
| dansmith | nova already keeps the N-2 job enabled in non-slurp releases | 17:20 |
| frickler | even if it was only nova for which there is a significant difference between SLURP and cycle-by-cycle, that would IMO make the whole concept not suited for the deployments I know about | 17:20 |
| dansmith | looks like this is the job for cinder that does that: cinder-grenade-mn-sub-volbak (and friends) | 17:21 |
| dansmith | frickler: I'm not sure I follow.. I'm saying nova already tests/proves that slurp-to-slurp works live.. do you mean it's not useful if it's *only* nova? | 17:23 |
| frickler | dansmith: I'm saying that it is not useful if it is not guaranteed by the definition of SLURP | 17:24 |
| dansmith | well, FWIW, our customers are all basically on FFUs (i.e. non-live) upgrades, so SLURP brings a halving of the steps required to do that | 17:25 |
| dansmith | but like I said, if the majority of the projects can do live upgrades, we definitely should make that be the SLURP requirement/expectation | 17:26 |
| dansmith | nova already does it voluntarily to make sure we *can* do it at any moment | 17:26 |
| JayF | dansmith: Ironic does rolling upgrades, we have all the flags in place, I think nearly-zero people actually did it in practice fwiw | 17:29 |
| JayF | dansmith: but we do maintain the infrastructure required for such things | 17:30 |
| JayF | (and usually well across 2 6 month release boundries) | 17:30 |
| dansmith | JayF: ack, I think it's actually not that common in practice even for nova, mostly because people are never staying current (with a few exceptions) | 17:31 |
| dansmith | so by the time they're doing it, it's multiple upgrades and it's less impactful to take a 30 minute outage than four 15 minute ones | 17:32 |
| dansmith | or four 15-minute service-is-kinda-degraded periods I mean | 17:32 |
| dansmith | JayF: to be clear, you have the flags and docs in place, but do you actually test that way? | 17:34 |
| JayF | There is no testing, we have the flags in place, the docs in place, and we ensure we keep the metadata about RPC versions in place | 17:35 |
| JayF | So like I said, we maintain the infrastructure required but nearly zero people (including us, in ci) do it in practice aiui | 17:35 |
| dansmith | ah, okay | 17:35 |
| fungi | so it sounds like the upshot is that slurp upgrades are intended to be no more (or less) impactful as not-slurp upgrades, right? and that if something is guaranteed in not-slurp upgrades we would guarantee that in slurp upgrades too? | 17:43 |
| dansmith | well, I think the assumption is/was that most projects are not actually providing (not testing means not providing) live upgrades, so SLURP upgrades are effectively the same for those projects, but with longer timelines or fewer lillypads for FFUers | 17:46 |
| dansmith | I kinda expect that most deployments are *not* doing fully live upgrades and thus most people get the benefit of fewer and less frequent steps | 17:46 |
| dansmith | if that's wrong and/or the majority of projects *can* test and guarantee live upgrades across SLURPs, then I definitely think that's worth adding to the definition of slurp | 17:47 |
| fungi | right, what i was trying to get at is we said live slurp upgrades won't work because we similarly don't expect not-slurp live upgrades to be seamless either? | 17:47 |
| dansmith | right | 17:48 |
| fungi | (depending on which projects you're upgrading amyway) | 17:48 |
| dansmith | the other way to look at that is: if you're doing live upgrades every six months now, then slurp probably isn't interesting to you anyway | 17:50 |
| dansmith | it's more for the people that were already doing FFUs every 12-18 months and reducing the steps they needed to do to make that happen.. adding live to that would certainly be nice, but... | 17:51 |
| JayF | #startmeeting tc | 18:00 |
| opendevmeet | Meeting started Tue Oct 31 18:00:19 2023 UTC and is due to finish in 60 minutes. The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot. | 18:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 18:00 |
| opendevmeet | The meeting name has been set to 'tc' | 18:00 |
| JayF | #topic Roll Call | 18:00 |
| dansmith | o/ | 18:00 |
| knikolla | o/ | 18:00 |
| JayF | Welcome to the weekly meeting of the OpenStack Technical Committee. A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct. | 18:00 |
| gmann | o/ | 18:00 |
| JayF | Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee. | 18:00 |
| JayF | #info Two noted absenses: spotz[m] and slaweq | 18:00 |
| rosmaita | o/ | 18:01 |
| frickler | \o | 18:01 |
| JayF | Alright, I think that's everyone accounted for then, moving on to the agenda. | 18:01 |
| JayF | #topic Follow up on tracked action items | 18:01 |
| JayF | there is one; it's mine; and I've not done it yet due to running out of hours in the day | 18:01 |
| JayF | #action JayF Before next video meeting, write up a short document on pros/cons of moving TC video meetings to jitsi-meet. | 18:02 |
| JayF | I wanted to have this done by this meeting, but between PTG and other responsibilities I did not have time. I'll try to have something written up in an etherpad by the end of the calendar-week. | 18:02 |
| JayF | #topic Gate Health Check | 18:02 |
| JayF | Anything on the gate? | 18:02 |
| dansmith | so, | 18:02 |
| dansmith | things are not terrible, but not great either, IMHO | 18:03 |
| dansmith | this morning I debugged an issue which ended up being a *glance* OOM, where it swole up to >2G while uploading a tiny snapshot to swift | 18:03 |
| dansmith | which of course took out a bunch of other tests | 18:03 |
| dansmith | there have also been some unstable devstack or node setup type things I've seen, but haven't chased to any conclusions | 18:04 |
| clarkb | we were temporarily running without rackspace due to an openstacksdk update that broke nodepool's ability to talk to that cloud. This should be temporarily resolved wtih a more proper fix on its way through merge processes | 18:04 |
| dansmith | but basically I think we're in that same post-big-fix-push where stuff starts to rot out the minute we stop making a concerted effort :/ | 18:04 |
| gmann | a few of failure i also observed but other than that it was ok. got a few of the changes merged in tempest, devstack, and neutron quickly | 18:05 |
| JayF | Is there any specific action we need to take or encourage other than constant vigilance? | 18:05 |
| dansmith | I've been rechecking an upgrade fix series in nova for 24 hours already.. merged one piece, but 2/3 still waiting for rechecks | 18:06 |
| gmann | dansmith: was oom in import job or other glance job? | 18:06 |
| dansmith | gmann: non-import | 18:06 |
| gmann | k | 18:06 |
| dansmith | we also did merge the devstack change to use cached tokens, which is nice and reduced some of the keystone calls by a lot | 18:06 |
| gmann | ++ | 18:07 |
| dansmith | still way too many of course, but it means we hit keystone less hard in the setup phase, which is good | 18:07 |
| rosmaita | nice | 18:07 |
| dansmith | basically client-side optimization which helps overall, but doesn't solve the actual problem | 18:07 |
| dansmith | anyway, nothing else specific to call out | 18:08 |
| JayF | Thanks for staying on top of that. | 18:09 |
| JayF | #topic Leaderless Projects | 18:09 |
| JayF | gmann: is there anythign worth chatting about here that is new since PTG? | 18:09 |
| gmann | nothing else from what we discussed on Thursady/friday | 18:09 |
| gmann | I will work on the action we discussed in PTG | 18:09 |
| JayF | Ack; please tc-members review the related PRs from the etherpad | 18:09 |
| JayF | #link https://etherpad.opendev.org/p/2024.1-leaderless | 18:10 |
| gmann | yeah | 18:10 |
| JayF | #topic PTG Follow-ups | 18:10 |
| JayF | #link https://etherpad.opendev.org/p/tc-ptg-october-2023 | 18:10 |
| JayF | #link https://etherpad.opendev.org/p/tc-leaders-interaction-2024-1 | 18:10 |
| JayF | #link https://etherpad.opendev.org/p/oct2023-ptg-os-dbperf-crossproject | 18:10 |
| JayF | I have not had time to significantly sort through TC PTG notes and break out any specific action items | 18:10 |
| JayF | I wanted to have this agenda item in place for us to note the etherpads and mark any actions that may be seen as urgent | 18:11 |
| JayF | otherwise my goal will be to have actions broken out into the TC tracker in the next 2 weeks, including an email summary of TC PTG to the list this week. | 18:11 |
| JayF | Looks like no interest in more PTG talk, just days after the PTG. I'm stunned! :D | 18:14 |
| JayF | #topic Open Discussion and Reviews | 18:14 |
| JayF | Are there any topics for open discussion? Most of the reviews we discussed at the PTG, I will not re-enumerate them here -- please review things open in governance repo. | 18:15 |
| frickler | one topic: should yoga go into EM instead of Unmaintained? | 18:15 |
| JayF | I followed the discussion earlier, and I mainly wonder if it does any harm to delay action on yoga until documentation is completed? I know this has been pushed a couple times already in general (finalizing this documentation), so I'm not certain we should, but it's one possible option. | 18:17 |
| dansmith | super unhelpful, but I don't have a strong opinion | 18:17 |
| gmann | it should not matter if we delay yoga to EM or now. question stay same | 18:17 |
| gmann | yoga and also for zed in future | 18:18 |
| rosmaita | well, the current EM branches will have to be moved to Unmaintained/deleted at some point, so might as well have yoga join them until we have a clear strategy | 18:18 |
| JayF | I don't have a strong opinion other than that it seems silly to ask the releases team to essentially do something 2x (automation -> EM, automation -> new UM process) | 18:18 |
| JayF | rosmaita: that's an extremely good point that cancels out mine entirely | 18:18 |
| rosmaita | :) | 18:19 |
| gmann | there is resolution passed to have no EM and move to Unmaintained so I think we should follow that even implementation of it in doc/policy is dleayed | 18:19 |
| JayF | I also think that from the perspective of what tonyb brought up at the PTG, about how/where decisions are made, that doing the thing in documentation until something else is in documentation seems to be the most consistent thing we can do. | 18:19 |
| JayF | hmm again a good point, same thought but different outcome | 18:19 |
| gmann | that does not stop us to discuss here right ? :) | 18:20 |
| JayF | No it doesn't, but it's extra awkward because we're in a middle ground | 18:20 |
| gmann | I think let's review p-t-g as soon as we can and if we can merge it before yoga EM time comes | 18:21 |
| JayF | where our written stuff says two things; our governance docs which are official say one, and the release docs which users/operators look to say another | 18:21 |
| JayF | Can someone ensure we get an ML thread started about this? frickler since you brought it up, would you be willing? | 18:21 |
| frickler | moving to Unmaintained is an optional process that needs to be documented and implemented, according to the resolution the default action would be to move yoga to EOL | 18:22 |
| gmann | yeah that we need to fix that is why I am saying let's do that as first step and as soon as we can | 18:22 |
| JayF | gmann: can you put a link to that change in here and put it in the minutes if you have it at hand? | 18:22 |
| gmann | #link https://review.opendev.org/c/openstack/project-team-guide/+/897505 | 18:23 |
| gmann | here ^^ | 18:23 |
| frickler | yoga EM time is in two days according to the tentative schedule | 18:23 |
| frickler | so not much time for an ML thread | 18:23 |
| JayF | Is there anyone who would object to punting that a minimum of one week, so we can have an ML thread and an agenda item for next week about how to adjudicate this, and hopefully also potentially merge the p-t-g doc in the meantime to make the entire situation less sticky? | 18:24 |
| gmann | frickler: but we did not say about yoga or anything, resolution mentioned last 3 active EM to automatically move to Unmaintained and rest all to EOL https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branches.html#transition | 18:24 |
| frickler | also I don't like to send any mail to the list | 18:24 |
| gmann | if yoga moving to EM before we implement the policy I think it make yoga to fall into those last three EM | 18:24 |
| frickler | gmann: the resolution says: no more EM, to me that also says: no more projects transition to EM. right now. | 18:24 |
| JayF | gmann's interpretation more closely matches mine; but I also care more about ensuring we do the sensible thing than I do about following the exact letter of the rule down to the minute of when yoga's next phase should be acted upon. | 18:25 |
| fungi | keep in mind that the release team has taken em/eol dates on the schedule as "on or sometime as soon after this date as is convenient" | 18:25 |
| gmann | yeah but policy are not implemented yet so... do not know I think it applies when we implied it? | 18:26 |
| fungi | my point being, if there's a delay of a week or two in switching branches to em or eol, that's not unusual based on past transitions anyway | 18:26 |
| JayF | I propose that the TC request the releases team delay action on the yoga stable branches at least one week, and that in that week we attempt to get the p-t-g docs landed and start asynchronous communications about the implementation of unmaintained branch policy | 18:26 |
| rosmaita | that sounds reasonable to me | 18:27 |
| gmann | +1. if we are doing that let's delay till we merge p-t-g change just to avoid postponing it again | 18:28 |
| JayF | gmann: I do not want to say it that way because I think we've shown without a time pressure we may delay too much. I prefer feeling the pressure of needing to land that in a week, then dealing with it next week if it does not. | 18:28 |
| gmann | sure, if we deadline that change to merge it is great. agree | 18:29 |
| JayF | I'm trying to check to ensure we have consensus, frickler does not seem on board with this, and we are down 2 members for this meeting, that makes 3-1 essentially, with 7 people attending, I don't feel like this represents TC consensus yet. | 18:29 |
| frickler | ack. there's another related question: if we tell people now or in a week "speak up if you want to prevent stable/yoga for project xyz from going eol", how much time do we give for responses | 18:29 |
| gmann | I think knikolla fixed my comment there so will re-review today | 18:29 |
| frickler | I'm +1 on the delay | 18:29 |
| gmann | cool | 18:29 |
| JayF | frickler: thank you for being explicit; didn't want to stomp on your concern from earlier :) | 18:29 |
| dansmith | delay is fine with me if it means we can move on | 18:29 |
| JayF | frickler: I was thinking about this that a 1 month period seems reasonable, we know from election nominations that 2 weeks can often miss people, and EOL'ing a branch is much more expensive to undo, technologically, than a late candidacy. | 18:30 |
| JayF | This is a detail we likely need documented in the p-t-g. I'll ensure that gets in my PR feedback. | 18:30 |
| JayF | (and IMO is best argued in that venue) | 18:31 |
| frickler | I already mentioned that there, just wanted to give it more visibility | 18:31 |
| JayF | Are there other comments on the topic in open discussion of stable/yoga branch adjudication, or generally on unmaintained branch support? | 18:31 |
| JayF | #info TC will request releases team delay at least 1 week in adjudication of stable/yoga branch to allow TC to complete documentation of unmaintained branch implementation. | 18:32 |
| JayF | Anything else, generally, for open discussion? | 18:32 |
| frickler | there is a reqs patch that may profit from tc feedback | 18:33 |
| JayF | Can you link it into the minutes? | 18:34 |
| frickler | https://review.opendev.org/c/openstack/requirements/+/898699 | 18:34 |
| frickler | #link https://review.opendev.org/c/openstack/requirements/+/898699 | 18:34 |
| JayF | Alright, that looks interesting for sure and has been added to my review queue. | 18:35 |
| JayF | Is there anything further to comment about that, or other topics for open discussion? | 18:35 |
| JayF | Last call for TC meeting? | 18:36 |
| JayF | #endmeeting | 18:37 |
| opendevmeet | Meeting ended Tue Oct 31 18:37:32 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 18:37 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/tc/2023/tc.2023-10-31-18.00.html | 18:37 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/tc/2023/tc.2023-10-31-18.00.txt | 18:37 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/tc/2023/tc.2023-10-31-18.00.log.html | 18:37 |
| JayF | Thanks for attending all o/ | 18:37 |
| frickler | o/ | 18:37 |
| *** elodilles is now known as elodilles_pto | 20:29 | |
| -opendevstatus- NOTICE: Gerrit on review.opendev.org will be restarted to pick up a configuration change required as part of Gerrit 3.8 upgrade preparations. | 22:02 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!