| *** tetsuro has joined #openstack-tc | 00:22 | |
| *** ricolin has quit IRC | 02:08 | |
| *** tetsuro has quit IRC | 03:02 | |
| *** tetsuro has joined #openstack-tc | 03:36 | |
| *** tetsuro has quit IRC | 03:44 | |
| *** tetsuro has joined #openstack-tc | 03:51 | |
| *** tetsuro has quit IRC | 03:52 | |
| *** tetsuro has joined #openstack-tc | 04:02 | |
| *** evrardjp has quit IRC | 04:36 | |
| *** evrardjp has joined #openstack-tc | 04:36 | |
| *** KeithMnemonic has quit IRC | 05:14 | |
| *** slaweq has joined #openstack-tc | 07:08 | |
| *** e0ne has joined #openstack-tc | 07:29 | |
| *** tosky has joined #openstack-tc | 07:31 | |
| *** rpittau|afk is now known as rpittau | 07:33 | |
| *** ricolin has joined #openstack-tc | 07:52 | |
| *** witek_ is now known as witek | 09:05 | |
| *** rpittau is now known as rpittau|bbl | 10:18 | |
| *** tetsuro has quit IRC | 10:29 | |
| *** lpetrut has joined #openstack-tc | 11:25 | |
| *** dklyle has quit IRC | 11:43 | |
| *** tkajinam has quit IRC | 12:01 | |
| *** rpittau|bbl is now known as rpittau | 12:15 | |
| *** ijolliffe has joined #openstack-tc | 12:20 | |
| *** lpetrut has quit IRC | 12:36 | |
| *** lpetrut has joined #openstack-tc | 12:48 | |
| cloudnull | o/ | 12:55 |
|---|---|---|
| *** KeithMnemonic has joined #openstack-tc | 13:43 | |
| ttx | o/ | 13:46 |
| *** lpetrut has quit IRC | 13:46 | |
| njohnston | o/ | 13:51 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/governance master: Appoint Javier Peña as rpm-packaging PTL https://review.opendev.org/728088 | 13:54 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/governance master: Appoint Javier Peña as rpm-packaging PTL https://review.opendev.org/728088 | 13:55 |
| evrardjp | woopsie | 13:55 |
| gmann | o/ | 14:00 |
| jungleboyj | o/ | 14:03 |
| knikolla | o/ | 14:09 |
| *** dklyle has joined #openstack-tc | 14:26 | |
| *** evrardjp has quit IRC | 14:55 | |
| *** evrardjp has joined #openstack-tc | 14:56 | |
| *** diablo_rojo has joined #openstack-tc | 14:59 | |
| diablo_rojo | o/ | 15:00 |
| mnaser | tc-members: bonjour | 15:00 |
| jungleboyj | Happy Thursday. | 15:00 |
| njohnston | hello! | 15:00 |
| gmann | o/ | 15:00 |
| ricolin | o/ | 15:01 |
| knikolla | mirëdita o/ | 15:01 |
| mnaser | How’s everyone doing | 15:01 |
| mnaser | Wild idea. How about we get on Jitsi? | 15:01 |
| jungleboyj | Good. | 15:01 |
| mnaser | The opendev hosted one | 15:01 |
| ricolin | Doing great:) | 15:01 |
| jungleboyj | I will need to drop in 30 if we do that as I have another meeting conflict. | 15:02 |
| diablo_rojo | I won't have my video on as I literally just got out of bed.. | 15:02 |
| mnaser | Or not everyone is enthusiastic about talking over audio? | 15:02 |
| evrardjp | I like the idea, but I prefer if it was planned: here I would lose 5 minutes into making sure my pulseaudio and my webcam works | 15:03 |
| njohnston | +1 to planned | 15:03 |
| jungleboyj | :-) | 15:03 |
| mnaser | Oh I aimed for audio only | 15:03 |
| evrardjp | tbh I trust more my webcam sharing than pulseaudio | 15:03 |
| jungleboyj | Yeah, sleep was not good to my hair last night. | 15:03 |
| mnaser | Y’all don’t want to see this mess. But, fair! | 15:03 |
| evrardjp | jungleboyj: lol | 15:03 |
| njohnston | I've never used Jitsi; I assume it works pretty much like Discord but I don't have the first clue | 15:03 |
| evrardjp | mine either | 15:03 |
| knikolla | i like the idea | 15:03 |
| gmann | i prefer audio but it might not be good for everyone from all countries ? | 15:03 |
| ricolin | I'm fine either way:) | 15:03 |
| evrardjp | njohnston: I think it's just webrtc behind the scenes? | 15:04 |
| gmann | typing is very energy consuming for me:) | 15:04 |
| evrardjp | arguing is even more time consuming :) | 15:04 |
| mnaser | Can I get a quick -1 vs +1 vote? | 15:04 |
| knikolla | +1 | 15:04 |
| evrardjp | 0! | 15:04 |
| mnaser | For audio only | 15:04 |
| jungleboyj | +1 | 15:04 |
| gmann | you mean for TC ? for meeting only ? office hour? | 15:04 |
| jungleboyj | If we can be done in 30 min. :-) | 15:05 |
| gmann | or for today ? | 15:05 |
| mnaser | For today | 15:05 |
| evrardjp | I am surprised nobody asked if my answer was factorial 0, or 0 with an exclamation. | 15:05 |
| njohnston | if ou mean for today then: -1 for voting on community goals because I think that should be recorded in the usual place, but +1 for after that | 15:05 |
| evrardjp | agreed with njohnston | 15:05 |
| *** slaweq has quit IRC | 15:06 | |
| mnaser | Ok fair. Let’s see if Jitsi can record in the future | 15:06 |
| jungleboyj | njohnston: ++ That makes sense. | 15:06 |
| gmann | +1 | 15:06 |
| gmann | so start here then ? | 15:07 |
| mnaser | yep! | 15:07 |
| ricolin | +1 with the record point | 15:07 |
| mnaser | lets get started. so given we spoke about wanting a record of all of this | 15:07 |
| knikolla | yep | 15:07 |
| mnaser | #startmeeting tc | 15:07 |
| openstack | Meeting started Thu May 14 15:07:53 2020 UTC and is due to finish in 60 minutes. The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:07 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:07 |
| *** openstack changes topic to " (Meeting topic: tc)" | 15:07 | |
| jungleboyj | :-) | 15:07 |
| mnaser | let's keep that open just to record | 15:07 |
| openstack | The meeting name has been set to 'tc' | 15:07 |
| njohnston | +1 thanks | 15:08 |
| mnaser | so afaik all proposed goals have landed | 15:08 |
| gmann | yeah, we have two proposed goal merged - https://governance.openstack.org/tc/goals/proposed/index.html | 15:08 |
| gmann | #link https://governance.openstack.org/tc/goals/proposed/index.html | 15:08 |
| diablo_rojo | The log here are definitely a good idea. We should send anything we decide to the ML for people not in attendance too though.. and give it 24 hours to really finally decide since not everyone could make it and this is kinda impromptu? | 15:08 |
| gmann | one goal zuulv3 migration is already selected fr V cycle, we need to select one more | 15:08 |
| mnaser | diablo_rojo: i think what we should do is propose a change with those goals we select, and then send out that review which will be under formal-vote rules too | 15:09 |
| diablo_rojo | mnaser, not quite, there are still a handful of projects that didn't finish their docs in time | 15:09 |
| mnaser | so technically we have no choice but waiting for 7 days to land it anyways, which gives us indirectly community approval? | 15:09 |
| diablo_rojo | openstackansible being one of them ;) | 15:09 |
| mnaser | diablo_rojo: the patches are up there i promise D: | 15:09 |
| gmann | njohnston: any idea if we can get ralonsoh here to know his opinion on single or multi cycle goal for rootwrap ? | 15:09 |
| diablo_rojo | mnaser, oh okay I mustve missed them in last audit | 15:10 |
| mnaser | diablo_rojo: they came up in the past few days | 15:10 |
| *** ralonsoh has joined #openstack-tc | 15:10 | |
| ralonsoh | hi | 15:10 |
| gmann | rootwrap goal is much user benefits and mock goal is easy and doable. | 15:10 |
| gmann | ralonsoh: hi thanks for joining | 15:10 |
| mnaser | FWIW, on this topic, njohnston proposed the following change wrt goal proposal, so something to think about -- https://review.opendev.org/#/c/724142/ | 15:10 |
| diablo_rojo | mnaser, ah alright, cool. Thank you :) I think we are still missing a couple projects though. So its close, but not completed. | 15:10 |
| jungleboyj | Yeah, we had talked about rootwrap and mock before and those both seemed good. | 15:11 |
| *** slaweq has joined #openstack-tc | 15:11 | |
| gmann | ralonsoh: we were discussing last time also and today. will rootwrap can be single cycle goal or need multi cycle ? | 15:11 |
| mnaser | fwiw, there's nothing stopping us from having 3 goals in total. | 15:11 |
| ralonsoh | multi cycle | 15:11 |
| jungleboyj | Most likely multi | 15:11 |
| ralonsoh | even for one single project, like neutron, we'll need more time | 15:11 |
| ricolin | mock goal isn't that hard but require 10+ patches in single project to fix all | 15:12 |
| diablo_rojo | Neither one of those is particularly user facing (which I know is a thing we've considered in the past) but I think both are good options. | 15:12 |
| mnaser | ralonsoh: i think the good thing is that many projects have already started using privsep though, so the infrastructure is there | 15:12 |
| ralonsoh | mnaser, exactly | 15:12 |
| ralonsoh | and that makes migration process easier | 15:12 |
| gmann | true | 15:12 |
| mnaser | something to note about privsep is that it's possible it may be a lot less of a case where we don't control our own destiny | 15:12 |
| mnaser | for example, if you migrate something to privsep, and use a library instead of a bunch of system shell outs | 15:13 |
| mnaser | and then that library is missing a feature or something, it may block us from progressing forward | 15:13 |
| ralonsoh | you can still use a CLI command but with privsep | 15:13 |
| mnaser | ralonsoh: right, but to an extent that might not be _that_ much more different than rootwrap, but maybe i may be missing some points | 15:14 |
| ralonsoh | exactly | 15:14 |
| mnaser | so it wouldn't be security hardening other than renaming to use something else | 15:14 |
| jungleboyj | I believe, from what I know, that even using privsep with a CLI is better than rootwrap. I could be wrong though. | 15:15 |
| ralonsoh | well, the security in rootwrap is just because you parse the command | 15:15 |
| njohnston | I could imagine an exception list for issues such as that as they come up, much as we have an exception list for projects that need to stay on py27 | 15:15 |
| ralonsoh | with privsep you are forcing the linux capatibilities to use | 15:15 |
| ralonsoh | that's an improvement | 15:15 |
| mnaser | actually that reminds me of a post that mikal wrote a while back | 15:15 |
| mnaser | https://www.madebymikal.com/how-to-make-a-privileged-call-with-oslo-privsep/ | 15:16 |
| gmann | yeah and new way present for all projects is at least good | 15:16 |
| mnaser | maybe we can reference that somewhere | 15:16 |
| ralonsoh | mnaser, I'll take a look at this article | 15:16 |
| diablo_rojo | Woudl be a good thing to include in the goal proposal | 15:17 |
| fungi | yes, directly translating rootwrap rules to privsep rules doesn't appreciably improve security | 15:17 |
| mnaser | dhellman seems to have suggested a while back to add this to docs https://twitter.com/mikal/status/993672440470372352 :) | 15:17 |
| mnaser | i want to look at nova nd see how much rootwrap usage they have | 15:18 |
| fungi | but at least if the rules are using privsep, that's a step toward more easily tightening them up | 15:18 |
| gmann | +1 oslo have these info will be helpful | 15:18 |
| jungleboyj | Yeah. | 15:18 |
| clarkb | privsep is also a performance benefit too iirc because it isn't forking a new python process each call? | 15:19 |
| mnaser | clarkb: technically we had rootwrap-daemon to do the same | 15:19 |
| ralonsoh | yes, we spawn a daemon the first time it's called | 15:19 |
| ralonsoh | and then we use a socket to this daemon | 15:19 |
| clarkb | ah I thought all that work went into privsep, didn't realize rootwrap got similar | 15:19 |
| mnaser | but rootwrap daemon couldn't call privilged functions, only shell outto things | 15:19 |
| mnaser | so from a performance benefit of "can i use pyroute2 in privsep and use internal apis vs fork out "ip link set..."" | 15:20 |
| mnaser | it benefits a aton | 15:20 |
| njohnston | Given that smcginnis is committed to making the mock change - and it's already well underway - I think that it makes more sense to get started with the privsep goal. This seems especially true for projects that have a very small contributor base and may need a long time to work on changing patterns from a simple CLI call to a library. | 15:20 |
| mnaser | https://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters -- does this tell me rootwrap is fully out of nova btw? | 15:20 |
| jungleboyj | njohnston: ++ | 15:20 |
| mnaser | https://github.com/openstack/nova/commit/909d0de68edcb232c0d0ca28b755806f7f3780bc | 15:20 |
| ttx | privsep is faster that the daemon, as it does not spawn a new process for the command called | 15:20 |
| mnaser | this patch says nova is alread ydone | 15:20 |
| ttx | so rootwrap (python+command) < rootwarp-daemon (command) < privsep (inline python) | 15:21 |
| mnaser | off the top of my head, nova and neutron are the two big targets because they need to do things as root. other projects probably should run in user space anyways? | 15:21 |
| gmann | njohnston: +1 | 15:22 |
| mnaser | maybe cinder too for operations that involve mounting things and doing copies | 15:22 |
| ttx | yes nova is fully privsepped. Just needs the phase 2 work (properly rewrite commands into python) | 15:22 |
| gmann | nova is done yea, but neutron too is completely migrated ? | 15:22 |
| mnaser | https://github.com/openstack/neutron/tree/master/etc/neutron/rootwrap.d -- neutron has not even moved to privsep | 15:22 |
| njohnston | mnaser: I wonder if nova just needs to clean some docs up: http://codesearch.openstack.org/?q=rootwrap&i=nope&files=&repos=openstack/nova | 15:22 |
| jungleboyj | Cinder is impacted as well. We are part way through the migration though. | 15:22 |
| mnaser | sorry, let me correct | 15:22 |
| mnaser | neutron still has rootwrap usage | 15:23 |
| ralonsoh | gmann, no, not neutron | 15:23 |
| mnaser | ok, how about we split this goal into 2? goal #1 -- move everything into privsep but not necessarily have to worry about reimplementing it with libraries | 15:23 |
| mnaser | goal #2 -- remove all usage of exec() unless explicitly necessary | 15:23 |
| mnaser | and #1 should be easy and if nova did it then anyone can do it :P | 15:23 |
| ttx | os-brick is also privsepped, but with execs() | 15:23 |
| mnaser | (and we have an example of a project which did it too) | 15:23 |
| diablo_rojo | That seems like a cleaner approach to making it all one goal and making that span two releases. | 15:24 |
| ttx | mnaser: yeah that is what I was thinking | 15:24 |
| jungleboyj | mnaser: ++ | 15:24 |
| jungleboyj | I like that approach. | 15:24 |
| diablo_rojo | Only to find out people wait to do part 1 until the end of the second release.. | 15:24 |
| ttx | You can do a one-cycle goal of moving stuff to privsep that does not involve rewriting every command | 15:24 |
| mnaser | cause looking at some of the stuff that neutron will have to do will require a lot of time | 15:24 |
| mnaser | lots of dnsmasq interactions and what not | 15:24 |
| ttx | then a goal of getting rid of all bad usage of privsep once you are closer that getting that completed | 15:24 |
| ttx | and in the mean time, you can spend a few cycles getting the large ones closer to phase 2 | 15:25 |
| njohnston | ralonsoh: what do you think about reframing the goal in this way? | 15:25 |
| ralonsoh | njohnston, ok with this | 15:25 |
| mnaser | yep. i like that a lot. it also makes it a lot less intimidating in terms of security and trying to think about how to do it without execs | 15:25 |
| ralonsoh | about neutron, just the 1:1 conversion will take more time | 15:25 |
| ttx | they do not have to be done one right after the other | 15:25 |
| gmann | this also sounds good idea or targeting set of projects in goal phases ? | 15:25 |
| ttx | So if be end of phase 1 we identify some very large amounts of work, maybe we can get it started on specific projects before setting it as a goal | 15:26 |
| ttx | Goals are good for getting everyone to a given level, not so great when amount of work is vastly different across projects | 15:26 |
| mnaser | given the situation we're in, if we split this out to "just convert to privsep and keep execs (unless you want to eliminate them)" + mock + zuul. i think that's a fair "load" at our contributors | 15:26 |
| njohnston | ^^ excellent point ttx | 15:26 |
| mnaser | the main reason being many projects have accomplished these goals already to varying degrees | 15:26 |
| ttx | I think phase 1 in one cycle is totally doable | 15:27 |
| mnaser | afaik mock efforts were already mostly underway in nova, most of nova already have zuul v3 jobs, an privsep already done | 15:27 |
| jungleboyj | mnaser: ++ | 15:27 |
| gmann | mnaser that will be too much for one cycle. mock can go without goal in that case | 15:27 |
| diablo_rojo | Then maybe a gap before phase two to get others with more work ready for it. | 15:27 |
| ttx | phase 2... it's doable in one cycle *if* we get a few cycle to have the largest targets caught up | 15:27 |
| diablo_rojo | But that can be decided exactly later. | 15:27 |
| ttx | diablo_rojo: ++ | 15:27 |
| gmann | zuul migration can be lot of work for many projects | 15:27 |
| mnaser | i dont want to stress our teams but i also feel like some teams literally won't have to do these projects | 15:27 |
| mnaser | like i dont think MOST of our services will be affected by rootwrap | 15:28 |
| ttx | gmann: hopefully not the same as the privsep one :) | 15:28 |
| jungleboyj | ++ | 15:28 |
| ttx | Like privsep phase 1 is basically done for nova | 15:28 |
| mnaser | i.e. i dont think heat ever needed to run things as root ever :) | 15:28 |
| gmann | yeah, and adding mock there might be lot of work for them | 15:28 |
| mnaser | and heat already had zuul v3 jobs | 15:28 |
| ricolin | I think phase1 + zuul +mock indeed sounds like fair load as mnaser mentioned, and we need more discussion on what's detail for phase two IMO | 15:28 |
| gmann | in term of review and all. | 15:28 |
| ricolin | mnaser, yes | 15:28 |
| jungleboyj | Cinder should be able to get phase 1 done pretty easily. | 15:28 |
| mnaser | well perhaps we should investigate the affect projects and we might realize that the overall load per project won't be big | 15:28 |
| mnaser | cause i think that for the most part, projects will end having to do only 1/3 | 15:29 |
| mnaser | (cause the other two are likely already done for example) | 15:29 |
| mnaser | like i think we need to assume that most project will be noop to 1 or 2 out of these 3. if most projects will not be noop to these 3 goals, we need to drop one | 15:30 |
| gmann | "convert to privsep and keep execs (unless you want to eliminate them)" + zuul' can be good try though there are still chance project might miss but we are giving them goals in start of cycle | 15:30 |
| gmann | there are lot of legacy jobs for many projects. In my xenail->bionic migration it was around >50 % if i remember correctly | 15:31 |
| mnaser | right but how many have already moved to zuulv3? and how many even use rootwrap in the first place | 15:31 |
| mnaser | gmann: i think maybe we might have a different perception given that you're much closer in your QA interaction though, so i'll take your word on our zuulv3 progress | 15:31 |
| mnaser | the way i see it is we have 1 goal everyone will most likely have to do (zuulv3) and two that most people will likely have to do 1 out of | 15:32 |
| mnaser | i'm looking at rootwrap usage and it seems minimal as i scroll through projects | 15:32 |
| mnaser | a lot of projects list it in their dependencies for some reason, but no actual usage | 15:32 |
| mnaser | i just wanted to pick a smaller project like zun that interacts with the system and even they use privsep: https://opendev.org/openstack/zun/src/branch/master/etc/zun/rootwrap.d/zun.filters | 15:34 |
| fungi | technically, the privsep phase 1 goal can really be the "get rid of rootwrap" goal, because there might be cases where projects were doing things as root which would be better done in other ways, and don't actually need either rootwrap or privsep | 15:35 |
| gmann | zun is one of most active and up to date one you checked :) | 15:35 |
| mnaser | magnum has no rootwrap usage, heat has no rootwrap usage, watcher has no rootwrap usage | 15:35 |
| mnaser | i'm trying to pick random arbitrary projects, i may suck at naming random ones but most of them have no actual need to use rootwrap with the exception of system interacting services such as nova/cinder/neutron | 15:35 |
| diablo_rojo | Easy goal for them then lol | 15:35 |
| njohnston | mnaser: You have to look for utils.execute with run_as_root as true http://codesearch.openstack.org/?q=utils.execute&i=nope&files=&repos=openstack/zun | 15:35 |
| mnaser | and nova is already done with it | 15:35 |
| jungleboyj | :-) | 15:36 |
| mnaser | njohnston: does that mean they have inaccurate filter file? oh | 15:36 |
| njohnston | oh, sorry, I did not see it was the filter file you linked to | 15:37 |
| mnaser | njohnston: they do use it, it seems: https://opendev.org/openstack/zun/src/branch/master/zun/common/privileged.py | 15:37 |
| mnaser | they just have zun.common as a whole privileged space | 15:37 |
| mnaser | (not the best idea but at least rootwrap ain't htere) | 15:37 |
| mnaser | that's why i feel like adding the phase of rootwrap isn't that much more load, IMHO. | 15:37 |
| mnaser | which really leaves 2 goals, mock and zuulv3 | 15:37 |
| gmann | also if we divide it in two phase, i think doing it in consecutive cylce will be good to get he flow going otherwise it is easy to miss the second phase | 15:38 |
| mnaser | I agree. | 15:38 |
| njohnston | we might want to add to the goal proposal saying 'best practice is to strictly scope your PrivContext space' | 15:38 |
| gmann | and we select the phase2 as the W cycle goal in advance? or preselect | 15:38 |
| * fungi suggests "recommended practice" or "standard practice" over "best practice" | 15:38 | |
| njohnston | fungi: sure, sounds good | 15:39 |
| mnaser | gmann: I don’t have an opinion on that right now but not opposed to that | 15:39 |
| fungi | the term "best practice(s)" always makes me twitch | 15:39 |
| gmann | mnaser: yeah something we can discuss in PTG for W. | 15:40 |
| mnaser | I agree. We can put it there and nothing stops us from changing it later | 15:41 |
| njohnston | manila is an instance where there is a fair - but not overwhelming - number of instances to fix, for example: https://opendev.org/openstack/manila/src/branch/master/etc/manila/rootwrap.d/share.filters http://codesearch.openstack.org/?q=utils.execute&i=nope&files=&repos=openstack/manila | 15:41 |
| mnaser | njohnston: ah yes that’s another system interacting one inmissed | 15:41 |
| mnaser | Do they use zuul v3? | 15:41 |
| mnaser | Like if it’s just one project thatmight be hit harder... | 15:42 |
| gmann | ph one more things in V cycle, migration to Ubuntu 20.04. this also need community level effort | 15:42 |
| mnaser | 8@ driving to the dealership now | 15:42 |
| gmann | though it will be easy after all projects move to zuulv3 | 15:42 |
| mnaser | Oops. Wrong window. | 15:42 |
| gmann | but if we need to do Ubuntu 20.04 migration in parallel that is also some work on projects side | 15:43 |
| ricolin | gmann, yeah, that's something better done after zuulv3 | 15:43 |
| fungi | oh, yep, folks were previously in favor of calling a switch of default job node major version a cycle goal | 15:43 |
| gmann | ricolin: but we have to do it in V cycle and cannot do late during release | 15:43 |
| fungi | and victoria would by our current rules, need to switch from ubuntu 18.04 lts to 20.04 lts | 15:43 |
| gmann | yeah, or can we move it to W cycle so that easy thing to do after zuulv3 goal | 15:44 |
| mnaser | I think the system components are a lot more lax now though | 15:44 |
| ricolin | gmann, what I proposed is to consider make the dependency between two goals if we need to do both of them in V | 15:44 |
| mnaser | It’s not like we switched to systemd | 15:44 |
| gmann | ricolin: +1 but doing dependent goal in single cycle is difficult. | 15:45 |
| fungi | well, we'd need to revise the rules by which we select our test platform if we push the ubuntu focal switch out an additional cycle | 15:45 |
| gmann | humm | 15:45 |
| ricolin | gmann, that's true | 15:45 |
| gmann | in that case, who about keep "zuulv3 + Ubuntu 20.04 migration" for V and rest all to later cycle | 15:46 |
| gmann | s/who/how | 15:46 |
| fungi | https://governance.openstack.org/tc/reference/runtimes/victoria.html already lists "Ubuntu 20.04" | 15:46 |
| njohnston | I am good with zuulv3 + 20.04 for V, rootwrap/privsep for W and X, and mock done informally | 15:47 |
| fungi | it doesn't *have* to be positioned as a cycle goal, but it's the same amount of work either way | 15:47 |
| gmann | yeah, we can either move it to W cycle or select this as second goal. coordination of thus migration with zuulv3 might be difficutl though | 15:47 |
| fungi | hopefully the focal migration won't be that rough, we already have established patterns for splitting xenial and bionic to different branches | 15:48 |
| gmann | fungi: yeah, it need same effort as goal and i am in favor of doing it as goal so that we get enough attention. with zuulv3 it will be more of testing carefully before migration | 15:48 |
| mnaser | My vote is to picking both of those and if teams express concern we can revisit it | 15:48 |
| mnaser | I don’t want us to plan too far out and start working that far back. We can adjust and adapt | 15:49 |
| fungi | also, the focal switch should probably happen asap so teams have all cycle to work out any bugs exposed by it | 15:50 |
| gmann | that is something difficult as it will be dependent on zuulv3 migration | 15:50 |
| gmann | migrating legacy job to focal and then convert them to zuulv3 is duplicate work | 15:51 |
| jungleboyj | mnaser: ++ | 15:51 |
| fungi | gmann: like so we can avoid having to update legacy tools/jobs to support running on focal? | 15:51 |
| gmann | that is my main worry, when we finish zuulv3 say m-3 then we migrate to focal and we will have risk of breaking things at release time | 15:51 |
| fungi | we could tie them together without delaying the default nodeset change | 15:52 |
| gmann | fungi: you mean zuulv3 base job to focal and while migrating legycy jobs to zuulv3 will automatic to focal ? | 15:52 |
| fungi | just say legacy jobs still run on bionic in master, but v3-native jobs use focal in master | 15:52 |
| gmann | yeah, sounds good | 15:53 |
| fungi | so switching to v3 native in master also means switching to run on focal | 15:53 |
| gmann | +1 | 15:53 |
| fungi | and provides added incentive for projects with lingering legacy jobs to switch them out by the end of the cycle (otherwise they're not testing on the current ubuntu lts) | 15:53 |
| njohnston | and if a project is not sure if a job failure is because of the focal switch or. azuulv3 switch they can always temporarily specify a bionic nodeset for that job to troubleshoot | 15:54 |
| gmann | njohnston: yeah, those failure will be less but it can work like you suggested to differentiate the failure | 15:55 |
| njohnston | just to preempt the usual engineer objection to changing two things at once | 15:55 |
| gmann | ok, 5 min remaining. i need to move to nova meeting after that. | 15:55 |
| ricolin | tosky, ^^^ | 15:56 |
| gmann | what is consensus? | 15:56 |
| njohnston | I vote for zuulv3 + ubuntu 20.04 for V | 15:56 |
| ricolin | +1 on making ubuntu 20.04 as part of zuul v3 goal if that's possible:) | 15:57 |
| evrardjp | I second that | 15:57 |
| jungleboyj | I think it makes sense if it is possible. | 15:58 |
| gmann | ricolin: not part of zuulv3 but as separate goal. goal1 - zuulv3 + goal2 ubuntu 20.04 for V | 15:59 |
| ricolin | Okay, then +1 on zuulv3 + ubuntu 20.04 for V | 15:59 |
| gmann | because that need separate coordination and testing more on all projects | 16:00 |
| ricolin | and pre-select mock, privsep for W | 16:00 |
| njohnston | I think we just leave privsep in proposed should be good enough, without overly binding a future TC | 16:01 |
| *** rpittau is now known as rpittau|afk | 16:01 | |
| gmann | yeah, we can add that in etherpad to disucss for W. i mean in PTG etherpad | 16:01 |
| njohnston | I believe mock is happening with or without the goal status | 16:01 |
| njohnston | in V | 16:01 |
| gmann | i also think so | 16:01 |
| ricolin | sure, as we need to figure out what we gonna do for W in PTG anyway:) | 16:02 |
| ricolin | njohnston, yes it is | 16:02 |
| tosky | I concur that it's probably better to port the legacy jobs to non-legacy first, and gain fossa support from the base job automagically | 16:02 |
| tosky | some tuning in the jobs may be needed, but in the worst case people can temporarily pin the nodeset to a bionic one | 16:03 |
| gmann | tosky: yeah, we will not do legacy job to focal | 16:03 |
| ricolin | make sense | 16:03 |
| ricolin | we need a champion for ubuntu 20.04 one | 16:05 |
| ricolin | And a proposal | 16:06 |
| gmann | i can do that. | 16:06 |
| ricolin | gmann, cool:) | 16:08 |
| jungleboyj | Thanks gmann | 16:08 |
| njohnston | thanks gmann! | 16:08 |
| njohnston | do we have a consensus that this is the direction we want to go? | 16:09 |
| njohnston | or do we need to wait for the proposal doc and then rediscuss next week? | 16:09 |
| jungleboyj | It think we have more or less reached consensus, but it also seems that people have wandered off. | 16:10 |
| gmann | waiting for mnaser, he seems away. | 16:10 |
| ricolin | we can do official vote on patch which propose ubuntu 20.04 as v goal under https://governance.openstack.org/tc/goals/selected/victoria/ | 16:10 |
| gmann | ok | 16:11 |
| gmann | we need to end meeting also | 16:11 |
| njohnston | thanks all! | 16:11 |
| ricolin | but yeah, let mnaser do to call:) | 16:11 |
| jungleboyj | Yeah, we should probably propose and vote to make it official. | 16:11 |
| njohnston | #endmeeting | 16:11 |
| *** openstack changes topic to "OpenStack Technical Committee office hours: Tuesdays at 09:00 UTC, Wednesdays at 01:00 UTC, and Thursdays at 15:00 UTC | https://governance.openstack.org/tc/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-tc/" | 16:11 | |
| openstack | Meeting ended Thu May 14 16:11:45 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:11 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tc/2020/tc.2020-05-14-15.07.html | 16:11 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tc/2020/tc.2020-05-14-15.07.txt | 16:11 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tc/2020/tc.2020-05-14-15.07.log.html | 16:11 |
| jungleboyj | Thank you everyone! | 16:11 |
| ricolin | thanks everyone | 16:12 |
| *** witek has quit IRC | 16:31 | |
| *** evrardjp has quit IRC | 16:33 | |
| *** evrardjp has joined #openstack-tc | 16:33 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/governance master: Add goal to migrate CI/CD jobs to Ubuntu Focal https://review.opendev.org/728158 | 17:24 |
| *** diablo_rojo has quit IRC | 18:14 | |
| *** diablo_rojo has joined #openstack-tc | 18:21 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/governance master: Add goal to migrate CI/CD jobs to Ubuntu Focal https://review.opendev.org/728158 | 18:21 |
| gmann | njohnston: ^^ updated | 18:21 |
| *** ralonsoh has quit IRC | 18:21 | |
| gmann | tc-members: this is ready to review https://review.opendev.org/728158 | 18:21 |
| njohnston | Looks great, thanks gmann! | 18:22 |
| smcginnis | Isn't that just a zuul job configuration thing? What is the community-wide action we need that this would be a community goal? | 18:23 |
| njohnston | All the projects need to be aware that breakage may happen and be prepared to deal with it | 18:24 |
| smcginnis | Isn't that just part of life in OpenStack? | 18:24 |
| jungleboyj | :-) | 18:27 |
| njohnston | Yes, to a certain extent that just means it's a day ending in "y". The discussion in the TC meeting went towards it being a community goal in order to ensure projects have extra capacity set aside for greater-than-usual issues, is my read on the situation. | 18:28 |
| fungi | so when we switched from precise to trusty the infra team "just did it" and projects who saw breakage were stuck until they fixed whatever bugs that exposed. when we switched from trusty to xenial the infra team decided to make it possible for projects to do it individually at their own pace, so some projects broke each others integration tests and some projects just didn't get around to switching at | 18:28 |
| fungi | all until the next cycle and had to backport testing changes to stable | 18:29 |
| fungi | after those two experiences and getting users mad at them both ways, the infra team decided to let the tc take care of deciding how and when to switch openstack projects to a new default node distro release | 18:29 |
| fungi | so the tc handled the xenial to bionic switch as a cycle goal | 18:30 |
| smcginnis | When was that? https://governance.openstack.org/tc/goals/selected/index.html | 18:30 |
| fungi | ahh, right, so i think it was handled late because the tc couldn't agree on how to go about it, and then decided after that experience that it *should have* been handled as an actual goal | 18:31 |
| fungi | (the bionic default went in very late in the cycle) | 18:31 |
| fungi | there were e-mail threads, i should be able to find | 18:32 |
| smcginnis | I agree it is something that should be done, and something that the TC should help drive. It just seems like an odd community goal. | 18:33 |
| fungi | heh, also this is an amusing proposal in light of today's discussion: http://lists.openstack.org/pipermail/openstack-dev/2018-April/129576.html | 18:36 |
| fungi | "I'd like to propose we do not add legacy-ubuntu-bionic nodesets into openstack-zuul-jobs. Projects should be working towards moving away from the legacy format..." | 18:37 |
| fungi | so here'e the start of the thread from when the devstack switch finally came (over 6 months later): http://lists.openstack.org/pipermail/openstack-dev/2018-November/136316.html | 18:41 |
| *** ijolliffe has quit IRC | 18:43 | |
| fungi | it continued here across the ml move from -dev to -discuss: http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000168.html | 18:45 |
| fungi | and into the december archive here: http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000610.html | 18:46 |
| fungi | here's the point where we finally gave up saying there would be no legacy jobs on bionic: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/003129.html | 18:51 |
| fungi | continuing in march: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003479.html (neutron specific, but third message also indicates they're one of the first projects to actually do it) | 18:53 |
| fungi | and HERE's the thread where we finally started talking about switching the default node type: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003584.html | 18:54 |
| fungi | note this is nearly a full year after bionic was available for projects to use in our nodepool/zuul infrastructure | 18:54 |
| fungi | and the thread continuing the legacy job migration discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003598.html (non-neutron-specific) | 18:59 |
| fungi | linking to this ad hoc tc meeting where goal-related discussion occurred: http://eavesdrop.openstack.org/meetings/tc_python3/2019/tc_python3.2019-03-07-21.00.log.html | 19:01 |
| fungi | then this thread providing a status report on migration progress noting "we are close to Stein release" http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004113.html | 19:07 |
| clarkb | ya we did it forcefully, no one liked it. We gave people more control and no on did it. So we decided it would be better if the governing body took more control of it | 19:08 |
| clarkb | in the way back when it was less of an issue because ubuntu releases came with at least 18 months of support so we actually tried to update every release | 19:08 |
| clarkb | but then they dropped that to 9 months which wasn't enough for our stable branches | 19:09 |
| gmann | true, and starting the work in start of cycle with testing in front will keep things smooth. | 19:09 |
| clarkb | (and so before the trusty lts switch it was just a constant bumping thing that happened rather than every 2 year jumps) | 19:09 |
| fungi | i think that was before precise, not trusty? | 19:11 |
| fungi | i remember the precise->trusty migration we just said "we're switching the default node type now, deal with it" | 19:12 |
| fungi | (that was not popular, but in retrospect it was better than the alternatives we tried later) | 19:12 |
| clarkb | fungi: ya I think before precise we did the constant updates | 19:13 |
| clarkb | then after precise the support period changed | 19:13 |
| clarkb | so we had to do trusty as our first big leap | 19:13 |
| fungi | yep, we were stable on precise, then we just upgraded everyone to trusty and let the chips fall where they may, and some folks were unhappy. so then for xenial we found a way to let them switch at their own pace, and folk were still unhappy | 19:16 |
| fungi | so for bionic we said, to heck with it, let the openstack tc take care of the logistics | 19:16 |
| fungi | ...and it took more than a year | 19:17 |
| fungi | the other takeaway i get from revisiting these threads is we should absolutely not back away from the resolve for dropping legacy migrated v2 jobs in the focal switch. most of the hassle in the xenial to bionic transition turned out to be trying to get the legacy jobs working on bionic after we had said we weren't going to do that | 19:18 |
| clarkb | ++ | 19:18 |
| gmann | ++, if someone suggest then he/she needs to maintain legacy job and so d-g :) | 19:21 |
| fungi | we did say all these same things when we upgraded from xenial to bionic, so hopefully that taught us a lesson that we should actually follow through this time around | 19:22 |
| *** e0ne has quit IRC | 19:34 | |
| *** slaweq has quit IRC | 19:51 | |
| *** slaweq has joined #openstack-tc | 19:59 | |
| *** diablo_rojo has quit IRC | 20:55 | |
| *** diablo_rojo has joined #openstack-tc | 21:42 | |
| *** slaweq has quit IRC | 22:02 | |
| *** slaweq has joined #openstack-tc | 22:19 | |
| *** slaweq has quit IRC | 22:24 | |
| *** slaweq has joined #openstack-tc | 22:29 | |
| * gouthamr finally catches up with the discussion here wrt manila being one of those projects which has a ton to do wrt zuulv3 and rootwrap-->privsep.. | 22:31 | |
| gouthamr | I confirm that this is True, and we got started (finally) on the zuulv3 conversions! We're halfway there for openstack/manila; and we'll get there soon for python-manilaclient and manila-ui.. There's a ton of cruft that we're working through - but this goal seems absolutely doable to me. | 22:31 |
| gouthamr | However, the rootwrap--->privsep migration might not be - noone i know has looked at it; and we were internally prioritizing two other candidates: getting OSC support completed as well as starting to look at system scoped policies and secure policy defaults. So, unless we have someone who's willing to help with rootwrap-->privsep, we'll be one of those teams that might miss the goal if that were chosen. | 22:31 |
| gouthamr | Also support the bionic--->focal conversion! We'll be happy to get started on that right away, while we're in this drive to get testing and CI revamped (read zuulv3) | 22:31 |
| *** slaweq has quit IRC | 22:33 | |
| *** tkajinam has joined #openstack-tc | 22:58 | |
| *** tosky has quit IRC | 23:05 | |
| *** jamesmcarthur has joined #openstack-tc | 23:29 | |
| *** jamesmcarthur has quit IRC | 23:39 | |
| *** jamesmcarthur has joined #openstack-tc | 23:44 | |
| *** jamesmcarthur has quit IRC | 23:46 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!