Monday, 2018-10-01

*** efried has quit IRC00:38
*** efried has joined #openstack-placement00:38
*** tetsuro has joined #openstack-placement01:44
*** tetsuro has quit IRC04:03
*** rubasov has joined #openstack-placement05:45
*** takashin has quit IRC07:07
*** takashin has joined #openstack-placement07:15
*** helenafm has joined #openstack-placement07:26
*** tssurya has joined #openstack-placement08:11
*** giblet is now known as gibi08:19
*** finucannot is now known as stephenfin08:43
*** ttsiouts has joined #openstack-placement08:55
*** ttsiouts has quit IRC08:56
*** ttsiouts has joined #openstack-placement08:56
*** e0ne has joined #openstack-placement08:59
openstackgerritTheodoros Tsioutsias proposed openstack/nova-specs master: Add PENDING vm state  https://review.openstack.org/55421209:13
*** alex_xu has quit IRC09:38
*** alex_xu has joined #openstack-placement09:40
*** tetsuro has joined #openstack-placement09:52
*** ttsiouts has quit IRC10:28
*** ttsiouts has joined #openstack-placement11:04
openstackgerritTheodoros Tsioutsias proposed openstack/nova-specs master: Enable rebuild for instances in cell0  https://review.openstack.org/55421811:11
openstackgerritTheodoros Tsioutsias proposed openstack/nova-specs master: Add PENDING vm state  https://review.openstack.org/55421211:15
*** tetsuro has quit IRC11:30
*** e0ne has quit IRC12:01
*** e0ne has joined #openstack-placement12:08
*** ttsiouts has quit IRC12:26
*** ttsiouts has joined #openstack-placement12:30
*** jroll has quit IRC12:54
*** jroll has joined #openstack-placement12:55
openstackgerritChris Dent proposed openstack/placement master: DNM: Experimenting with integration gabbits  https://review.openstack.org/60161413:06
openstackgerritChris Dent proposed openstack/placement master: WIP: Add placeload to integration test  https://review.openstack.org/60248413:06
*** ttsiouts has quit IRC13:21
*** ttsiouts has joined #openstack-placement13:27
*** cdent has joined #openstack-placement13:28
cdento/13:28
openstackgerritChris Dent proposed openstack/placement master: WIP: Add placeload to integration test  https://review.openstack.org/60248413:45
efriedn-sch/placement meeting in 7 minutes in #openstack-meeting-alt13:53
*** tetsuro has joined #openstack-placement13:55
*** leakypipes is now known as jaypipes14:15
*** ttsiouts has quit IRC14:28
*** zaneb has joined #openstack-placement14:28
*** ttsiouts has joined #openstack-placement14:28
*** tetsuro has quit IRC14:42
*** takashin has left #openstack-placement15:01
*** ttsiouts has quit IRC15:02
*** mriedem has joined #openstack-placement15:07
openstackgerritChris Dent proposed openstack/placement master: DNM: Experimenting with integration gabbits  https://review.openstack.org/60161415:12
openstackgerritChris Dent proposed openstack/placement master: WIP: Add placeload to integration test  https://review.openstack.org/60248415:12
mriedemcdent: not sure if you noticed yet, but your devstack patch to start extracted placement, which depends on my grenade patch, is failing to start up the placement service on the new side after the upgrade,15:20
mriedemhttp://logs.openstack.org/62/600162/5/check/neutron-grenade/13bde35/logs/screen-placement-api.txt.gz15:20
mriedemi can't figure out why it's actually failing to start though15:21
cdentmriedem: i hadn't noticed yet, no15:21
mriedemit looks like it might not be starting with placement.conf?15:21
cdentI started late today so still getting situated15:21
* cdent looks15:21
cdentmriedem: it's using nova-placement-api as it's wsgi script, not placement-api15:22
cdentit's looking at /etc/nova/placement-uwsgi.ini15:22
cdentwhich is the wrong thing15:22
cdentso presumably part of the upgrade process is to change the system unit file (or just create a new one via lib/placement)15:23
cdentnova-placement-api (from the nova code) continues to look at /etc/nova/nova.conf.15:24
mriedem"new one via lib/placement" after your devstack change?15:25
mriedembecause we can't do that right now - the devstack change depends on the grenade change15:25
cdentthen sed-ing the existing one ought to be doable but a couple of caveats:15:28
cdentactually if you give me a minut to turn on a devstack vm, I can't speak a bit more clearly about this, one sec...15:28
mriedemsure, no rush, i just got online15:29
jaypipescdent: whatcha currently waiting on to remove the WIP on the placeload thingie?15:35
cdentjaypipes: its parent doesn't work quite right yet15:35
jaypipesah, k.15:35
cdenttrying to make it "modern zuul" opened up many works15:36
cdentworms15:36
jaypipesack15:36
jaypipescdent: be grateful. you could instead be having to develop a Screwdriver pipeline for it... :(15:36
cdenthttps://www.youtube.com/watch?v=ZdJ5e70Q8mw15:37
jaypipesheh15:37
jrolljaypipes: the open source version of screwdriver (v4) is pretty awesome, tbh15:47
cdentokay mriedem I'm spooled up now. Basically we've got three separate placements where placement been started on some variation of the "nova-placement" theme:15:52
cdent  /etc/systemd/system/devstack@placement-api.service refers to /etc/nova/placement-uwsgi.ini refers to /var/run/uwsgi/nova-placement-api.socket and /usr/local/bin/nova-placement-api15:55
cdentthe *.socket file is mentioned in /etc/apache2/sites-enabled/nova-placement-api.conf15:55
*** ttsiouts has joined #openstack-placement15:56
*** ttsiouts has quit IRC15:56
*** ttsiouts has joined #openstack-placement15:56
cdentthe bare minimum change to change /et/cnova/placement-uwsgi.ini to replace "nova-placement-api" with "placement-api"15:56
cdentthe rest of it is just names15:57
mriedemcdent: ok can you -1 my grenade change with those details?15:57
cdentyeah, will do, but before I head off there: are we after a bare minimum change here or a more correct change? Or is it cool to iterate to absolute perfection?15:58
*** e0ne has quit IRC15:58
jaypipesjroll: I respectfully disagree :)16:00
mriedemcdent: as in are we ok to merge the grenade change now?16:01
mriedemcdent: it's not really useful at this point since we can't start placement on the new side16:01
*** ttsiouts has quit IRC16:01
cdentmriedem: no, I mean we can get placement to start correctly by doing the bare minimum thing I just described above16:01
mriedemif all i'm lacking is a 1 line sed, then i think we can wait16:01
mriedemoh16:01
cdentbut that leave's some files with bad names16:02
cdent(sigh, I don't want to ever type again)16:02
jaypipeshehe16:02
mriedemi.e. should we recreate from scratch /etc/placement/placement-uwsgi.ini16:02
cdent /etc/nova/*uwsgi* will still, but it's in the "wrong" place16:02
cdentwhat I'm wondering is: do the bare min, get things to merge, then tune it up with more robust file fixes16:03
mriedemi think the grenade change should do whatever we expect/want real deployment tools to do for the placement extraction upgrade16:03
mriedemso we can just point people at the single grenade patch16:03
* cdent thinks on that16:03
mriedemthis is already going to be confusing for a bunch of people, so i don't want to add to that confusion by saying, "here is a series of changes to make extracted placement upgrade work, you figure out what you need"16:04
mriedemgoing back to your question, since we said we shouldn't just copy/rename nova.conf to placement.conf in grenade, i suppose we shouldn't just do that for the wsgi config either16:05
mriedemmaybe i need a patch to devstack to provide override paths/names for setting up the placement uwsgi config16:06
mriedemthat would make the grenade patch simpler16:06
cdentmriedem: lemme finish writing up the bits on the review and then we can see what we think16:08
*** helenafm has quit IRC16:12
cdentmriedem: done. It's basically 4 fairly simple actions that need to happen16:19
mriedemok, thanks16:24
cdentmriedem: another option is a time machine, if you got one spare16:26
openstackgerritChris Dent proposed openstack/placement master: DNM: Experimenting with integration gabbits  https://review.openstack.org/60161416:45
openstackgerritChris Dent proposed openstack/placement master: WIP: Add placeload to integration test  https://review.openstack.org/60248416:45
mriedemi don't16:46
*** tssurya has quit IRC16:54
*** mriedem has quit IRC17:04
edleafeThis trip is some serious $$. Around £200/night per room, and with our 19-yo daughter, we can't always fit her in our room, so many nights it's closer to £40017:22
*** e0ne has joined #openstack-placement17:24
* edleafe types in wrong window17:24
jaypipesefried: was I present for the discussion of https://review.openstack.org/#/c/602160/? I must be losing my mind in old age. I have no recollection of that topic whatsoever :(17:30
efriedI don't remember who was in the room. But it was in the room, wasn't like a hallway or dinner discussion or whatever.17:31
jaypipesedleafe: cdent probably can rent a room for less than 200 quid a night :P17:31
jaypipesefried: cyborg room or nova room?17:31
efriednova17:31
edleafejaypipes: sure, but who wants to be out in the boonies?17:32
jaypipesheh17:32
efriedjaypipes: https://etherpad.openstack.org/p/nova-ptg-stein L16617:32
* cdent has the international openstackers hostel, cornwall edition17:33
efriededleafe: you too, fyi ---^17:33
edleafeefried: that was something agreed for Nova, not placement17:37
*** e0ne has quit IRC17:37
efriedeh?17:37
efriedIt was agreed to use a standard trait (i.e. os-traits) to indicate the service owner of a provider (so not just nova, which is the whole point)17:37
efriedno changes in the placement API17:38
efriedunless you're counting os-traits.17:38
efriedwhich I guess counts17:38
edleafeit counts for me17:38
efriedbecause it affects whan you get back from /traits out of the box17:38
efriedanyway, it's pretty unambiguous as it was discussed in the room and as it's written down in the etherpad (by melwitt I think, based on the color).17:39
jaypipesefried: not sure how a standard os-trait would ever indicate >1 bit of information in the trait string? i.e. you'd need to have "standard" traits that look like OWNER_NOVA, OWNER_NEUTRON, OWNER_CYBORG, etc... is that what you want?17:39
efriedyes, precisely, look at the review.17:39
efriedthough to say it's what "I" want isn't really a good way to say it. It's the way *we* (the nova room) agreed to solve the problem at hand.17:40
*** e0ne has joined #openstack-placement17:40
efriedI'm just proposing the patch that will implement what we agreed upon.17:41
*** e0ne has quit IRC17:41
jaypipesefried: sorry, I definitely didn't agree on that and don't remember that one line of the etherpad.17:41
jaypipesefried: if we're already putting a "partition" or "shard" key into the providers table (for multi-nova use case), why not add owner field in there at the same time?17:42
jaypipesefried: my thought is that an owner is a conceptually significant attribute for a provider and as such should be a real attribute on the provider model, not tucked away as a trait string.17:43
edleafeI would prefer a db change to messing with traits like this17:43
jaypipes(I also thought this about the sharing thing, but that ship has sailed)17:43
efriedpartition/shard sounds only vaguely familiar. But if it's an option to start putting explicit keys onto the provider object, shrug, wfm.17:43
jaypipesefried: partition/shard is the consumer type thingie17:43
efriedI think I wasn't paying attention for that discussion :(17:44
jaypipesefried: line 126 in that etherpad...17:44
jaypipesefried: no worries :)17:44
cdentdid we get into the implementation side of shards? We noodled some ideas, but I had the impression we liked the concept but hadn't nailed down the details. Also: I thought that we had shard and consumer type as something different17:45
cdentshard was effecftively supporting N>1 openstack-like-things from one placement17:45
jaypipesyep17:45
cdentbut consumer type "this is an instance"17:45
jaypipescdent: yeah, two different things. one being on the provider, the other being on the consumer, you're right.17:46
cdentpresumably one option (of several) for shard would something like an "auto-aggregate"17:46
cdentbecause conceptually that's what it is17:46
cdent(a group of providers)17:47
jaypipescdent: the provider one was originally referred to as "consumer type" because it was like "which nova is the thing that is consuming" but then we started referring to that as like a "consumer source" or something like that, and then we just referred to it as a provider shard/partition17:47
efriedokay, so again we've got a big and complicated patch (including db migrations, a new schema/microversion for payloads on the /resource_providers/* routes, etc.) which is going to take a while to come to fruition. So if we need to solve the ownership issue in the meantime... can we use CUSTOM_OWNER_* as a temporary measure?17:47
jaypipesefried: and delay the big mess to being a giant data migration.17:47
efriedyeah, fair point.17:48
efriedrock, hard place17:48
jaypipesack17:48
cdentno data migration strictly required. new stuff could do the new way. old way continues to work because traits aren't going to stop working.17:49
cdent(not saying we should do it, just thinking that perhaps the cost isn't _that_ dire)17:49
efriednot sure I see a way to avoid *some* kind of data migration.17:49
efriedIf we have no way at all to indicate ownership in the interim17:49
cdentno, I meant: do CUSTOM_OWNER_*17:49
jaypipesright. it's just about where you encode the tribal knowledge and for how long.17:49
cdentbut also have options for the future17:50
efriedyeah, if we're not indicating ownership explicitly, we have to encode/hardcode an implicit owner.17:50
efried...and only one of those17:50
efriedmeaning we can't actually let cyborg own device providers17:50
efriedor17:50
efriedwe can't put devices in placement until cyborg is ready.17:51
efriedand once the provider owner prop is ready, we're still "migrating" in a sense, it's just a little bit easier because we know any provider with ownership unset belongs to nova.17:51
cdentquestion: why does placement care who an owner is?17:51
cdentis it just the clients that do?17:52
efriedafaict yes17:52
efriedThe advantage to having it as a provider prop is that it's enforced as unary.17:52
efriedCUSTOM_OWNER_CYBORG and CUSTOM_OWNER_NOVA could coexist and placement wouldn't be the wiser.17:53
efriedBut that's all over the place,17:53
efried.17:53
cdentand presumably placement continues to not care17:53
efriedyup.17:53
efriedBut I don't really forsee needing to make a scheduling decision based on ownership17:53
efriedwhich is kind of what traits are limited to atm.17:54
cdentif ownership doesn't drive "placement" why put the info in placement?17:54
efriedrather, this is a marker that would be used by a service to determine whether it owns a provider and thus has the authority to muck with it.17:54
efriedoh, because ^17:54
cdentpresumably a service already know is owns something because it has its own database: this uuid is in me17:54
cdents/is owns/it owns/17:55
efriedI... guess?17:55
efriedDo we not have to be able to survive data loss?17:55
cdentso once again: ownership is a bit like an aggregate: a group of uuids which the client database "owns"17:55
jaypipescdent: it's primarily for quota usage context.17:56
cdentefried: which we do you mean?17:56
jaypipescdent: at least, that's what I understand.17:56
cdentjaypipes: I though consumer (allocation uuid) type was the crux of quota ownership, not resource provider ownership?17:57
jaypipescdent: i.e. cyborg wants to use placement usage records to satisfy quota constraint logic. but if, say, nova owns a device AND cyborg owns a device, there's no way to differentiate it.17:57
jaypipesor that's my recollection of the use case..17:57
jaypipescdent: yes, sorry, I thought you were asking/referring about consumer type?17:57
jaypipessorry I misread you.17:57
cdentno, we're talking about CONSUMER_OWNER_CY*17:58
efriedMy understanding (of provider ownership) was simply that the service gets to twiddle inventory of providers it owns, and not of providers it doesn't own.17:58
cdentsorry s/CONSUMER/CUSTOM/17:58
cdent(it's all melting together)17:58
jaypipeslike a giant melted Crunchie bar.17:58
efriedIf cyborg and nova both think they own a provider, they thrash updating details and the top wobbles off the table.17:58
cdentefried: why would that ever happy?17:58
cdenthappen17:58
cdent(jesus, it just keeps getting worse)17:59
efriedwhich, both thinking they own a provider, or both trying to update the inventory?17:59
cdent(the typing)17:59
efried(find a happen place, cdent)17:59
cdentboth thinking they own a provider (and thus trying to update inventory)17:59
efriedIf both of them have a charter like, "I own devices".18:00
cdentI (perhaps naively) think that both nova and cyborg should only take action on resource providers that they can identify (by looking in _their own databases_) are theirs18:00
cdentnova knows it can writing on a compute node rp because it has a local record18:01
efriedlet's follow that.18:01
cdentnested gets harder, but nested _always_ gets harder18:01
efriedWhen nova owns a device, discovery is done by...?18:01
efriedThe virt driver.18:01
cdentright18:01
efriedSo the virt driver looks on the system and discovers a device18:01
efriedThen it looks in the provider tree to see whether a provider for that device already exists.18:02
efriedAnd what if it does? Was it created by me on the last iteration? Or was it created by cyborg and I should leave it alone?18:02
efriedWhat database am I looking at to figure that out?18:02
jaypipesI agree fully with cdent on the above statements.18:03
efriedRemembering that, without some mind shift, the virt driver method has a pretty restricted view of the universe. In particular, lack of access to the database.18:03
jaypipes(I'm in meeting mode now, so responses will be delayed)18:03
cdentRight, efried, that's the bug. We don't want virt drivers to be dumb and restricted.18:03
cdentthat constraint is causing many of the hurdles we are trying to get over18:04
cdentMind you: I actually have no problem with a trait that says "cyborg is managing this rp"18:05
efriedOkay, okay, but now you're talking about breaking down the virt driver walls, which is philosophically even huger than implementing a key/value primitive in placement.18:05
cdentI came along this path because I was trying to figure out why placement should care and why some kind of control and enforcement is required _in placement_.18:05
efried"required" is a strong word18:06
cdentlet's gets something out of the way so that we don't have to belabor it:18:06
edleafecdent: one problem with a trait that says "cyborg owns this" is that you could also add a trait that says "nova owns this".18:08
efriededleafe: Mentioned above, that problem exists everywhere in the API18:08
efriedbecause we have rp generations, we can avoid races as long as consumers are well-behaved in terms of "claiming ownership".18:09
cdentin my head, except for the need for things like enforcing unary control, there is no fundamental difference (TO ME) between a sequence of bytes that we label a trait and see as "HAS_RAID_5" and a sequence of bytes that we label a key value pair and means "RAID=5", when we assert that placement has no awareness of the meanings.18:09
edleafeefried: yeah, but cdent just said he has no problem with that.18:09
cdentI recognize that this is an unrealistic flaw on my part, but that's the level of abstraction my brain is operating at about 50% of the time when we have these discussions18:09
cdentsomebody says "key value pair" and somebody else says "a string" and I'm all "okay some ordered bytes, whatev"18:09
efriedIt's just that, once again, we find ourselves discussing these universe-bending shifts in entrenched principles, with changes that will take multiple cycles to implement, and have implications for years/ever, when it would be fully effective and immediately usable to add a handful of OWNER_* traits and use them.18:10
cdentI realized this is a problem, and I'm trying to fix it, and it is why I keep asking questions about what people are actually trying to do18:10
cdentbecause it is the _constraints_ that people are trying to enforce that really matter in all this, not so much what they want to enable18:11
cdentwe can able lots and lots of things with what amounts to faceted tags18:11
efriedyes18:11
cdentand if there is agreement amongst the parties involved it will "just work"18:11
efriedeasily18:11
efriedyes18:12
efriedwhereas doing things a different way is (usually) hard18:12
cdentso one of the concerns that some people seem to have unary control, right? Can we make a reasonable case that allows us to dispense with that concern? If we can, what's another concern?18:13
efriedIMO we can dispense with unary control. Generations and well-behaved consumers. If we can't count on that, the whole thing melts.18:13
efrieddispense with unary control as a problem, that is.18:13
cdentI agree. We operate in an "any admin can do anything, so we rely on well behaved consumers for the whole deal"18:14
*** e0ne has joined #openstack-placement18:14
efriedThat's the thing. The only other concern I (still) am able to see is "architectural purity".18:14
efriedI have a really hard time accepting that the tradeoff is worth it nearly as much of the time as we end up going that way.18:14
*** e0ne has quit IRC18:14
efriedand by "that way" I mean half the time we engage in these super awkward workarounds, and half the time we simply go nowhere at all because there *isn't* a workaround.18:15
cdenton unary good behavior: does that mean that clients are expected to get all the traits of a resource provider, look for ONWER_* traits and wave off if found?18:15
efriedcdent: That would be one way.18:15
cdentif so, that will be considered architecturally impure by some, presumably, but I agree that it is a small cosst18:16
efriedIf it freaks us out to be parsing the string, could also just look for all the values in os-traits under the owner. namespace.18:16
efriednot sure if we run into problems with os-traits lib version there.18:16
efriedbut if we manage versions carefully, that shouldn't be a problem.18:16
efriedLike, make sure lower-constraint is always maxed if we add an owner trait or something.18:17
jaypipesI don't know what you mean by "unary control"18:17
efriedjaypipes: Can "owner" have multiple values at once?18:17
cdentbrb18:17
jaypipesefried: oh... no, I don't think I know of a use case for that.18:17
efriedRight, we don't want that to happen18:17
efriedjaypipes: With traits (custom or standard) it can, theoretically, but with well-behaved consumers using generations properly, it's a non-issue.18:18
efriedwith an explicit provider property or (some variants of) key/value primitive, you can enforce that 'owner' only have one value.18:18
jaypipesefried: sorry, you lost me on that... you mean if we have an owner field on consumer, right? and consumers don't have traits decorating them...18:19
efriedWe're talking about owners on providers, still.18:19
jaypipesoh...18:19
*** mriedem has joined #openstack-placement18:20
cdentefried: for sake of mind shaking about and noodling what about [t 1sHq]18:33
purplerbot<cdent> so once again: ownership is a bit like an aggregate: a group of uuids which the client database "owns" [2018-10-01 17:55:42.087707] [n 1sHq]18:33
efriedPredicated on having access to said database18:33
efriedAlso assuming something like hot unplug doesn't leave you with no database entry, but a stale provider that *someone* has to get rid of. Theoretically the same entity removing the db entry would be responsible for removing the provider - but again, if nova (non-virt) manages the db and virt manages the providers (well, non-virt does the actual updates, but without introspecting what the virt driver twiddled) then we h18:37
*** e0ne has joined #openstack-placement18:37
cdentfair point18:38
openstackgerritChris Dent proposed openstack/placement master: DNM: Experimenting with integration gabbits  https://review.openstack.org/60161420:35
openstackgerritChris Dent proposed openstack/placement master: WIP: Add placeload to integration test  https://review.openstack.org/60248420:35
*** mriedem has quit IRC20:42
*** e0ne_ has joined #openstack-placement20:54
*** e0ne has quit IRC20:54
jrollefried: I like the general idea of these config blobs being able to affect scheduling like that21:41
efriedjroll: Yeah, I like the idea as well. What I don't like is that it's going to take like three person-cycles to get it implemented.21:42
jrollefried: well, hopefully we can agree the end goal and make smaller steps to get there21:42
efriedjroll: I, for one, would love to see the first step be, "use what we've got, acking that it's temporary until we get the real thing in place"21:43
efriedbecause that we can get done in three weeks.21:43
cdentwhich idea is the "config blobs" one?21:43
efriedcdent: http://lists.openstack.org/pipermail/openstack-dev/2018-October/135319.html21:44
jrollefried: right, but without the end goal, we won't know if the temporary thing is a step forward on that path, ya know?21:44
cdentthanks efried21:44
jrollcdent: I was referring to jay's suggestion, but eric builds on it21:44
efriedcdent: actually, here is where jaypipes introduced the idea: http://lists.openstack.org/pipermail/openstack-dev/2018-October/135300.html21:44
efriedjroll: I dig it.21:45
efriedI'm guessing this is the spec jaypipes is working on, which he alluded to in the "intended purpose" thread.21:46
jrollindeed21:46
efriedoh, yeah, he says so at the bottom.21:46
cdentso that would be a chance of some kind to the compute api, yes?21:47
cdent(along with other things)21:47
jrollcdent: yes21:48
* cdent goes to bed 21:51
cdentg'night all21:51
*** cdent has quit IRC21:51
*** e0ne_ has quit IRC22:04
*** e0ne has joined #openstack-placement22:05
*** e0ne has quit IRC22:06
*** mriedem has joined #openstack-placement22:36
*** efried has quit IRC22:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!