| *** tetsuro has joined #openstack-placement | 00:12 | |
| *** tetsuro_ has joined #openstack-placement | 00:14 | |
| *** tetsuro has quit IRC | 00:16 | |
| *** ttsiouts has joined #openstack-placement | 00:19 | |
| *** ttsiouts_ has joined #openstack-placement | 00:26 | |
| *** ttsiouts has quit IRC | 00:28 | |
| *** mriedem has quit IRC | 00:30 | |
| *** ttsiouts has joined #openstack-placement | 00:34 | |
| *** ttsiouts_ has quit IRC | 00:36 | |
| *** ttsiouts has quit IRC | 00:41 | |
| *** ttsiouts has joined #openstack-placement | 00:43 | |
| *** tetsuro_ has quit IRC | 00:50 | |
| *** ttsiouts has quit IRC | 00:51 | |
| *** takashin has joined #openstack-placement | 01:43 | |
| *** dklyle has quit IRC | 02:45 | |
| *** dklyle has joined #openstack-placement | 02:46 | |
| *** ttsiouts has joined #openstack-placement | 02:48 | |
| *** takashin has left #openstack-placement | 03:03 | |
| *** ttsiouts has quit IRC | 03:21 | |
| *** ttsiouts has joined #openstack-placement | 05:18 | |
| *** ttsiouts_ has joined #openstack-placement | 05:43 | |
| *** ttsiouts has quit IRC | 05:45 | |
| *** ttsiouts has joined #openstack-placement | 05:47 | |
| *** ttsiouts_ has quit IRC | 05:49 | |
| *** ttsiouts has quit IRC | 05:52 | |
| openstackgerrit | Surya Seetharaman proposed openstack/placement master: [WIP] Spec: Support Consumer Types https://review.opendev.org/654799 | 06:15 |
|---|---|---|
| *** ttsiouts has joined #openstack-placement | 07:22 | |
| *** helenafm has joined #openstack-placement | 08:08 | |
| *** tssurya has joined #openstack-placement | 08:12 | |
| tssurya | cdent: good morning | 08:14 |
| tssurya | let me know if you have some time, I had a couple of questions for the consumer types spec | 08:15 |
| *** ttsiouts_ has joined #openstack-placement | 08:21 | |
| *** ttsiouts has quit IRC | 08:23 | |
| *** e0ne has joined #openstack-placement | 08:39 | |
| *** gibi_off is now known as gibi | 08:47 | |
| *** ttsiouts has joined #openstack-placement | 08:53 | |
| *** ttsiouts_ has quit IRC | 08:56 | |
| sean-k-mooney | o/ | 10:35 |
| sean-k-mooney | edleafe: efried are either of ye about at the moment? | 10:35 |
| sean-k-mooney | quick question for ye. | 10:35 |
| sean-k-mooney | i was talking to bauzas eairlier today and a question came up which bauzas is going to look into later | 10:36 |
| sean-k-mooney | when we have nested resocue providers will placement return muliple allocation candiates for the same host | 10:36 |
| sean-k-mooney | e.g. if i request rescource from both the root rp and a resouce that can be provide by either of two child rps will i get 2 allocation candiates | 10:37 |
| sean-k-mooney | 1 with cpus/ram from the root and the other resocue form child RP1 and a second with the other resoucres form child rp2? | 10:38 |
| sean-k-mooney | this is imporant for several of the numa/bandwith related feature but it also improtant for sharing resouce providers usecase too. | 10:39 |
| sean-k-mooney | in the shareing case its not really about child resouce providers but more this is a member of 2 aggrates for differend shard disk providers. do i get an allcoation candiate for both possible configuration or jsut one of them | 10:40 |
| sean-k-mooney | anyway if anyone know the answer to the above ^ let me know | 10:41 |
| gibi | sean-k-mooney: placement returns multiple candidates per compute if possible. See for example the gabbit case https://github.com/openstack/placement/blob/931a9e124251a0322550ff016ae1ad080cd472f3/placement/tests/functional/gabbits/allocation-candidates.yaml#L602 | 10:47 |
| sean-k-mooney | gibi: ok that is good to know and is there a way to limit the candiate per host without limiting the total set | 10:48 |
| gibi | sean-k-mooney: I don't think so it is possible | 10:48 |
| sean-k-mooney | ok that might be useful as things get more nested | 10:49 |
| sean-k-mooney | if you have a low limit set like cern has then it could be problemaic | 10:49 |
| sean-k-mooney | they limit to like 20 or 50 allcoation candiates but if they get 5 allcotaions per host then thats only 10 hosts instead fo 50 | 10:50 |
| sean-k-mooney | or 4 i guess | 10:50 |
| gibi | sean-k-mooney: they will know how nested their deployment will be so they can addjust the limit accordingly | 10:52 |
| sean-k-mooney | not really | 10:52 |
| sean-k-mooney | they set the limit for performacne reasons | 10:52 |
| sean-k-mooney | so they cant increase it but we could ask placment to only return 1 candiate per host or 3 instead of all possible combinaitons | 10:53 |
| gibi | OK, I see | 10:53 |
| sean-k-mooney | just takign th eband with case if i install 2 4x10G nics in a host that give me 8 pfs | 10:54 |
| gibi | alternatively we can change placement to return one candidate for each possible host before retuns the second candidate for the first host (e.g. orderd the candidates differently) | 10:54 |
| sean-k-mooney | assuming all were on the same physnet and had capastiy left that would result in 8 allocation canidate | 10:54 |
| sean-k-mooney | gibi: perhaps that has other issue too | 10:55 |
| sean-k-mooney | namely numa affinity | 10:55 |
| sean-k-mooney | i think this is a larger topic that is worth discussing with more people | 10:55 |
| gibi | OK so probably one ordering will not be good for all case | 10:55 |
| *** tetsuro has joined #openstack-placement | 10:56 | |
| gibi | sean-k-mooney: agree about discussing it with others | 10:56 |
| sean-k-mooney | ya i think this is just another pair of axis that we need to consider in the wider "richer request syntax" discussion | 10:56 |
| gibi | sean-k-mooney: as an extra problem, nova only considers the first candidate per host | 10:57 |
| sean-k-mooney | gibi: yes today. | 10:57 |
| sean-k-mooney | if we want to do wigheing/filtering on allcoation candiate instead of hosts in the future that woudl change | 10:58 |
| gibi | sean-k-mooney: I agree | 10:58 |
| sean-k-mooney | any way i realised while talking to bauzas that we had assumed that there would be multiple allcoation candiates for the same host but i had never check | 10:59 |
| sean-k-mooney | thanks for pointing out that test | 10:59 |
| sean-k-mooney | im not sure if its also the case for nested | 11:00 |
| sean-k-mooney | its definelty assertign the behavior for local and shareing providers | 11:00 |
| gibi | sean-k-mooney: there are nested cases in the same file | 11:00 |
| sean-k-mooney | so in theory nova should allready be handeling it | 11:00 |
| gibi | sean-k-mooney: nova handling the multiple candidates per host by taking the first candidate | 11:01 |
| sean-k-mooney | ah yes like these https://github.com/openstack/placement/blob/931a9e124251a0322550ff016ae1ad080cd472f3/placement/tests/functional/gabbits/allocation-candidates.yaml#L538-L557 | 11:02 |
| gibi | sean-k-mooney: yes | 11:03 |
| *** tetsuro has quit IRC | 11:03 | |
| sean-k-mooney | gibi: ya taking the first is fine until we start considering numa | 11:03 |
| gibi | sean-k-mooney: yes, for numa affinity the nova scheduler should start understanding allocation candidates | 11:04 |
| sean-k-mooney | actully we could have issue with tacking the first one for sharing providers also right? | 11:05 |
| sean-k-mooney | actully maybe not | 11:05 |
| gibi | sean-k-mooney: if we want to prefer sharing over local disk for a compute that has both, then yes | 11:05 |
| sean-k-mooney | since a host will either be configured ot used a local sotrage or the shared for ephemeral storge | 11:05 |
| sean-k-mooney | i would have to think about it some more but there is proably some unexpeced behavior lingering there. | 11:08 |
| sean-k-mooney | im not sure how we handel bfv vs non-bfv or local storage vs rbd image backend for ephmeral storage | 11:09 |
| sean-k-mooney | or bfv on a host with rbd configured for epheral storage. as in that case we coudl have 2 sharing resouce providers of storage from different ceph cluster. | 11:10 |
| sean-k-mooney | in other words i should poably learn how we do storage with placment at some point :) | 11:11 |
| *** belmoreira has joined #openstack-placement | 11:45 | |
| *** mriedem has joined #openstack-placement | 12:11 | |
| edleafe | tssurya: cdent is off this week | 12:34 |
| tssurya | edleafe: oh thanks for telling me | 12:35 |
| tssurya | so I guess I will catch him at the ptg then | 12:36 |
| edleafe | yeah, he'll be there | 12:36 |
| *** belmoreira has quit IRC | 13:07 | |
| *** belmoreira has joined #openstack-placement | 13:12 | |
| efried | sean-k-mooney: I'm here now, catching up... | 13:40 |
| efried | sean-k-mooney: Having only read your intro, yes, it's absolutely possible to get multiple candidates from the same host, in both nested and sharing cases. We have quite a few gabbi tests that demonstrate this behavior. | 13:41 |
| efried | okay, I see that's where gibi pointed you. | 13:41 |
| sean-k-mooney | efried: yes | 13:42 |
| sean-k-mooney | efried: i had assumed it would | 13:42 |
| efried | I see the issue is again how to limit the number of candidates we get. | 13:42 |
| efried | and/or dictate the order in which they arrive | 13:43 |
| efried | We have a placement config to randomize | 13:43 |
| sean-k-mooney | ya if you want to do weighing/filtering on the nova side with allocation candiates | 13:43 |
| efried | Randomize will get you close-ish to "spread" behavior. Non-randomize *might* get you close-ish to "pack" behavior. Neither is all the way there. | 13:43 |
| sean-k-mooney | using limit might result in failure becaue not enough host were condiserd | 13:44 |
| efried | Right, assuming there's some reason for the hosts at the top of the list to be unsuitable. | 13:44 |
| sean-k-mooney | yes which is likely if you are using anything not tracked in placement | 13:44 |
| efried | We should in general keep striving to tailor the placement queries to make fewer allocation candidates unsuitable. | 13:44 |
| sean-k-mooney | like sriov or pci passthough or numa or pinning or ... | 13:45 |
| efried | yup | 13:45 |
| efried | swhy we're trying to get all that stuff into placement. | 13:45 |
| sean-k-mooney | yep | 13:45 |
| sean-k-mooney | im just wondering if we need an allocation_per_host query arg or config setting | 13:46 |
| sean-k-mooney | untill we get to the point of tracking all that stuff in placemnt | 13:46 |
| sean-k-mooney | although even tracking every thing in placement wont fix it | 13:47 |
| sean-k-mooney | we will still have a combintorial problem if there are multiple RP of the same resouces | 13:47 |
| sean-k-mooney | e.g. multipe PFs | 13:47 |
| sean-k-mooney | if you ask for 1 vf, 1 cpu 1G of ram and 10G of disk but have 8 PF you will get 8 allocation candiates for the same host | 13:48 |
| sean-k-mooney | if you add in a request for a vGPU and you have 2 pGPUs on the host that becoems 16 alloication canidates | 13:49 |
| sean-k-mooney | tracking numa in plamcnet might reduce that set or it might increase it | 13:50 |
| sean-k-mooney | (beacue you have muliple sets fo cpus/memory) | 13:50 |
| sean-k-mooney | so its not a proablem in stien but it might become one in train if we dont have some logic for this | 13:51 |
| sean-k-mooney | edleafe: by the way when efried was joking about weigher in placment on thursday was that actully a thing? | 13:52 |
| edleafe | sean-k-mooney: ideas like that have been suggested, and pretty much laughed off | 13:52 |
| efried | If you mean "preferred traits" that's been semi-seriously considered. | 13:52 |
| edleafe | that's different | 13:53 |
| sean-k-mooney | correct | 13:53 |
| edleafe | that's more like "if you have some with those traits, great, otherwise I'll take what you got" | 13:53 |
| efried | No, preferred traits is a weigher-in-placement. | 13:53 |
| sean-k-mooney | i mean a weigher that sorts the allication candiate | 13:53 |
| sean-k-mooney | efried: prefered traits when i propsoed it first was in nova | 13:54 |
| sean-k-mooney | edleafe: yes more or less | 13:54 |
| sean-k-mooney | edleafe: prefered traits was when we get to the nova weigher look at the llocation candiates and chose the one with the most tratis that are in the prefered list | 13:55 |
| edleafe | yeah, that's different than having a bunch of random criteria sorting ACs | 13:55 |
| sean-k-mooney | ya | 13:55 |
| efried | I guess you could implement it as, "if results with these traits, return only those results and skip the rest." I had been thinking of it as, "Find all the candidates as usual, but put the ones with these traits at the front of the list." | 13:55 |
| sean-k-mooney | weighers in placment would be on the placmeent api side | 13:55 |
| sean-k-mooney | before we return the allcoation canidates reposcne to nova | 13:55 |
| sean-k-mooney | efried: yes that is how prefered traits is intedn to work. but it was proposed as a nova only feature | 13:56 |
| edleafe | efried: that's what people were asking for, and we had agreed that was best done in the scheduler | 13:56 |
| sean-k-mooney | the placement request woudl nto have any perfered tratis in it | 13:56 |
| efried | I don't remember that being agreed to; I thought it was just that it hadn't made the cut yet. | 13:57 |
| efried | ...and we would continue to weigh in the scheduler until it did. | 13:57 |
| efried | because the only problem with doing it in the scheduler is the limit thing. | 13:57 |
| edleafe | efried: It was rejected for exactly that reason: it already exists in nova, and it's a nova concept | 13:58 |
| sean-k-mooney | well there was certenly an element of there are more important things to do first | 13:58 |
| edleafe | sean-k-mooney: yep. Re-implementing nova in placement is not a priority :) | 13:58 |
| sean-k-mooney | edleafe: yes although sorting result is a general usecse that goes beyond nova | 13:59 |
| efried | ++ | 13:59 |
| efried | Agree that placement currently *doesn't* handle sorting, except for random vs "natural" | 13:59 |
| sean-k-mooney | anyway ill assume it will continue to be not a priority for train | 13:59 |
| efried | but that doesn't mean that it never should, that it's a concept anathema to the DNA of placement. | 13:59 |
| edleafe | sean-k-mooney: sure, but it should be a generic solution | 14:00 |
| sean-k-mooney | that is spereate form the problem i was describing above | 14:00 |
| sean-k-mooney | edleafe: yep totoally aggre that if placment ever provides allocation sorting it shoudl be a general solution | 14:00 |
| efried | How generic? | 14:01 |
| efried | Seems to me like we need to support a finite set of sorting policies | 14:01 |
| sean-k-mooney | initally i woudl start with assending/decending based on available of resouce clase on a host | 14:01 |
| sean-k-mooney | sort=VCPU> | 14:02 |
| efried | right, most/least available resource by percentage/absolute of $RC | 14:02 |
| sean-k-mooney | ya something like that | 14:02 |
| efried | or [$RC, ...] (in order) | 14:02 |
| efried | so "least available by absolute" would be a "pack" strategy for example. | 14:03 |
| efried | It also sounds like "spread roots" would be another desired policy. | 14:03 |
| sean-k-mooney | the other intersting soritng critira coudl be on traits but i woudl defer that until way later | 14:03 |
| efried | Yeah, without trying to weigh traits relative to each other, it would just be: pass in a [$list of preferred traits] and sort based on which candidate has the most of them. | 14:04 |
| sean-k-mooney | yes | 14:05 |
| efried | weighing traits relative to each other would be tricky, and would potentially require a whole other policy, like "rarest" traits are "heavier". That seems like going too far to me. | 14:05 |
| sean-k-mooney | there are more complex thing you could do but baby steps | 14:05 |
| edleafe | I'm not sure why those types of sorts are better done in placement than in the calling service | 14:06 |
| efried | because the calling service needs to be able to not need to deal with tens of thousands of results | 14:06 |
| efried | CERN case. | 14:06 |
| sean-k-mooney | edleafe: the issue is with the use of limmit | 14:06 |
| sean-k-mooney | cern do like limit=20 ro limit50 | 14:07 |
| efried | ^ so they don't have to deal with tens of thousands of results | 14:07 |
| sean-k-mooney | yep | 14:07 |
| edleafe | So we do the filtering *and* weighing in placement so that we only have to return the 20 "best"? | 14:07 |
| sean-k-mooney | which to be fair randomisation helps with but it distorst the set that nova can then filter/sort | 14:08 |
| efried | edleafe: Right, you would say sort_policy=foo&limit=20 | 14:08 |
| efried | with a generic ?sort_policy=<enum val> you could start small, implementing "pack" and "spread" based on the total available resource. IMO this is not a nova-ism at all. "Use the open jug of milk from the fridge before starting a new jug". | 14:09 |
| edleafe | efried: sure, I get that. Weighers in Nova aren't always that simple, though | 14:10 |
| sean-k-mooney | efried: pack is easy as it jsut an order by rp_uuid | 14:10 |
| sean-k-mooney | spread is harder | 14:10 |
| efried | sean-k-mooney: pack isn't sort by rp_uuid. | 14:10 |
| efried | that assumes all the resource and trait requests are the same. | 14:11 |
| sean-k-mooney | sorry yoru are right | 14:11 |
| efried | pack is sort by least available remaining resource. and spread is sort by *most* available remainin gresource. | 14:11 |
| sean-k-mooney | i was think of something related but different | 14:11 |
| sean-k-mooney | yes | 14:11 |
| sean-k-mooney | that is what i ment by vcpu> or memoy_mb< | 14:12 |
| efried | Yes, that's generic and complicated and powerful | 14:12 |
| efried | to keep it simple, you could just average the percentage of remaining amount of all the resources in the request | 14:12 |
| efried | (keep it simple for the client, that is :) | 14:12 |
| efried | so the client just says "pack" or "spread". | 14:12 |
| sean-k-mooney | perhaps. | 14:13 |
| efried | it's not perfect, but it gets real close while being vastly simpler for the caller. | 14:13 |
| sean-k-mooney | anyway i just wanted to raise the posible problem we could have with limit as the combinitorics of allocation candiates increase with nested rps | 14:14 |
| edleafe | pack/spread can be by percent of free resources, number of instances, number per switch, etc. We will have to keep this *very* limited. | 14:14 |
| sean-k-mooney | efried: it does not solve my origninal problem however | 14:14 |
| efried | no, for that you need a policy like "split roots". | 14:14 |
| sean-k-mooney | im not sure we want to go that route | 14:15 |
| sean-k-mooney | i feel it will be limiting | 14:15 |
| sean-k-mooney | but anyway its just somehting for use to be thinking about | 14:15 |
| efried | perhaps I'm not understanding the specific use cases you're trying to solve. | 14:15 |
| efried | I thought you were trying to get it so the list of candidates didn't have umpteen results for the same host clustered together at the top of the list. | 14:16 |
| sean-k-mooney | all host return 8 possible allocation candiates. i want ot limit that to a max of 2 | 14:16 |
| efried | ...in case for some reason that host as a whole is unsuitable and we run off the end of our limit before we get to a suitable one. | 14:16 |
| sean-k-mooney | efried: yes | 14:17 |
| efried | Randomize results gets you pretty close to solving that problem though. | 14:17 |
| efried | randomize+limit | 14:17 |
| efried | until there's some other criterion in play, it's almost as good as "spread roots". | 14:17 |
| sean-k-mooney | but the second we track numa in placement that does not work | 14:17 |
| efried | eh, why not? | 14:17 |
| sean-k-mooney | because we cant express the numa toploty relationship to placemnet | 14:18 |
| sean-k-mooney | so while one of the allction candiate may be vaildi to the host we dont know which one | 14:18 |
| sean-k-mooney | if we randomise it then we will get false no valid hosts | 14:19 |
| efried | oh, okay, I was assuming when you said "track numa in placement" that you meant we were expressing the topology relationship. | 14:19 |
| efried | yeah, so this is where being able to express topology and affinity matters. | 14:19 |
| sean-k-mooney | i just mean track rps in a tree | 14:19 |
| efried | both of those blueprints are proposed for Train. | 14:19 |
| sean-k-mooney | without the richer query syntax randomisation is not going to help | 14:20 |
| sean-k-mooney | yep they are. | 14:20 |
| sean-k-mooney | will both land in train | 14:20 |
| efried | randomization will *help*, but it won't *solve*. It will still sometimes break. | 14:22 |
| efried | will both land in train <== is that a question? | 14:22 |
| efried | One could hope so. But it seems quite unlikely that we'll be able to land anything for affinity on the placement side and also consume it on the nova side within Train. | 14:23 |
| sean-k-mooney | yes :) although perhaps a slightly biased one | 14:23 |
| sean-k-mooney | ya | 14:23 |
| sean-k-mooney | but im not going to looese hope that train will deliver good improvemnt in general | 14:24 |
| *** tetsuro has joined #openstack-placement | 14:59 | |
| *** tetsuro has quit IRC | 15:01 | |
| *** ttsiouts has quit IRC | 15:27 | |
| *** ttsiouts has joined #openstack-placement | 15:27 | |
| *** ttsiouts has quit IRC | 15:32 | |
| *** helenafm has quit IRC | 15:37 | |
| *** e0ne has quit IRC | 15:50 | |
| *** belmoreira has quit IRC | 16:03 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack/os-traits master: Add CPU traits for Meltdown/Spectre mitigation https://review.opendev.org/655193 | 16:45 |
| *** altlogbot_0 has quit IRC | 16:50 | |
| *** altlogbot_3 has joined #openstack-placement | 16:56 | |
| *** e0ne has joined #openstack-placement | 17:20 | |
| *** e0ne has quit IRC | 17:24 | |
| *** tssurya has quit IRC | 18:07 | |
| *** tssurya has joined #openstack-placement | 18:10 | |
| *** e0ne has joined #openstack-placement | 18:29 | |
| *** amodi has joined #openstack-placement | 18:43 | |
| *** amodi has quit IRC | 18:55 | |
| *** e0ne has quit IRC | 19:01 | |
| -openstackstatus- NOTICE: the zuul scheduler is being restarted now in order to address a memory utilization problem; changes under test will be reenqueued automatically | 19:09 | |
| *** tssurya has quit IRC | 19:39 | |
| *** ttsiouts has joined #openstack-placement | 21:42 | |
| *** ttsiouts has quit IRC | 23:14 | |
| *** ttsiouts has joined #openstack-placement | 23:15 | |
| *** ttsiouts has quit IRC | 23:19 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!