| openstackgerrit | Akihito Takai proposed openstack/storlets master: WIP: create the configration file for the agent side https://review.openstack.org/436263 | 05:01 |
|---|---|---|
| openstackgerrit | Akihito Takai proposed openstack/storlets master: WIP: create the configration file for the agent side https://review.openstack.org/436263 | 06:44 |
| *** kota_ has quit IRC | 08:11 | |
| *** kota_ has joined #openstack-storlets | 08:11 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets master: Set concurrency 2 for less gate failures with 35 mins Timeout https://review.openstack.org/436912 | 08:32 |
| kota_ | rebased off to the unique container patch set 4 | 08:32 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets master: Cleanup deploy_storlet and the tests https://review.openstack.org/422470 | 08:47 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets master: Set concurrency 2 for less gate failures with 35 mins Timeout https://review.openstack.org/436912 | 08:54 |
| kota_ | it resolved my mis-rebasing | 08:56 |
| *** sagara has joined #openstack-storlets | 08:59 | |
| eranrom | Hi | 08:59 |
| eranrom | Hi | 08:59 |
| sagara | hi | 08:59 |
| *** akihito_ has joined #openstack-storlets | 08:59 | |
| sagara | I also thinking multi tenant use case. when many account in the same node, they might consume almost all host resource | 08:59 |
| kota_ | sagara: yeah, that's worst case | 09:00 |
| sagara | am I thinking too much? | 09:00 |
| eranrom | sagara: no! :-) | 09:00 |
| kota_ | sagara: I'm feeling the resource limitation has complex metrics to think of, so I'd like to solve the tangled problems step by step | 09:01 |
| eranrom | I think the point here is to give the admin enough control to make sure that even in a multi-tenant env, Swift can still work | 09:01 |
| kota_ | eranrom: yes | 09:01 |
| kota_ | that is absolutely first step | 09:02 |
| kota_ | ah | 09:02 |
| kota_ | no- it's goal | 09:02 |
| kota_ | and even if it's not enough, the first step is enabling some resourece could be limited by operator somehow (ulimit? docker limitation?) | 09:02 |
| sagara | sure, I agree, we need to do it step by step | 09:02 |
| kota_ | and the point for development, where we should start with, | 09:03 |
| kota_ | it's my question | 09:03 |
| kota_ | account level? app level? | 09:03 |
| eranrom | I would even say this: | 09:04 |
| kota_ | the worst case as sagara described, current storlets can eat any resource we should reserve for the backend swift works | 09:04 |
| kota_ | what I'd like to address first, any way is welcome if it's possible | 09:05 |
| kota_ | my opinion | 09:05 |
| kota_ | eranrom: sorry for interapting you :/ | 09:05 |
| eranrom | kota_: np. I am not sure what I was about to say... | 09:06 |
| eranrom | ok, here is the thought: | 09:06 |
| eranrom | Ideally we could cap the maximum resources used by all Docker containers in a way that more resource can be used only if available (not sure this is possible) | 09:06 |
| eranrom | once all containers are capped to a limited set of resources, we can split those between them either on an app or account level | 09:07 |
| kota_ | eranrom: nnnnnnnnice | 09:07 |
| eranrom | here is an example for CPU: | 09:08 |
| eranrom | so the admin can define the set of CPUs on which docker can run | 09:08 |
| sagara | thanks, I think it's nice too. | 09:08 |
| eranrom | say we are on a 24 cores system and the admin has a way to say, docker can use cores 0-17 (just an example) | 09:09 |
| kota_ | cool | 09:09 |
| eranrom | then in a more fined grained level each tenant gets a subset of those, say tenant 1 gets 0-4 | 09:10 |
| eranrom | The problem here is that this is too rigid | 09:10 |
| eranrom | and does not allow over provisioning. Perhaps this is here we can switch to nice / io-nice | 09:10 |
| eranrom | I mean: | 09:10 |
| eranrom | all docker containers will use cores 0-17. | 09:11 |
| eranrom | between the containers we will use nice, according to some service level agreement | 09:11 |
| eranrom | or even a combination: | 09:12 |
| eranrom | a customer that pays a lot will get dedicated reosurces (e.g. cores 0-3) | 09:12 |
| eranrom | others can get a better nice values | 09:12 |
| kota_ | the priorities are adressed in the 0-17, right? | 09:12 |
| eranrom | correct | 09:12 |
| kota_ | and 18-24 are still available for swift | 09:12 |
| kota_ | great | 09:13 |
| eranrom | technically, this means that all containers are executed with a limiting affinity to cores 0-17 (should be supported by docker) | 09:13 |
| sagara | if we can use docker '--cpu-shares=0' or '--blkio-weight=' in capped all docker containes, I think it will resolve cap problem | 09:13 |
| eranrom | given that we have the management hooks in place... | 09:13 |
| eranrom | sagara: right! I was not aware of this option. Its been some time since I looked at this :-) | 09:15 |
| eranrom | sagara: do you know if this is a CGroup feature? | 09:15 |
| sagara | maybe it can. I will confirm | 09:15 |
| eranrom | sagara: thanks! | 09:16 |
| eranrom | so maybe a first step would be to define the APIs or policies that an admin can define - just a thought | 09:16 |
| eranrom | also, back to kota's point, seems like we can have both priorities between containers and perhaps between apps: | 09:18 |
| eranrom | different containers get different --cpu-share | 09:18 |
| eranrom | different apps get different nice values | 09:18 |
| eranrom | where --cpu-share is defined by the admin based on SLA, and different apps by the tenant himself | 09:19 |
| eranrom | again just a thought :-) | 09:19 |
| kota_ | sounds reasonable to me | 09:21 |
| eranrom | sorry if I got carried away and "spoke" too much :-| | 09:22 |
| kota_ | sagara could find the first step, maybe? does it make you sense sagara? | 09:22 |
| kota_ | eranrom: np | 09:23 |
| sagara | yes, of course, thank you | 09:24 |
| sagara | I think app priorities seems to be ditto. | 09:33 |
| sagara | In a certain (proxy or object) host, we can control that resource with multi-cgroup. | 09:34 |
| sagara | e.g. CGroup(host / storlet) - (CGroup (tenant wight)) - CGroup (app wight) - app. | 09:34 |
| sagara | so we can expose CGroup (app wight) API for tenant user. tenant user can adjust their app weight as they like | 09:35 |
| sagara | but app wight may not be so effective, it is valid just only same host app | 09:36 |
| eranrom | sagara: I think this makes sense. This assumes that CGroup supports weights - right? | 09:45 |
| sagara | eranrom: yes. I saw docker reference, docker has weight feature, so Cgroup maybe also have it | 09:47 |
| sagara | #link: https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources | 09:47 |
| sagara | I need to confirm it | 09:48 |
| eranrom | sagara: right! I have just checked out some CGroups docs. It surely has changed a lot since I looked at it 3 years ago... | 09:48 |
| eranrom | Apparently, for CPU it was introduced in the 3.2 kernel... | 09:52 |
| *** openstackgerrit has quit IRC | 10:33 | |
| sagara | eranrom, kota_: many thanks today! I will continue to consider that | 10:38 |
| eranrom | sagara: Thank you for sharing your thoughts! | 10:39 |
| *** sagara has quit IRC | 13:12 | |
| *** akihito_ has quit IRC | 13:13 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!