| *** igordc has quit IRC | 00:09 | |
| *** lcastell has joined #kata-general | 00:25 | |
| *** lcastell has quit IRC | 00:26 | |
| *** lcastell has joined #kata-general | 00:59 | |
| *** igordc has joined #kata-general | 01:26 | |
| *** lcastell has quit IRC | 01:29 | |
| *** sameo has quit IRC | 02:09 | |
| *** igordc has quit IRC | 02:40 | |
| *** igordc has joined #kata-general | 02:40 | |
| kata-irc-bot | <kmacleod> Hi folks. Using Kata 1.5, Kube 1.12. I’m trying to manipulate my pod spec such that kata allocates 8 vCPUs to the pod’s VM, but the relationship between the pod spec’s CPU resource limits and the number of allocate CPus isn’t obvious from experimentation. | 05:26 |
|---|---|---|
| *** tmhoang has joined #kata-general | 07:26 | |
| *** sgarzare has joined #kata-general | 07:56 | |
| kata-irc-bot | <xwlpt> Kata container reserved one cpu and 2G memory by default for VM | 08:25 |
| *** gwhaley has joined #kata-general | 08:49 | |
| *** igordc has quit IRC | 09:25 | |
| *** sameo has joined #kata-general | 09:44 | |
| kata-irc-bot | <graham.whaley> hi @kmacleod - there is an algo, but it is maybe slightly more complex than one expects as we try to take into account CPU resource needed by the VM/kata itself as well as handling different CPU requirements across multiple containers in the same pod (VM). There are some docs at https://github.com/kata-containers/documentation/blob/master/constraints/cpu.md that hopefully help explain. If you think we can improve them, edits | 10:46 |
| kata-irc-bot | most welcome! | 10:46 |
| *** stackedsax has quit IRC | 13:53 | |
| *** stackedsax has joined #kata-general | 13:54 | |
| *** igordc has joined #kata-general | 13:56 | |
| *** igordc has quit IRC | 15:11 | |
| *** tmhoang has quit IRC | 16:12 | |
| *** sgarzare has quit IRC | 17:18 | |
| *** tmhoang has joined #kata-general | 17:22 | |
| *** igordc has joined #kata-general | 19:31 | |
| *** gwhaley has quit IRC | 19:52 | |
| *** sameo has quit IRC | 22:22 | |
| kata-irc-bot | <kmacleod> Yeah, I figured as much. I did read that page previously, but it’s rather light on detail. A few more concrete, non-trivial examples would go a long way top explain things. I’ve been trying to deduce the algorithm empirically but it’s hard to get a grip on. | 22:59 |
| kata-irc-bot | <kmacleod> For example, if I have a pod spec that has a cpu limit of “8”, I get 4 vcpus | 23:08 |
| kata-irc-bot | <eric.ernst> @kmacleod agreed! | 23:12 |
| kata-irc-bot | <eric.ernst> I'll work to update this, with some example yaml, and resulting VM specs. | 23:13 |
| kata-irc-bot | <eric.ernst> i don't expct 4 vCPU if you have a limit of 8, btw. | 23:13 |
| kata-irc-bot | <eric.ernst> Is this just an example you made up or something you observed? | 23:13 |
| kata-irc-bot | <eric.ernst> And ... thanks for the feedback -- much appreciated! | 23:13 |
| kata-irc-bot | <kmacleod> does it depend on the physical CPUs on the host? | 23:13 |
| kata-irc-bot | <eric.ernst> @julio.montes will know best. | 23:14 |
| kata-irc-bot | <eric.ernst> it doesn't map 1:1, unless you are using somethinng like CPU sets (cpu manager with static) | 23:15 |
| kata-irc-bot | <eric.ernst> its just the number of hypervisor threads (vcpus) which are started. | 23:15 |
| kata-irc-bot | <eric.ernst> Realistically, you don't want to start more threads than CPUs, but i'm not sure if we specificalyl capped that. | 23:16 |
| kata-irc-bot | <eric.ernst> your k8s scheduler, however, should take this into account | 23:16 |
| kata-irc-bot | <kmacleod> sure | 23:16 |
| kata-irc-bot | <eric.ernst> Kenny, you based out of Australia? | 23:17 |
| kata-irc-bot | <kmacleod> Here’s an example podspec, for example. 3 containers, one with a cpu limit of 8, one with 4, another with no limit. When I exec into container1 and `cat /proc/cpuinfo` I see 5 vCPUs. | 23:18 |
| kata-irc-bot | <kmacleod> yeah, australia | 23:18 |
| kata-irc-bot | <eric.ernst> interesting. to verify -- you're on 1.5 kata? | 23:19 |
| kata-irc-bot | <kmacleod> If I change those values, the number of vCPUs does go up and down, just not by a number I can predict | 23:19 |
| kata-irc-bot | <kmacleod> yeah, 1.5 | 23:19 |
| kata-irc-bot | <eric.ernst> that's concerning. | 23:19 |
| kata-irc-bot | <eric.ernst> @jose.carlos.venegas.m ^ you have any ideas here? | 23:19 |
| kata-irc-bot | <eric.ernst> it shoudl just be the sum of the container requests + default | 23:20 |
| kata-irc-bot | <eric.ernst> default being from the configuration.toml for kata. | 23:20 |
| kata-irc-bot | <kmacleod> yeah, I beliebe the default is still 1 | 23:20 |
| kata-irc-bot | <eric.ernst> so, I would espect you get 13 vCPU | 23:20 |
| kata-irc-bot | <kmacleod> IIRC, when I last looked at this under kata 1.3, it behaved as you describe | 23:20 |
| kata-irc-bot | <eric.ernst> well, we did work to improve things. | 23:21 |
| kata-irc-bot | Action: jose.carlos.venegas.m looks | 23:21 |
| kata-irc-bot | <kmacleod> yeah, I understand there were improvements related to not over-provisioning when there were fractional values used | 23:21 |
| kata-irc-bot | <eric.ernst> before, in 1.3, if you had 1 containr which requsts 500 mCPU and another that requests 200 mCPU, it would result in 3 (1 default, 1 for ctr1, 1 for ctr2). this was wrong, imo, and we should have fixed this. | 23:21 |
| kata-irc-bot | <eric.ernst> yeap. | 23:21 |
| kata-irc-bot | <eric.ernst> I have a cluster up and give this a test, but I won't be able to dive into it for a few more hours. | 23:22 |
| kata-irc-bot | <eric.ernst> It's a bit early in your day (i don't math good, so correct me :)), so hopefully i'll have more news before EOD for you | 23:22 |
| kata-irc-bot | <eric.ernst> if we're lucky Carlos, @jose.carlos.venegas.m, may have input sooner. | 23:22 |
| kata-irc-bot | <kmacleod> that would great if you could, thanks. not massively urgent at this point, but it will become a blocker for us in the next week or so | 23:22 |
| kata-irc-bot | <eric.ernst> Yea, undertsood. | 23:23 |
| kata-irc-bot | <kmacleod> at this point I’d settle for just being able to predict the result, even if I can’t make it do what I want | 23:23 |
| kata-irc-bot | <eric.ernst> :slightly_smiling_face: | 23:23 |
| kata-irc-bot | <eric.ernst> The current behavior is certainly not expected, and I'll need to dig into our CI to make sure this is covered already (it should be...) | 23:23 |
| kata-irc-bot | <eric.ernst> thanks for bringing it up. | 23:24 |
| kata-irc-bot | <kmacleod> :thumbsup: thanks for the help | 23:24 |
| kata-irc-bot | <eric.ernst> If you like open source karam bonus points, if you can open a github issue it'd be awesome | 23:24 |
| kata-irc-bot | <eric.ernst> https://github.com/kata-containers/runtime/issues/new | 23:24 |
| kata-irc-bot | <kmacleod> will do | 23:25 |
| kata-irc-bot | <kmacleod> just a sanity check - I should see the same number of vCPUs regardless of which container in the pod I exec into, right? | 23:25 |
| kata-irc-bot | <eric.ernst> Yah, it should show all of the available ones in the Virtual Machine. | 23:26 |
| kata-irc-bot | <kmacleod> yeah | 23:26 |
| kata-irc-bot | <eric.ernst> but the container would be constrained to just use what it is allocated via cgroups | 23:27 |
| kata-irc-bot | <kmacleod> also, are we talking podspec cpu requests or cpu limits here? My example uses limits. If I change the requested value, it doesn’t seem to have any effect. | 23:27 |
| kata-irc-bot | <kmacleod> although I’ll do a bit more fiddling with it to verify that | 23:27 |
| kata-irc-bot | <eric.ernst> you can just set one; it gives you a guaranteed pod (if you just say limits, it sets req and limits to the same) | 23:32 |
| kata-irc-bot | <eric.ernst> these are container spec | 23:32 |
| kata-irc-bot | <kmacleod> ah ok | 23:32 |
| kata-irc-bot | <eric.ernst> there actually isn't a limits/requests for resourcs in the podspec, just container | 23:32 |
| kata-irc-bot | <kmacleod> yep | 23:32 |
| kata-irc-bot | <eric.ernst> (though there are talks of adding this ...) | 23:33 |
| kata-irc-bot | <eric.ernst> (i undrstand that was an interesting use case for atlassian before) | 23:33 |
| kata-irc-bot | <kmacleod> yeah, for our use case it would be nicer if we could specify memory and cpu limits at the pod level, to remove the need for our users to partition the memory | 23:33 |
| kata-irc-bot | <kmacleod> https://github.com/kata-containers/runtime/issues/1292 | 23:36 |
| kata-irc-bot | <eric.ernst> Yeah, i think other sig-node folks agree. | 23:36 |
| kata-irc-bot | <eric.ernst> I'm working on a KEP which may add this to the scope. | 23:36 |
| kata-irc-bot | <kmacleod> nice | 23:36 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!