| *** auk_ has joined #kata-general | 01:38 | |
| *** auk has quit IRC | 01:41 | |
| *** saline_retry__ has joined #kata-general | 01:46 | |
| *** auk_ has quit IRC | 01:49 | |
| *** saline_retry__ has quit IRC | 01:53 | |
| *** lpetrut has quit IRC | 06:55 | |
| *** gwhaley has joined #kata-general | 08:55 | |
| kata-irc-bot3 | <william> @s.s.filatov94 Did you figure it out? I have been struggling to get k8s+kata+firecracker to work during the past week. I am unable to getthe kata-deploy to work under either CRI-O or Containerd. | 10:28 |
|---|---|---|
| kata-irc-bot3 | <graham.whaley> @william - for fc+CRI-O, what storage backing are you using - fc requires block based storage (devicemapper et. al.) afaik. If you are still struggling, post some details of your setup and ideally the error you are seeing from k8s logs, and somebody here should be able to help. | 10:31 |
| kata-irc-bot3 | <william> @graham.whaley I could get back with that. Does there exist any additional documentation for what is required on the machine and by the kubernetes cluster that the kata-deploy yaml are applied to? Or if there's a reference setup it is being tested towards? | 10:36 |
| kata-irc-bot3 | <graham.whaley> The only doc/ref I know of for kata/fc requirements is on the initial wiki page at https://github.com/kata-containers/documentation/wiki/Initial-release-of-Kata-Containers-with-Firecracker-support | 10:37 |
| kata-irc-bot3 | <graham.whaley> General requirement for Kata is 'must support virtualisation', as it is running VMs. Same holds for Kata using either qemu or fc (or any other hypervisor/VM) as a backend. | 10:38 |
| kata-irc-bot3 | <graham.whaley> overall then, kata-qemu supports a number of common configs (overlay, CNI etc.), but fc has stricter requirements aiui. | 10:39 |
| kata-irc-bot3 | <s.s.filatov94> @william yes, CRI-O did the thing for us. Though overall k8s + kata + fc works really slow, but we haven't tried to find the root cause yet. | 10:45 |
| kata-irc-bot3 | <william> @s.s.filatov94 did you set up devicemapper in advance then initiate a kubernetes cluster with crio followed by running the kata-deploy yaml? | 10:47 |
| kata-irc-bot3 | <s.s.filatov94> I just invited my colleague who did the setup to the kata space, you'll be able to PM him once he's up. | 10:53 |
| kata-irc-bot3 | <anuriq> nope. we've configured K8s cluster with crio runtime first. you can configure crio to use devicemapper as storage_type | 10:55 |
| kata-irc-bot3 | <anuriq> but order does not matter that much | 10:55 |
| kata-irc-bot3 | <anuriq> it's better to make sure, that k8s with traditional containers works fine and then start to implement Kata runtime classes | 10:59 |
| kata-irc-bot3 | <anuriq> btw would be great to have an article from Kata team about how kata-runtime bootstraps firecracker in details. that would help to understand how to speed up that for specific workflows | 11:00 |
| kata-irc-bot3 | <anuriq> btw would be great to have an article from Kata team about how kata-runtime bootstraps firecracker in details. that would help to understand how to speed up that for specific workflows | 11:01 |
| kata-irc-bot3 | <william> Thank you. I'll make sure to update when and if I make progress. I'd preferrably like to get it to work with containerd and it's latest devmapper snapshotter from version 1.13.0 but at this point if it works with CRI-O I'm happy to start with that :slightly_smiling_face: | 11:02 |
| *** gwhaley has quit IRC | 12:09 | |
| *** gwhaley has joined #kata-general | 13:42 | |
| kata-irc-bot3 | <eric.ernst> It should work with both, @william | 15:18 |
| kata-irc-bot3 | <eric.ernst> Hardest part is enabling the block based snapshotter and switching to containerd or crio | 15:18 |
| kata-irc-bot3 | <eric.ernst> If you can characterize, I'd like to understand what you mean by slow - time to boot? IO performance? | 15:19 |
| kata-irc-bot3 | <william> @eric.ernst I have attempted to get devmapper for containerd to work with fc using both https://github.com/projectatomic/container-storage-setup and through the steps at clearlinux https://github.com/clearlinux/cloud-native-setup/blob/master/clr-k8s-examples/containerd_devmapper_setup.sh without success. | 15:23 |
| kata-irc-bot3 | <eric.ernst> 'attempted' Any success? | 15:24 |
| kata-irc-bot3 | <william> No success. rootfs won't be set up for the container | 15:25 |
| kata-irc-bot3 | <eric.ernst> To clarify, can you boot a traditional container using this snapshotter? | 15:28 |
| kata-irc-bot3 | <william> Yes. runc and qemu. Not FC | 15:28 |
| kata-irc-bot3 | <eric.ernst> vhost_vsock is necessary as well. | 15:30 |
| kata-irc-bot3 | <eric.ernst> you modprobe this? | 15:30 |
| kata-irc-bot3 | <eric.ernst> and can you clarify which version of kata you are using? | 15:30 |
| kata-irc-bot3 | <eric.ernst> Sorry it isn't working out of the box for ou. | 15:30 |
| kata-irc-bot3 | <william> Yes i'm rocking ```cat > /etc/modules-load.d/containerd.conf <<EOF vhost_vsock br_netfilter EOF modprobe vhost_vsock modprobe br_netfilter ``` | 15:34 |
| *** igordc has joined #kata-general | 15:36 | |
| kata-irc-bot3 | <william> ``` williamviktorsson@nested-template-1:~$ sudo ctr run --runtime io.containerd.run.runc.v1 -t --rm docker.io/library/busybox:latest hello sh / # exit williamviktorsson@nested-template-1:~$ sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh / # / # ls bin dev etc home proc root run sys tmp usr var / # exit williamviktorsson@nested-template-1:~$ sudo ctr run --runtime | 15:38 |
| kata-irc-bot3 | io.containerd.run.katafc.v2 -t --rm docker.io/library/busybox:latest hello sh ctr: rootfs (/run/kata-containers/shared/containers/hello/rootfs) does not exist: unknown williamviktorsson@nested-template-1:~$ ``` | 15:38 |
| kata-irc-bot3 | <william> Works very well with runc, kata-qemu but kata-fc is struggling. I was having similar issues under my CRI-O attempts. | 15:40 |
| kata-irc-bot3 | <eric.ernst> let me check something locally, I have a cluster up.. | 15:43 |
| kata-irc-bot3 | <william> I am using this static binary: ```kata-static-1.9.0-x86_64.tar.xz``` | 15:43 |
| kata-irc-bot3 | <william> Thank's a lot. I really appreciate it. This is part of my master thesis which vill evaluate the performance tradeoffs of some of the resent hardened container runtimes. | 15:43 |
| *** igordc has quit IRC | 15:47 | |
| kata-irc-bot3 | <eric.ernst> sweet. | 15:48 |
| kata-irc-bot3 | <eric.ernst> The error makes it sounds like devmapper isn't actually being used. | 15:48 |
| kata-irc-bot3 | <eric.ernst> In ctr, you can explicitly say what snapshotter to use on a per pod basis, right? | 15:48 |
| kata-irc-bot3 | <eric.ernst> @gmmaharaj IIRC we ran into this before? | 15:48 |
| kata-irc-bot3 | <eric.ernst> The error makes it sound like devmapper isn't actually being used. | 15:49 |
| kata-irc-bot3 | <eric.ernst> Perhaps you can start up a kata-qemu based on. | 15:49 |
| kata-irc-bot3 | <eric.ernst> Do an exec and check mount | 15:49 |
| kata-irc-bot3 | <eric.ernst> see how the rootfs is mounted in the pod's container.. | 15:49 |
| kata-irc-bot3 | <william> Will check into it. | 15:50 |
| kata-irc-bot3 | <william> My crictl info does report devmapper as the snapshotter but I will look at the mounts. | 15:53 |
| kata-irc-bot3 | <william> @eric.ernst Nice catch, ```sudo ctr run --snapshotter devmapper --runtime io.containerd.run.katafc.v2 -t --rm docker.io/library/busybox:latest hello sh``` Did the trick. | 16:05 |
| kata-irc-bot3 | <william> Now if I can just get k8s to utilize it I should be golden. | 16:05 |
| kata-irc-bot3 | <eric.ernst> nice, glad to hear. | 16:07 |
| kata-irc-bot3 | <eric.ernst> yeah, the kata part is generally not htat hard. | 16:07 |
| kata-irc-bot3 | <eric.ernst> just deploy the kata-deploy daemonset. | 16:07 |
| kata-irc-bot3 | <eric.ernst> Its getting the rest of it configured that requires pain/suffering, imo. | 16:08 |
| kata-irc-bot3 | <william> I have been all over the place to get fc to work under k8s. A lot of different approaches. | 16:08 |
| kata-irc-bot3 | <eric.ernst> interesting. It shoudl just work if you have containerd defaulting to devmapper. | 16:09 |
| kata-irc-bot3 | <william> kata + fc under docker worked instantaneously | 16:09 |
| kata-irc-bot3 | <william> but that wont do for k8s | 16:09 |
| kata-irc-bot3 | <eric.ernst> This is how I've seen other folks do it, for reference, for CRIO: https://raw.githubusercontent.com/clearlinux/cloud-native-setup/master/clr-k8s-examples/setup_kata_firecracker.sh | 16:11 |
| kata-irc-bot3 | <eric.ernst> For containerd: https://raw.githubusercontent.com/clearlinux/cloud-native-setup/master/clr-k8s-examples/containerd_devmapper_setup.sh | 16:12 |
| kata-irc-bot3 | <eric.ernst> can you share snippet of your /etc/containerd/config.toml? | 16:12 |
| kata-irc-bot3 | <william> Yeah the second one I linked earlier. I've tried the prior one for CRI-O without success. It's just the past 2 days i've been understanding devmapper / devicemapper is missing. | 16:13 |
| kata-irc-bot3 | <william> ``` [plugins] [plugins.devmapper] pool_name = "contd-thin-pool" base_image_size = "512MB" [plugins.cri.containerd] snapshotter="devmapper" [plugins.cri.containerd.runtimes.kata] runtime_type = "io.containerd.kata.v2" [plugins.cri.containerd.runtimes.kata.options] ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration.toml" | 16:14 |
| kata-irc-bot3 | [plugins.cri.containerd.runtimes.kata-fc] runtime_type = "io.containerd.kata-fc.v2" [plugins.cri.containerd.runtimes.kata-fc.options] ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-fc.toml" [plugins.cri.containerd.runtimes.kata-qemu] runtime_type = "io.containerd.kata-qemu.v2" [plugins.cri.containerd.runtimes.kata-qemu.options] ConfigPath = | 16:14 |
| kata-irc-bot3 | "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml" [plugins.cri.containerd.runtimes.kata-qemu-virtiofs] runtime_type = "io.containerd.kata-qemu-virtiofs.v2" [plugins.cri.containerd.runtimes.kata-qemu-virtiofs.options] ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu-virtiofs.toml" [plugins.cri.containerd.runtimes.kata-nemu] runtime_type = | 16:14 |
| kata-irc-bot3 | "io.containerd.kata-nemu.v2" [plugins.cri.containerd.runtimes.kata-nemu.options] ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-nemu.toml" ``` | 16:14 |
| kata-irc-bot3 | <eric.ernst> what version of containerd are you using? 1.3? | 16:54 |
| *** igordc has joined #kata-general | 17:13 | |
| kata-irc-bot3 | <william> 1.3.0 | 17:14 |
| kata-irc-bot3 | <william> I got it to work under k8s now, containerd+kata+fc I'll try to pinpoint what's been the issue and share the manual steps. | 17:58 |
| *** gwhaley has quit IRC | 18:00 | |
| *** igordc has quit IRC | 20:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!