| kata-irc-bot | <escita> @anastassios.nanos Hi Ananos, I managed to setup the device mapper and I am able to run kata container with firecracker VMM. How do you configure network for the containers? I tried to do a host pass-through using `--net-host` it doesn't seem to be working. Do you have any links I can refer to? Thanks. `sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm --net-host docker.io/library/ubuntu:latest kata-c bin/bash` | 06:55 |
|---|---|---|
| kata-irc-bot | <tim.rice> ill test that! thanks for the feedback so far | 07:10 |
| kata-irc-bot | <anastassios.nanos> thats great news! regarding the network, I think that net-host is something not supported in kata [https://github.com/kata-containers/kata-containers/blob/main/docs/Limitations.md#docker---nethost] | 07:22 |
| kata-irc-bot | <escita> oh thanks, what can we currently do about the network setup? :grinning: | 07:36 |
| kata-irc-bot | <tim.rice> unfortunately still getting the errors. ive confirmed the socket is where it should be but still getting ```Sep 03 07:46:01 pumped-gator kata[89372]: time="2021-09-03T07:46:01.731023858Z" level=error msg="post event" error="failed to connect: dial unix: missing address" name=containerd-shim-v2 pid=89372 sandbox=b9260f33d83fc563661bfae787d2a149bef3a00d326287d1f0ca8ab65c864135 source=containerd-kata-shim-v2``` | 07:52 |
| kata-irc-bot | <tim.rice> ill try installing kata via the kata-deploy daemonset, didnt realise that existed :slightly_smiling_face: | 07:52 |
| kata-irc-bot | <anastassios.nanos> hmm, not entirely sure what's the proposed way to handle this -- on our side we have been using this setup just for local / runtime testing. Networking is handled by the upper layers when integrated with k8s. Without being an expert on containerd/ctr I suspect it has to do with the json spec passed to the shim (can't find a `--net` option in `ctr`, other than `--net-host`). @fidencio who's the most relevant person to ping? | 08:02 |
| kata-irc-bot | <anastassios.nanos> @shuo.chen can you share the running hypervisor options? I'm curious what's the virtio-scsi parameters passed to the VM | 08:03 |
| kata-irc-bot | <fidencio> @archana.m.shinde is *the* person you want to talk to. :slightly_smiling_face: | 08:31 |
| kata-irc-bot | <fidencio> Tim, let us know if that works bettter. | 08:32 |
| kata-irc-bot | <tim.rice> it doesnt unfortunatley | 08:33 |
| kata-irc-bot | <tim.rice> ```Sep 03 08:32:25 pumped-gator kata[113928]: time="2021-09-03T08:32:25.635577654Z" level=error msg="post event" error="failed to connect: dial unix: missing address" name=containerd-shim-v2 pid=113928 sandbox=aae801800f627ec1f5870112c0d125d91859bc39458b3111f2c243594f40c285 source=containerd-kata-shim-v2 Sep 03 08:32:25 pumped-gator crio[47547]: time="2021-09-03T08:32:25.635577654Z" level=error msg="post event" error="failed to | 08:33 |
| kata-irc-bot | connect: dial unix: missing address" name=containerd-shim-v2 pid=113928 sandbox=aae801800f627ec1f5870112c0d125d91859bc39458b3111f2c243594f40c285 source=containerd-kata-shim-v2 Sep 03 08:32:25 pumped-gator kata[113928]: time="2021-09-03T08:32:25.668711945Z" level=error msg="post event" error="failed to connect: dial unix: missing address" name=containerd-shim-v2 pid=113928 sandbox=aae801800f627ec1f5870112c0d125d91859bc39458b3111f2c243594f40c285 | 08:33 |
| kata-irc-bot | source=containerd-kata-shim-v2 Sep 03 08:32:25 pumped-gator crio[47547]: time="2021-09-03T08:32:25.668711945Z" level=error msg="post event" error="failed to connect: dial unix: missing address" name=containerd-shim-v2 pid=113928 sandbox=aae801800f627ec1f5870112c0d125d91859bc39458b3111f2c243594f40c285 source=containerd-kata-shim-v2 Sep 03 08:32:26 pumped-gator crio[47547]: time="2021-09-03 08:32:26.143595976Z" level=info msg="Finish piping | 08:33 |
| kata-irc-bot | \"stdout\" of container exec \"e5dcf949ad93724b094e5c0ea9daae26488144046f6c95f0a7640d79700e7281\"" Sep 03 08:32:26 pumped-gator crio[47547]: time="2021-09-03 08:32:26.143707015Z" level=info msg="Container exec \"e5dcf949ad93724b094e5c0ea9daae26488144046f6c95f0a7640d79700e7281\" stdin closed" Sep 03 08:32:26 pumped-gator kata[113928]: time="2021-09-03T08:32:26.14410838Z" level=error msg="post event" error="failed to connect: dial unix: missing | 08:33 |
| kata-irc-bot | address" name=containerd-shim-v2 pid=113928 sandbox=aae801800f627ec1f5870112c0d125d91859bc39458b3111f2c243594f40c285 source=containerd-kata-shim-v2 Sep 03 08:32:26 pumped-gator crio[47547]: time="2021-09-03T08:32:26.144108380Z" level=error msg="post event" error="failed to connect: dial unix: missing address" name=containerd-shim-v2 pid=113928 sandbox=aae801800f627ec1f5870112c0d125d91859bc39458b3111f2c243594f40c285 | 08:33 |
| kata-irc-bot | source=containerd-kata-shim-v2 Sep 03 08:32:26 pumped-gator crio[47547]: time="2021-09-03 08:32:26.451671104Z" level=info msg="Checking image status: fluent/fluent-bit:1.7.1" id=2d5c558b-03c4-412d-831d-0093387c1928 name=/runtime.v1alpha2.ImageService/ImageStatus Sep 03 08:32:26 pumped-gator crio[47547]: time="2021-09-03 08:32:26.451988095Z" level=info msg="Resolving \"fluent/fluent-bit\" using unqualified-search registries | 08:33 |
| kata-irc-bot | (/etc/containers/registries.conf)" Sep 03 08:32:26 pumped-gator crio[47547]: time="2021-09-03 08:32:26.455811563Z" level=info msg="Image status: &{0xc0005c6700 map[]}" id=2d5c558b-03c4-412d-831d-0093387c1928 name=/runtime.v1alpha2.ImageService/ImageStatus Sep 03 08:32:26 pumped-gator kata[113928]: time="2021-09-03T08:32:26.636926996Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:26 | 08:33 |
| kata-irc-bot | pumped-gator crio[47547]: time="2021-09-03T08:32:26.636926996Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:26 pumped-gator kata[113928]: time="2021-09-03T08:32:26.669283551Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:26 pumped-gator crio[47547]: time="2021-09-03T08:32:26.669283551Z" level=error msg="forward event" error="failed to | 08:33 |
| kata-irc-bot | connect: dial unix: missing address" Sep 03 08:32:27 pumped-gator kata[113928]: time="2021-09-03T08:32:27.145103908Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:27 pumped-gator crio[47547]: time="2021-09-03T08:32:27.145103908Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:28 pumped-gator kata[113928]: | 08:33 |
| kata-irc-bot | time="2021-09-03T08:32:28.63741895Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:28 pumped-gator crio[47547]: time="2021-09-03T08:32:28.637418950Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:28 pumped-gator kata[113928]: time="2021-09-03T08:32:28.669816501Z" level=error msg="forward event" error="failed to connect: dial unix: missing | 08:33 |
| kata-irc-bot | address" Sep 03 08:32:28 pumped-gator crio[47547]: time="2021-09-03T08:32:28.669816501Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:29 pumped-gator kata[113928]: time="2021-09-03T08:32:29.145523144Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:29 pumped-gator crio[47547]: time="2021-09-03T08:32:29.145523144Z" level=error msg="forward | 08:33 |
| kata-irc-bot | event" error="failed to connect: dial unix: missing address" Sep 03 08:32:31 pumped-gator kata[113928]: time="2021-09-03T08:32:31.637713845Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:31 pumped-gator crio[47547]: time="2021-09-03T08:32:31.637713845Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:31 pumped-gator kata[113928]: | 08:33 |
| kata-irc-bot | time="2021-09-03T08:32:31.670108274Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:31 pumped-gator crio[47547]: time="2021-09-03T08:32:31.670108274Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 03 08:32:32 pumped-gator kata[113928]: time="2021-09-03T08:32:32.145970856Z" level=error msg="forward event" error="failed to connect: dial unix: missing | 08:33 |
| kata-irc-bot | address" Sep 03 08:32:32 pumped-gator crio[47547]: time="2021-09-03T08:32:32.145970856Z" level=error msg="forward event" error="failed to connect: dial unix: missing address"``` | 08:33 |
| kata-irc-bot | <tim.rice> still this | 08:33 |
| kata-irc-bot | <tim.rice> everything seems to be fine | 08:33 |
| kata-irc-bot | <tim.rice> but that ^ occurs on container startup/shutdown | 08:33 |
| kata-irc-bot | <tim.rice> also when execing into get a shell for the container | 08:33 |
| kata-irc-bot | <tim.rice> but the container seems to be working fine, has network etc | 08:34 |
| kata-irc-bot | <tim.rice> ```Sep 03 08:36:09 pumped-gator kata[113885]: time="2021-09-03T08:36:09.384480599Z" level=error msg="evicting /tasks/exit from queue because of retry count" Sep 03 08:36:09 pumped-gator crio[47547]: time="2021-09-03T08:36:09.384480599Z" level=error msg="evicting /tasks/exit from queue because of retry count" Sep 03 08:36:09 pumped-gator kata[113885]: time="2021-09-03T08:36:09.664890039Z" level=error msg="evicting /tasks/exit from | 08:36 |
| kata-irc-bot | queue because of retry count" Sep 03 08:36:09 pumped-gator crio[47547]: time="2021-09-03T08:36:09.664890039Z" level=error msg="evicting /tasks/exit from queue because of retry count" Sep 03 08:36:09 pumped-gator kata[113885]: time="2021-09-03T08:36:09.796945841Z" level=error msg="evicting /tasks/delete from queue because of retry count" Sep 03 08:36:09 pumped-gator crio[47547]: time="2021-09-03T08:36:09.796945841Z" level=error msg="evicting | 08:36 |
| kata-irc-bot | /tasks/delete from queue because of retry count"``` this also occurs when shutting down pods | 08:36 |
| kata-irc-bot | <tim.rice> | 08:39 |
| kata-irc-bot | <tim.rice> took a stacktrace from the shim, while getting a shell in the container | 08:39 |
| kata-irc-bot | <fidencio> So, wait, the problem is to start a pod or to exec into a pod? | 11:33 |
| kata-irc-bot | <fidencio> (anyways it is a problem, just trying to narrow down the scope of the problem) | 11:34 |
| kata-irc-bot | <fidencio> @fgiudici, would you have time to help @tim.rice to debug this on the CRI-O side? I am with my hands very full at this moment | 11:34 |
| kata-irc-bot | <tim.rice> the pod starts, execing into it works, stopping pods works, but every time those things happen the crio log is spammed with variations of "dial unix: missing address" error level log events | 11:41 |
| kata-irc-bot | <tim.rice> thanks for your time so far :slightly_smiling_face: | 11:41 |
| kata-irc-bot | <fidencio> Hmmm. That's interesting, quite interesting. Would you mind to open an issue on our github about that? I know, it's a boring process, but it'll help us and other who'll face the same issue that you're facing. | 11:43 |
| kata-irc-bot | <tim.rice> will do, i was hoping that this was something simple and i was making some obvious error hence coming to slack first :slightly_smiling_face: | 11:43 |
| kata-irc-bot | <fidencio> I wish, it looks like you're hitting a true issue. Out of curiosity, which CRI-O version are you using? | 11:45 |
| kata-irc-bot | <tim.rice> i did notice that there were some open file handlers from the shim under /run/containerd/s/<id> .... might not be relevant bu thought it was weird as we dont have containerd | 11:45 |
| kata-irc-bot | <tim.rice> 1.21.2 | 11:45 |
| kata-irc-bot | <fidencio> Yeah, shimv2 is a containerd thing, we even vendor some containerd bits on CRI-O in order to properly interact with that. | 11:48 |
| kata-irc-bot | <tim.rice> ah ok | 11:49 |
| kata-irc-bot | <fgiudici> Ok, so I found the cause of that error | 13:13 |
| kata-irc-bot | <tim.rice> oh? | 13:19 |
| kata-irc-bot | <fgiudici> The address arg in the katav2 shim binary is used to give a connection back to the containerd socket, connecting via a containerd client. The containerd client protocol is not spoken by CRI-O, and that part was never used in CRI-O. So, nothing should affect CRI-O - kata workloads. The address arg requires ... and address value. As there is no address speaking containerd client protocol in CRI-O the address is missing. | 13:21 |
| kata-irc-bot | <fgiudici> This was just ignored in 2.1 | 13:21 |
| kata-irc-bot | <fgiudici> in 2.2 we updated the vendored containerd code to v1.5.4 | 13:21 |
| kata-irc-bot | <fgiudici> When kata uses the containerd shim code to publish the event, now the check on the address is performed (it wasn't before) | 13:22 |
| kata-irc-bot | <fgiudici> and that error pops out. | 13:22 |
| kata-irc-bot | <tim.rice> ah, ok! makes sense i guess. so i can safely ignore these errors for now? | 13:23 |
| kata-irc-bot | <fgiudici> absolutely! | 13:24 |
| kata-irc-bot | <tim.rice> :relieved: | 13:24 |
| kata-irc-bot | <tim.rice> thanks very much. i didnt get around to creating an issue yet, would you like me to do that still? | 13:25 |
| kata-irc-bot | <fgiudici> Yeah, I think would be good, thanks Tim! Maybe we will need to deal (and track) this on the CRI-O side, but not sure yet :S So, please track the bug on kata and from there we will see where to address that :slightly_smiling_face: | 13:26 |
| kata-irc-bot | <fgiudici> :thinking_face: mmm, but honestly maybe better to open on the CRI-O project | 13:28 |
| kata-irc-bot | <fidencio> I think it'd be better to directly open this on the CRI-O side. | 13:28 |
| kata-irc-bot | <fgiudici> agreed | 13:28 |
| kata-irc-bot | <fidencio> @fgiudici, thanks a whole lot for jumping in and debugging the issue! | 13:28 |
| kata-irc-bot | <fgiudici> Thanks to @tim.rice for reporting it and providing info and logs! | 13:29 |
| kata-irc-bot | <fgiudici> Good to get this early | 13:29 |
| kata-irc-bot | <tim.rice> ok, ill open a issue at the CRI-O repo within a few hours :slightly_smiling_face: thanks again! | 13:36 |
| kata-irc-bot | <rlk> I haven't played with virtio-scsi (we're using virtiofs in OpenShift), but you're going to need to look at what's going on on the host side as well as the guest side. I suspect iostat works with virtio-scsi, since it presents a block device to the guest. You could use `iostat 5` on the guest to see how much I/O it's doing to `/dev/xda`. You might also want to look at `iostat` on the host, and `pidstat` on the host to see how busy the | 13:53 |
| kata-irc-bot | virtio-scsi helper is during both scenarios. | 13:53 |
| kata-irc-bot | <tim.rice> https://github.com/cri-o/cri-o/issues/5276 :thumbsup: | 17:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!