| *** fuentess has quit IRC | 02:46 | |
| *** sameo has joined #kata-dev | 06:18 | |
| kata-irc-bot | <samuel.ortiz> @fidencio overlayfs means we need to use virtiofs to expose the container image, while devicemapper means virtio-blk. | 06:26 |
|---|---|---|
| kata-irc-bot | <samuel.ortiz> @fidencio | 06:26 |
| *** dklyle has quit IRC | 07:16 | |
| *** sgarzare has joined #kata-dev | 08:01 | |
| kata-irc-bot | <fidencio> I was missing that piece, makes sense. Thanks @samuel.ortiz. | 08:05 |
| *** jodh has joined #kata-dev | 08:19 | |
| *** davidgiluk has joined #kata-dev | 09:01 | |
| *** pmores has joined #kata-dev | 09:06 | |
| pmores | Hi! I'm looking into merging https://github.com/kata-containers/runtime/pull/3127 . There are errors reported in CI logs (e.g. http://jenkins.katacontainers.io/job/kata-containers-runtime-ubuntu-18-04-virtiofs-PR/851/console). Some of them do look interesting (to me at least) however they either don't seem to cause the tests fail or don't seem to be related to the PR in question. | 09:13 |
| pmores | How to proceed in such a case? | 09:13 |
| kata-irc-bot | <fidencio> Let's take a look at the error: ```18:45:15 docker: Error response from daemon: OCI runtime create failed: virtiofs daemon /usr/bin/virtiofsd returned with error: fork/exec /usr/bin/virtiofsd: no such file or directory: unknown.``` | 09:21 |
| kata-irc-bot | <fidencio> Looking way up in the logs it says: ```18:42:05 INFO: Installing cached QEMU 18:42:06 0.1% 1.1% ###### 9.2% ################################ 45.2% | 09:24 |
| kata-irc-bot | ########################################################## 81.7% ######################################################################## 100.0% 18:42:06 kata-static-qemu.tar.gz: OK``` | 09:24 |
| kata-irc-bot | <fidencio> Then, taking a look at the tar.gz, from http://jenkins.katacontainers.io/job/qemu-nightly-x86_64/, we can see: ```fidencio@quino ~ $ tar tf ~/Descargas/kata-static-qemu.tar.gz | grep virtiofsd usr/share/kata-qemu/qemu/vhost-user/50-qemu-virtiofsd.json usr/libexec/kata-qemu/virtiofsd``` | 09:25 |
| kata-irc-bot | <fidencio> So, at least now we know the reason why it fails. | 09:25 |
| kata-irc-bot | <fidencio> Now, going back to your original question, "How to proceed in such a case?" I'd recommend opening an issue against the tests repo, explicitly mentioning which tests is failing, from which CI, and with the info provided above. | 09:26 |
| kata-irc-bot | <fidencio> Then wait till it's solved before merging #3127 | 09:27 |
| kata-irc-bot | <fidencio> Does this sound reasonable? | 09:27 |
| kata-irc-bot | <fidencio> Folks, if I can get a quick review on https://github.com/kata-containers/community/pull/199, I'd greatly appreciate! | 09:54 |
| fidencio | jodh: ^ | 09:55 |
| kata-irc-bot | <fidencio> @fupan ^^ :slightly_smiling_face: | 09:55 |
| kata-irc-bot | <fupan> ? | 09:56 |
| kata-irc-bot | <fidencio> If you can give me a quick review on https://github.com/kata-containers/community/pull/199 | 09:57 |
| kata-irc-bot | <fidencio> Just to get the process going :slightly_smiling_face: | 09:57 |
| fidencio | jodh: thanks! | 09:57 |
| pmores | fidencio: okay, so the policy is don't merge even if errors aren't related, right? | 09:59 |
| fidencio | pmores: the policy is to have the issues reported, at least. | 10:00 |
| fidencio | pmores: I'd prefer not merging, some may prefer to merge | 10:01 |
| fidencio | pmores: but the issue being reported is a must | 10:01 |
| pmores | I mean, besides the one you mentioned, there's a lot of other errors related to e.g. tracing, TestCleanupContainer cannot remove rootfs dir etc. | 10:02 |
| kata-irc-bot | <fupan> Done | 10:03 |
| fidencio | pmores: we have to understand what are the errors causing the abortion of the CI, and what are the logged "unharmful" content | 10:03 |
| kata-irc-bot | <fidencio> Lovely, thanks! | 10:03 |
| pmores | (Of course I'll report the error, no question about that. Just trying to understand the overall process.) | 10:03 |
| fidencio | pmores: I guess it takes some time to navigate through the CI errors, and figure out what's causing the breakage | 10:04 |
| fidencio | pmores: it took for me, at least | 10:04 |
| fidencio | pmores: that's one of the proposals that Cameron raised, about improving that for contributors to easily see what went wrong with the CI | 10:05 |
| fidencio | but people didn't have cycle to start working on that, at least not yet | 10:05 |
| pmores | Yeah, got it. | 10:05 |
| pmores | BTW how about the last reported error: ERROR: kata-log-parser: missing pid: {Count:0 TimeDelta:0 Filename:/tmp/jenkins/workspace/kata-containers-runtime-ubuntu-18-04-virtiofs-PR/go/src/github.com/kata-containers/tests/kata.log Line:1 Time:2021-02-23 17:43:33.932314193 +0000 UTC Pid:0 Level:warning Msg:Feature to allow containers to share PID namespace with the agent has been enabled. Please understand this has security implications | 10:06 |
| pmores | and should only be used for debug purposes Source: Name: Container: Sandbox: Data:map[]} | 10:06 |
| pmores | Isn't this the one that actually caused failure? | 10:06 |
| fidencio | no, that's just the teardown failing | 10:11 |
| kata-irc-bot | <fidencio> For instance, when we look at: `````` | 10:11 |
| kata-irc-bot | <fidencio> ```18:45:15 docker: Error response from daemon: OCI runtime create failed: virtiofs daemon /usr/bin/virtiofsd returned with error: fork/exec /usr/bin/virtiofsd: no such file or directory: unknown. 18:45:15 Makefile:237: recipe for target 'conformance' failed 18:45:15 make: *** [conformance] Error 127 18:45:15 Failed at 121: sudo -E PATH="$PATH" bash -c "make conformance" 18:45:15 Build step 'Execute shell' marked build as failure``` | 10:11 |
| pmores | ah okay, thanks | 10:11 |
| kata-irc-bot | <fidencio> We see this: ```18:45:15 Build step 'Execute shell' marked build as failure``` | 10:11 |
| kata-irc-bot | <fidencio> This is important information, this is what marked the whole job as a failure | 10:12 |
| kata-irc-bot | <fidencio> Whatever happens after that, that's not that important (generally speaking) | 10:12 |
| kata-irc-bot | <fidencio> Makes sense? | 10:12 |
| fidencio | (and sorry for mixing IRC and Slack, some things are easier to type on IRC, some others on Slack) | 10:13 |
| pmores | Yeah, makes sense. | 10:13 |
| pmores | fidencio: hm, I somehow can't find the failing test in kata-containers or tests repos ('main' branch in both cases)... | 10:27 |
| fidencio | pmores: yep, things are not so easy to find / navigate. First thing we have to keep in mind, this test case comes from kata-containers/runtime, which means kata-containers/tests should be using `master` branch (yeah, confusing, I know) | 10:29 |
| fidencio | from the logs, seems that what's failing is `make conformance` | 10:30 |
| fidencio | `make conformance` is called from `kata-containers/tests/.ci/run.sh` | 10:31 |
| fidencio | from `master` branch | 10:31 |
| pmores | alright, at least I can now see the Dockerfile that the test uses! :-) | 10:33 |
| fidencio | pmores: so, let's go a little bit deepr | 10:34 |
| fidencio | *deeper | 10:34 |
| fidencio | pmores: we know the failure comes from `make conformance`, if yoiu open the `.ci/run.sh` script and search for that, you'll see it in several places | 10:34 |
| fidencio | you also pointed out that the failing job is "virtiofs-PR", which leads us to the line 118, the "VIRTIOFS" case in that switch. | 10:35 |
| fidencio | and, as you alread pointed out, from the Makefile you can get to the `conformance` folder | 10:36 |
| fidencio | where you can find the Dockerfile | 10:36 |
| fidencio | So, my *guess* here, and it's just a *guess* | 10:36 |
| pmores | hang on, I think I'm confused - is this the 2.0 kata-containers repo, or the 1.x tests repo? | 10:37 |
| fidencio | `master` branch, is 1.x | 10:38 |
| fidencio | pmores: what are you confused with? | 10:39 |
| pmores | fidencio: you said I should see 'make conformance' several times in .ci/run.sh. I can see it only once in my run.sh. So I'm probably looking at a different one. | 10:41 |
| fidencio | pmores: https://github.com/kata-containers/tests/blob/master/.ci/run.sh | 10:42 |
| pmores | I'm looking at 1.x tests/.ci/run.sh, branch 'master'. | 10:42 |
| fidencio | how old is your clone? | 10:42 |
| pmores | I've just pulled. I'm looking into it, give me a second... | 10:44 |
| pmores | okay, I think I'm OK now (not quite sure what was wrong but apparently it was nothing that 'git reset --hard' and 'git pull' couldn't solve ;-)) | 10:47 |
| fidencio | pmores: so, shall I proceed? | 10:48 |
| pmores | yes please :-) | 10:48 |
| fidencio | pmores: basically, what I *think* that's going wrong here, and that's just a *guess*, is that when we're building Kata Containers, we're not passing the correct value for virtiofsd location | 10:50 |
| fidencio | this is handled here: https://github.com/kata-containers/tests/blob/master/.ci/install_runtime.sh | 10:51 |
| pmores | Right. That's a good hint, I can take it from here and investigate. | 10:51 |
| fidencio | See that it uses runtime_config_path="${SYSCONFDIR}/kata-containers/configuration.toml" | 10:51 |
| fidencio | I don't know what's the content of that file, or where that file comes from (most like from `make ` | 10:52 |
| pmores | cool, don't bother fixing it all for me, I'll have a look and only ask questions if I get stuck - thanks a lot! | 10:53 |
| fidencio | pmores: np. last tip: https://github.com/kata-containers/runtime/blob/59e227336903383fcb04e0075e0b55cbd98c42bb/cli/config/configuration-qemu-virtiofs.toml.in#L123-L124 | 10:54 |
| fidencio | pmores: that's the part that contains the wrong information | 10:54 |
| fidencio | pmores: something sets DEFVIRTIOFSDAEMON as /usr/bin/virtiofsd, while we saw the binary is actually at /usr/libexec/kata-qemu/virtiofsd | 10:55 |
| fidencio | I'd recommend you check the whole chain, as it surprises me quite a lot that it's looking for the correct path for QEMU itself. | 10:55 |
| fidencio | pmores: have fun! :-) | 10:55 |
| pmores | so you won't rest... ;-D | 10:56 |
| fidencio | pmores: by the way, I also think that the questions you are asking should have been asked by some sort of FAQ for the herders | 10:57 |
| fidencio | pmores: and I'd appreciate if you could take some time and prepare that | 10:57 |
| *** devimc has joined #kata-dev | 13:04 | |
| davidgiluk | devimc: Get anywhere with your seg? | 13:20 |
| devimc | davidgiluk, nop :(, I can't reproduce it using `strace -vv -tt --ff virtiofsd` nor virtiofsd-debug (binary with debug symbols), maybe the problem is the optimization / and or a race condition .. ? | 13:31 |
| devimc | davidgiluk, I'm still on that | 13:31 |
| devimc | davidgiluk, https://paste.centos.org/view/453b003c | 13:54 |
| *** fuentess has joined #kata-dev | 14:04 | |
| *** crobinso has joined #kata-dev | 14:21 | |
| davidgiluk | devimc: Ah so you're running with the dax world, which is worth noting | 14:51 |
| devimc | davidgiluk, no, dax is disabled | 14:52 |
| davidgiluk | ok | 14:52 |
| devimc | davidgiluk, I'm still trying to get a coredump but centos is kidd(ll)ing me | 14:54 |
| davidgiluk | devimc: Explain the problem to centos in an assertive manner | 14:57 |
| davidgiluk | devimc: You could try using -o sandbox=chroot and attaching gdb to the daemon | 14:57 |
| devimc | davidgiluk, really hard to do that, since this happen randomly | 14:58 |
| davidgiluk | devimc: Ah, I thought it was happening on every pod run | 14:59 |
| devimc | davidgiluk, no :( | 15:00 |
| davidgiluk | devimc: Is your core problem where it lands or what? | 15:01 |
| devimc | davidgiluk, yeah, I don't see it | 15:03 |
| devimc | davidgiluk, I followed this guide https://pve.proxmox.com/wiki/Enable_Core_Dump_systemd | 15:03 |
| devimc | but nothing - nada | 15:03 |
| davidgiluk | devimc: I'm an old school type, try setting /proc/sys/kernel/core_pattern to /tmp/core_%p | 15:04 |
| davidgiluk | devimc: The problem is that may land you in the /tmp of the container or of the chroot or ...wth knows | 15:05 |
| devimc | davidgiluk, # cat /proc/sys/kernel/core_pattern | 15:06 |
| devimc | /var/lib/systemd/coredump/core-%e-%s-%u-%g-%p-%t | 15:06 |
| davidgiluk | devimc: Are you sure you didn't miss a '|' at the beginning of that? | 15:07 |
| davidgiluk | oh hmm | 15:07 |
| davidgiluk | that's different from mine | 15:07 |
| davidgiluk | devimc: OK, sthe systemd way sseems to be |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h | 15:08 |
| davidgiluk | devimc: then I guess it pops out in coredumpctl | 15:08 |
| devimc | davidgiluk, # coredumpctl | 15:09 |
| devimc | No coredumps found. | 15:09 |
| devimc | davidgiluk, # systemctl --failed | 15:14 |
| devimc | UNIT LOAD ACTIVE SUB DESCRIPTION | 15:14 |
| devimc | ● kdump.service loaded failed failed Crash recovery kernel arming | 15:14 |
| devimc | is this the issue? | 15:14 |
| devimc | do I need kdump ? | 15:14 |
| davidgiluk | not unless you manage to nuke your host | 15:15 |
| davidgiluk | devimc: Which centos version? | 15:16 |
| devimc | davidgiluk, 7 | 15:16 |
| devimc | the infamous 7 | 15:16 |
| davidgiluk | hmm that's a bit crusty | 15:17 |
| davidgiluk | devimc: I've not tested on 7 at all; I normally test on fedora and colleagues depend on it on rhel 8 | 15:19 |
| kata-irc-bot | <eric.ernst> “Infamous 7” :D | 15:30 |
| kata-irc-bot | <eric.ernst> Updated kernel at least devimc? | 15:30 |
| devimc | @<eric.ernst I will try that | 15:31 |
| *** dklyle has joined #kata-dev | 15:46 | |
| pmores | Where can I find the latest kata-static nightly tarball for 2.0? | 16:18 |
| *** devimc has quit IRC | 16:24 | |
| *** devimc has joined #kata-dev | 16:24 | |
| *** sgarzare has quit IRC | 17:08 | |
| *** dklyle has quit IRC | 17:53 | |
| *** jodh has quit IRC | 18:05 | |
| *** dklyle has joined #kata-dev | 18:13 | |
| *** sameo has quit IRC | 19:04 | |
| devimc | davidgiluk, around? | 19:15 |
| devimc | finally got a core | 19:16 |
| devimc | 1.6 G | 19:16 |
| *** sameo has joined #kata-dev | 19:27 | |
| davidgiluk | devimc: Not bad! | 19:34 |
| davidgiluk | devimc: Now, can you get a backtrace? | 19:35 |
| devimc | davidgiluk, https://paste.centos.org/view/1b1cb8c5 | 19:35 |
| devimc | davidgiluk, I can compress it and send you | 19:36 |
| devimc | davidgiluk, 491K compressed | 19:36 |
| davidgiluk | that's a surprising thing to fail isn't it | 19:37 |
| devimc | davidgiluk, yeah, a segfault is too bad | 19:37 |
| devimc | davidgiluk, this is the frame 1 | 19:37 |
| devimc | https://paste.centos.org/view/807724f3 | 19:37 |
| davidgiluk | devimc: can you get the output of t a a bt full | 19:38 |
| devimc | davidgiluk, https://paste.centos.org/view/1649c93a | 19:40 |
| davidgiluk | devimc: OK, that makes a lot more sense | 19:40 |
| devimc | davidgiluk, https://paste.centos.org/view/e541129e | 19:41 |
| devimc | this looks better | 19:41 |
| devimc | without pager | 19:41 |
| davidgiluk | devimc: Is this happening just as the 2nd container in the pod starts up? | 19:42 |
| devimc | davidgiluk, yes | 19:43 |
| devimc | incepthi2-fpnhp-deployment-6f749d84b4-x8zzm 1/2 CrashLoopBackOff 8 32m | 19:43 |
| devimc | davidgiluk, the first container (workload/midroservice) is ready | 19:43 |
| davidgiluk | devimc: OK, so I see what's happening - but I need to go and figure out whether we thought we had code in there already to stop it :-) | 19:43 |
| davidgiluk | devimc: What's happening is that in one thread you're procesisng a request to open a file, and then passing the result back on the virtqueue - boring normal | 19:43 |
| davidgiluk | devimc: On the other thread, the hypervisor has just told you that someone has hotplugged some memory and your memory map has changed | 19:44 |
| davidgiluk | devimc: that other thread is just in the process of remapping all the queues at the same time as the 1st thread is writing into it | 19:44 |
| devimc | ouch! | 19:45 |
| devimc | davidgiluk, it's good to know what's happening :) | 19:46 |
| davidgiluk | devimc: I suspect it's more likely to happen with CloudHypervisor because there was a new call (that I don't think CH got yet) that means 'here's an extra mapping' rather than 'hey here's all your mappings' | 19:46 |
| devimc | and that you know how to fix it :) | 19:46 |
| devimc | davidgiluk, you are right, this is easier to reproduce with CLH | 19:47 |
| davidgiluk | devimc: Yeh, so I thought we had a lock somewhere that we took when we were nuking the queues; but perhaps I'm remembering thinking that we should have a lock.... | 19:48 |
| davidgiluk | devimc: I'm going to create a gitlab issue for it for now and then I need to look at our locking | 19:51 |
| devimc | davidgiluk, thanks | 19:52 |
| davidgiluk | devimc: https://gitlab.com/virtio-fs/qemu/-/issues/25 | 19:52 |
| devimc | davidgiluk, cool | 19:53 |
| devimc | thanks again | 19:53 |
| davidgiluk | devimc: the setmemtable call is a *horrible* interface; it tells you all of your mappings; when you hotplug you get a new setmemtable call, but no indication of whether there's a correspondence between old/new | 19:54 |
| *** davidgiluk has quit IRC | 20:23 | |
| kata-irc-bot | <fidencio> @jose.carlos.venegas.m, @salvador.fuentes, have you ever seen those errors before? ```"error opening storage: /dev/sdb is already part of a volume group \"storage\": must remove this device from any volume group or provide a different device"``` | 20:30 |
| kata-irc-bot | <fidencio> This is happening with CRI-O CI | 20:30 |
| kata-irc-bot | <fidencio> http://jenkins.katacontainers.io/job/kata-containers-crio-PR/16570/console | 20:31 |
| kata-irc-bot | <fidencio> Looks like, for some reason, we're not deleting the configuration of the storage | 20:33 |
| *** th0din has quit IRC | 21:01 | |
| *** th0din has joined #kata-dev | 21:05 | |
| kata-irc-bot | <jose.carlos.venegas.m> I think I have seen that in the past | 21:11 |
| kata-irc-bot | <fidencio> We noticed this now that we switched the CRI-O CI to use Ubuntu 18.04 | 21:12 |
| kata-irc-bot | <jose.carlos.venegas.m> @salvador.fuentes isn't an attached disk in the VM ? | 21:12 |
| kata-irc-bot | <fidencio> It is! | 21:12 |
| kata-irc-bot | <jose.carlos.venegas.m> @fidencio oh before was fedora? or ... centos? | 21:13 |
| kata-irc-bot | <fidencio> ```export LVM_DEVICE=/dev/sdb``` | 21:13 |
| kata-irc-bot | <jose.carlos.venegas.m> then it may be the VM config ? | 21:13 |
| kata-irc-bot | <fidencio> It was Ubuntu 16.04 | 21:13 |
| kata-irc-bot | <fidencio> I just updated the image from 16.04 to 18.04 | 21:14 |
| kata-irc-bot | <jose.carlos.venegas.m> @fidencio let me ping @salvador.fuentes or @gabriela.cervantes.te | 21:14 |
| kata-irc-bot | <gabriela.cervantes.te> no I have not see that error before | 21:18 |
| kata-irc-bot | <jose.carlos.venegas.m> @fidencio see in http://jenkins.katacontainers.io/configureClouds/ | 21:21 |
| kata-irc-bot | <fidencio> But that was a good guess. :slightly_smiling_face: | 21:26 |
| kata-irc-bot | <eric.ernst> devimc and davidgulik - nice work! | 21:28 |
| kata-irc-bot | <fidencio> I will keep debugging this Tomorrw, thanks Carlos, Chava, and Gaby! | 21:40 |
| kata-irc-bot | <jose.carlos.venegas.m> @fidencio chat with you tomorrow! | 21:41 |
| *** devimc has quit IRC | 22:02 | |
| *** sameo has quit IRC | 22:05 | |
| *** crobinso has quit IRC | 22:11 | |
| *** pmores has quit IRC | 22:44 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!