Friday, 2021-01-22

« Thursday, 2021-01-21 Index Saturday, 2021-01-23 »
*** fuentess has quit IRC00:31
*** dklyle has quit IRC02:54
*** david-lyle has joined #kata-dev02:54
*** auk has joined #kata-dev03:02
*** david-lyle has quit IRC03:38
kata-irc-bot1<fidencio> @bergwolf, let's get those branches renamed? :slightly_smiling_face: I'm going to send an email asking for no more merges to 2.0 and will drop a comment on every open PR about what we're going to do.  Once I do that, I'd like to ask you to just double-check you have all the commits from 2.0-dev plus yours, and we start the dance.  Sounds reasonable?06:35
kata-irc-bot1<bergwolf> yup, let's do it!06:35
kata-irc-bot1<fidencio> cool, let me brew some coffee and start with the email / comments!06:36
*** EricAdamsZNC has quit IRC06:52
kata-irc-bot1<fidencio> @bergwolf, may I merge to 2.0 the PRs that already already ready to be merged, with acks, and all tests passing and you do a last rebase when I finish going through the PRs?06:52
kata-irc-bot1<bergwolf> sure06:52
kata-irc-bot1<bergwolf> let me know when you are done06:52
*** auk has quit IRC07:09
kata-irc-bot1<fidencio> @bergwolf, `kata-containers/kata-containers` is ready to be switched, please, do the switch and I'll check the CI bits then07:12
*** fgiudici has joined #kata-dev07:12
kata-irc-bot1<bergwolf> cool, I've changed `kata-containers/kata-containers` default branch to `main` , and pushed `stable-2.0` there07:24
kata-irc-bot1<bergwolf> what about the `tests` repo? do we switch now or later?07:25
kata-irc-bot1<fidencio> It's okay to do it now as well, as I just finished commenting in the opened PRs for the affected branches.07:26
kata-irc-bot1<bergwolf> just saw this07:30
kata-irc-bot1<bergwolf>07:30
kata-irc-bot1<bergwolf> so we should just use the rename feature, it helps with existing PRs07:30
kata-irc-bot1<fidencio> Can we do that for the `kata-containers/kata-containers` as well?07:31
kata-irc-bot1<fidencio> If so, that would help *a lot*07:31
kata-irc-bot1<bergwolf> yeah, done on both branch07:33
kata-irc-bot1<bergwolf> both repos07:33
kata-irc-bot1<bergwolf> for `tests`, I updated the default branch to `2.0-dev` and then renamed it to `main`07:33
kata-irc-bot1<bergwolf> for `kata-containers`, I deleted the just pushed `main` branch, and then renamed `2.0-dev` to `main`07:34
kata-irc-bot1<bergwolf> please help to see how CI goes with the change07:34
kata-irc-bot1<fidencio> right, I'm doing the changes in the jenkins right now07:35
kata-irc-bot1<bergwolf> PRs are updated automatically, sweet~07:36
kata-irc-bot1<fidencio> That's amazing!07:36
kata-irc-bot1<fidencio> It'll take 30 minutes or so, on my side as it's really tedious work07:36
kata-irc-bot1<fidencio> and then I'll check whether the CI is working or not, and try to debug it07:36
kata-irc-bot1<bergwolf> finger crossed :slightly_smiling_face:07:37
kata-irc-bot1<fidencio> I know it's late for you, but I hope we have everything working in the next 2~3 hours07:37
kata-irc-bot1<bergwolf> nop, it's just 15:44 for me. very much early07:44
kata-irc-bot1<bergwolf> oops, just realised that now that I do the `rename` magic, `main` doesn't have my local commits...07:44
kata-irc-bot1<bergwolf> Should I push it directly or through PRs? now that `main` is protected07:45
kata-irc-bot1<bergwolf> @fidencio ^^07:46
kata-irc-bot1<fidencio> Go through the PRs as we can use that to test the CI :slightly_smiling_face:07:46
kata-irc-bot1<fidencio> Just, please, do not run `/test`  yet07:47
kata-irc-bot1<bergwolf> there you go https://github.com/kata-containers/kata-containers/pull/1311 https://github.com/kata-containers/tests/pull/318207:53
kata-irc-bot1<fidencio> I wonder if I should say "let's see if CI passes"  or "let's see which CI I forgot to change"07:54
kata-irc-bot1<bergwolf> would you be able to trigger CI on the tests repo PR? It has a depend-on label so it is supposed to be able to test all bits07:54
kata-irc-bot1<bergwolf> let's see if CI likes it ;)07:54
kata-irc-bot1<fidencio> done!07:55
kata-irc-bot1<fidencio> now we sit, wait, and take more coffee :slightly_smiling_face:07:55
*** sgarzare has joined #kata-dev07:55
kata-irc-bot1<fidencio> I've figured out that I should *also* put `main` in the list of branches were we don't run 1.x CI :slightly_smiling_face:08:04
kata-irc-bot1<fidencio> And here we go again08:04
kata-irc-bot1<bergwolf> ```15:57:49 /tmp/jenkins/workspace/kata-containers-tests-cri-containerd-PR/go/src/github.com/kata-containers/tests/.ci/resolve-kata-dependencies.sh: line 40: repos_found[@]: unbound variable``` depends-on label parsing is broken?08:07
kata-irc-bot1<bergwolf> hmm, working if I take out these lines and test locally08:11
kata-irc-bot1<bergwolf> any idea what's wrong with CI?08:11
kata-irc-bot1<fidencio> Don't act on that yet, cri-containerd was not supposed to be run against 2.x08:13
kata-irc-bot1<fidencio> But I forgot to do a `s/2.0-dev/main` in the blocked branches08:13
kata-irc-bot1<fidencio> Now I'm going through all the tests jobs we have for 1.x doing this08:13
kata-irc-bot1<fidencio> And I will re-run the CI08:13
kata-irc-bot1<bergwolf> ok08:14
kata-irc-bot1<fidencio> It's just a slow and tedious clicket-clacket process08:14
kata-irc-bot1<bergwolf> yup, got to test/fix/test/fix/test/fix/test/fix08:15
*** sameo has joined #kata-dev08:29
*** jodh has joined #kata-dev08:38
kata-irc-bot1<fidencio> @bergwolf, what happened was: For the 2.x CIs we had `2.0-dev` branch as part of an allow-list, while for the 1.x CIs we had `2.0-dev` branch as part of a block-list. I knew about the former, but not about the latter, and the latter is what caused all those failures in the test branch.08:39
kata-irc-bot1<fidencio> I'm not sure if a simple  `/test` will be able to remove those jobs that were wrongly triggered from the list or if a new PR would be needed to do so08:40
kata-irc-bot1<fidencio> I guess I know a workaround08:42
kata-irc-bot1<fidencio> ah, no, closing and reopening the PR doesn't fix this08:43
kata-irc-bot1<bergwolf> oh, then how can we fix the latter?08:48
kata-irc-bot1<bergwolf> who knows how to the latter config?08:49
kata-irc-bot1<fidencio> I do know and I changed it already08:50
kata-irc-bot1<fidencio> But the failed jobs won't disappear from the existing PR08:50
kata-irc-bot1<fidencio> would you mind to close the current PR you have and open a new one with the same content, just so I can try that `/test` does what it's supposed to do? :slightly_smiling_face:08:51
kata-irc-bot1<fidencio> (only for the `kata-containers/test` repo08:51
kata-irc-bot1<bergwolf> sure, let's try that08:59
kata-irc-bot1<bergwolf> https://github.com/kata-containers/tests/pull/318309:01
kata-irc-bot1<bergwolf> those are still there09:01
kata-irc-bot1<fidencio> argh, mind if I try to open a PR with your patches? (Of course, I'll keep your authorship)09:05
kata-irc-bot1<bergwolf> please just go ahead!09:06
kata-irc-bot1<bergwolf> I don't mind it at all :slightly_smiling_face:09:06
kata-irc-bot1<fidencio> https://github.com/kata-containers/tests/pull/318409:11
kata-irc-bot1<fidencio> seems that github was smart enough to detect your new PR as the old PR09:11
kata-irc-bot1<fidencio> Now the list there seems correct09:11
kata-irc-bot1<fidencio> \o/  Let's just wait and see :slightly_smiling_face:09:11
*** iamweswilson has quit IRC09:14
*** iamweswilson has joined #kata-dev09:15
*** ailan__ has quit IRC09:18
kata-irc-bot1<fidencio> I had missed ubuntu-initrd ones, fixed now.09:48
kata-irc-bot1<fidencio> And one bit in the vfio ci.09:54
kata-irc-bot1<fidencio> Running those again, and we should be good to go!09:54
kata-irc-bot1<bergwolf> bravo!10:01
kata-irc-bot1<bergwolf> you rock, man!10:01
kata-irc-bot1<fidencio> A whole bunch of Kudos to be also distributed to @gabriela.cervantes.te, @jose.carlos.venegas.m, and @salvador.fuentes for previously sharing the CI knowledge \o/10:05
kata-irc-bot1<bergwolf> vfio ci is still checking out 2.0-dev ```18:26:35 ++ cd /home/jenkins/workspace/workspace/kata-containers-2.0-vfio-PR/go/src/github.com/kata-containers/kata-containers 18:26:35 ++ git checkout 2.0-dev 18:26:35 error: pathspec '2.0-dev' did not match any file(s) known to git.```10:30
kata-irc-bot1<bergwolf> rebuilt now that we have merged the tests PR. let's see if that fixes it10:31
kata-irc-bot1<fidencio> Exactly what I was about to say :slightly_smiling_face:10:32
kata-irc-bot1<bergwolf> //hand10:32
kata-irc-bot1<fidencio> I think it was happening only because we didn't have the tests PR merged10:32
kata-irc-bot1<bergwolf> yup, TBH I was surprised that we can pass that much CIs on kata-containers repo without merging the tests PR first.10:33
kata-irc-bot1<fidencio> That's because the vfio was the only one explicitly referring to 2.0-dev branch10:35
kata-irc-bot1<fidencio> The other jobs, AFAIU, they were only marked to run a few of specific branch, and that I changed on the jenkins side10:35
kata-irc-bot1<fidencio> Food arrived here, I'll have lunch, merge that PR once I come back, fire another email to the ML, and go through the PRs that I've commented on.10:36
kata-irc-bot1<fidencio> Technically we're back to business, socially we need one or two hours. :slightly_smiling_face:10:36
kata-irc-bot1<bergwolf> cool, it's smoother than I had expected. thanks man! I'll drop off for dinner as well. ttyl.10:38
kata-irc-bot1<fidencio> @bergwolf, did you rename the stable-2.0.0 to stable-2.0 on both tests and kata-containers repos?12:04
kata-irc-bot1<fidencio> Looking at the tests repo I can see both stable-2.0.0 and stable-2.0, is this expected?12:06
kata-irc-bot1<fidencio> because old PRs still point to stable-2.0.012:06
kata-irc-bot1<bergwolf> ah, rename only works for default branch. I have to delete these stable-2.0.0 branches12:08
kata-irc-bot1<bergwolf> and we need to ask the PR authors to reopen targeting the new stable-2.0 branch12:09
kata-irc-bot1<fidencio> Oh, okay, I didn't know about that.12:10
kata-irc-bot1<fidencio> Let me comment on those PRs targeting the stable-2.0 branch12:10
kata-irc-bot1<fidencio> wait ...12:11
kata-irc-bot1<fidencio> can we make stable-2.0.0 the default one? rename it, and then make `main` the default one again?12:11
kata-irc-bot1<fidencio> @bergwolf, if we can do that, that would be better12:12
kata-irc-bot1<bergwolf> oops, already deleted12:12
kata-irc-bot1<fidencio> okay, that's fine :slightly_smiling_face:12:12
kata-irc-bot1<bergwolf> hopefully we don't have many pending PRs to the stable branch12:12
kata-irc-bot1<bergwolf> we have just tagged the stable releases12:13
kata-irc-bot1<fidencio> yep, there's one PR from @wmoschet that's still valid and one that may not be valid from @salvador.fuentes.12:15
kata-irc-bot1<bergwolf> ok, sorry for the trouble.12:15
kata-irc-bot1<fidencio> not a trouble at all!12:16
kata-irc-bot1<fidencio> last but not least, I'll drop an email to kata-dev ML and announce the changes and how they can fix their stuff locally12:16
kata-irc-bot1<fidencio> @bergwolf, sincerely, thanks a lot for the help on this!12:16
kata-irc-bot1<bergwolf> yeah, sounds good to me. and thank you for working on most of it ;)12:17
kata-irc-bot1<bergwolf> please send the mail and if no one complains, we are all set :)12:17
*** devimc has joined #kata-dev12:58
*** crobinso has joined #kata-dev14:22
*** fuentess has joined #kata-dev14:52
*** mugsie has joined #kata-dev15:09
*** mugsie has quit IRC15:18
*** devimc has quit IRC15:31
*** devimc has joined #kata-dev15:32
*** dklyle has joined #kata-dev15:44
*** mugsie has joined #kata-dev15:51
*** egernst has joined #kata-dev15:56
*** egernst has quit IRC16:06
*** egernst has joined #kata-dev16:06
kata-irc-bot1<christophe> Does `ctr` returning `ctr: Failed to check if grpc server is working: unexpected EOF: unknown`  ring a bell for anyone? Looking at the logs, I see ```level=info msg="scanner return error: read unix @->/run/vc/vm/test-kata/qmp.sock: use of closed network connection" pid=282133 sandbox=test-kata source=virtcontainers subsystem=qmp```16:13
egernstcmdline?16:14
kata-irc-bot1<christophe> After that some complaints about failing to cleanup `/var/run/netns/cnitest-...` "No such file or directory".16:14
kata-irc-bot1<christophe> `ctr --debug run --rm --runtime io.containerd.kata.v2 -t $image test-kata cat /proc/cmdline`16:14
kata-irc-bot1<christophe> @eric.ernst ^16:14
kata-irc-bot1<christophe> It works when I use `runc.v1`  or `runc.v2`16:15
egernst...seems like an issue in your configuration -- that seems pretty basic "run a container" command....16:18
egernst...let me check..16:18
kata-irc-bot1<christophe> I guess it's a configuration issue, but I'm not sure which part is misconfigured. The qemu command line looks sane. ```launching /usr/bin/qemu-kvm with: [-name sandbox-test-kata -uuid 3fa55063-9c80-47c4-960e-5a0d302c95bb -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/test-kata/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=8698M -device16:19
kata-irc-bot1pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile=,max_ports=2 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/test-kata/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile=16:19
kata-irc-bot1-device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1010924195,guest-cid=1010924195,romfile= -chardev socket,id=char-0b4e08a92af937f8,path=/run/vc/vm/test-kata/vhost-fs.sock -device vhost-user-fs-pci,chardev=char-0b4e08a92af937f8,tag=kataShared,romfile= -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object16:19
kata-irc-bot1memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1 -kernel /usr/lib/modules/5.10.0-0.rc4.20201117git9c87c9f41245.79.fc34.x86_64/vmlinuz -initrd /var/cache/kata-containers/osbuilder-images/5.10.0-0.rc4.20201117git9c87c9f41245.79.fc34.x86_64/fedora-kata-5.10.0-0.rc4.20201117git9c87c9f41245.79.fc34.x86_64.initrd -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=116:19
kata-irc-bot1i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=4 scsi_mod.scan=none systemd.unified_cgroup_hierarchy=0 -pidfile /run/vc/vm/test-kata/pid -smp 1,cores=1,threads=1,sockets=4,maxcpus=4```16:19
kata-irc-bot1<christophe> Oh, I know what it might be. I think I may have a v1 image in place instead of v2 (I'm switching back and forth between the two). Let me check that.16:21
kata-irc-bot1<christophe> Nope. Rebuilt the image, same result.16:24
kata-irc-bot1<christophe> OK. Need to go for a doctor appointment. Will keep investigating. Add that to my list "better errors in case of misconfiguration" wishlist ;)16:25
*** pcaruana has quit IRC16:26
kata-irc-bot1<eric.ernst> virtio-fs / lack of inotify :thread:17:05
kata-irc-bot1<eric.ernst> JFYI: started a thread on ML to bring up what @manohar.r.castelino had highlighted last year -- lack of inotify support for virtiofs.17:05
kata-irc-bot1<eric.ernst> https://www.redhat.com/archives/virtio-fs/2021-January/msg00065.html17:05
kata-irc-bot1<eric.ernst> I think we should either clearly mark this as a limitation, or we should come up with sugggested workarounds. Ideally this can be a stop gap and someone in the greater community can work to update FUSE to support this.17:06
*** sgarzare has quit IRC17:07
kata-irc-bot1<efoster> This would definitely be a nice thing to have! fluent-bit gets very unhappy when it doesn't have inotify to inform it of file changes. You either need to compile it without inotify or move to that follows files more traditionally, like Vector.17:14
kata-irc-bot1<fidencio> @eric.ernst, do we have this tracked on Kata side?17:39
kata-irc-bot1<eric.ernst> just here: https://github.com/kata-containers/runtime/issues/150517:40
kata-irc-bot1<fidencio> I've looped Dave and Vivek there, thanks!17:42
*** EricAdamsZNC has joined #kata-dev17:44
*** jodh has quit IRC17:57
*** fgiudici has quit IRC18:16
kata-irc-bot1<eric.ernst>19:32
kata-irc-bot1<eric.ernst> That's great -- nice work @fidencio, @bergwolf19:32
*** crobinso has quit IRC21:06
*** devimc has quit IRC21:51
*** sameo has quit IRC22:21
*** fuentess has quit IRC23:04
« Thursday, 2021-01-21 Index Saturday, 2021-01-23 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!