Thursday, 2019-04-04

« Wednesday, 2019-04-03 Index Friday, 2019-04-05 »
*** lcastell has quit IRC00:21
*** igordc has quit IRC00:36
*** auk has joined #kata-general01:00
*** sameo has joined #kata-general04:49
*** tmhoang has joined #kata-general06:45
*** sameo has quit IRC06:46
*** auk has quit IRC06:51
*** sgarzare has joined #kata-general06:57
*** auk has joined #kata-general07:01
*** auk has quit IRC07:08
*** sameo has joined #kata-general07:28
kata-irc-bot<voje> hey, this is a very general question so I'll post it here in general :slightly_smiling_face:08:07
kata-irc-bot<voje> how do kata containers perform in cloud (tunnelled kvm)08:08
kata-irc-bot<voje> are they usable inside VMs or is the preferred use case to be run on bare-metal?08:08
kata-irc-bot<voje> thanx08:08
*** gwhaley has joined #kata-general08:17
kata-irc-bot<graham.whaley> Hi @voje - they should perform fine in nested-kvm mode, as long as you have nesting enabled ;) iirc, the nested VM won't be able to use the EPT (Extended Page Tables) VM acceleration, so memory intensive workloads might not be quite as optimal. Functionally, nesting kata in a cloud vm host works just the same as on bare metal.08:36
kata-irc-bot<voje> @graham.whaley that's good to know, thatx for the info :slightly_smiling_face:09:34
kata-irc-bot<xu> some data for you, the penalty on memory by nested-vt09:46
*** tmhoang_ has joined #kata-general11:01
*** gwhaley has quit IRC11:02
*** tmhoang has quit IRC11:04
*** tmhoang_ has quit IRC11:05
*** tmhoang_ has joined #kata-general11:06
*** tmhoang__ has joined #kata-general11:08
*** tmhoang_ has quit IRC11:12
*** gwhaley has joined #kata-general12:18
*** sgarzare has quit IRC12:25
*** sgarzare has joined #kata-general12:29
*** tmhoang__ is now known as tmhoang12:33
*** altlogbot_3 has quit IRC13:55
*** gwhaley has quit IRC14:30
*** gwhaley has joined #kata-general14:43
kata-irc-bot<eric.ernst> @voje here'sa presentation from OSS last year on the topic of nested: https://events.linuxfoundation.org/wp-content/uploads/2017/12/Eric-Ernst_KY-Srinivasan_OSS-slides-kata-nested.pdf15:42
*** sameo has quit IRC15:51
*** lcastell has joined #kata-general15:57
*** lcastell has quit IRC15:57
*** tmhoang has quit IRC16:17
*** igordc has joined #kata-general16:33
*** sgarzare has quit IRC16:49
*** gwhaley has quit IRC17:08
*** sameo has joined #kata-general17:21
*** sameo has quit IRC18:28
kata-irc-bot<david532> So I found a project that is what I call preschool Kata containers - my question is in this case I have to specify memory specs for the KVM - how does Kata handle memory increases/allocation? :thinking_face:18:48
*** lcastell has joined #kata-general19:30
*** lcastell has quit IRC21:22
*** lcastell has joined #kata-general21:23
kata-irc-bot<greg.bock> didn't help, seems like virtiofsd fails/exits21:31
*** lcastell has joined #kata-general21:31
kata-irc-bot<sebastien.boeuf> wow the daemon fails... Do you have the logs related to this?21:31
kata-irc-bot<sebastien.boeuf> run the daemon with `-d` option if you want to see more logs21:32
kata-irc-bot<greg.bock> it is running from docker kata runtime21:33
kata-irc-bot<greg.bock> not sure i can use -f -d on it21:33
kata-irc-bot<sebastien.boeuf> hack the parameters used to run the daemon from the kata runtime21:34
kata-irc-bot<sebastien.boeuf> ah no I understand you will loose the stdout there21:35
kata-irc-bot<greg.bock> i can try21:35
kata-irc-bot<sebastien.boeuf> ok the only easy way to do this is to write a simple wrapper around the daemon21:35
kata-irc-bot<greg.bock> so writing to files works21:35
kata-irc-bot<greg.bock> i'll attach strace to virtiofsd real quick and kick off a yum update21:35
kata-irc-bot<sebastien.boeuf> ok so at least the fix fixed somthing right?21:35
kata-irc-bot<sebastien.boeuf> ok21:36
kata-irc-bot<greg.bock> that worked before too21:36
kata-irc-bot<greg.bock> just writing out /dev/zero to files with dd21:36
kata-irc-bot<sebastien.boeuf> and the yum install is still failing here?21:36
kata-irc-bot<greg.bock> `17070 write(2, "fv_panic: libvhost-user: virtio: invalid address for buffers\n", 61) = 61`21:37
kata-irc-bot<sebastien.boeuf> oh interesting, something is wrong about the guest memory21:39
kata-irc-bot<sebastien.boeuf> are you doing some memory hotplug?21:39
kata-irc-bot<greg.bock> not deliberately21:39
kata-irc-bot<sebastien.boeuf> well I mean are you setting up memory limits?21:40
kata-irc-bot<greg.bock> pm'ing you config21:40
kata-irc-bot<sebastien.boeuf> from your docker command>21:40
kata-irc-bot<sebastien.boeuf> ?21:40
kata-irc-bot<greg.bock> `docker run --runtime kata-runtime -it centos:7 /bin/bash`21:41
kata-irc-bot<greg.bock> ```[root@localhost system]# systemctl cat docker # /usr/lib/systemd/system/docker.service [Unit] Description=Docker Application Container Engine Documentation=http://docs.docker.com After=network.target docker-containerd.service Wants=docker-storage-setup.service Requires=docker-containerd.service rhel-push-plugin.socket registries.service  [Service] Type=notify EnvironmentFile=/run/containers/registries.conf21:41
kata-irc-botEnvironmentFile=-/etc/sysconfig/docker EnvironmentFile=-/etc/sysconfig/docker-storage EnvironmentFile=-/etc/sysconfig/docker-network Environment=GOTRACEBACK=crash ExecStart=/usr/bin/dockerd-current \           --add-runtime oci=/usr/libexec/docker/docker-runc-current \           --default-runtime=oci \           --authorization-plugin=rhel-push-plugin \           --containerd /run/containerd.sock \           --exec-opt21:41
kata-irc-botnative.cgroupdriver=systemd \           --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \           --init-path=/usr/libexec/docker/docker-init-current \           --seccomp-profile=/etc/docker/seccomp.json \           $OPTIONS \           $DOCKER_STORAGE_OPTIONS \           $DOCKER_NETWORK_OPTIONS \           $ADD_REGISTRY \           $BLOCK_REGISTRY \           $INSECURE_REGISTRY \           $REGISTRIES ExecReload=/bin/kill -s21:41
kata-irc-botHUP $MAINPID TasksMax=8192 LimitNOFILE=1048576 LimitNPROC=1048576 LimitCORE=infinity TimeoutStartSec=0 Restart=on-abnormal KillMode=process  [Install] WantedBy=multi-user.target  # /etc/systemd/system/docker.service.d/kata-containers.conf [Service] Type=simple ExecStart= ExecStart=/usr/bin/dockerd-current -D \           --add-runtime oci=/usr/libexec/docker/docker-runc-current \           --add-runtime21:41
kata-irc-botkata-runtime=/opt/virtiofs/bin/kata-runtime \           --default-runtime=oci \           --containerd /run/containerd.sock \           --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \           --init-path=/usr/libexec/docker/docker-init-current \           $OPTIONS \           $DOCKER_STORAGE_OPTIONS \           $DOCKER_NETWORK_OPTIONS \           $ADD_REGISTRY \           $BLOCK_REGISTRY \           $INSECURE_REGISTRY \21:41
kata-irc-bot     $REGISTRIES ```21:41
kata-irc-bot<greg.bock> I hate that you can't do snippets in a thread on slack ...21:42
kata-irc-bot<sebastien.boeuf> looks fine, I don't understand what's wrong here21:42
kata-irc-bot<sebastien.boeuf> and this error only happens when doing yum?21:42
kata-irc-bot<greg.bock> haven't tried other reproducers yet21:43
kata-irc-bot<sebastien.boeuf> ok21:43
kata-irc-bot<greg.bock> sent you the full strace in pm if you care to peek at it21:44
kata-irc-bot<sebastien.boeuf> so basically the guest driver is giving some address to the virtiofsd daemon, but this address (after translation into a host virtual address) is not mmap'ed21:44
kata-irc-bot<greg.bock> let me try this without hugepages21:45
kata-irc-bot<greg.bock> been following https://virtio-fs.gitlab.io/howto-kata.html plus my changes to make it work with 1.521:45
kata-irc-bot<sebastien.boeuf> oh yes please try to disable hugepages21:46
kata-irc-bot<sebastien.boeuf> maybe some weird behavior there because the mmap is not performed with hugepages21:46
kata-irc-bot<sebastien.boeuf> if MAP_HUGETLB is not used by QEMU, then this might not work21:47
kata-irc-bot<greg.bock> give me a bit, need to bounce the server21:51
kata-irc-bot<greg.bock> seems to not want to even start without hubepages21:53
kata-irc-bot<greg.bock> ```[root@localhost ~]# ps wwwaux | grep '[v]irtiofsd\|[q]emu' root      2391  0.0  0.0      0     0 ?        Z    21:52   0:00 [virtiofsd] <defunct> root      2394  0.0  0.0      0     0 ?        Zs   21:52   0:00 [qemu-system-x86] <defunct> root      2396  7.5  0.2 4653420 1211792 ?     Sl   21:52   0:05 /opt/virtiofs/bin/qemu-system-x86_64 -name sandbox-b2373de5c227735a799d1a9509e75b77909e7cfe829d62240c69e0aa61c7182d -uuid21:54
kata-irc-botf06d2d65-0c11-4427-a914-2f3ccd8252ac -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/vm/b2373de5c227735a799d1a9509e75b77909e7cfe829d62240c69e0aa61c7182d/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=516906M -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev21:54
kata-irc-botsocket,id=charconsole0,path=/run/vc/vm/b2373de5c227735a799d1a9509e75b77909e7cfe829d62240c69e0aa61c7182d/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng,rng=rng0,romfile= -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev21:54
kata-irc-botsocket,id=charch0,path=/run/vc/vm/b2373de5c227735a799d1a9509e75b77909e7cfe829d62240c69e0aa61c7182d/kata.sock,server,nowait -chardev socket,id=char-f885740e6cd768f2,path=/run/vc/sbs/b2373de5c227735a799d1a9509e75b77909e7cfe829d62240c69e0aa61c7182d/vhost-fs.sock -device vhost-user-fs-pci,chardev=char-f885740e6cd768f2,tag=kataShared,cache-size=2G -netdev tap,id=network-0,vhost=on,vhostfds=3,fds=4 -device21:54
kata-irc-botdriver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-modern=false,mq=on,vectors=4,romfile= -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /opt/virtiofs/vmlinuz -initrd /opt/virtiofs/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc021:54
kata-irc-botconsole=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=56 systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket -smp 1,cores=1,threads=1,sockets=1,maxcpus=56 root      2398  0.0  0.0   5896   184 ?        Ss   21:52   0:00 /opt/virtiofs/bin/virtiofsd -o21:54
kata-irc-botvirtio_socket=/run/vc/sbs/b2373de5c227735a799d1a9509e75b77909e7cfe829d62240c69e0aa61c7182d/vhost-fs.sock -o source=/run/kata-containers/shared/sandboxes/b2373de5c227735a799d1a9509e75b77909e7cfe829d62240c69e0aa61c7182d -o cache=none / ```21:54
kata-irc-bot<greg.bock> console never returns. I'll have to make a new initrd to try to watch kernel boot or get a shell in the HV21:54
kata-irc-bot<sebastien.boeuf> could you push your code to some branch so that someone on our side can try to reproduce?21:55
kata-irc-bot<greg.bock> https://github.com/greg-bock/runtime/tree/1.5.1-virtiofs21:55
kata-irc-bot<greg.bock> https://github.com/greg-bock/agent/tree/1.5.1-virtiofs21:55
kata-irc-bot<greg.bock> https://github.com/greg-bock/osbuilder/tree/1.5.1-virtiofs21:56
kata-irc-bot<greg.bock> my fork from upstream might be a bit old I'll try to get it back in sync tonight if i can21:56
kata-irc-bot<sebastien.boeuf> thx21:56
kata-irc-bot<greg.bock> building on f29 as per the kata howto on gitlab21:56
kata-irc-bot<sebastien.boeuf> I'm on ubuntu21:57
kata-irc-bot<sebastien.boeuf> but should not matter21:57
kata-irc-bot<greg.bock> thp off hugepages on: same behavior22:01
kata-irc-bot<greg.bock> also, with huge off one virtiofsd always zombies at start22:02
kata-irc-bot<greg.bock> ```root@localhost ~]# ps wwwaux | grep [v]irtiofsd root      4280  0.0  0.0      0     0 ?        Z    22:02   0:00 [virtiofsd] <defunct> root      4287  0.0  0.0   5896   184 ?        Ss   22:02   0:00 /opt/virtiofs/bin/virtiofsd -o virtio_socket=/run/vc/sbs/9d282418d9fefe64502feb9410c9561b411fe86a3973d233fb222d3ea0085815/vhost-fs.sock -o22:02
kata-irc-botsource=/run/kata-containers/shared/sandboxes/9d282418d9fefe64502feb9410c9561b411fe86a3973d233fb222d3ea0085815 -o cache=none / ```22:02
kata-irc-bot<greg.bock> i try a build of the 1.4 branches from gitlab too and see if it is the same behavior22:03
kata-irc-bot<sebastien.boeuf> ok let me know, we'll try on our side, as I want to make sure this works22:05
*** lcastell has quit IRC22:33
*** igordc has quit IRC22:34
*** lcastell has joined #kata-general22:35
*** lcastell has quit IRC22:36
kata-irc-bot<greg.bock> same on 1.4.022:37
kata-irc-bot<greg.bock> `[pid  7393] write(2, "fv_panic: libvhost-user: virtio: invalid address for buffers\n", 61) = 61`22:37
kata-irc-bot<sebastien.boeuf> wait! I'm looking at your QEMU command line that you posted22:40
kata-irc-bot<sebastien.boeuf> you're missing `-object memory-backend-file,id=mem,size=2G,mem-path=/dev/shm,share=on -numa node,memdev=mem`22:40
kata-irc-bot<sebastien.boeuf> otherwise you don't back the guest RAM with any file22:40
kata-irc-bot<sebastien.boeuf> try to add this to kata-runtime where the QEMU command line is created22:41
kata-irc-bot<sebastien.boeuf> and make sure the size of the shared backend here is the same as the size of the guest RAM22:41
kata-irc-bot<greg.bock> this in on 1.4.022:45
kata-irc-bot<greg.bock> `-object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/hugepages,share=on,prealloc=on `22:45
kata-irc-bot<greg.bock> let me see on 522:45
kata-irc-bot<greg.bock> yup, not there22:47
kata-irc-bot<sebastien.boeuf> change mem-path to be /dev/shm and try to remove prealloc22:47
kata-irc-bot<sebastien.boeuf> wow ok so if it's not there, there's no way this will work in 1.522:47
kata-irc-bot<sebastien.boeuf> and the option used for 1.4 might not work22:47
kata-irc-bot<greg.bock> this is all through govmm so it might take me a bit22:49
kata-irc-bot<greg.bock> where runtime build the qemu command22:49
kata-irc-bot<sebastien.boeuf> no it's through the runtime repo directly22:50
kata-irc-bot<sebastien.boeuf> and if you need to hack govmm for some reason, do it from the vendor directory under the runtime repo22:50
kata-irc-bot<sebastien.boeuf> quicker to get things tested22:51
kata-irc-bot<greg.bock> I think i just need to fill out the qemu config22:51
kata-irc-bot<greg.bock> just adding options isn't as easy as it would seem22:51
kata-irc-bot<greg.bock> due to the abstraction22:51
kata-irc-bot<greg.bock> I went down that path with trying to get debug console working (unless I missed something easy, even wrapping was tough due to arg parsing)22:52
kata-irc-bot<greg.bock> doh, that was the test when I had it turned off in the conf22:56
kata-irc-bot<greg.bock> let me rebuild 1.5 and see what it really has for args22:57
kata-irc-bot<sebastien.boeuf> ok22:57
kata-irc-bot<greg.bock> ```/opt/virtiofs/bin/qemu-system-x86_64 -name sandbox-f1d0c57f84d07522ef650c474a87a46958095830a3bd0f4cf91be53653b93302 -uuid a62dbc1a-0052-4f67-8036-486b28ec11da -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/vm/f1d0c57f84d07522ef650c474a87a46958095830a3bd0f4cf91be53653b93302/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=516906M -device23:10
kata-irc-botpci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/f1d0c57f84d07522ef650c474a87a46958095830a3bd0f4cf91be53653b93302/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom23:10
kata-irc-bot-device virtio-rng,rng=rng0,romfile= -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/f1d0c57f84d07522ef650c474a87a46958095830a3bd0f4cf91be53653b93302/kata.sock,server,nowait -chardev socket,id=char-9797e0d54bac5d8b,path=/run/vc/sbs/f1d0c57f84d07522ef650c474a87a46958095830a3bd0f4cf91be53653b93302/vhost-fs.sock -device23:10
kata-irc-botvhost-user-fs-pci,chardev=char-9797e0d54bac5d8b,tag=kataShared,cache-size=2G -netdev tap,id=network-0,vhost=on,vhostfds=3,fds=4 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-modern=false,mq=on,vectors=4,romfile= -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/hugepages,share=on,prealloc=on -numa23:10
kata-irc-botnode,memdev=dimm1 -kernel /opt/virtiofs/vmlinuz -initrd /opt/virtiofs/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=56 systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service23:10
kata-irc-botsystemd.mask=systemd-networkd.socket -smp 1,cores=1,threads=1,sockets=1,maxcpus=56 ```23:10
kata-irc-bot<sebastien.boeuf> ok, does this work?23:10
kata-irc-bot<raravena80> @david532 based on the vmm that it's using, qemu or firecracker.   There's also ongoing work for memory ballooning: https://github.com/kata-containers/runtime/issues/79023:11
kata-irc-bot<sebastien.boeuf> if not, change the line `-object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/hugepages,share=on,prealloc=on` into `-object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on`23:11
kata-irc-bot<sebastien.boeuf> hopefully that should do it23:11
kata-irc-bot<greg.bock> yeah, digging now to see how to do it23:11
kata-irc-bot<david532> In the case of qemu?23:12
kata-irc-bot<greg.bock> ```[root@localhost runtime]# git diff diff --git a/vendor/github.com/intel/govmm/qemu/qemu.go b/vendor/github.com/intel/govmm/qemu/qemu.go index 6c47c2f..9e45ca3 100644 --- a/vendor/github.com/intel/govmm/qemu/qemu.go +++ b/vendor/github.com/intel/govmm/qemu/qemu.go @@ -1686,7 +1686,7 @@ func (config *Config) appendMemoryKnobs() {         if config.Knobs.HugePages == true {                 if config.Memory.Size != "" {23:14
kata-irc-bot           dimmName := "dimm1" -                       objMemParam := "memory-backend-file,id=" + dimmName + ",size=" + config.Memory.Size + ",mem-path=/dev/hugepages,share=on,prealloc=on" +                       objMemParam := "memory-backend-file,id=" + dimmName + ",size=" + config.Memory.Size + ",mem-path=/dev/shm,share=on"                         numaMemParam := "node,memdev=" + dimmName                           config.qemuParams =23:14
kata-irc-botappend(config.qemuParams, "-object") ```23:14
kata-irc-bot<greg.bock> should do it23:14
kata-irc-bot<sebastien.boeuf> :+1:23:14
kata-irc-bot<greg.bock> ```/opt/virtiofs/bin/qemu-system-x86_64 -name sandbox-b0dd03bc47c9286af392b729e870f36bf8799f1f3d91f175a75ea753d0b15e52 -uuid 5d35ae45-c514-4fa1-a301-f303fbdde42e -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/vm/b0dd03bc47c9286af392b729e870f36bf8799f1f3d91f175a75ea753d0b15e52/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=516906M -device23:16
kata-irc-botpci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/b0dd03bc47c9286af392b729e870f36bf8799f1f3d91f175a75ea753d0b15e52/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom23:16
kata-irc-bot-device virtio-rng,rng=rng0,romfile= -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/b0dd03bc47c9286af392b729e870f36bf8799f1f3d91f175a75ea753d0b15e52/kata.sock,server,nowait -chardev socket,id=char-b73ddccd93b1d29d,path=/run/vc/sbs/b0dd03bc47c9286af392b729e870f36bf8799f1f3d91f175a75ea753d0b15e52/vhost-fs.sock -device23:16
kata-irc-botvhost-user-fs-pci,chardev=char-b73ddccd93b1d29d,tag=kataShared,cache-size=2G -netdev tap,id=network-0,vhost=on,vhostfds=3,fds=4 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-modern=false,mq=on,vectors=4,romfile= -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on -numa node,memdev=dimm123:16
kata-irc-bot-kernel /opt/virtiofs/vmlinuz -initrd /opt/virtiofs/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=56 systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket -smp23:16
kata-irc-bot1,cores=1,threads=1,sockets=1,maxcpus=56 ```23:16
kata-irc-bot<greg.bock> still died23:16
kata-irc-bot<greg.bock> same error `[pid  5155] write(2, "fv_panic: libvhost-user: virtio: invalid address for buffers\n", 61) = 61`23:18
kata-irc-bot<greg.bock> it take a few seconds after I run yum update -y to die23:18
kata-irc-bot<sebastien.boeuf> ok so stay with this config, and now hack the vhost-user-fs-pci from the qemu command line again. Try to add `disable-modern=false` to make sure there is no virtio backward compatibility error23:21
kata-irc-bot<greg.bock> odd that when the container/vm is running it isn't using any hugepages23:22
kata-irc-bot<greg.bock> ```root@localhost runtime]# grep -i huge /proc/meminfo  AnonHugePages:      2048 kB ShmemHugePages:        0 kB HugePages_Total:    8192 HugePages_Free:     8192 HugePages_Rsvd:        0 HugePages_Surp:        0 Hugepagesize:       2048 kB Hugetlb:        16777216 kB ```23:22
kata-irc-bot<sebastien.boeuf> forget what I said about disable-modern, there's no such option for this device23:23
kata-irc-bot<sebastien.boeuf> well about hugepages I'm not sure here23:24
kata-irc-bot<sebastien.boeuf> I mean about the meaning of those not being used23:24
kata-irc-bot<greg.bock> yeah, not sure either, it doesn't boot the HV if there aren't any free23:25
kata-irc-bot<sebastien.boeuf> can you print the dmesg inside the VM?23:26
kata-irc-bot<greg.bock> setting thp back to never didn't change anything after recent changes either23:27
kata-irc-bot<greg.bock> sure, one se23:27
kata-irc-bot<sebastien.boeuf> I'm wondering if we wouldn't see some driver issues23:27
kata-irc-bot<greg.bock> snippet in pm23:27
kata-irc-bot<raravena80> you specify it in the `configuration.toml` file, and basically kata-containers call qemu with the memory option. Does that make sense?23:28
kata-irc-bot<greg.bock> wonder if I should disable kpt isolation in host and/or guest23:29
kata-irc-bot<sebastien.boeuf> what's kpt?23:30
kata-irc-bot<greg.bock> kernel page table isolation23:31
kata-irc-bot<greg.bock> spectre and whatnot23:31
kata-irc-bot<greg.bock> just as a test23:31
kata-irc-bot<sebastien.boeuf> I doubt but yeah try it23:32
kata-irc-bot<sebastien.boeuf> Do you have my patch in your kernel?23:32
kata-irc-bot<sebastien.boeuf> the one to support also virtio-mmio?23:32
kata-irc-bot<greg.bock> just this one23:33
kata-irc-bot<greg.bock> ```[root@localhost linux]# git diff diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 4cb8c8a8011c..cde18dad46c9 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -206,8 +206,9 @@ static struct fuse_dax_mapping *alloc_dax_mapping(struct fuse_conn *fc)    out_kick:         /* If number of free ranges are below threshold, start reclaim */ -       free_threshold = (fc->nr_ranges * FUSE_DAX_RECLAIM_THRESHOLD)/100; -       if23:33
kata-irc-bot(free_threshold > 0 && fc->nr_free_ranges < free_threshold) { +       free_threshold = max((fc->nr_ranges * FUSE_DAX_RECLAIM_THRESHOLD)/100, +                               (unsigned long)1); +       if (fc->nr_free_ranges < free_threshold) {                 pr_debug("fuse: Kicking dax memory reclaim worker. nr_free_ranges=0x%ld nr_total_ranges=%ld\n", fc->nr_free_ranges, fc->nr_ranges);                 queue_delayed_work(system_long_wq,23:33
kata-irc-bot&fc->dax_free_work, 0);         } ```23:33
kata-irc-bot<greg.bock> grab this too? https://gitlab.com/virtio-fs/linux/merge_requests/123:33
kata-irc-bot<sebastien.boeuf> no no don't23:33
kata-irc-bot<sebastien.boeuf> I just wanted to make sure you didn't use it23:34
kata-irc-bot<sebastien.boeuf> just in case23:34
kata-irc-bot<greg.bock> ```[root@localhost linux]# git log --oneline -n4 4cd4e655955c (HEAD -> virtio-fs, origin/virtio-fs, origin/HEAD) fuse: fix fuse_permission() for the default_permissions case daf7c1bd47bf fuse: shared version cleanups f4bc6fda8f86 fuse: add shared version support (virtio-fs only) 4f46ede86709 fuse: don't crash if version table is NULL ```23:35
*** auk has joined #kata-general23:35
kata-irc-bot<greg.bock> plus above23:35
kata-irc-bot<greg.bock> fresh clone from today23:35
kata-irc-bot<sebastien.boeuf> I don't know...23:36
kata-irc-bot<sebastien.boeuf> we need to reproduce internally, that's all I can do now23:36
kata-irc-bot<sebastien.boeuf> cause I out of ideas :(23:36
kata-irc-bot<greg.bock> no worries, thanks for all the time so far23:36
kata-irc-bot<sebastien.boeuf> no problem, and I hope we'll get this sorted out soon!23:37
*** igordc has joined #kata-general23:39
kata-irc-bot<david532> Yup! That's what I was looking for! I'm just trying to compare it to runq, which is much more lightweight (and wokring for my use case).  Ultimately getting FUSE to work became a headache for me with Kata ,:)23:57
« Wednesday, 2019-04-03 Index Friday, 2019-04-05 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!