| *** bpradipt has joined #kata-dev | 04:34 | |
| *** Yarboa has joined #kata-dev | 06:18 | |
| *** ailan has joined #kata-dev | 07:03 | |
| *** sgarzare has joined #kata-dev | 07:05 | |
| *** fgiudici has joined #kata-dev | 07:10 | |
| *** snir_ has quit IRC | 07:35 | |
| *** snir_ has joined #kata-dev | 07:36 | |
| *** sgarzare has quit IRC | 07:44 | |
| *** sgarzare has joined #kata-dev | 08:04 | |
| *** davidgiluk has joined #kata-dev | 08:05 | |
| *** sgarzare has quit IRC | 10:47 | |
| *** sgarzare has joined #kata-dev | 10:47 | |
| *** sgarzare_ has joined #kata-dev | 11:16 | |
| *** sgarzare has quit IRC | 11:18 | |
| *** sgarzare_ has quit IRC | 11:31 | |
| *** sgarzare_ has joined #kata-dev | 11:32 | |
| *** sgarzare_ has quit IRC | 11:45 | |
| *** sgarzare_ has joined #kata-dev | 12:36 | |
| *** devimc has joined #kata-dev | 13:04 | |
| *** devimc has quit IRC | 13:35 | |
| *** fgiudici has quit IRC | 14:06 | |
| *** fgiudici has joined #kata-dev | 14:16 | |
| *** devimc has joined #kata-dev | 14:57 | |
| *** dklyle has joined #kata-dev | 15:18 | |
| *** sgarzare_ has quit IRC | 16:40 | |
| *** fgiudici has quit IRC | 16:45 | |
| *** Yarboa has quit IRC | 17:38 | |
| *** devimc has quit IRC | 17:56 | |
| *** Yarboa has joined #kata-dev | 18:05 | |
| *** Yarboa has quit IRC | 18:24 | |
| *** devimc has joined #kata-dev | 18:31 | |
| *** Yarboa has joined #kata-dev | 18:38 | |
| *** devimc_ has joined #kata-dev | 18:42 | |
| *** devimc has quit IRC | 18:46 | |
| *** devimc_ has quit IRC | 18:55 | |
| *** devimc has joined #kata-dev | 18:59 | |
| *** davidgiluk has quit IRC | 19:14 | |
| *** devimc has quit IRC | 19:29 | |
| *** devimc has joined #kata-dev | 19:40 | |
| *** devimc has quit IRC | 19:51 | |
| *** devimc has joined #kata-dev | 19:54 | |
| *** devimc has quit IRC | 20:47 | |
| kata-irc-bot1 | <eric.ernst> Doing some builds within pods run with Kata in CLH and see "clock skew detected" | 20:59 |
|---|---|---|
| kata-irc-bot1 | <eric.ernst> @archana.m.shinde IIRC we ran into time sync issues in the past. Was the solution VMM specific? | 20:59 |
| kata-irc-bot1 | <eric.ernst> @archana.m.shinde IIRC we ran into time sync issues in the past. Was the solution VMM specific? | 20:59 |
| kata-irc-bot1 | <eric.ernst> @archana.m.shinde IIRC we ran into time sync issues in the past. Was the solution VMM specific? | 20:59 |
| kata-irc-bot1 | <archana.m.shinde> @eric.ernst Yes, it was VMM specific | 21:12 |
| kata-irc-bot1 | <archana.m.shinde> I had to enable the chronyd deamon inside the guest, which was vmm agnostic of course | 21:14 |
| kata-irc-bot1 | <eric.ernst> ok. | 21:24 |
| kata-irc-bot1 | <archana.m.shinde> @eric.ernst its dependent on KVM | 21:25 |
| kata-irc-bot1 | <archana.m.shinde> so I take back, it should not depend on clh supporting it | 21:25 |
| kata-irc-bot1 | <archana.m.shinde> wondering if we included the change in the kernel fragments | 21:26 |
| kata-irc-bot1 | <archana.m.shinde> let me check | 21:26 |
| *** ailan has quit IRC | 21:32 | |
| *** vgoyal has joined #kata-dev | 21:36 | |
| kata-irc-bot1 | <eric.ernst> sure. | 21:38 |
| kata-irc-bot1 | <eric.ernst> Note, this is using the virtio-fs kernel... | 21:39 |
| kata-irc-bot1 | <chen.bo> @eric.ernst That's new to me. Honestly, I didn't try any workloads like 'build' for kata+clh. Thanks for reporting another issue. | 21:52 |
| kata-irc-bot1 | <chen.bo> Wondering do you observe it only with clh not for qemu w/ kata, right? | 21:53 |
| *** vgoyal has quit IRC | 22:33 | |
| kata-irc-bot1 | <eric.ernst> ...testing on qemu now... | 22:49 |
| kata-irc-bot1 | <eric.ernst> Another change in behavior: is there a different wrt memory between QEMU/CLH? More specifically, on demand paging? | 22:50 |
| kata-irc-bot1 | <archana.m.shinde> hey back from meeting, do you not see the behaviour with qemu | 22:51 |
| kata-irc-bot1 | <archana.m.shinde> was just checking the kernel config required for clock | 22:51 |
| *** Yarboa has quit IRC | 22:52 | |
| kata-irc-bot1 | <archana.m.shinde> this is the one that I added : https://github.com/kata-containers/packaging/blob/594ce2fee7dc011e91dece409311f0fa08ff97d0/kernel/configs/x86_64_kata_kvm_4.14.x#L1866 | 22:52 |
| kata-irc-bot1 | <archana.m.shinde> CONFIG_PTP_1588_CLOCK | 22:52 |
| kata-irc-bot1 | <eric.ernst> not seeing it with QEMU | 22:54 |
| kata-irc-bot1 | <archana.m.shinde> but I dont see it in the fragments for x86 | 22:55 |
| kata-irc-bot1 | <eric.ernst> I can't see the kernel config options, but vmlinuz v vmlinux | 22:55 |
| kata-irc-bot1 | <archana.m.shinde> let me switch to my dev machine to verify | 22:55 |
| kata-irc-bot1 | <eric.ernst> thx | 22:55 |
| kata-irc-bot1 | <chen.bo> @eric.ernst Assuming you mean the memory footprint, I think now clh is consuming more memory than qemu . The number is like 270MB vs 135MB, based on @jose.carlos.venegas.m recent work on the metrics CI. | 22:56 |
| kata-irc-bot1 | <eric.ernst> ! | 22:56 |
| kata-irc-bot1 | <eric.ernst> ok. | 22:57 |
| kata-irc-bot1 | <chen.bo> Thanks for confirming kata+qemu is working fine. @archana.m.shinde if you find a easy way to reproduce, please open an issue and we can take over and debug from there. Thanks ! | 22:57 |
| kata-irc-bot1 | <chen.bo> @eric.ernst Right. The number is not good. And @jose.carlos.venegas.m is debugging on that and understanding where is the memory consumption from. | 22:58 |
| kata-irc-bot1 | <eric.ernst> okay. is there an issue open for that? On CLH side or .. ? | 22:58 |
| kata-irc-bot1 | <eric.ernst> also, is there a result for Firecracker VMM as well? | 22:59 |
| kata-irc-bot1 | <jose.carlos.venegas.m> @eric.ernst I have not issues opened yet, I can create one | 22:59 |
| kata-irc-bot1 | <chen.bo> AFAIK, I don't think so. But CLH folks are involved for thie debugging. | 22:59 |
| kata-irc-bot1 | <jose.carlos.venegas.m> @eric.ernst I have not check firecracker at all | 23:00 |
| kata-irc-bot1 | <eric.ernst> okay, issue would be helpful just to track. | 23:00 |
| kata-irc-bot1 | <chen.bo> @jose.carlos.venegas.m do you know we include FC for our metrics CI? | 23:01 |
| kata-irc-bot1 | <archana.m.shinde> @eric.ernst | 23:01 |
| kata-irc-bot1 | <chen.bo> just wondering. | 23:01 |
| kata-irc-bot1 | <archana.m.shinde> `CONFIG_PTP_1588_CLOCK is not set` | 23:01 |
| kata-irc-bot1 | <archana.m.shinde> for x86 | 23:02 |
| kata-irc-bot1 | <jose.carlos.venegas.m> @chen.bo no we dont track any metrics about FC :S | 23:02 |
| kata-irc-bot1 | <archana.m.shinde> so you should see time sync issue exists for qemu as well as clh | 23:02 |
| kata-irc-bot1 | <archana.m.shinde> dont know why you didnt see the message for qemu | 23:02 |
| kata-irc-bot1 | <archana.m.shinde> I suppose this was a kernel log message sent from the agent | 23:03 |
| kata-irc-bot1 | <archana.m.shinde> ^@egernst | 23:03 |
| kata-irc-bot1 | <eric.ernst> it was a message I see when doing a make. | 23:03 |
| kata-irc-bot1 | <eric.ernst> ie, was make -j6 on the Linux kernel | 23:04 |
| *** Yarboa has joined #kata-dev | 23:04 | |
| kata-irc-bot1 | <archana.m.shinde> hmm.. not sure if the make fixed the issue when you ran with qemu | 23:09 |
| kata-irc-bot1 | <archana.m.shinde> does a consequent build with clh again show the error? | 23:09 |
| kata-irc-bot1 | <archana.m.shinde> I am assuming what you are building may come from a volume on host | 23:09 |
| kata-irc-bot1 | <jose.carlos.venegas.m> @eric.ernst qemu+virtiofs? or vanilla qemu? | 23:09 |
| kata-irc-bot1 | <eric.ernst> I saw it consistently w/ QEMU | 23:10 |
| kata-irc-bot1 | <eric.ernst> To be clear, the sources are in the container - so it should be idompotent | 23:10 |
| kata-irc-bot1 | <eric.ernst> So, no sharing w/ the host. | 23:10 |
| kata-irc-bot1 | <eric.ernst> all self containerd, in the rootfs itself. | 23:10 |
| kata-irc-bot1 | <eric.ernst> qemu+virtiofs | 23:10 |
| kata-irc-bot1 | <archana.m.shinde> can you provide the worload image/yaml you are using so that I can reproduce at my end | 23:11 |
| kata-irc-bot1 | <eric.ernst> yeah, 1 minute. | 23:21 |
| kata-irc-bot1 | <eric.ernst> ```--- apiVersion: batch/v1 kind: Job metadata: name: kbuild spec: template: spec: containers: - name: kbuild image: egernst/kernel-build command: ["bash"] args: ["-c", "make allyesconfig && make -j6"] resources: requests: cpu: 1 memory: 5Gi limits: cpu: 8 memory: 10Gi restartPolicy: Never | 23:24 |
| kata-irc-bot1 | runtimeClassName: kata-clh``` | 23:24 |
| kata-irc-bot1 | <eric.ernst> container image is very large, sorry :slightly_smiling_face: | 23:24 |
| kata-irc-bot1 | <archana.m.shinde> oh vow..you are doing kernel builds! | 23:28 |
| kata-irc-bot1 | <archana.m.shinde> ok, let me give that a try | 23:28 |
| kata-irc-bot1 | <archana.m.shinde> I'll submit a PR to add the ptp support in any case, as that is something we missed in the fragments | 23:29 |
| kata-irc-bot1 | <eric.ernst> kernel builds are just nice for exercising filesystem :slightly_smiling_face: | 23:31 |
| kata-irc-bot1 | <eric.ernst> poor mans stability test. | 23:31 |
| kata-irc-bot1 | <eric.ernst> I'm still distrubed when I run a test and see SOOOO many virtiofsd threads. | 23:39 |
| kata-irc-bot1 | <archana.m.shinde> yeah I did notice | 23:43 |
| kata-irc-bot1 | <archana.m.shinde> how many are you seeing? | 23:43 |
| kata-irc-bot1 | <eric.ernst> 131? | 23:45 |
| kata-irc-bot1 | <eric.ernst> ps -aTef | grep virtiofsd | grep -v grep | wc | 23:45 |
| kata-irc-bot1 | <eric.ernst> I had chatted w/ @jose.carlos.venegas.m in the past on this. | 23:46 |
| kata-irc-bot1 | <eric.ernst> i'm still trying to recall who did the most in-depth testing here -- this doesn't seem like an ideal amount | 23:46 |
| kata-irc-bot1 | <archana.m.shinde> @eric.ernst open an issue, Vivek and David can chime in | 23:49 |
| kata-irc-bot1 | <archana.m.shinde> btw, https://github.com/kata-containers/packaging/pull/1104 please ack | 23:50 |
| kata-irc-bot1 | <eric.ernst> ok if I edit your issue to reflect what i was seeing? | 23:52 |
| kata-irc-bot1 | <archana.m.shinde> if i can recall, the ptp daemon was added to keep the time synced over the course of time | 23:53 |
| kata-irc-bot1 | <archana.m.shinde> as it started drifting | 23:53 |
| kata-irc-bot1 | <eric.ernst> okay; so only helpful if agent != init process? | 23:53 |
| kata-irc-bot1 | <archana.m.shinde> the time should be in sync when the VM is started | 23:53 |
| kata-irc-bot1 | <archana.m.shinde> yes | 23:54 |
| kata-irc-bot1 | <archana.m.shinde> are you using agent as init? | 23:54 |
| kata-irc-bot1 | <archana.m.shinde> in your tests | 23:54 |
| kata-irc-bot1 | <eric.ernst> no | 23:56 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!