| kata-irc-bot | <david532> So I got the console finally. I think my debug options weren't set... | 00:06 |
|---|---|---|
| kata-irc-bot | <dangot> @david532 - we’ve been trying to get a console too without success. What do you mean your debug options weren’t set ? | 00:11 |
| kata-irc-bot | <david532> @dangot if you're using the debug sed to change the parameters, appears it doesn't seem to write the file back. I just went through and enabled all but netmon and proxy per the docs. | 00:13 |
| kata-irc-bot | <david532> First time getting the shell | 00:13 |
| kata-irc-bot | <david532> But can see my issue that fuse was not properly there. | 00:13 |
| *** lcastell has joined #kata-general | 00:30 | |
| *** lcastell has quit IRC | 00:35 | |
| *** spotz has quit IRC | 01:12 | |
| kata-irc-bot | <david532> @dangot does socat connect but not show anything? | 01:18 |
| kata-irc-bot | <dangot> yep that’s what happens | 01:18 |
| kata-irc-bot | <david532> If you follow my script (just make sure you make the debug service file too and change kernel options if needed it should get you there. Just make sure to enable debug but proxy and verify you're starting with the modified images. | 01:20 |
| kata-irc-bot | <david532> "script". I'm using packet.net spot instances for baremetal. | 01:22 |
| kata-irc-bot | <david532> If you're lucky you can get a beefy one for under ten cents. | 01:22 |
| *** igordc has quit IRC | 01:29 | |
| *** auk has joined #kata-general | 02:58 | |
| *** auk has quit IRC | 03:04 | |
| kata-irc-bot | <eric.ernst> @dangot @david532 - Hey Davids -- can you help with a PR to improve the guide? Or suggestion? | 04:06 |
| kata-irc-bot | <eric.ernst> Glad its working for you. | 04:06 |
| kata-irc-bot | <david532> I got it once but I need to replicate again. The instructions are mostly clear - I think it's more when you start introducing your own flavors that it strays from the path. | 04:15 |
| kata-irc-bot | <dangot> So we built kata with `virtio-fs`, and while performance are better than 9p - they’re still nowhere close to what we get without volumes (devicemapper) | 04:16 |
| *** igordc has joined #kata-general | 04:18 | |
| kata-irc-bot | <dangot> For sharing data between containers (EmptyDir in k8s), would it be possible to not to have 9p or virtiofs ? i.e. keeping the volume inside the VM ? | 04:19 |
| kata-irc-bot | <eric.ernst> are the containers in the same pod, @dangot? | 04:27 |
| kata-irc-bot | <dangot> yes | 04:27 |
| kata-irc-bot | <eric.ernst> we should figure something out then | 04:27 |
| kata-irc-bot | <eric.ernst> will have to brainstorm how we can set this up, but it would be ideal to not go back to host unnecessarily. @bergwolf @fupan FYI | 04:28 |
| kata-irc-bot | <eric.ernst> @krsna1729 ^ | 04:29 |
| kata-irc-bot | <dangot> Or maybe we could add a `kata` volume type in Kubernetes ? | 04:34 |
| kata-irc-bot | <krsna1729> Wait I thought emptydir with memory worked | 04:43 |
| kata-irc-bot | <dangot> it does work | 04:44 |
| kata-irc-bot | <krsna1729> Two pods should be able to share | 04:44 |
| kata-irc-bot | <dangot> but it has serious limitation | 04:44 |
| kata-irc-bot | <dangot> We have people pulling 20+G of data for their build… | 04:45 |
| kata-irc-bot | <krsna1729> Ok | 04:45 |
| kata-irc-bot | <krsna1729> Host local volumes... Need to read up on that | 04:45 |
| kata-irc-bot | <krsna1729> @dangot seems like this one https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner | 05:13 |
| kata-irc-bot | <krsna1729> disclaimer: zero experience | 05:13 |
| kata-irc-bot | <krsna1729> quesiton for @eric.ernst @archana.m.shinde @manohar.r.castelino - is emptydir implementation dependent? does kata-runtime have a say on the behavior? | 05:16 |
| kata-irc-bot | <krsna1729> or is it determined by runtime | 05:16 |
| *** tmhoang has joined #kata-general | 05:19 | |
| kata-irc-bot | <krsna1729> @dangot thanks for bringing these usecases. learnt new things last week :slightly_smiling_face: | 05:19 |
| kata-irc-bot | <aprice> @krsna1729 :wave: (I’m in the same team as @dangot) Could emptyDir make use of this ephemeralType driver? https://github.com/kata-containers/agent/blob/master/mount.go#L200 | 05:21 |
| kata-irc-bot | <krsna1729> hehe i dont work on kata side :P thats why pointed to them ;) | 05:21 |
| kata-irc-bot | <krsna1729> more on k8s side | 05:21 |
| kata-irc-bot | <aprice> ah no worries then :P | 05:22 |
| kata-irc-bot | <krsna1729> lol looks like we came full circle https://github.com/kata-containers/runtime/pull/458 | 05:27 |
| kata-irc-bot | <krsna1729> the descriptions seems like exactly what you want | 05:30 |
| kata-irc-bot | <krsna1729> and what you pointed to in code | 05:30 |
| kata-irc-bot | <aprice> Yea that looks good, but the issue we have is that we want the emptyDir to be a directory on the VM filesystem, rather than a tmpfs | 05:30 |
| kata-irc-bot | <aprice> our VM filesystem isn’t 9pfs, but using device mapper | 05:31 |
| kata-irc-bot | <krsna1729> got it | 05:36 |
| *** auk has joined #kata-general | 05:46 | |
| kata-irc-bot | <dangot> https://github.com/kata-containers/runtime/issues/1341 | 05:50 |
| kata-irc-bot | <dangot> So this is why we never encountered the problem initially … EmptyDir were `tmpfs` | 05:51 |
| kata-irc-bot | <dangot> @aprice - how about your first contribution to Kata ? | 05:53 |
| kata-irc-bot | <dangot> I’ve opened https://github.com/kata-containers/runtime/issues/1472 | 06:19 |
| kata-irc-bot | <raravena80> @dangot what's the virtio-fs config that you used in the `configuration.toml` file? (For my reference) | 06:19 |
| kata-irc-bot | <dangot> cc @aprice ^ | 06:19 |
| kata-irc-bot | <dangot> I believe we’ve followed this guide https://virtio-fs.gitlab.io/howto-kata.html | 06:21 |
| kata-irc-bot | <aprice> Yea we’ve used the config file from that guide, but with one minor change to get `--privileged` docker containers working - `block_device_driver = "virtio-blk"` | 06:22 |
| kata-irc-bot | <raravena80> ahh cool, thx! | 06:23 |
| *** sgarzare has joined #kata-general | 06:37 | |
| *** sgarzare has quit IRC | 06:45 | |
| *** sgarzare has joined #kata-general | 06:47 | |
| *** auk has quit IRC | 07:13 | |
| *** auk has joined #kata-general | 07:14 | |
| *** auk has quit IRC | 07:19 | |
| *** auk has joined #kata-general | 07:19 | |
| *** auk has quit IRC | 07:36 | |
| *** sameo has joined #kata-general | 07:49 | |
| *** igordc has quit IRC | 08:04 | |
| *** tmhoang_ has joined #kata-general | 10:28 | |
| *** tmhoang_ has quit IRC | 10:28 | |
| *** tmhoang_ has joined #kata-general | 10:28 | |
| *** tmhoang__ has joined #kata-general | 10:29 | |
| *** tmhoang has quit IRC | 10:29 | |
| *** tmhoang__ is now known as tmhoang | 10:30 | |
| *** tmhoang_ has quit IRC | 10:33 | |
| *** igordc has joined #kata-general | 14:01 | |
| kata-irc-bot | <eric.ernst> :slightly_smiling_face: | 14:42 |
| kata-irc-bot | <eric.ernst> I want to reduce the amount of innovation necessary to make it work well (so, not just functional, but usability too). Thx for sticking with it @david532 | 14:42 |
| *** tmhoang_ has joined #kata-general | 15:53 | |
| *** lcastell has joined #kata-general | 15:54 | |
| *** tmhoang has quit IRC | 15:55 | |
| *** tmhoang__ has joined #kata-general | 15:55 | |
| *** tmhoang__ is now known as tmhoang | 15:57 | |
| *** tmhoang_ has quit IRC | 15:59 | |
| *** sgarzare has quit IRC | 16:24 | |
| *** lcastell has quit IRC | 16:27 | |
| kata-irc-bot | <archana.m.shinde> @dangot just catching up, empty-dirs with "memory" medium are backed by tmpfs in Kata, those with "default" medium are directories on host that are shared with Kata through 9p | 18:11 |
| *** igordc has quit IRC | 18:14 | |
| *** igordc has joined #kata-general | 18:50 | |
| kata-irc-bot | <archana.m.shinde> @dangot I recently went through the process of having a debug shell with ubuntu, maybe you can share your contents of configuration.toml | 19:03 |
| kata-irc-bot | <archana.m.shinde> there are a couple of sanity checks that I usually run, mount the image manually using losetup, make sure bash is present and one can start a shell with it, and check for the systemd targets | 19:04 |
| *** sameo has quit IRC | 19:43 | |
| *** tmhoang has quit IRC | 19:52 | |
| kata-irc-bot | <greg.bock> @dangot I've had issues with things going into d state and locking up when testing kata 1.5 + virtiofs (also saw similar with 1.4). Which version were you testing against? | 20:05 |
| kata-irc-bot | <greg.bock> it could just be the agent locking up, I haven't started a container with remote access or used the debug shell to poke around from within the HV | 20:11 |
| kata-irc-bot | <eric.ernst> @greg.bock can you clarify d-state? | 20:13 |
| kata-irc-bot | <greg.bock> uninterruptible sleep | 20:14 |
| kata-irc-bot | <greg.bock> at least iirc, I had to repurpose my test box. | 20:14 |
| kata-irc-bot | <greg.bock> disk r/w with things like dd seemed fine, but something like a yum list or yum update would hang (this was over a exec shell so it could be that interface locking up and not the container process) | 20:18 |
| *** igordc has quit IRC | 20:18 | |
| kata-irc-bot | <aprice> @greg.bock We were using all the versions in https://virtio-fs.gitlab.io/howto-kata.html, so 1.4. I noticed that running `apt update` in an ubuntu container caused it to lock up too. | 20:41 |
| *** igordc has joined #kata-general | 20:59 | |
| kata-irc-bot | <greg.bock> very odd | 21:07 |
| kata-irc-bot | <greg.bock> was hoping to install benchmarking tools, guess I'll just build an image with the packages already in it | 21:07 |
| kata-irc-bot | <greg.bock> was that over exec or using something like ssh to access it over network? | 21:07 |
| kata-irc-bot | <eric.ernst> @sebastien.boeuf ^^ | 21:09 |
| kata-irc-bot | <eric.ernst> @gmmaharaj - in case you have your virio-fs setup up. | 21:09 |
| kata-irc-bot | <eric.ernst> right now its even more experimental than experimental, but a heads up. | 21:09 |
| kata-irc-bot | <greg.bock> i did finally get virtiofs forward ported for 1.5, had silly scoping issue and it needed chroot vs pivot root | 21:10 |
| kata-irc-bot | <gmmaharaj> @eric.ernst not yet. working on it with kata rihgt now. | 21:11 |
| kata-irc-bot | <gmmaharaj> but yes, will keep that in mind. | 21:11 |
| kata-irc-bot | <sebastien.boeuf> @greg.bock you mean benchmark in the context of Kata or purely testing virtio-fs with QEMU. The former is definitely more complex than the latter one. https://github.com/rhvgoyal/virtiofs-tests are the tests used by Redhat folks to try virtio-fs | 21:15 |
| kata-irc-bot | <greg.bock> kata | 21:15 |
| kata-irc-bot | <sebastien.boeuf> Ok I have no clue there since I haven't tried to evaluate how this behave with Kata (from a perf standpoint) | 21:17 |
| kata-irc-bot | <eric.ernst> s/perf/functional | 21:17 |
| kata-irc-bot | <eric.ernst> ? | 21:17 |
| kata-irc-bot | <greg.bock> perf secondary to stability but just getting it working in general. Interesting that Alex also had issues when using the package manager for the OS | 21:18 |
| kata-irc-bot | <sebastien.boeuf> oh you haven't got that working yet | 21:18 |
| kata-irc-bot | <sebastien.boeuf> got it | 21:18 |
| kata-irc-bot | <greg.bock> well it works but trying to use yum caused something to lockup | 21:19 |
| kata-irc-bot | <greg.bock> I had to switch focus and tear down my lab temporarily | 21:19 |
| kata-irc-bot | <sebastien.boeuf> ok, I'd suggest that you post your findings somewhere with the logs that you get once you get some time | 21:20 |
| kata-irc-bot | <sebastien.boeuf> if we can find some non functional use case, we need to get RH folks involved on those | 21:21 |
| kata-irc-bot | <greg.bock> need to get it running with the debug console so I can hop into the HV to poke around | 21:21 |
| kata-irc-bot | <sebastien.boeuf> ok | 21:21 |
| kata-irc-bot | <greg.bock> all access to the container from exec locks | 21:21 |
| kata-irc-bot | <sebastien.boeuf> but you can play from the container directly, from a `docker run ...` | 21:22 |
| kata-irc-bot | <sebastien.boeuf> ? | 21:22 |
| kata-irc-bot | <greg.bock> ``` https://github.com/greg-bock/runtime/compare/stable-1.5...greg-bock:1.5.1-virtiofs https://github.com/greg-bock/agent/compare/stable-1.5...greg-bock:1.5.1-virtiofs https://github.com/greg-bock/osbuilder/commit/64488a633c852cc13c198e8028938e8815306ec6 ``` | 21:23 |
| kata-irc-bot | <greg.bock> that will lock up too iirc | 21:23 |
| kata-irc-bot | <greg.bock> so if i run /bin/bash, then get a few more shells over exec, then run yum list or yum install then they all lock | 21:23 |
| kata-irc-bot | <greg.bock> on a base centos:7 container image | 21:24 |
| kata-irc-bot | <sebastien.boeuf> ok can you try a quick kernel patch (unless you have it with you) | 21:24 |
| kata-irc-bot | <greg.bock> Going to try to find a spare server to get resetup on so i can get some better data | 21:24 |
| kata-irc-bot | <sebastien.boeuf> ``` | 21:25 |
| kata-irc-bot | <sebastien.boeuf> ``` commit edba201cacf6ba8da34ad2948085ae51dcc6f019 (HEAD -> extend_virtio-mmio) Author: Sebastien Boeuf <sebastien.boeuf@intel.com> Date: Mon Mar 11 12:41:23 2019 -0700 Vivek's patch fixing remapping Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com> diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 4cb8c8a8011c..cde18dad46c9 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -206,8 +206,9 @@ static | 21:25 |
| kata-irc-bot | struct fuse_dax_mapping *alloc_dax_mapping(struct fuse_conn *fc) out_kick: /* If number of free ranges are below threshold, start reclaim */ - free_threshold = (fc->nr_ranges * FUSE_DAX_RECLAIM_THRESHOLD)/100; - if (free_threshold > 0 && fc->nr_free_ranges < free_threshold) { + free_threshold = max((fc->nr_ranges * FUSE_DAX_RECLAIM_THRESHOLD)/100, + (unsigned long)1); + if | 21:25 |
| kata-irc-bot | (fc->nr_free_ranges < free_threshold) { pr_debug("fuse: Kicking dax memory reclaim worker. nr_free_ranges=0x%ld nr_total_ranges=%ld\n", fc->nr_free_ranges, fc->nr_ranges); queue_delayed_work(system_long_wq, &fc->dax_free_work, 0); } ``` | 21:25 |
| kata-irc-bot | <greg.bock> I think i tried that one | 21:25 |
| kata-irc-bot | <greg.bock> have a pr open on gitlab virtiofs for it already? | 21:25 |
| kata-irc-bot | <greg.bock> ahh, not the one I was thinking of: https://gitlab.com/virtio-fs/linux/merge_requests/1/diffs | 21:26 |
| kata-irc-bot | <greg.bock> I'll try it as soon as I can | 21:26 |
| kata-irc-bot | <sebastien.boeuf> oh no the one I wrote is about supporting virtio-mmio, but you don't care in case of Kata, that was for something else | 21:26 |
| kata-irc-bot | <sebastien.boeuf> but yes please try the one from Vivek (RH folks) | 21:27 |
| kata-irc-bot | <greg.bock> I found a few patches i tried can't remember if that was one of them | 21:27 |
| kata-irc-bot | <sebastien.boeuf> because I had some locking issues without this patch. The first mmap will work but then the guest will not do the right thing, preventing from further mapping | 21:27 |
| *** lcastell has joined #kata-general | 22:04 | |
| kata-irc-bot | <manohar.r.castelino> @gmmaharaj ^ | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!