| kata-irc-bot | <ssheribe> Hi, does anyone has idea why jenkins-vfio-host:ubuntu18-04-vm:fedora32 is failing on stable? i wonder if there's something that is not backported | 07:45 |
|---|---|---|
| kata-irc-bot | <fidencio> Let's take a look at the error. | 07:47 |
| kata-irc-bot | <fidencio> ```07:54:53 Set up environment 07:54:53 Trying to install containerd from static tarball 07:55:04 Download tarball from https://storage.googleapis.com/cri-containerd-release/v1.3.7/cri-containerd-cni-1.3.7-linux-amd64.tar.gz 07:55:04 % Total % Received % Xferd Average Speed Time Time Time Current 07:55:04 Dload Upload Total Spent Left Speed 07:55:05 0 0 0 0 0 | 07:47 |
| kata-irc-bot | 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 07:55:05 curl: (22) The requested URL returned error: 404 07:55:05 Failed to download tarball from https://storage.googleapis.com/cri-containerd-release/v1.3.7/cri-containerd-cni-1.3.7-linux-amd64.tar.gz 07:55:05 Trying to install containerd from source 07:55:05 error: pathspec 'v1.3.7' did not match any file(s) known to | 07:47 |
| kata-irc-bot | git 07:55:05 Failed at 44: git checkout "${cri_containerd_tarball_version}" 07:55:05 Failed at 48: ( cd "${GOPATH}/src/${cri_containerd_repo}" >> /dev/null; git fetch; git checkout "${cri_containerd_tarball_version}"; make BUILDTAGS="${BUILD_TAGS:-}" cri-cni-release; tarball_name="cri-containerd-cni-${cri_containerd_version}-${CONTAINERD_OS}-${CONTAIENRD_ARCH}.tar.gz"; sudo tar -xvf "./releases/${tarball_name}" -C / ) 07:55:05 ERROR: containerd | 07:47 |
| kata-irc-bot | not installed``` | 07:48 |
| kata-irc-bot | <fidencio> This error rings a bell and it was happening before we did the cointainerd bump. | 07:48 |
| kata-irc-bot | <fidencio> Now, backporting the whole content to support newer versions of containerd is rather intrusive for a stable release. | 07:49 |
| kata-irc-bot | <fidencio> Maybe just bumping the containerd version for the next patch release would fix the issue (and the tarball would be present on their repo? | 07:49 |
| kata-irc-bot | <fidencio> That's the path I'd take to start investigating how to fix the issue on the stable branch | 07:50 |
| kata-irc-bot | <fidencio> Does this make sense, @ssheribe? | 07:50 |
| kata-irc-bot | <fidencio> For instance, maybe trying to use v1.3.10? https://github.com/containerd/containerd/releases/tag/v1.3.10 | 08:08 |
| kata-irc-bot | <fidencio> @ssheribe, https://github.com/kata-containers/kata-containers/pull/2222 | 08:25 |
| kata-irc-bot | <ssheribe> @fidencio sorry for the delay, yes make sense, thanks, lets see | 08:30 |
| kata-irc-bot | <ssheribe> BTW does the failures in s390x and arm are known? | 08:32 |
| kata-irc-bot | <fidencio> s390x is not supposed to work, to be honest. @jakob.naucke is doing a really amazing work on getting it up and green, but I don't think we reached that point yet. | 08:32 |
| kata-irc-bot | <fidencio> About the ARM, that's a good question and I'd ping @jianyong.wu about that :slightly_smiling_face: | 08:33 |
| kata-irc-bot | <ssheribe> thanks! well i'm pretty sure it's failing regardless to the static checks fix | 08:35 |
| kata-irc-bot | <jakob.naucke> s390x is sometimes successful, I think https://github.com/kata-containers/tests/pull/3706 will fix it | 08:35 |
| kata-irc-bot | <ssheribe> @jakob.naucke https://github.com/kata-containers/kata-containers/pull/2193 do you think this one should be ported to stable? @fidencio https://github.com/kata-containers/kata-containers/pull/2203 this one? is it related to the static check failure in #2222 | 08:53 |
| kata-irc-bot | <fidencio> @ssheribe, 2203 shouldn't be backported to stable unless we backport https://github.com/kata-containers/tests/pull/3646 as well. The main point the CI was failing was because we were running the stable-2.1 content against the `main` branch | 08:54 |
| kata-irc-bot | <ssheribe> @fidencio fails:( newer version maybe? | 09:15 |
| kata-irc-bot | <fidencio> It failed because the URL generated in the tests repo was wrong ... I've changed the tests repo to cope with the new URL, let's see if we have better results now | 09:17 |
| kata-irc-bot | <fidencio> Regardless, why only the vfio is failing for you? | 09:17 |
| kata-irc-bot | <ssheribe> why? you mean, why not others? | 09:19 |
| kata-irc-bot | <fidencio> I'd expect the same failures on other CIs, when getting the containerd tarball | 09:22 |
| kata-irc-bot | <fidencio> This is what happens in another CI: ```15:42:22 Trying to install containerd from static tarball 15:42:22 Download tarball from https://storage.googleapis.com/cri-containerd-release/cri-containerd-1.3.7.linux-amd64.tar.gz 15:42:22 % Total % Received % Xferd Average Speed Time Time Time Current 15:42:22 Dload Upload Total Spent Left Speed 15:42:22 0 0 0 0 0 0 | 09:25 |
| kata-irc-bot | 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 15:42:22 curl: (22) The requested URL returned error: 404 15:42:22 Failed to download tarball from https://storage.googleapis.com/cri-containerd-release/cri-containerd-1.3.7.linux-amd64.tar.gz 15:42:22 Trying to install containerd from source 15:42:23 Note: checking out 'f864905c93b97db15503c217dc9a43eb65670b53'.``` This is | 09:25 |
| kata-irc-bot | what happens with VFIO: ```07:54:53 Trying to install containerd from static tarball 07:55:04 Download tarball from https://storage.googleapis.com/cri-containerd-release/v1.3.7/cri-containerd-cni-1.3.7-linux-amd64.tar.gz 07:55:04 % Total % Received % Xferd Average Speed Time Time Time Current 07:55:04 Dload Upload Total Spent Left Speed 07:55:05 0 0 0 0 0 0 0 | 09:25 |
| kata-irc-bot | 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 07:55:05 curl: (22) The requested URL returned error: 404 07:55:05 Failed to download tarball from https://storage.googleapis.com/cri-containerd-release/v1.3.7/cri-containerd-cni-1.3.7-linux-amd64.tar.gz 07:55:05 Trying to install containerd from source 07:55:05 error: pathspec 'v1.3.7' did not match any file(s) known to git``` | 09:25 |
| kata-irc-bot | <fidencio> So, fixing the version used is just papering over the real issue. | 09:26 |
| kata-irc-bot | <fidencio> The question we should ask is ... why the scripts are falling back to a commit hash in some cases and to a specific tag in different cases? | 09:27 |
| kata-irc-bot | <ssheribe> What is this hash? latest commit? | 09:29 |
| kata-irc-bot | <fidencio> No, and it's actually ... interesting ... ```fidencio@quino ~/go/src/github.com/containerd/cri $ git tag 0.0.0 0.1.0-alpha.0 v0.1.0 v0.1.0-alpha.1 v1.0.0 v1.0.0-alpha.0 v1.0.0-alpha.1 v1.0.0-beta.0 v1.0.0-beta.1 v1.0.0-rc.0 v1.0.0-rc.1 v1.0.0-rc.2 v1.0.1 v1.0.2 v1.0.3 v1.0.4 v1.0.5 v1.11.0 v1.11.1 v1.19.0``` | 09:34 |
| kata-irc-bot | <fidencio> Okay, lunch time. | 09:39 |
| kata-irc-bot | <ssheribe> I saw this tarball existed | 09:40 |
| kata-irc-bot | <ssheribe> I wonder how link is different | 09:40 |
| kata-irc-bot | <ssheribe> bon appetit | 09:40 |
| kata-irc-bot | <jakob.naucke> > @jakob.naucke https://github.com/kata-containers/kata-containers/pull/2193 do you think this one should be ported to stable? @ssheribe so that on its own fixes local snap builds for s390x (and maybe arm & ppc but I can't test those), but builds on Snapcraft won't work at least for s390x/ppc except hopefully with https://github.com/kata-containers/kata-containers/pull/2219, which @julio.montes will be able to test later. So yes, | 10:26 |
| kata-irc-bot | would be good to have, but much better with #2219, and only if that actually fixes it. | 10:26 |
| kata-irc-bot | <jakob.naucke> > The question we should ask is ... why the scripts are falling back to a commit hash in some cases and to a specific tag in different cases? | 11:23 |
| kata-irc-bot | <fidencio> Maybe, I think we have everything worked around right now (waiting for the latest run to ensure everything is working as expected) | 11:27 |
| kata-irc-bot | <fidencio> But we need to improve how we handle our CI wrt the stable branches, we really do need that. | 11:28 |
| kata-irc-bot | <fidencio> Unless someone does weekly backports, we face the very same breakage on pretty much every stable release. | 11:28 |
| kata-irc-bot | <fidencio> @ssheribe, another thing to ensure (and backport things if needed), is that `cargo vendor` does work for the stable branch | 11:29 |
| kata-irc-bot | <fidencio> Otherwise it'll hit us quite hard downstream | 11:29 |
| kata-irc-bot | <ssheribe> @fidencio i'm not sure i got you, you mean to check that stable branch is actually run with stable agent? | 11:34 |
| kata-irc-bot | <fidencio> That was broken for the last release, I know it was fixed for `main`, but I'm not sure whether the patch was backported to `stable-2.1` | 11:36 |
| kata-irc-bot | <ssheribe> ahh ok | 11:37 |
| kata-irc-bot | <fidencio> Basically, ensure that aa264f915fc9a52d8b4063568f712209341dbdcb gets backported | 11:37 |
| kata-irc-bot | <fidencio> And we should have a test for this, I guess I can add something to the static checks soon | 11:38 |
| kata-irc-bot | <ssheribe> That's indeed would be great, maybe worth filing an issue already.. | 11:41 |
| kata-irc-bot | <fidencio> I think I did :slightly_smiling_face: | 11:42 |
| kata-irc-bot | <fidencio> https://github.com/kata-containers/kata-containers/issues/2159 | 11:43 |
| kata-irc-bot | <fidencio> But -ENOTIME to work on that, as usual | 11:43 |
| kata-irc-bot | <ssheribe> maybe kata's magical leprechaun will make someone to pick it up | 11:48 |
| kata-irc-bot | <fidencio> leprechaun | 11:49 |
| kata-irc-bot | <fidencio> While here, let me actually work on this | 12:04 |
| *** ricolin_ is now known as ricolin | 12:48 | |
| fidencio | where's Julio? I really would like to have his PR adding support to go 1.16.x merged sooner than later :-) | 13:19 |
| kata-irc-bot | <jakob.naucke> fidencio, he's up now :slightly_smiling_face: | 13:38 |
| kata-irc-bot | <ssheribe> Hi, I backported things i thought might be necessary for the release, more suggestions & reviews would be appreciated: https://github.com/kata-containers/kata-containers/pull/2226 https://github.com/kata-containers/tests/pull/3717 (I missed first part of the meeting but i saw it was agreed to postpone 2.1.2 till CI/#2222 is fixed) | 16:08 |
| fidencio | devimc: http://jenkins.katacontainers.io/job/kata-containers-2.0-metrics-ubuntu-20-04-PR/949/console ... it makes no sense to me, sincerely :-/ | 20:30 |
| devimc | fidencio, uhmm let me fix it | 20:57 |
| fidencio | devimc: I've opened a PR on the tests repo | 20:57 |
| fidencio | devimc: but I really don't understand the reason why it got so low | 20:58 |
| fidencio | devimc: https://github.com/kata-containers/tests/pull/3718 | 20:58 |
| devimc | fidencio, in that PR I'm fixing some field alignments | 20:58 |
| devimc | so, this means once I have fixed all the structures, memory footprint will be smaller | 20:59 |
| devimc | ~6M smaller per shim | 20:59 |
| devimc | maybe more | 21:00 |
| fidencio | devimc: I'd expect that packing better the fields would help, but then I'd expect two scenarios | 21:01 |
| fidencio | 1. the difference is consistent between all the memory-footprint tests, not only ksm | 21:01 |
| fidencio | 2. the difference would be less brutal (but, hey, that's me not being optmistic) | 21:02 |
| devimc | fidencio, let me revert that | 21:02 |
| devimc | I'll raise a different pr for that (fixing alignments) | 21:02 |
| devimc | fidencio, wdyt? | 21:03 |
| fidencio | devimc: I just commented in the PR suggesting exactly the same | 21:03 |
| fidencio | devimc: leaving all the alignment changes for an upcoming PR | 21:04 |
| fidencio | devimc: thanks! | 21:04 |
| devimc | ok | 21:04 |
| fidencio | devimc: ~6M smaller per shim is ... brutal | 21:06 |
| kata-irc-bot | <eric.ernst> :eyes: | 21:06 |
| devimc | fidencio, done | 21:06 |
| kata-irc-bot | <eric.ernst> our shim shrunk or grew that much? | 21:06 |
| fidencio | it'll shrink that much, or even more, once we have all the fixes for the field alignments | 21:07 |
| devimc | nada yet | 21:07 |
| devimc | fidencio, now you can close it | 21:09 |
| devimc | thx | 21:09 |
| fidencio | I'm not complaining about such an improvement, but it really surprised me | 21:09 |
| fidencio | devimc: gracias! | 21:10 |
| devimc | fidencio, almost *all* structures are misaligned, the thing here is why the compiler is not able to align them ? | 21:10 |
| fidencio | devimc: that's quite frustrating, isn't it? | 21:12 |
| devimc | yes | 21:13 |
| devimc | pita | 21:13 |
| fidencio | devimc: it's a severe pain for a developer to align the fields, and it gets rather complicated on some structs that are basically composed by n other structs | 21:13 |
| devimc | right, that's the painful part of this | 21:14 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!