| openstackgerrit | Merged zuul/nodepool master: Use python3 for building releases. Everywhere. https://review.opendev.org/742761 | 00:07 |
|---|---|---|
| ianw | damn it, i did something that gave useful scroll in the debugger/inspect, but then reloaded. now i can't figure out what i did again :/ | 00:28 |
| *** jamesmcarthur has joined #zuul | 01:01 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: web: restore scrollbars and scroll behaviour https://review.opendev.org/742759 | 01:10 |
| ianw | clarkb/corvus: perhaps have a look at ^ in your browser of choice when zuul returns and see if it fixes things for you too. i won't claim it's right, but nothing gets people interested like an obviously incorrect patch :) | 01:12 |
| ianw | does anyone edit the webui with emacs? i find that npm/react/whatever dies every time emacs writes out a temp file | 01:14 |
| *** masterpe has quit IRC | 01:16 | |
| ianw | https://github.com/facebook/create-react-app/issues/9056 - Development server errors due to emacs file lock | 01:16 |
| *** masterpe has joined #zuul | 01:20 | |
| ianw | basically, there's no solution it seems. great | 01:26 |
| *** jamesmcarthur has quit IRC | 01:26 | |
| corvus | ianw: i edited the webpack config file -- it's a temporary solution, but i expect it to hold for a while (i don't plan on blowing that dir away). tristanC turned of tempfiles. | 01:30 |
| corvus | off | 01:30 |
| *** jamesmcarthur has joined #zuul | 01:30 | |
| *** jamesmcarthur has quit IRC | 02:02 | |
| *** jamesmcarthur has joined #zuul | 02:03 | |
| *** jamesmcarthur has quit IRC | 02:08 | |
| *** jamesmcarthur has joined #zuul | 02:13 | |
| *** rlandy|bbl is now known as rlandy | 02:16 | |
| *** sgw has quit IRC | 02:20 | |
| ianw | https://0a753ebf90e04a6fa838-d3b9488846fd47e876cca24d70452d35.ssl.cf2.rackcdn.com/742759/2/check/zuul-build-dashboard-opendev/bfe802e/npm/html/t/openstack/status | 02:22 |
| ianw | page with scrolling from above for reference | 02:22 |
| *** bhavikdbavishi has joined #zuul | 02:36 | |
| *** jamesmcarthur has quit IRC | 02:38 | |
| *** jamesmcarthur has joined #zuul | 02:38 | |
| *** jamesmcarthur has quit IRC | 02:43 | |
| *** bhavikdbavishi1 has joined #zuul | 02:44 | |
| *** bhavikdbavishi has quit IRC | 02:45 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:45 | |
| *** bhavikdbavishi has quit IRC | 03:33 | |
| *** jamesmcarthur has joined #zuul | 03:45 | |
| *** sgw has joined #zuul | 03:46 | |
| *** bhavikdbavishi has joined #zuul | 03:52 | |
| *** rfolco has quit IRC | 03:53 | |
| *** bhavikdbavishi has quit IRC | 04:13 | |
| *** bhavikdbavishi has joined #zuul | 04:15 | |
| *** smyers has quit IRC | 04:55 | |
| *** smyers has joined #zuul | 04:56 | |
| *** sgw has quit IRC | 04:56 | |
| *** sgw has joined #zuul | 05:07 | |
| *** saneax has joined #zuul | 05:25 | |
| *** jamesmcarthur has quit IRC | 05:49 | |
| *** jamesmcarthur has joined #zuul | 05:50 | |
| *** jamesmcarthur has quit IRC | 05:54 | |
| *** bhavikdbavishi1 has joined #zuul | 05:58 | |
| *** bhavikdbavishi has quit IRC | 06:00 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:00 | |
| *** jamesmcarthur has joined #zuul | 06:13 | |
| *** jamesmcarthur has quit IRC | 06:14 | |
| *** jamesmcarthur has joined #zuul | 06:17 | |
| *** jamesmcarthur has quit IRC | 06:22 | |
| *** bhavikdbavishi has quit IRC | 06:23 | |
| *** jamesmcarthur has joined #zuul | 06:27 | |
| *** dmellado has quit IRC | 06:27 | |
| *** jamesmcarthur has quit IRC | 06:40 | |
| *** jamesmcarthur has joined #zuul | 06:40 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: build-python-release: default to Python 3 https://review.opendev.org/742799 | 06:41 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: build-python-release: default to Python 3 https://review.opendev.org/742799 | 06:42 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: build-python-release: default to Python 3 https://review.opendev.org/742799 | 06:43 |
| *** jamesmcarthur has quit IRC | 06:44 | |
| *** sgw1 has quit IRC | 06:53 | |
| openstackgerrit | Merged zuul/zuul master: Re-add temporarily removed pending release notes https://review.opendev.org/742354 | 06:58 |
| *** jcapitao has joined #zuul | 07:09 | |
| *** jpena|off is now known as jpena | 07:11 | |
| *** jamesmcarthur has joined #zuul | 07:13 | |
| *** bhavikdbavishi has joined #zuul | 07:13 | |
| *** jamesmcarthur has quit IRC | 07:20 | |
| *** tosky has joined #zuul | 07:37 | |
| *** harrymichal has joined #zuul | 07:37 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 08:14 |
| *** dmellado has joined #zuul | 08:20 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 08:25 |
| *** avass has quit IRC | 08:27 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role https://review.opendev.org/742732 | 08:29 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 08:29 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Merge upload logs modules into common role https://review.opendev.org/742732 | 08:32 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 08:32 |
| *** nils has joined #zuul | 08:59 | |
| *** bhavikdbavishi has quit IRC | 09:20 | |
| *** bhavikdbavishi has joined #zuul | 09:21 | |
| *** zbr is now known as zbr|ruck | 09:26 | |
| *** nils has quit IRC | 09:32 | |
| *** bhavikdbavishi has quit IRC | 09:46 | |
| *** bhavikdbavishi has joined #zuul | 09:53 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support promote via tenant scoped rest api https://review.opendev.org/742848 | 10:24 |
| *** antony-king has joined #zuul | 10:27 | |
| zbr|ruck | i wonder how hard to would be to enable annotations in zuul/gerrit using the same format as one from github actions, https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#setting-an-error-message | 10:31 |
| *** harrymichal has quit IRC | 10:48 | |
| *** harrymichal has joined #zuul | 10:48 | |
| *** holser has joined #zuul | 10:57 | |
| tobiash | zbr|ruck: you mean reporting file comments? | 10:59 |
| *** harrymichal has quit IRC | 11:02 | |
| *** jcapitao is now known as jcapitao_lunch | 11:04 | |
| antony-king | Hi, We are struggling with our zuul installation as it is now refusing to start up the scheduler with the following error "alembic.util.exc.CommandError: Can't locate revision identified by '269691d2220e'" - any ideas how to clear it ? | 11:13 |
| zbr|ruck | tobiash: YES | 11:13 |
| tobiash | zbr|ruck: zuul already supports leaving file based comments, e.g. see https://review.opendev.org/#/c/742736/4 | 11:14 |
| zbr|ruck | on gh actions/workflows it is enough to produce this format on console and you get annotations for free. | 11:14 |
| zbr|ruck | i know that zuul added ability to add comments on gerrit reviews recently, but i doubt is as easy | 11:15 |
| zbr|ruck | maybe we can improve it to make it compatible | 11:15 |
| tobiash | imho zuul's way is more generic as you can parse the output of any tool and return it | 11:17 |
| *** holser_ has joined #zuul | 11:17 | |
| zbr|ruck | i wonder if these are also working when referring to lines not covered inside review | 11:17 |
| tobiash | but yes, that currently comes at the expense of some extra complexity | 11:17 |
| tobiash | nope, that doesn't work | 11:17 |
| tobiash | gerrit and github both only accept comments for files that are part of the change/pr | 11:18 |
| tobiash | thus zuul filters those | 11:18 |
| zbr|ruck | that is a problem: think about a change bumps the linter and produce errors in various places | 11:18 |
| tobiash | that cannot be solved in zuul | 11:19 |
| tobiash | in this case you have to look at the job log | 11:19 |
| zbr|ruck | i am 100% github adds annotation to any file, unrelated to touched files | 11:19 |
| zbr|ruck | but i do realise this is more gerrit than zuul | 11:19 |
| *** holser_ has quit IRC | 11:19 | |
| tobiash | maybe github, I didn't test that but gerrit refuses certainly | 11:19 |
| *** holser_ has joined #zuul | 11:20 | |
| *** holser has quit IRC | 11:20 | |
| zbr|ruck | but ti does make sense what I say? sometimes the error is not in touched code | 11:21 |
| *** holser_ has quit IRC | 11:21 | |
| tobiash | yes sure, but gerrit is limiting that and the linter log is just one click away ;) | 11:21 |
| *** holser has joined #zuul | 11:22 | |
| zbr|ruck | with current implementation it may take a while for the newbie to find the error, especially if the linter is verbose and error was somewhere before the "42" character limit | 11:22 |
| *** harrymichal has joined #zuul | 11:23 | |
| zbr|ruck | in fact in older zuul versions is even worse, rdo still lacks the "more" link. | 11:23 |
| antony-king | Client: HexChat 2.14.1 • OS: Ubuntu "bionic" 18.04 • CPU: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz (800MHz) • Memory: Physical: 15.1 GiB Total (7.3 GiB Free) Swap: 953.1 MiB Total (640.7 MiB Free) • Storage: 179.0 GB / 269.1 GB (90.1 GB Free) • VGA: Intel Corporation Skylake GT2 [HD Graphics 520] @ Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers • Uptime: 1w 1d 13h 13m 51s | 11:23 |
| tobiash | zbr|ruck: for that it's always useful to tee the linter's output into a separate file and register it as an artifact in zuul | 11:24 |
| zbr|ruck | that reminds me another thing i was looking to implement. making an html report visible as a "tab". | 11:25 |
| zbr|ruck | now when you produce a html file as an artifact, it is accessible from middle of summary page, not quite strait in the face | 11:26 |
| tobiash | that would be interesting | 11:26 |
| zbr|ruck | if we can expose some artifacts as tabs (unittest results), it would improve the browsing experience | 11:26 |
| zbr|ruck | we can likely even load them as an iframe to avoid loosing the navigation context | 11:27 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 11:27 |
| *** holser has quit IRC | 11:27 | |
| tobiash | we might be able to judge on the artifact type and if it's of type html or so add it as a tab | 11:27 |
| zbr|ruck | yep, or have another attribute that tells it to expose it as a tab | 11:28 |
| *** holser has joined #zuul | 11:28 | |
| * zbr|ruck bit worried that some users may produce 20 html files, and that they may not want 20 tabs. | 11:28 | |
| tobiash | then judge on type html_for_inline_tab | 11:30 |
| *** holser has quit IRC | 11:30 | |
| *** holser_ has joined #zuul | 11:30 | |
| *** holser_ has quit IRC | 11:31 | |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** holser has joined #zuul | 11:32 | |
| *** bhavikdbavishi has quit IRC | 11:48 | |
| *** holser_ has joined #zuul | 11:48 | |
| *** holser has quit IRC | 11:50 | |
| *** jamesmcarthur has joined #zuul | 11:56 | |
| *** holser has joined #zuul | 11:58 | |
| *** holser has quit IRC | 11:58 | |
| *** holser has joined #zuul | 11:59 | |
| *** holser_ has quit IRC | 11:59 | |
| *** jamesmcarthur has quit IRC | 12:00 | |
| *** bhavikdbavishi has joined #zuul | 12:01 | |
| tristanC | antony-king: what versions of zuul are you using? | 12:02 |
| *** holser has quit IRC | 12:03 | |
| *** holser has joined #zuul | 12:03 | |
| *** rfolco has joined #zuul | 12:03 | |
| antony-king | tristanC: we are using the latest 3.19 release from the docker environment (I am using the version tagged with "3") | 12:05 |
| tristanC | antony-king: that's odd, because revision '269691d2220e' doesn't seems to be released yet, it got introduced by the `Report retried builds via sql reporter.` changeid I5960df8a997c7ab81a07b9bd8631c14dbe22b8ab | 12:05 |
| antony-king | tristanC: Ah, I was using the latest tag but thought I would be "smart" and use a blessed version - I did a docker-compose pull and then everything broke, hence switching to a tagged release. How do I get hold of the correct id ? | 12:07 |
| *** holser has quit IRC | 12:08 | |
| *** holser has joined #zuul | 12:08 | |
| tristanC | antony-king: that's a good question... i don't think zuul can do database rollback | 12:08 |
| tristanC | perhaps there is a magical sql expression that can fool alembic to use your existing database | 12:10 |
| antony-king | tristanC: well I can move to latest again - I have no problems with that. I have tried forcing a rebuild of the alembic_version table but I get different error - "relation "alembic_version_pkc" already exists" | 12:11 |
| tristanC | antony-king: another solution would be to dump the database, remove the added field, drop the database and reimport | 12:12 |
| tristanC | antony-king: otherwise you might want to stick to master until the next release | 12:12 |
| antony-king | TBH this is getting beyond my knowledge of database recovery (up until now it was a bit of a magic box); your suggestions may be the only solution. I will give them a go. | 12:14 |
| *** jcapitao_lunch is now known as jcapitao | 12:15 | |
| *** bhavikdbavishi has quit IRC | 12:17 | |
| tristanC | antony-king: i'm not an sql expert either, but running `mysqldump --databases zuul | gzip` periodically is quite handy for such situation | 12:17 |
| antony-king | tristanC: The docker-compose pull of the latest tag seems to have done the trick, fingers crossed !. Maybe when I did the pull yesterday (when I broke the system), I got an inconsistent set. Thanks for the suggestions. Have a good day. | 12:19 |
| tristanC | antony-king: that's good to hear, have a good day too! | 12:20 |
| *** antony-king has quit IRC | 12:29 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support promote via tenant scoped rest api https://review.opendev.org/742848 | 12:36 |
| *** jpena|lunch is now known as jpena | 12:39 | |
| *** harrymichal has quit IRC | 12:43 | |
| *** harrymichal has joined #zuul | 12:44 | |
| tobiash | zuul-maint: could you please add https://review.opendev.org/718157 and child to your review list? This implements executor lifecycle improvements (graceful shutdown) that made our lives much much easier. | 12:47 |
| *** avass has joined #zuul | 12:56 | |
| *** nils has joined #zuul | 13:24 | |
| felixedel | zuul-maint: Regarding the scroll behaviour with pageup/down and spacebar in the new UI: I just took a look and the problem seems to be the drawer component which we use to display config errors. When I remove the drawwer component (which is kind of a wrapper around the page content) the scrolling works as expected. | 13:27 |
| felixedel | Though I'm not sure if that is the expected behaviour of the drawer or if I used that component the wrong way :D | 13:35 |
| felixedel | There are also a few issues on GitHub https://github.com/patternfly/patternfly/issues?q=is%3Aissue+is%3Aopen+drawer regarding the drawer component (which is officially in "beta" state), so there might also be some fixes for that in the future | 13:36 |
| *** bhavikdbavishi has joined #zuul | 13:45 | |
| *** jamesmcarthur has joined #zuul | 13:46 | |
| felixedel | But apart from that, everything I've read so far says that scrolling via spacebar or pageup/down is native browser behaviour. So to me that looks like the drawer is messing with this in some way | 13:48 |
| corvus | felixedel: does it need to wrap the page, or can we just include it in in the banner? | 13:48 |
| *** harrymichal has quit IRC | 13:49 | |
| *** harrymichal has joined #zuul | 13:50 | |
| felixedel | corvus: Not sure, currently it is inside the page, but wraps the content, so Page -> Drawer -> <content>. I think this was necessary to make the drawer show up next to the content without moving the content down below the drawer panel when opening it | 13:50 |
| felixedel | But I can check that again | 13:50 |
| *** harrymichal has quit IRC | 13:55 | |
| mnaser | ok, i'm actually digging down to fix the multiarch image fails | 13:56 |
| felixedel | I will check that on Monday | 13:57 |
| mnaser | so this is building on opendev -- "time="2020-07-24T12:33:42Z" level=fatal msg="Error initializing image from source docker://127.0.0.1:58547/vexxhost/node-labeler:latest: unsupported docker v2s2 media type: \"application/vnd.oci.image.layer.v1.tar+gzip\""" | 13:57 |
| mnaser | is it zuul-registry that runs on 127.0.0.1 ? | 13:58 |
| mnaser | i see https://github.com/containers/buildah/issues/2266#issuecomment-608416639 | 13:58 |
| mnaser | well, or i guess its the intermediate registry because that's probably the socat'd port | 13:59 |
| corvus | mnaser: it could be the temporary registry; do you have a link to the log? | 13:59 |
| mnaser | corvus: https://review.opendev.org/#/c/742276/ and the failed job https://zuul.opendev.org/t/vexxhost/build/3efbe52cd48f4738ba97c9fcf3aa5d88 | 13:59 |
| mnaser | corvus: ok it is the socat, because the slurp task, when base64 decoding returned "2020/07/24 12:32:59 socat[18] N listening on AF=2 0.0.0.0:58547" | 14:01 |
| mnaser | which is the exact port i'm trying to push to, so i think this is zuul-registry (i.e. intermediate registry) that's not being happy | 14:01 |
| corvus | it's the buildset registry | 14:03 |
| corvus | mnaser: the path is buildx -> temp registry -> buildset registry -> intermediate registry | 14:03 |
| mnaser | ah, in my case buildset registry is running on the same host as im doing a single docker image and not consuming it later (at least for now) | 14:04 |
| corvus | all three of those are zuul-registry | 14:04 |
| mnaser | corvus: would it then be useful to maybe add collect-docker-logs to the build docker image jobs, for debugging things like this in hte future | 14:05 |
| corvus | mnaser: yes | 14:05 |
| corvus | mnaser: i'm not convinced this is a registry problem yet | 14:05 |
| mnaser | corvus: i agree that there's no leads on where the issue is right now, i just don't have any weird things here, and im inheriting the opendev build jobs which i think zuul/nodepool also consumes | 14:06 |
| mnaser | i remain with the 'feeling' that it might be because of the way the golang parent image is built | 14:06 |
| mnaser | so some of the layers dont have the 'docker' format, so those layers fail to be accepted | 14:06 |
| mnaser | but i feel like we'd see the same failure happen with non-multiarch images too | 14:07 |
| *** sgw1 has joined #zuul | 14:07 | |
| corvus | mnaser: did we talk about adding the docker flag to buildx? | 14:08 |
| mnaser | corvus: i think we did but at the time i didn't dig toooooo much into it as i was busy with a bunch of other stuff, but i plan slash need to get to the bottom of this today :> | 14:09 |
| mnaser | Open10K8S is going to work on a zuul-jobs patch and then add a depends-on | 14:09 |
| corvus | mnaser: or maybe it was the '--format=docker' arg to skopeo | 14:10 |
| *** dpawlik2 has quit IRC | 14:11 | |
| corvus | mnaser: my gut says that if we examined all the registry traffic, we'd see this: https://github.com/containers/buildah/issues/2266#issuecomment-626579548 | 14:12 |
| corvus | mnaser: i'm guessing the bug is in skopeo. like 60% confident there, but enough that i think i'd start by adding that option and see if it works | 14:13 |
| mnaser | corvus: i think so too. ok, let me throw a `--format=docker` attempt at it | 14:13 |
| mnaser | agreed | 14:13 |
| *** bhavikdbavishi1 has joined #zuul | 14:19 | |
| *** bhavikdbavishi has quit IRC | 14:20 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 14:20 | |
| *** jamesmcarthur has quit IRC | 14:26 | |
| *** Goneri has joined #zuul | 14:26 | |
| *** jamesmcarthur has joined #zuul | 14:26 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add docker format option to skopeo in push-to-intermediate-registry role https://review.opendev.org/742892 | 14:27 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add docker format option to skopeo in push-to-intermediate-registry role https://review.opendev.org/742892 | 14:28 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Modify push-to-intermediate-registry role https://review.opendev.org/742892 | 14:29 |
| *** nils has quit IRC | 14:41 | |
| corvus | tristanC: https://review.opendev.org/718157 and child are ready for your +3 | 14:42 |
| *** jamesmcarthur has quit IRC | 14:50 | |
| *** jamesmcarthur has joined #zuul | 14:51 | |
| mnaser | corvus: same failure, but, just realized that this happens in post in a trusted job | 14:52 |
| mnaser | so i dont think our depends-on is going to do much | 14:52 |
| mnaser | cause it ran skopeo without it | 14:52 |
| mnaser | i think the best path might be to request a node hold | 14:53 |
| mnaser | try and run it with --format=docker, and then if that works.. that's 'proof'? i dunno ;( | 14:53 |
| corvus | mnaser: yeah, we can do that; we also have full testing of this in zuul-jobs, you could probably modify the test to use the problematic images? | 14:57 |
| corvus | mnaser: but i'm happy to hold a node after the opendev rename maintenance | 14:57 |
| mnaser | corvus: oh yeah we could do that.. let's do the held node so i can try and see if i can make a very tiny reproducer on that system too | 14:58 |
| mnaser | corvus: for later, reviewing Open10K8S change https://review.opendev.org/#/c/742895/2 shows up an interesting problem, if our push to intermediate registry fails, we don't end up gathering logs so :( | 14:58 |
| mnaser | turns out we _do_ capture the logs, but if the post fails there, we stop | 14:59 |
| corvus | oops; we should separate that out into a different playbook | 15:04 |
| corvus | (we run all the post playbooks) | 15:04 |
| corvus | that way the push can fail, we'll then run the logs playbook, and the job will still fail | 15:04 |
| *** jcapitao has quit IRC | 15:04 | |
| corvus | (we don't have to do anything tricky like save the error code and re-raise) | 15:04 |
| *** sgw is now known as sgw_away | 15:06 | |
| *** holser_ has joined #zuul | 15:14 | |
| *** holser has quit IRC | 15:15 | |
| *** holser_ has quit IRC | 15:19 | |
| *** holser has joined #zuul | 15:19 | |
| -openstackstatus- NOTICE: We are renaming projects in Gerrit and review.opendev.org will experience a short outage. Thank you for your patience. | 15:23 | |
| *** holser_ has joined #zuul | 15:36 | |
| *** holser__ has joined #zuul | 15:39 | |
| *** holser has quit IRC | 15:39 | |
| *** holser_ has quit IRC | 15:40 | |
| mnaser | corvus: can you have multiple post playbooks? that might be a TIL for me | 15:56 |
| clarkb | mnaser: yes it takes a list iirc | 15:56 |
| clarkb | looking for docs on that | 15:56 |
| clarkb | https://zuul-ci.org/docs/zuul/reference/job_def.html#attr-job.post-run yup it takes a list | 15:57 |
| mnaser | ahhh ok perfect | 15:57 |
| mnaser | corvus: dang, skopeo runs on the executor :( | 16:18 |
| mnaser | so my hold is not really productive | 16:19 |
| *** holser__ has quit IRC | 16:19 | |
| *** holser has joined #zuul | 16:19 | |
| clarkb | mnaser: what you can do is run skopeo on the test node against the registry there | 16:20 |
| clarkb | mnaser: and at least confirm ti work if you add the flag | 16:20 |
| mnaser | clarkb: oh yes, that's a good idea | 16:20 |
| mnaser | thank you | 16:20 |
| *** jpena is now known as jpena|off | 16:21 | |
| corvus | that should work, but also the skopeo process was copying to the intermediate registry using our secret creds | 16:22 |
| corvus | mnaser: so i think you'll want to maybe just copy both from and to the buildset registry | 16:22 |
| corvus | mnaser: hopefully that should still trigger the problem | 16:22 |
| *** jamesmcarthur has quit IRC | 16:22 | |
| mnaser | ok cool, will do, let me pull down skopeo | 16:22 |
| corvus | i have to prepare for my hazmat shopping expidition; i'll be back in a little while and will finish up the nodepool release then and check back in with mnaser | 16:23 |
| *** holser has quit IRC | 16:24 | |
| mnaser | corvus: thanks, stay safe! and i'll be reporting here | 16:24 |
| mnaser | https://www.irccloud.com/pastebin/wSLlQ4AK/ | 16:25 |
| mnaser | FATA[0000] unknown format "docker". Choose one of the supported formats: 'oci', 'v2s1', or 'v2s2' | 16:26 |
| mnaser | format is defined as: "MANIFEST TYPE (oci, v2s1, or v2s2) to use when saving image to directory using the 'dir:' transport (default is manifest type of source)" | 16:27 |
| *** SpamapS has quit IRC | 16:27 | |
| *** SpamapS has joined #zuul | 16:29 | |
| mnaser | so it sounds to me like the image that's pushed to 'temp' registry is a 'source' format that cannot be consumed | 16:33 |
| *** jamesmcarthur has joined #zuul | 16:39 | |
| *** zbr|ruck is now known as zbr | 16:41 | |
| mnaser | i dont seem to be able to run docker buildx | 16:41 |
| corvus | mnaser: i'm really about to leave :) but you might try skopeo copying the upstream images too, to see if they're in an unusable format upstream | 16:41 |
| mnaser | corvus: ah good call ,i can try that too | 16:41 |
| mnaser | `skopeo --insecure-policy copy --all docker://docker.io/golang:1.13 docker://insecure-ci-registry.opendev.org:5000/vexxhost/node-labeler:d830455b9d714906bb5b8d1e7fdaaaf5_latest` didn't blow up | 16:42 |
| mnaser | i mean, it complained at 'FATA[0001] Error trying to reuse blob sha256:31dd5ebca5efc5e96a425402fa85e492b02c8fe757dfd3edfdea2a7c67322909 at destination: unable to retrieve auth token: invalid username/password: unauthorized: authentication required ' but thats expected | 16:42 |
| mnaser | oh i need DOCKER_CLI_EXPERIMENTAL=enabled | 16:44 |
| openstackgerrit | Merged zuul/zuul master: Implement graceful termination for the executor https://review.opendev.org/718157 | 17:02 |
| openstackgerrit | Merged zuul/zuul master: Ensure deletion of the command socket https://review.opendev.org/719776 | 17:02 |
| *** sgw_away is now known as sgw | 17:23 | |
| *** hamalq has joined #zuul | 17:29 | |
| *** jamesmcarthur has quit IRC | 17:30 | |
| corvus | mnaser: try copying it to the buildset registry instead of the intermediate registry | 17:37 |
| mnaser | corvus: oh, so from docker.io to buildset, the buildset to intermediate ? yeah that makes sense as a repro | 17:38 |
| corvus | mnaser: you won't be ably to copy anything to intermediate | 17:38 |
| corvus | mnaser: we'll just have to use the buildset registry to stand in for the intermediate registry | 17:39 |
| corvus | intermediate has production creds we need to keep private, but you can do whatever you want with the buildset registry, or the temp registry | 17:39 |
| *** AJaeger has quit IRC | 17:45 | |
| mnaser | ok, finally around to take this | 17:49 |
| *** bhavikdbavishi has quit IRC | 17:52 | |
| mnaser | corvus: i'm really sorry but i'm really struggling to troubleshoot. i can add your keys to help, but in this case buildset registry needs me to maybe do auth first? | 17:53 |
| mnaser | https://www.irccloud.com/pastebin/PJUl2IE1/ | 17:53 |
| mnaser | trying to copy to the temp registry which seems to be on 0.0.0.0 | 17:54 |
| mnaser | :5000 leads to `Get https://zuul-jobs.buildset-registry:5100/v2/: http: server gave HTTP response to HTTPS client ` | 17:54 |
| mnaser | i have to use zuul-jobs.buildset-registry cause if i use localhost or anything else, it complains about ssl cert validity | 17:54 |
| corvus | mnaser: what's the ip? | 17:54 |
| mnaser | `error pinging docker registry localhost:15000: Get https://localhost:15000/v2/: x509: certificate is valid for zuul-jobs.buildset-registry, 198.72.124.203, not localhost ` | 17:54 |
| mnaser | 198.72.124.203 | 17:54 |
| mnaser | and https://zuul.opendev.org/t/vexxhost/build/d830455b9d714906bb5b8d1e7fdaaaf5 is the build if we need to 'rebuild' things | 17:55 |
| mnaser | i'm happy to watch in a screen :) | 17:55 |
| *** jamesmcarthur has joined #zuul | 17:56 | |
| corvus | mnaser: this is working for me: skopeo --insecure-policy copy --all docker://docker.io/golang:1.13 docker://zuul-jobs.buildset-registry:5000/golang:1.13 | 17:56 |
| corvus | mnaser: note the port | 17:56 |
| corvus | 5000 not 15000 | 17:56 |
| mnaser | ah, i was using 5100 because `docker ps` seemed to tell me 5100 was going to the registry | 17:57 |
| corvus | hrm that is confusing | 17:59 |
| corvus | mnaser: oh the socat is on 5000 | 18:01 |
| mnaser | yeah, i think that is for the scenario where buildset is hosted on another vm | 18:01 |
| mnaser | but in this case, it is hosted on teh same system, so its a 'noop' | 18:01 |
| corvus | mnaser: so it's localhost:5000(socat) -> localhost:15000(buildset registry) | 18:01 |
| mnaser | but i guess the 'docker login' happened. for port 5000, so thats why it works with auth | 18:01 |
| corvus | mnaser: exactly | 18:01 |
| mnaser | yeah ~/.docker/config.json confirms that, ok, well that helps a bit | 18:02 |
| corvus | mnaser: i started that copy, and it was running, but i aborted after 3 images -- i'll leave it to you to continue debugging | 18:02 |
| mnaser | corvus: yes, that's helpful, i'll try to do something of doing copy => copy and see if thats a possible failure case, thank you | 18:02 |
| corvus | mnaser: and if we need to get more info from the buildset registry, we can -- it's pretty easy to edit it in place and print debug output, or inspect the storage, etc (since it's all local files) | 18:03 |
| corvus | mnaser: i notice that the test registry (localhost:5100) is the docker registry software, not zuul-registry -- that may be helpful for testing too | 18:04 |
| mnaser | right, yeah, so i think this also means that we are at a state where copying the same image is failing | 18:05 |
| mnaser | `skopeo --insecure-policy copy --all docker://zuul-jobs.buildset-registry:5000/vexxhost/node-labeler:latest docker://zuul-jobs.buildset-registry:5000/vexxhost/node-labeler:d830455b9d714906bb5b8d1e7fdaaaf5_latest` fails | 18:06 |
| corvus | i've re-enqueued the nodepool tag so we can re-run the python release | 18:06 |
| mnaser | interesting, it's 'resolving' the image for both arm64 and amd64 to be the same one... this could be 'ok' though, i'm digging https://www.irccloud.com/pastebin/ti7J0Hxy/ | 18:12 |
| clarkb | corvus: nodepool python release succeeded | 18:12 |
| *** gmann is now known as gmann_lunch | 18:13 | |
| corvus | huzzah | 18:17 |
| corvus | email sent | 18:19 |
| mnaser | oh | 18:28 |
| mnaser | unless im missing something it looks like no projects are doing multiarch builds.. except for python base image right now? | 18:28 |
| mnaser | searching for `linux/arm64` doesn't yield anything except system-config | 18:28 |
| clarkb | mnaser: system-config/zuul.d/docker-images/python.yaml: | 18:29 |
| clarkb | er drop the : suffix | 18:29 |
| mnaser | yeah, hence 'except system-config' i am looking at the build jobs there now to see if the build logs look.. different | 18:30 |
| clarkb | those are doing it. We have been trying to get nodepool to do it but the compiles for cryptography, bcrypt, and pynacl are just too slow so we've disabled it | 18:30 |
| clarkb | mnaser: you can look at what nodepool did look like I got it working with hacks to use openstack constraints and python wheel cache | 18:30 |
| *** jamesmcarthur has quit IRC | 18:30 | |
| clarkb | mnaser: https://review.opendev.org/#/c/741973/ is the end of that stack and should show you what it looks like | 18:31 |
| mnaser | clarkb: oh i see, yeah in my case it's a golang project and it builds in 30s, so i'm good on that front, except for the publishing, but looking at 'working' publish logs will be very helpful | 18:31 |
| mnaser | ok so something interesting is that in my failure case, my buildset-registry is on teh same host | 18:35 |
| *** jamesmcarthur has joined #zuul | 18:41 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: dnm: test multiarch https://review.opendev.org/742967 | 18:46 |
| *** jamesmcarthur has quit IRC | 18:47 | |
| *** jamesmcarthur has joined #zuul | 18:50 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Allow ara-report to run on any node https://review.opendev.org/742971 | 18:52 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Allow ara-report to run on any node https://review.opendev.org/742971 | 18:56 |
| mnaser | i learned something today | 19:02 |
| mnaser | you don't need to do apt clean all | 19:02 |
| mnaser | https://github.com/moby/moby/blob/03e2923e42446dbb830c654d0eec323a0b4ef02a/contrib/mkimage/debootstrap#L82-L105 | 19:02 |
| fungi | i assume you mean specifically when doing docker container builds | 19:11 |
| fungi | judging from that code snippet | 19:11 |
| mnaser | fungi: yes, correct | 19:18 |
| mnaser | also -- tried another much simpler build and not seeing that same issue -- https://review.opendev.org/#/c/742976/ | 19:18 |
| *** gmann_lunch is now known as gmann | 19:19 | |
| mnaser | it must be something to do either with base image or multistage | 19:19 |
| mnaser | but nodepool does multistage, so that narrows it down to the image | 19:23 |
| mnaser | OOOH, i use FROM scratch for the final image | 19:25 |
| mnaser | i wonder if that has to do with it | 19:25 |
| corvus | mnaser: oooh that sounds plausible. also, neat. :) | 19:28 |
| mnaser | corvus: i will try to rebuild that same thing with buildx but not using from scratch | 19:28 |
| mnaser | corvus: well, that did it.. | 19:38 |
| mnaser | changing the FROM scratch to FROM golang:1.13 .. fixed it ;\ | 19:38 |
| mnaser | im gonna guess that maybe its literally taking FROM scratch.. or something | 19:42 |
| corvus | mnaser: okay, so from scratch, with buildx, produces something that skopeo doesn't like? | 19:42 |
| mnaser | corvus: yeah, let me try to make an even smaller reproducer | 19:42 |
| mnaser | ok i have a really small reproducer | 19:46 |
| mnaser | corvus: http://paste.openstack.org/show/796298/ this is a 100% fail | 19:47 |
| mnaser | so i think the thing here is 'scratch' might actually exist in registry as `application/vnd.oci.image.layer.v1.tar+gzip` format | 19:48 |
| openstackgerrit | Jimmy McArthur proposed zuul/zuul-website master: Add new Supporter Logos https://review.opendev.org/742984 | 19:55 |
| corvus | mnaser: huh, i was thinking that maybe buildx built it in that format | 19:55 |
| corvus | jamesmcarthur: i think you may be missing some "git add"s in that change | 19:57 |
| mnaser | corvus: or that too, i don't know enough about the details to be honest :( | 19:57 |
| corvus | mnaser: can you retry your test case with the temporary registry? | 19:57 |
| mnaser | corvus: ah good idea! one second | 19:57 |
| corvus | if it fails, that should completely exclude zuul-registry | 19:58 |
| mnaser | "Error initializing source docker://127.0.0.1:5100/test:latest: error pinging docker registry 127.0.0.1:5100: Get https://127.0.0.1:5100/v2/: http: server gave HTTP response to HTTPS client " -- i was running into this earlier | 19:59 |
| corvus | mnaser: "127.0.0.1:5100" is the temp registry, and should be logged in | 19:59 |
| mnaser | let me find how to force http | 19:59 |
| mnaser | `--src-tls-verify=false` seems to be the trick | 20:00 |
| mnaser | `FATA[0000] Error initializing image from source docker://127.0.0.1:5100/test:latest: unsupported docker v2s2 media type: "application/vnd.oci.image.layer.v1.tar+gzip"` -- yep, ain't the registries' fault | 20:00 |
| corvus | w00t | 20:00 |
| mnaser | its a buildx/skopeo thing | 20:00 |
| corvus | mnaser: i think maybe that's enough to open a skopeo issue | 20:00 |
| mnaser | yeah. the reproducer is helpful | 20:01 |
| *** SpamapS has quit IRC | 20:03 | |
| mnaser | corvus: https://github.com/containers/skopeo/issues/991 | 20:05 |
| *** SpamapS has joined #zuul | 20:05 | |
| corvus | mnaser: ++ | 20:14 |
| mnaser | in the meantime my workaround is to do FROM debian:stable-slim .. good enough | 20:15 |
| mnaser | ah its so nice that this is working, the chrony image is already deployed and working across multiple archs :D | 20:15 |
| corvus | mnaser: want to leave that node held for a bit in case they have followups? | 20:15 |
| mnaser | corvus: i think that might be useful indeed | 20:15 |
| corvus | cool, will do | 20:15 |
| *** jamesmcarthur has quit IRC | 20:17 | |
| corvus | dmsimard: sent announcement http://lists.zuul-ci.org/pipermail/zuul-announce/2020-July/000080.html | 20:19 |
| *** jamesmcarthur has joined #zuul | 20:36 | |
| jamesmcarthur | Oh crap. | 20:38 |
| jamesmcarthur | Corvus: so true. | 20:38 |
| openstackgerrit | Jimmy McArthur proposed zuul/zuul-website master: Add new Supporter Logos https://review.opendev.org/742984 | 20:47 |
| jamesmcarthur | corvus: should be there now: https://review.opendev.org/#/c/742984/ | 20:47 |
| *** jamesmcarthur has quit IRC | 20:49 | |
| *** jamesmcarthur has joined #zuul | 20:59 | |
| openstackgerrit | Jimmy McArthur proposed zuul/zuul-website master: Add new Supporter Logos https://review.opendev.org/742984 | 21:01 |
| *** rfolco has quit IRC | 21:07 | |
| *** rfolco has joined #zuul | 21:08 | |
| *** rfolco has quit IRC | 21:12 | |
| tristanC | jamesmcarthur: hello, has the survey results been published somewhere? | 21:15 |
| jamesmcarthur | tristanC: not yet. We typically let the surveys run through late July/early August. Then we have to anonymize the data and do analysis. | 21:26 |
| jamesmcarthur | We're looking for community volunteers if anyone is interested. Only requirement is to sign an NDA to protect the anonymity of the companies that respond. | 21:26 |
| *** jamesmcarthur has quit IRC | 21:34 | |
| *** jamesmcarthur has joined #zuul | 21:48 | |
| *** jamesmcarthur has quit IRC | 21:55 | |
| *** rlandy has quit IRC | 22:14 | |
| mnaser | mm, this seems like a zuul bug | 22:21 |
| mnaser | https://review.opendev.org/#/c/742981/1 -- you can see that the `openstack-operator:images:upload:chronyd` job was actually removed (specifically https://review.opendev.org/#/c/742981/1/zuul.d/chronyd-jobs.yaml ) | 22:22 |
| mnaser | yet zuul still ... ran it? | 22:22 |
| mnaser | git grep shows no references when checking that out locally | 22:22 |
| clarkb | mnaser: for debugging that the zuul inventory for the job is a good start it shows you wher ethe job came from | 22:22 |
| *** _erlon_ has quit IRC | 22:23 | |
| mnaser | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2e2/742981/1/gate/openstack-operator:images:upload:chronyd/2e2ab0d/zuul-info/inventory.yaml | 22:23 |
| mnaser | the file it pulls it from has been deleted? | 22:24 |
| mnaser | <Job openstack-operator:images:upload:chronyd branches: None source: vexxhost/openstack-operator/zuul.d/chronyd-jobs.yaml@master#13> -> <Job openstack-operator:images:upload:chronyd branches: None source: vexxhost/openstack-operator/zuul.d/chronyd-jobs.yaml@master#26> | 22:24 |
| clarkb | ya that implies to me that it is using merged state | 22:24 |
| clarkb | does it use secrets maybe? | 22:24 |
| mnaser | ooou, `vexxhost-upload-docker-image` does use secrets | 22:25 |
| mnaser | but this job was running in gate so it .. should be ok? | 22:25 |
| clarkb | ya gate is post review, but I'm not sure if it uses speculative state for secrets? | 22:26 |
| clarkb | corvus: ^ | 22:26 |
| clarkb | I wonder if removals from job lists can be speculatively applied regardless (since it removes potentially dangerous things) | 22:26 |
| corvus | mnaser, clarkb: i suspect it's because of the file removal | 22:32 |
| corvus | and i agree, that's a little buggish | 22:32 |
| corvus | it might be due to how the caching is implemented | 22:32 |
| corvus | mnaser: can you try truncating the file first, then removing it in a second change? | 22:33 |
| corvus | i think that would confirm the bug/provide immediate relief | 22:33 |
| mnaser | corvus: ok will try that’s | 22:46 |
| mnaser | corvus: that did it. https://usercontent.irccloud-cdn.com/file/9q7ET5vc/image.png | 22:49 |
| *** rfolco has joined #zuul | 22:54 | |
| *** holser has joined #zuul | 22:57 | |
| *** rfolco has quit IRC | 22:58 | |
| *** hamalq has quit IRC | 23:10 | |
| *** holser has quit IRC | 23:19 | |
| *** holser has joined #zuul | 23:29 | |
| *** sgw1 has quit IRC | 23:35 | |
| *** tosky has quit IRC | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
