Tuesday, 2019-10-08

*** rlandy has quit IRC00:01
*** saneax has quit IRC00:25
*** jamesmcarthur has joined #zuul00:29
*** jamesmcarthur has quit IRC01:07
*** wxy-xiyuan_ has joined #zuul01:07
*** jamesmcarthur has joined #zuul01:07
*** jamesmcarthur has quit IRC01:12
*** jamesmcarthur has joined #zuul01:38
*** jamesmcarthur has quit IRC02:16
openstackgerritGabor Lekeny proposed zuul/zuul master: Decode k8s ServiceAccount bearer token  https://review.opendev.org/68710702:18
*** wxy-xiyuan_ is now known as wxy-xiyuan02:33
*** jamesmcarthur has joined #zuul02:36
*** jamesmcarthur has quit IRC02:41
*** igordc has quit IRC02:52
*** bhavikdbavishi has joined #zuul02:57
*** bhavikdbavishi1 has joined #zuul03:00
*** bhavikdbavishi has quit IRC03:01
*** bhavikdbavishi1 is now known as bhavikdbavishi03:01
*** rfolco has quit IRC03:12
*** igordc has joined #zuul03:19
openstackgerritMerged zuul/project-config master: Trigger gate after reverify  https://review.opendev.org/68693304:18
openstackgerritMerged zuul/project-config master: Make recheck alias reverify in gate  https://review.opendev.org/68713604:18
*** fungi has quit IRC04:21
*** fungi has joined #zuul04:26
*** badboy has joined #zuul05:16
*** bolg has joined #zuul05:42
*** swest has quit IRC06:44
*** swest has joined #zuul06:45
*** themroc has joined #zuul06:56
*** tosky has joined #zuul07:24
bolgShrews: Yes, Mac/Darwin uses kqueue. The issue in https://review.opendev.org/c/671674 is that Zuul grar uses Epoll (since it targets Linux machines) using some extended features over Poll (which has broader OS support). For development purposes (developer's machines) we want it to run localy and do not care so much about performance. Implementing KQueue would be significantly more effort since the API is completely different and there would be no benefit. We07:33
bolgare not running Zuul production on BSD/Darwin systems. Maybe I can improve the commit message.07:33
openstackgerritLuigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try)  https://review.opendev.org/67433407:38
*** jpena|off is now known as jpena07:47
openstackgerritLuigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try)  https://review.opendev.org/67433408:28
*** hashar has joined #zuul08:47
openstackgerritLuigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try)  https://review.opendev.org/67433408:51
openstackgerritLuigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try)  https://review.opendev.org/67433408:57
*** kerby has quit IRC09:03
*** bhavikdbavishi has quit IRC09:17
toskyso, https://review.opendev.org/674334 - now with working tests! ^^^ Shrews, AJaeger09:23
*** pcaruana has joined #zuul09:30
*** gtema has joined #zuul09:31
*** rfolco has joined #zuul09:32
AJaeger\o/09:33
openstackgerritFabien Boucher proposed zuul/zuul master: Pagure - Manage project connector refresh when EINVALIDTOK  https://review.opendev.org/68725910:01
*** gtema has quit IRC10:18
*** hashar has quit IRC10:28
openstackgerritSimon Westphahl proposed zuul/nodepool master: Assign static 'building' nodes in cleanup handler  https://review.opendev.org/68726110:31
*** jamesmcarthur has joined #zuul11:15
*** bolg has quit IRC11:15
openstackgerritSimon Westphahl proposed zuul/nodepool master: Sort waiting static nodes by creation time  https://review.opendev.org/68727111:18
*** jamesmcarthur has quit IRC11:19
*** sileht has quit IRC11:21
*** bolg has joined #zuul11:32
*** jpena is now known as jpena|lunch11:41
*** badboy has quit IRC12:19
*** jangutter_ has joined #zuul12:20
*** jangutter has quit IRC12:23
*** jamesmcarthur has joined #zuul12:26
*** jamesmcarthur has quit IRC12:31
*** jpena|lunch is now known as jpena12:37
*** rlandy has joined #zuul12:39
*** bhavikdbavishi has joined #zuul12:42
*** jamesmcarthur has joined #zuul12:51
*** pcaruana has quit IRC12:53
*** bhavikdbavishi has quit IRC13:15
*** jangutter has joined #zuul13:32
*** jangutter_ has quit IRC13:36
*** sileht has joined #zuul13:37
openstackgerritMerged zuul/nodepool master: Use real uuids in fake cloud resource IDs  https://review.opendev.org/68714413:53
*** fdegir has quit IRC14:06
*** jamesmcarthur has quit IRC14:06
*** fdegir has joined #zuul14:07
*** themroc has quit IRC14:14
flaper87I learned today that it's possible to pass a script as the tenant_config instead of a yaml file. Has anyone implemented a script to load as untrusted-project all the projects that Zuul Github's app is installed for?14:29
flaper87If not, is there an example of a script to load the tenant config? I'm assuming the output should be a yaml just like the config it expects14:29
pabelangerI think software factory might be using it, but dmsimard or tristanC may know more14:31
flaper87pabelanger: danke14:32
*** avass has joined #zuul14:35
fboflaper87: here is an example of script https://review.opendev.org/#/c/535878/18/tests/fixtures/config/tenant-parser/tenant_config_script.py. The format is yaml, the same as a classic main.yaml zuul tenant file14:36
*** jamesmcarthur has joined #zuul14:37
*** bolg has quit IRC14:43
*** pcaruana has joined #zuul14:46
openstackgerritKerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters  https://review.opendev.org/68320514:46
openstackgerritKerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters  https://review.opendev.org/68320514:50
openstackgerritKerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters  https://review.opendev.org/68320514:52
Shrewsfungi: your website change https://review.opendev.org/685799 makes the footer a bit weird. Your link text wraps: https://6dca5728c40d535db466-4fcaafdedb24be0c657932ab646595c9.ssl.cf2.rackcdn.com/685799/2/check/zuul-website-build/4120b5f/html/15:02
Shrewswhich is weird, because there is tons of space there15:02
fungiShrews: i was worried it might... want to figure out a phrase which people are likely to find when keyword-searching on a page15:04
fungiyet short enough to fit the flow of the other footer entries15:04
Shrewsfungi: i'm guessing it's just a css width/limit that needs increasing15:04
fungiyeah, could be. i'll try to fiddle with that in a bit15:04
corvusfbo, flaper87: if anyone does have/write that github script, feel free to put it in the tools/ dir to share15:05
Shrews<section class="3u 6u$(narrower) 12u$(mobilep)">15:05
Shrewssuch greek to me15:05
corvusShrews: those numbers are a sort of css construct that says how wide to make it -- the screen is 12 "units" wide, normally that should be made 3 "units" wide (ie 1/4 of screen width) unless the window is small, in which case it should be 1/2 the width, or really small (mobile), it should be the full width15:07
Shrewscorvus: ah, of course  :)15:08
Shrewsso possibly adjusting some of those numbers can help with the wrap around issue15:10
*** jamesmcarthur_ has joined #zuul15:23
*** jamesmcarthur has quit IRC15:26
*** avass has quit IRC15:43
*** mattw4 has joined #zuul16:09
*** bhavikdbavishi has joined #zuul16:13
openstackgerritFabien Boucher proposed zuul/zuul master: Pagure - add the enqueue_ref unit test  https://review.opendev.org/68735116:19
*** igordc has quit IRC16:25
*** rfolco is now known as not_rlandy16:29
*** not_rlandy is now known as rfolco16:29
SpamapS3.10.2-67-g510efce0 ... 67 commits? Is there something preventing a release?16:57
fungiwow, that's a lot of commits for having only been 3 weeks16:59
clarkbI think the gerrit checks api support needed to be stable enough(eg not break ssh gerrit?)17:00
clarkbprobably can be released nowish though17:00
fungii expect it'll be 3.11.0 just looking at the list of commits since the last release17:01
fungithe new autohold stuff, some new reporter actions, gerrit checks plugin support, gerrit robot comments support, http-only gerrit support, new event handlers for pagure, the change to no longer encrypt job-output.txt by default...17:04
Shrewsooh, that reminds me to check my holds...17:04
fungis/encrypt/compress/17:10
* fungi sighs17:10
fungier, i guess the job-output.txt compression default change was in zuul-jobs not in zuul, so the change in zuul was just about adapting the quickstart tests to deal with that17:11
fungibut the other stuff there still makes it at least a minor version bump17:12
SpamapSAgreed17:17
corvusyeah, the gerrit checks api is experimental -- no release notes or anything, but implementing it touched a bunch of other gerrit stuff which we needed to stabilize (it has been now).  it also ended up meaning that some things we did over ssh with gerrit we may now do over http -- we should make sure there's a relnote for that17:25
corvusalso, lots of flying around and conferences and stuff17:25
fungilooks like we've been running e6496fa in opendev since friday17:26
corvuslooks like we're missing the gerrit http relnote.  i'll add that, then i think we're probably set for a release unless others think of something else17:26
fungiand https://review.opendev.org/686853 is the only thing which has merged since opendev's zuul services were all restarted17:29
fungifairly unobtrusive17:29
Shrewscorvus: so, the quick start failures....  latest one is: https://zuul.opendev.org/t/zuul/build/57510d8c2bb745eba8bf2a0ffa742b89/log/job-output.txt#114917:29
Shrewscorvus: if i do: curl http://localhost:8080/a/changes/test1~master~I5ea67b22d2a5467b0747ea0507587051ed9de563//detail17:30
Shrewsi get "Unauthorized"17:30
Shrewsmaybe curl won't work there17:31
fungiShrews: any api path starting in a/ needs (digest i think?) auth performed17:32
fungiare you sure the changes api needs a/ in there?17:32
*** jpena is now known as jpena|off17:32
Shrewsfungi: oh, i see a username/password being specified in the playbook. yeah, i need "something" else there17:33
openstackgerritJames E. Blair proposed zuul/zuul master: Add a relnote about gerrit and http  https://review.opendev.org/68736517:33
fungiit used to be that gerrit api methods which required authentication were prefixed by a/ and those which didn't need authentication weren't, and methods were either one or the other (so you couldn't reach the anonymous methods under the a/ tree)17:34
fungii don't know whether that still holds true for the version of gerrit installed in the quickstart job17:34
corvusShrews: looking at the container logs, it looks like maybe zuul was in the process of running that job?17:34
corvuslike we hit a timeout too soon?17:35
corvusShrews: https://zuul.opendev.org/t/zuul/build/57510d8c2bb745eba8bf2a0ffa742b89/log/container_logs/executor.log17:35
corvusi have to run now; biab17:35
Shrewsthat task only waits 2 minutes, looks like17:37
daniel2So when building the images, its failing at the final point, which kind of makes no sense: https://shafer.cc/paste/view/raw/1a65a4c917:38
daniel2I'm curious where it's getting.r17:38
fungidaniel2: looks like that should be /var/lib/nodepool/images/bare-bionic-0000001650.tar not /var/lib/nodepool/images/bare-bionic-0000001650.r17:41
fungicurious how it's winding up with the latter17:41
daniel2I'm not sure where its getting that17:41
pabelangerwhat does nodepool.yaml look like for the diskimage?17:42
pabelangerIMAGE_TYPES looks to be wrong?17:42
daniel2I guess if I put the formats in the clouds.yaml file I don't need it in the nodepool.yaml file17:43
pabelangerpossible you formatted it in correct?17:44
pabelangerr is from raw?17:44
daniel2    image_format: 'raw' is whats in clouds.yaml17:44
fungii wonder if you have a string there instead of a list?17:45
fungiso it's iterating over the letters in the string "raw"17:45
pabelangeryah17:45
pabelangerthinking that too17:45
pabelangerin nodepool formats is a list17:45
daniel2ah17:45
pabelangerhttps://zuul-ci.org/docs/nodepool/configuration.html#attr-diskimages.formats17:46
daniel2So yeah, I dunno why I did that, it should just be raw without the quotes.17:46
daniel2image_format: raw17:46
*** igordc has joined #zuul17:46
pabelanger++17:46
daniel2| devstack-bionic-0000001651 | devstack-bionic | devops.boi.a10networks.com | a,r,raw,w | building | 00:00:25:51 |17:47
daniel2I dunno why I didn't see that.17:47
daniel2Thanks for the heads up.  Sometimes it just takes another set of eyes :)17:47
Shrewscorvus: oh, hrm... the uri 'content' looks like it's only partial (https://zuul.opendev.org/t/zuul/build/57510d8c2bb745eba8bf2a0ffa742b89/log/job-output.txt#1155). I bet that's messing up the 'until' part of the playbook since it can't parse it as json17:53
Shrewsoh, nm. that's taken into account18:01
*** jamesmcarthur_ has quit IRC18:02
*** jamesmcarthur has joined #zuul18:02
Shrewscorvus: there is, however, only a single message in the content.messages list. Running that task locally and the only entry is for "Uploaded patch set 1."18:04
openstackgerritKerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters  https://review.opendev.org/68320518:07
*** jamesmcarthur has quit IRC18:14
*** bhavikdbavishi has quit IRC18:18
*** daniel2 has quit IRC18:21
*** daniel2 has joined #zuul18:31
*** avass has joined #zuul18:51
*** hashar has joined #zuul18:52
*** ianw_pto is now known as ianw18:59
openstackgerritKerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters  https://review.opendev.org/68320519:07
openstackgerritMerged zuul/zuul master: Add a relnote about gerrit and http  https://review.opendev.org/68736519:13
*** jamesmcarthur has joined #zuul19:17
avassdoes the zuul_console daemon stop tracking the log file for the current job if the file it's reading from isn't updated often enough?19:49
fungiavass: it shouldn't19:51
fungiavass: though it does have to be restarted explicitly if stopped, like say because of a node reboot19:51
*** jamesmcarthur has quit IRC19:52
avassfungi: we have problem with a job where it doesn't output anything for a couple of seconds and it looks like that causes the daemon to stop tracking the file19:53
*** jamesmcarthur has joined #zuul19:53
fungiinteresting19:56
fungiwe have loads of jobs which are silent on their ansible output for more than "a couple of seconds" and haven't exhibited that behavior19:57
avassfungi: strange19:57
fungii've looked at some which go silent for an hour (particularly when a task gets "stuck" deadlocked on something and output doesn't resume until it's timed out)19:58
*** jamesmcarthur has quit IRC19:58
avassare those different tasks then?19:58
funginot sure which "those" you're referring to19:58
fungido you still get the full job-output.txt archived when the build completes? or only up to where you saw the live stream cease?19:59
avassstill get the output from ansible in the logs19:59
clarkbavass: could it be that your client is closing the connection (or some intermediate firewall?) Does reloading give you new output?20:00
avassit only happens if it's a single task that doesn't print anything for a couple of seconds, but it works for the next task20:00
fungisounds like the websocket could be getting prematurely terminated, yeah, like by an overzealous web proxy or firewall20:02
fungidoes reloading the live stream url show you more output after you see it cease?20:02
fungithat should reinitiate the websocket connection i think20:02
avassclarkb: nope20:02
avassfungi*20:03
avassclarkb: don't think it is since it's always the same task. the other ones are fine20:03
fungiahh, okay, so maybe not a disconnect in the path between the client and the zuul-web service... though could i suppose be something disconnecting between zuul-web and the executor's finger socket20:04
avassit also ends with a 'Timeout exception waiting for the logger'20:04
avassI think it was, or something like that error20:07
fungiyeah, it gets raised here https://opendev.org/zuul/zuul/src/branch/master/zuul/ansible/base/callback/zuul_stream.py#L12920:09
avassyeah exactly20:10
avasswas looking at that earlier20:10
avassin the zuul_console source it looks to me that if there's nothing new to read it sleeps for 0.5 seconds and then stops unless the file was truncated20:12
avasshttps://opendev.org/zuul/zuul/src/branch/master/zuul/ansible/base/library/zuul_console.py#L10620:13
avassunless I'm missing something20:13
*** jamesmcarthur has joined #zuul20:15
*** rlandy has quit IRC20:16
SpamapSsounds like maybe that timeout should be configurable20:24
*** pcaruana has quit IRC20:35
avassi'm probably just missing something20:35
SpamapSavass:worst case, you can wrap that command in something that spits out a log entry every 5 minutes or something.20:37
avassyeah I was thinking about doing something like that. but I really wanted to figure this out :)20:38
avassHave to look into it more tomorrow. I'll hop in if I found something20:40
fungii'll be surprised if it's not environment-specific since we stream output from jobs which goes silent for very long periods sometimes20:41
avassfungi: it probably is since that's the only ansible task we have that behaves that way20:43
*** igordc has quit IRC20:59
*** igordc has joined #zuul21:00
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Use JWT for authorization  https://review.opendev.org/68742221:20
corvusclarkb, fungi, tristanC: ^ that and two opendev system-config changes are a pre-req for shadowing21:22
corvusalso, makes anonymous access friendlier21:22
fungicorvus: i appreciate your code comments about the docker client ;)21:27
fungii continue to marvel at the insanity of that behavior21:27
corvusfungi: i thought you would, but i didn't want to spoil it for you by tipping you off :)21:27
corvusthat is a case where they have explicitly chosen to break compatability with the docker client.  that's something to file away for clarkb's appeal.21:31
corvuser, they=podman21:31
corvus(i saw a pr somewhere where they were like "we need to be really careful that we send the right creds to the registry".  which is a nice thing for them to think.)21:31
fungiclearly the docker client doesn't care if you leak your dockerhub creds to just any ol' mirror21:33
fungifoot meet gun21:33
mordredcorvus: bless it's little heart indeed21:33
corvusnext up in supporting buildset registry is one more patch to add registry namespace support, then the rest is fiddling with configuration in zuul-jobs roles.21:36
mordred\o/21:37
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Use JWT for authorization  https://review.opendev.org/68742221:47
corvusmordred, tristanC: pep8 fixes ^21:47
mordredcorvus: +221:49
SpamapSfungi: I've been trying to get to podman for a while now.. hopefully it doesn't footgun in such obvious ways. :-P21:54
tristanCcorvus: how do you test jwt auth?21:57
corvustristanC: 'docker login' / 'podman login'21:58
corvustristanC: it works the same way as basic auth did21:58
corvustristanC: the test job is exercising it here: https://zuul.opendev.org/t/zuul/build/5f8286fa5ae948b8868dcd93e96f5e59/console#2/1/11/ubuntu-bionic21:59
corvus(though we don't have a negative test for that)21:59
tristanCcorvus: using podman login i get AttributeError: module 'jwt' has no attribute 'encode'22:02
tristanCzuul_registry/main.py", line 12722:03
corvustristanC: did you 'pip install pyjwt' or 'pip install jwt'?22:03
tristanCoops, installed the wrong jwt library22:03
corvustristanC: yeah, ask me how i know that :)22:04
corvusit's uncool that the 'pyjwt' library imports as 'jwt'.22:04
*** jamesmcarthur has quit IRC22:04
fungimuch like the pyyaml library imports as 'yaml' i suppose22:04
tristanCalright, i was able to work, but i can't see the token in the conf in my auth.json, thus i'm not sure the code is actually tested22:05
tristanCpodman was able to login*22:05
corvustristanC: it should store the creds (but not the token) in /run/user/*/containers/auth.json22:06
corvustristanC: mine looks like http://paste.openstack.org/show/782274/22:08
corvus(that's testuser:testpass)22:08
tristanCsame here, ok perhaps we should add a test to do: `curl -k -u testuser:testpass https://localhost:9000/auth/token` which outputs a jwt token22:09
tristanCoh my bad, i see that's what the client does already22:10
corvusyeah, but we should add tests that try to push and pull without auth (pull should work, push should fail)22:11
openstackgerritMerged zuul/nodepool master: dib-functional : allow extra elements to be passed  https://review.opendev.org/68588422:11
tristanCit's just that doing `podman login` and `skopeo copy` worked without that jwt patch, thus it's unclear what this enable22:13
corvustristanC: 2 things: it allows you to do 'podman pull' and 'skopeo copy' without providing any credentials.  that will make it easier for folks to pull test images from the intermediate registry for debugging.22:14
corvustristanC: the other thing is that it will handle the case where we configure it as a buildset registry in front of dockerhub and the client is logged into dockerhub.  in that case, we need to accept the credentials provided, whatever they are, and allow read access.22:15
corvusalso, i guess this is a third thing, it is really the only way to have truly anonymous read access and authenticated write access.  which we need for the buildset registry because we don't want to require auth for pulling, but we do need auth for pushing.22:18
openstackgerritTristan Cacqueray proposed zuul/zuul-registry master: Add anonymous pull test  https://review.opendev.org/68743322:19
tristanCnot sure how to test the proxy thing, but here is a test for the anonymous pull ^22:20
corvustristanC: we're not ready to test proxying; that's next22:22
tristanCcorvus: could this happen by chaining two zuul-registries?22:22
corvustristanC: what?22:22
tristanCe.g. have a zuul-registry pretending to be the real dockerhub, and another one to be the proxy22:23
corvustristanC: we don't need to proxy -- the result of our conversation yesterday is that we decided to use registry mirrors to implement the buildset registry functionality.  that means we'll be relying on docker/podman mirror fallback logic.22:24
corvusbut i still need to write more code for zuul-registry before we're ready for that22:24
corvus(mostly around dealing with multiple namespaces)22:25
tristanCcorvus: not sure to understand how mirror fallback logic works and why the zuul-registry needs to implement jwt and accept credentials in that case. i guess the next patch will shed some light :)22:28
fungitristanC: at least the docker client will unconditionally authenticate to the fallback (better still, using the same credentials it used to authenticate to dockerhub)22:29
tristanCi thought this was because of the mitm solution.22:29
tristanCfungi: ha ok, got it22:29
fungiso if the fallback registry doesn't support (and eat) that authentication...22:30
corvusthis is not to support mitm, this is to support the protocol as designed :)22:31
*** jamesmcarthur has joined #zuul22:35
*** hashar has quit IRC22:35
*** jamesmcarthur has quit IRC22:36
*** jamesmcarthur has joined #zuul22:36
tristanCiiuc, this is going to assume image names used by jobs are not fully qualified? for podman, we can configure multiple registries and it will try them in order when doing "podman pull image-name:ref", but when doing "podman pull docker.io/image-name:ref" then it directly pull from dockerhub22:36
openstackgerritMerged zuul/zuul-registry master: Use JWT for authorization  https://review.opendev.org/68742222:37
*** jamesmcarthur has quit IRC22:39
tristanCwhich may be an issue as some user may hardcode the registry name when pulling, e.g. to avoid getting docker.io/fedora instead of registry.fedoraproject.org/fedora22:39
openstackgerritTristan Cacqueray proposed zuul/nodepool master: Ensure both kubernetes and openshift token are b64decoded  https://review.opendev.org/68743522:39
corvustristanC: i'm pretty sure i have this covered -- let me finish work on it and we'll see :)22:45
corvusi should have something by tomorrow22:45
*** jamesmcarthur has joined #zuul22:45
*** jamesmcarthur has quit IRC22:50
openstackgerritTristan Cacqueray proposed zuul/zuul-registry master: Add anonymous pull test  https://review.opendev.org/68743323:05
*** saneax has joined #zuul23:08
*** tosky has quit IRC23:10
*** armstrongs has joined #zuul23:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!