Thursday, 2019-11-21

*** tosky has quit IRC00:22
*** jamesmcarthur has joined #zuul00:35
*** mhu has quit IRC00:46
mnaserianw: you know what i think ive seen thatttttt00:48
mnaseri remember tracking it down to the exact spot00:49
mnaserlet me try to find it00:49
mnaserianw: i think disabling checksumming actually made it pass this https://github.com/openstack/diskimage-builder/blob/cfba9ea79d903574afe58bbe3964c050676ee8da/diskimage_builder/lib/common-functions#L68-L7200:50
mnaserim not sure if i fully understood that part00:50
ianwmnaser: yeah, that's exactly the point01:15
ianwthe only thing i think it can be waiting for is the outfilter.py that we run things through01:16
ianwwe could just wait for those pids ... i don't know if that is a fix or a horrible hack papering over the real problem01:18
mnaserianw: i remember looking into it and the outfilter having to do with it i think01:23
mnaserim sorry i cant do much other than "i confirm what you're seeing" heh01:23
ianwno that's super useful!  nice to know i'm not alone :)01:23
*** bhavikdbavishi has joined #zuul02:57
*** bhavikdbavishi has quit IRC02:59
*** jamesmcarthur has quit IRC03:09
*** jamesmcarthur has joined #zuul03:10
*** jamesmcarthur has quit IRC03:15
ianwi swear there's some sort of race condition with https://review.opendev.org/#/c/693464/2003:19
*** rlandy|rover|bbl has quit IRC03:20
ianwtwice it reported "This change depends on a change with an invalid configuration."03:20
ianwthen nothing changed, but the 3rd recheck and it started03:20
ianwhttps://paste.fedoraproject.org/paste/xnxogv52-DnCiJbc3CjMrA03:23
ianwit looks like it's https://review.opendev.org/#/c/694177/ in the chain it doesn't like03:25
ianwthat Depends-On: https://review.opendev.org/694175 ... which is a dib change ... this is cross-tenant03:25
*** jamesmcarthur has joined #zuul03:40
*** jamesmcarthur has quit IRC03:45
*** jamesmcarthur has joined #zuul04:01
*** jamesmcarthur has quit IRC04:14
*** jamesmcarthur has joined #zuul04:15
*** jamesmcarthur has quit IRC04:21
*** jamesmcarthur has joined #zuul04:46
*** jamesmcarthur has quit IRC04:52
*** bjackman has joined #zuul05:04
*** raukadah is now known as chandankumar05:09
*** igordc has quit IRC05:11
ianwhttps://zuul.opendev.org/t/zuul/build/9f38348bffaa43799c807e055e070974 ... it worked!  nodepool-launcher & builder containers in the functional job05:26
*** reiterative has quit IRC05:35
*** swest has joined #zuul06:25
*** jamesmcarthur has joined #zuul06:48
*** jamesmcarthur has quit IRC06:52
*** saneax has joined #zuul07:01
*** bjackman has quit IRC07:08
*** chandankumar is now known as chkumar|rover07:22
*** pcaruana has joined #zuul07:27
*** jamesmcarthur has joined #zuul07:28
*** tosky has joined #zuul07:31
*** jamesmcarthur has quit IRC07:34
*** bolg has joined #zuul07:48
*** avass is now known as Guest7250007:51
*** avass has joined #zuul07:51
*** adso78 has joined #zuul07:55
*** bolg has quit IRC08:02
*** bolg has joined #zuul08:03
*** bolg has quit IRC08:05
*** bolg has joined #zuul08:15
*** jamesmcarthur has joined #zuul08:15
*** bolg has quit IRC08:17
*** jamesmcarthur has quit IRC08:20
*** bolg has joined #zuul08:27
*** bolg has quit IRC08:29
*** jpena|off is now known as jpena08:35
*** reiterative has joined #zuul08:38
*** bolg has joined #zuul08:39
*** bolg has quit IRC08:41
*** jamesmcarthur has joined #zuul08:47
*** bolg has joined #zuul08:51
*** jamesmcarthur has quit IRC08:52
*** bolg has quit IRC08:55
*** bolg has joined #zuul09:02
*** bolg has quit IRC09:07
*** bolg has joined #zuul09:14
*** sshnaidm|ruck is now known as sshnaidm09:19
*** bolg has quit IRC09:20
*** bolg has joined #zuul09:26
*** bolg has quit IRC09:30
*** saneax has quit IRC09:44
*** saneax has joined #zuul09:44
*** sshnaidm has quit IRC10:01
*** sshnaidm has joined #zuul10:05
*** jamesmcarthur has joined #zuul10:49
*** jamesmcarthur has quit IRC10:53
*** rfolco has joined #zuul11:18
*** rfolco is now known as rfolco|doctor11:18
*** jamesmcarthur has joined #zuul11:25
*** jamesmcarthur has quit IRC11:31
*** avass has quit IRC12:00
*** jamesmcarthur has joined #zuul12:01
*** jpena is now known as jpena|lunch12:14
*** jamesmcarthur has quit IRC12:50
*** jamesmcarthur_ has joined #zuul12:50
*** rlandy has joined #zuul12:58
*** jpena|lunch is now known as jpena13:20
*** rfolco|doctor is now known as rfolco13:23
fungizuul-maint: an interesting problem... right now https://zuul.opendev.org/api/tenant/openstack/status has well over 1k entries in its "gate' pipeline, almost all of which have an empty heads list13:26
fungiif the dashboard attempts to render that content, it kills your browser (or your entire computer)13:27
fungii saved a snapshot since the last time folks reported similar browser symptoms they cleared up on their own eventually13:28
*** hashar has joined #zuul13:28
fungier, now that i think about it, was the builds dashboard chewing up folks' browsers last week not status13:29
AJaegerfungi: http://zuul.opendev.org/t/openstack/jobs kills my browser - is that calling to /status?13:29
fungioh, wow. i wouldn't think so13:30
*** hashar has quit IRC13:32
*** yolanda has quit IRC13:33
*** yolanda has joined #zuul13:36
tobiashfungi: interesting, I can observe the same thing in our deployment13:38
tobiashthanks for the note13:38
tobiashthat might even explain the memleak we still see in the scheduler...13:39
tobiashAJaeger: we have a similar problem with the jobs page, there the js seems to oom the browser13:41
tobiashbut the jobs page is a different problem13:41
AJaegertobiash: the jobs page is the problem, not status...13:42
tobiashAJaeger: fungi wrote about the status endpoint (which is used by the status page)13:42
tobiashso that are actually two problems13:42
tobiash* the status endpoint seems to contain old entries13:43
tobiash* the jobs page ooms the browser with many jobs or maybe jobs with special config13:44
*** mhu has joined #zuul13:45
fungiyeah, sorry, trying to debug it in here and in #openstack-infra at the same time13:46
tobiashok, looking more closely at the status endpoint it looks like the dependent pipeline doesn't delete empty change queues -> all earlier change queues are reported13:46
fungii misunderstood the initial problem report and thought it was /status but it's /jobs13:46
tobiashthat makes one per repo that was in use since the last scheduler startup13:46
tobiashah ok, we see the jobs problem as well in one of our bigger tenants13:47
tristanCadding some console.log in my local copy, it seems like the jobs page is having issue with the one named "opendev-uploader-docker-image"13:48
tobiashtristanC: so it might be a special job construct and not just the number of jobs?13:48
fungiyeah, i doubt it's number of jobs13:49
tristanCtobiash: i can't tell, reloading the page make the debug stop at another job name13:50
tobiashhrm13:50
fungii watched a "web content" subprocess in firefox 70 consume ~11gb of memory in a matter of a few minutes trying to load the jobs dashboard13:50
mhuhey there, could I get reviews on https://review.opendev.org/#/c/641099/ ? it's been in limbo for a while13:50
fungiso seems like something self-referential maybe causing runaway memory allocation13:50
fungiinfinite loop kinda territory13:50
openstackgerritMatthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry  https://review.opendev.org/64240813:51
tristanCoh wait no, the console wasn't scrolled after my refresh, it did stopped again at opendev-uploader-docker-image13:51
tobiashoh cool, now the question is, is there anything special with this job?13:51
tristanCI added a console.log("getNode(", job, ")") here https://opendev.org/zuul/zuul/src/branch/master/web/src/containers/jobs/Jobs.jsx#L6313:52
fungihuh, http://zuul.opendev.org/t/openstack/job/opendev-uploader-docker-image returns a 404 for http://zuul.opendev.org/api/tenant/openstack/job/opendev-uploader-docker-image13:53
tobiashI don't see it in the jobs endpoint as well13:54
fungidid you mean opendev-upload-docker-image maybe?13:55
fungihttp://zuul.opendev.org/t/openstack/job/opendev-upload-docker-image returns details13:55
fungithe parentage on it seems fairly straightforward too, if that's the one13:55
fungiand there are no builds for that job13:57
tobiashcould it be a problem that this job is listed as parent before it's listed itself? http://paste.openstack.org/show/786491/13:57
fungithough a side note, it might be nice if the job description renderer could handle sphinx include tags13:57
tristanCfungi: yeah, though i think it's the api that should resolve the include directive13:59
fungitristanC: ahh, good point, the dashboard lacks sufficient context to find those inclusions14:00
fungitobiash: i would be surprised if the dashboard code relied on jobs being sorted in ascending order of ancestry14:01
fungii would expect the problem to be far more prevalent since i doubt we intentionally sort that data (though i certainly could be mistaken)14:01
tristanCtobiash: it should'nt matter, the list is processed in multiple stage14:01
tobiashok14:01
tristanCi'm commenting parent resolution code trying to find what could be the issue, so far no luck14:02
AJaegeraccessing /jobs worked a few weeks ago - did we add so many more jobs or is this a regression from some other change?14:05
tristanCcommenting this block https://opendev.org/zuul/zuul/src/branch/master/web/src/containers/jobs/Jobs.jsx#L66-L73  makes the ui work14:07
*** Goneri has quit IRC14:09
*** jamesmcarthur_ has quit IRC14:10
fungithat loop does seem ripe for recursion issues14:11
fungithough i don't immediately see any self-referential calls there14:13
tristanCfungi: the parents list is processed later down the loop14:14
tristanCthe issue start with this job zuul-upload-image  (when adding a console.log here: https://opendev.org/zuul/zuul/src/branch/master/web/src/containers/jobs/Jobs.jsx#L12714:15
fungiaha, so you were effectively zeroing out the parents list14:16
fungicausing the processing loop to no-op14:17
*** jamesmcarthur has joined #zuul14:20
openstackgerritMatthieu Huin proposed zuul/zuul master: enqueue: make trigger optional  https://review.opendev.org/69544614:28
tristanCso the getNode doesn't seems to be the issue, i think it's the tree view component that is getting confused by a bad parents list, e.g. the page also works by commenting https://opendev.org/zuul/zuul/src/branch/master/web/src/containers/jobs/Jobs.jsx#L14214:30
*** tosky_ has joined #zuul14:31
tristanCthus i don't think it's related to the zuul-upload-image or opendev-upload-docker-image, it could be caused by another job14:32
*** tosky has quit IRC14:32
*** tosky_ is now known as tosky14:33
tristanCpreventing adding a jobnode to multiple parent also fixes the issue, here is another fix and a list of suspicious job: http://paste.openstack.org/show/786495/14:39
corvusso the issue is that the tree view, naturally, can't display nodes with multiple parents14:42
*** fdegir has quit IRC14:42
fungimakes sense14:43
corvusgiven that, this is always going to be an approximation, so i think that fix (use the parent of the first variant) is probably best14:43
*** fdegir has joined #zuul14:43
tristanCcorvus: iirc it used to work, we made that whole get each parent per variant to support the use-case where a job can inherit from multiple parent...14:43
corvustristanC: oh, did we show the job multiple times in the tree?14:44
corvusthat's the only other way i can think to handle this14:44
tristanCcorvus: i thought we did, perhaps i don't remember correctly.14:44
fungii suppose if we wanted to get fancy we could add some sort of indicator to nodes with additional parents (could even set an on-click handler to jump/highlight additional enumerated parents, though i doubt that would be particularly useful)14:46
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support  https://review.opendev.org/69505114:48
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: only display the first parent in the jobs tree  https://review.opendev.org/69545014:49
openstackgerritMatthieu Huin proposed zuul/zuul master: enqueue: make trigger optional  https://review.opendev.org/69544614:51
tristanCcorvus: in case multi parent never worked, here is the easy fix ^14:51
corvustristanC: i'm happy to +2 that (and that seems like a good quick fix for the current breakage), but also happy to support rendering the job multiple times if you prefer that as a followup.  thanks for digging in.  :)14:51
tristanCcorvus: yeah, i think it may have worked in simple case, but perhaps when the schema is more complex (e.g. a job attached to a multi-parented parent) then the component is unable to render the tree14:54
*** tosky has quit IRC14:54
tristanCthat could be fixed by using unique job's React.Fragment per attached parent14:56
tristanCi can have a look later14:56
*** tosky has joined #zuul14:57
*** igordc has joined #zuul15:01
*** pcaruana has quit IRC15:02
*** igordc has quit IRC15:05
*** igordc has joined #zuul15:06
*** igordc has quit IRC15:14
*** igordc has joined #zuul15:14
*** pcaruana has joined #zuul15:20
AJaegertristanC: thanks for the fix!15:25
*** jpena is now known as jpena|off15:41
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support  https://review.opendev.org/69505115:47
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: handle jobs that have multiple parent  https://review.opendev.org/69545015:55
tristanCcorvus: i think this is a better fix, but i'm not sure how to differentiate the branch based variants from the multi-parent ones15:56
tristanCcorvus: actually i can't find the example where two job variants inherit from different parent. iirc it was used by some docker-build?16:04
*** chkumar|rover is now known as raukadah16:08
tristanCoh found it, https://review.opendev.org/#/c/629983/3/.zuul.yaml . So if that's something we want to support, shouldn't the parent attribute be a list, so that we only have variants per branch?16:09
tobiashtristanC: we already discussed this a while ago, so atm we accidentally support multiple inheritance by this process and I think nobody objected making this official by making the parents a list16:10
tobiashbut I think nobody has implemented it yet16:10
tristanCtobiash: it seems like that would make the model simpler, then 'variants' would be equivalent to 'branches' iiuc16:11
openstackgerritMatthieu Huin proposed zuul/zuul master: [WIP] admin REST API: zuul-web integration  https://review.opendev.org/64353616:20
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support  https://review.opendev.org/69505116:26
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support  https://review.opendev.org/69505116:54
*** jamesmcarthur has quit IRC17:27
*** jpena|off is now known as jpena17:27
*** jamesmcarthur has joined #zuul17:28
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support.  https://review.opendev.org/69505117:41
*** michael-beaver has joined #zuul17:50
*** pcaruana has quit IRC17:51
*** mattw4 has joined #zuul17:51
openstackgerritTristan Cacqueray proposed zuul/zuul master: config: add tenant.toDict() method and REST endpoint  https://review.opendev.org/62134418:07
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: install-podman: also install uidmap  https://review.opendev.org/69554518:09
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support  https://review.opendev.org/69505118:10
*** pcaruana has joined #zuul18:14
*** tosky has quit IRC18:29
*** gmann is now known as gmann_afk18:38
*** saneax has quit IRC18:42
*** pcaruana has quit IRC18:42
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support  https://review.opendev.org/69505118:51
*** pcaruana has joined #zuul19:06
corvusoh sweet, that patch has gotten to the point where the errors are now actually in the 'run-buildset-registry' role :)19:25
fungiprogressive error development19:26
mordred\o/19:29
*** jpena is now known as jpena|off19:36
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support  https://review.opendev.org/69505119:37
*** openstackstatus has quit IRC19:50
openstackgerritMerged zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint  https://review.opendev.org/64109919:50
*** dtroyer has quit IRC19:50
*** dtroyer has joined #zuul19:51
*** openstackstatus has joined #zuul19:52
*** ChanServ sets mode: +v openstackstatus19:52
*** mattw4 has quit IRC19:54
*** mattw4 has joined #zuul19:54
*** ianw has quit IRC20:20
*** ssbarnea has quit IRC20:26
*** ianw has joined #zuul20:27
openstackgerritMerged zuul/zuul-jobs master: install-podman: also install uidmap  https://review.opendev.org/69554520:39
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support  https://review.opendev.org/69505120:39
ianwcorvus / modred: i got the nodepool functional job working with containerised builder & launcher yesterday ... which means as i start the process of getting it all reviewable i'll have to make a decision on the container namespace for dib20:42
ianwhttps://review.opendev.org/#/c/693971/20:43
ianwi don't really mind, but will have to choose something :)20:43
ianwit feels like opendev/diskimage-builder works?  but not moving it to opendev/ in git, to avoid making work for people20:44
*** gmann_afk is now known as gmann20:45
corvusianw: this is probably more of an infra question20:45
ianwcorvus: ok, so we can probably rule out zuul/diskimage-builder?20:46
corvusoh yeah, sorry.  i think so.  i think we considered that when we started the zuul project and decided it wasn't the right place.  i don't think that has changed.  so i think it's mostly an openstack/opendev question.20:47
ianwok; and from a nodepool point of view, inheriting from either doesn't really make a difference?20:48
corvusianw: not that i can think of.  now, part of me thinks it's weird for it to inherit from dib at all (since dib is one component of one part of one driver).  it seems like it should just be adding in what's needed for dib to work.20:50
corvuslike, what if we needed something else to build aws images, would we inherit from that?20:50
corvusbut i haven't thought about this much20:50
*** rfolco has quit IRC20:50
clarkbpersonally I think it should be ok to produce an image that works with many drivers20:51
corvusbut that seems easy to change later; i don't think getting the perfect inheritance ordering needs to block anything.20:51
clarkboptimizing in this particular space seems to always cause headaches for users when they realize they cant use $oyherthing because it isnt supported in the current special builf20:51
corvusit looks like the dockerfile for the dib image is just installing dib; that seems like something we could put in the dockerfile for nodepool-builder20:52
ianwi think the reason to do that was that it was installing all the dependencies, etc; stuff that we wanted isolated from the builder image20:53
ianwlike if dib has a new dependency, you don't have to update the builder image because it's layer on20:53
corvusmakes sense.  that works for this case, but as soon as we have a second thing like that, we're back in the same boat.  so it's probably unsafe to consider it permanent.20:54
ianwmordred did actually start more or less that suggestion (add dib to nodepool) https://review.opendev.org/#/c/693306/20:54
ianwwe already sort of do i guess - openstacksdk?  we want to add that into the mix for testing too20:55
corvusi wonder if we could run bindep on nodepool's bindep file20:55
corvuser dib's bindep file20:55
ianwthat would mean the python builder was cloning dib source?20:57
corvusyeah, or otherwise getting the source (or that file) into the build image.20:58
corvus(i'm flexible when it comes to dockerfile purity :)20:58
mordredcorvus: I thought a bit about doing that - but it seemed to me like we'd essentially be building something very similar to the dib image in the first phase of such a nodepool dockerfile - so why not just use the dib image as a base. I agree, it's a little weird :)20:59
*** openstack has joined #zuul21:18
*** ChanServ sets mode: +o openstack21:18
*** sshnaidm has quit IRC21:21
*** pcaruana has quit IRC21:25
clarkbalso re the specific example of aws I believe that dib is probably 95% capable of producing ec2 images already. There is a good chance that it might even work already?21:28
corvusyeah, i was just trying to come up with an example21:31
corvusthe specifics aren't important; what's important is that we explicitly designed the architecture so that dib isn't required.  it's just one implementation of one (could be more later) optional part of the system.21:32
corvusit should be part of the nodepool-builder image, but it shouldn't always be the only thing on there.21:33
clarkbyup I agree. I just didn't want people thinking for aws they had to use something other than dib21:33
corvus++21:33
clarkblooks like the aws image import tool (you upload image to s3 then run import-image against that object) supports raw images. In this case dib should be fine as is. Just need nodepool-builder bits that know how to do that process against aws21:34
corvusit's sort of looking like if i start a container with podman as a regular user, it doesn't show up in "podman ps" run as that user, but does with sudo21:34
corvusi don't understand that21:34
corvusthat's not the behavior i see locally, but i do see that in a zuul job21:35
clarkbcould it be that the ps process namespace is disjoint from the podman container process namespace?21:36
corvusin fact, locally, i don't see a user container when i run the command as root21:36
clarkbI would expect that could happen if both are running in containers, but if running in the root process namespace I would expect what you describe locally21:36
corvusoh, i see it21:37
corvusit was started as root; i just missed it because the "become: true" was like taskfile levels up21:37
corvusit would be cool if the become flag trickled down to the leaf node json data, so that the root tasks always showed that regardless of where it was set21:38
*** tjgresha has quit IRC22:12
*** sshnaidm has joined #zuul22:17
corvusremote:   https://review.opendev.org/695601 install-podman: also install slirp4netns22:29
*** tosky has joined #zuul22:35
*** lennyb has joined #zuul22:41
*** mhu has quit IRC23:34
*** jamesmcarthur has quit IRC23:52
*** jamesmcarthur has joined #zuul23:53
Shrewsslirp? wow, such memories of hacking up college internet access23:53
*** jamesmcarthur has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!