| *** rlandy has quit IRC | 00:17 | |
| *** harlowja has quit IRC | 01:05 | |
| tristanC | arg, a misclick and storyboard discarded my report :( | 03:23 |
|---|---|---|
| tristanC | corvus: the issue pabelanger mentioned was that the final attribute verification doesn't check for child job in other projects | 03:24 |
| tristanC | corvus: in our case, periodic job was depending on a parent job that got the final attribute set to True | 03:24 |
| tristanC | corvus: after zuul merged that change, periodic job wasn't triggered, and the only tell was a stacktrace in the scheduler log that mentioned the "Unable to freeze job graph: Unable to modify final job" error | 03:25 |
| tristanC | so... shouldn't the configloader check for freezing error before accepting such changes? | 03:26 |
| tristanC | and/or, periodic pipeline job graph errors shouldn't be reported to the config-errors list? | 03:26 |
| tristanC | err, shouln't* post-review job graph errors be reported to the config-errors list? | 03:36 |
| tobiash | tristanC: checking for freezing errors is not that simple as it depends on project, branch, changed files | 03:54 |
| tobiash | tristanC: but maybe we can/should report freezing errors to the user | 03:56 |
| corvus | tristanC, tobiash: the pipeline should report the error | 03:56 |
| tobiash | corvus: actually I thought it does but it might be a problem that this was a periodic pipeline | 03:58 |
| corvus | tristanC: does the periodic pipeline report anywhere? (email, etc?) | 03:58 |
| *** spsurya_ has joined #zuul | 03:59 | |
| corvus | i don't think the sql reporter is capable of displaying errors like that, so if that's the only reporter, that may explain why no one saw the msg | 03:59 |
| tobiash | corvus: maybe we could inject a failing fake job on a freezing error | 03:59 |
| tobiash | Then it would be included in a normal sql reporting and show up in the builds tab | 04:00 |
| corvus | tobiash: perhaps; or we could have the sql reporter store a buildset message | 04:01 |
| corvus | it's > eod, so i'm not going to think about which approach would be better :) | 04:01 |
| tobiash | Or that, I'm still at breakfast so I haven't thought very deeply about that | 04:02 |
| tobiash | corvus: I already wondered why you're responding at this time :) | 04:03 |
| corvus | poor virtual desktop organization ;) | 04:03 |
| *** harlowja has joined #zuul | 04:25 | |
| tristanC | corvus: the rdo's periodic pipeline only has sql reporter, and it didn't reported the issue | 04:33 |
| tristanC | corvus: it also can happen with any pipeline, i reproduced the issue for regular job, zuul doesn't prevent setting a final job, even though it's currently used as a parent job | 04:34 |
| tristanC | then, if that setting get merge, every user of that jobs are basically broken | 04:34 |
| tobiash | tristanC: I think we should solve that particular problem by imroving error reporting | 04:37 |
| tobiash | Checking for child jobs is not as trivial as it seems due to the various filters | 04:38 |
| tobiash | tristanC: and it shouldn't prevent you to make a job final if you detect a misuse by a child job in a different repo | 04:39 |
| tobiash | tristanC: in order to check this in the config loader I think you would have to freeze a job graph for every child job on every graph, various changed files... | 04:42 |
| tobiash | That's not realistic | 04:42 |
| tobiash | s/graph/branch | 04:43 |
| tobiash | gah, mobile keyboard... | 04:43 |
| *** harlowja has quit IRC | 04:44 | |
| tristanC | tobiash: sure, error reporting would be good enough... right now one have to dig GBs of logs to find the traceback and figure out what went wrong... | 04:45 |
| tobiash | tristanC: yes, that's not how it should be | 04:46 |
| tristanC | though, i don't understand why we couldn't do: "for each job, for each variant, for each parent, check if a final flag would prevent job graph to freeze" | 04:47 |
| tristanC | and be able to prevent the error early, instead of waiting for the pipeline to fail execution | 04:48 |
| *** ianychoi_ has joined #zuul | 05:27 | |
| *** ianychoi has quit IRC | 05:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: scheduler: fix enqueue event to use canonical project name https://review.openstack.org/580040 | 05:47 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: scheduler: return project_canonical in status page https://review.openstack.org/582451 | 05:54 |
| *** nchakrab has joined #zuul | 06:23 | |
| *** lennyb has quit IRC | 06:49 | |
| *** logan- has quit IRC | 06:50 | |
| *** logan- has joined #zuul | 06:50 | |
| *** lennyb has joined #zuul | 06:52 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: scheduler: fix enqueue event to use canonical project name https://review.openstack.org/580040 | 06:58 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: scheduler: return project_canonical in status page https://review.openstack.org/582451 | 06:58 |
| *** jimi|ansible has quit IRC | 07:06 | |
| *** gtema has joined #zuul | 07:18 | |
| *** electrofelix has joined #zuul | 07:37 | |
| *** hwoarang has quit IRC | 08:15 | |
| *** hwoarang has joined #zuul | 08:23 | |
| *** hwoarang has quit IRC | 08:23 | |
| *** hwoarang has joined #zuul | 08:23 | |
| *** GonZo2000 has joined #zuul | 08:52 | |
| *** hwoarang has quit IRC | 08:54 | |
| *** hwoarang has joined #zuul | 08:55 | |
| *** GonZo2000 has quit IRC | 08:56 | |
| *** sshnaidm|afk is now known as sshnaidm|off | 09:00 | |
| *** sshnaidm|off has quit IRC | 09:05 | |
| *** quiquell has joined #zuul | 09:06 | |
| quiquell | Hello | 09:06 |
| quiquell | Question, the jobs specified at "required-jobs" are cloned fom master or from the change that triggered the build ? | 09:06 |
| *** jimi|ansible has joined #zuul | 09:50 | |
| *** sshnaidm|off has joined #zuul | 09:55 | |
| *** GonZo2000 has joined #zuul | 10:01 | |
| *** GonZo2000 has quit IRC | 10:01 | |
| *** GonZo2000 has joined #zuul | 10:01 | |
| *** nchakrab_ has joined #zuul | 10:34 | |
| *** nchakrab has quit IRC | 10:37 | |
| *** nchakrab has joined #zuul | 10:55 | |
| *** nchakrab has quit IRC | 10:57 | |
| *** nchakrab has joined #zuul | 10:57 | |
| *** nchakrab_ has quit IRC | 10:58 | |
| mordred | quiquell: the change that triggered the build and/or any depends-on patches that are relevant | 11:00 |
| quiquell | mordred: ack | 11:03 |
| *** nchakrab_ has joined #zuul | 11:11 | |
| *** nchakrab has quit IRC | 11:15 | |
| *** abelur has quit IRC | 11:15 | |
| *** abelur has joined #zuul | 11:16 | |
| *** abelur has quit IRC | 11:16 | |
| *** abelur has joined #zuul | 11:17 | |
| *** nchakrab_ has quit IRC | 11:30 | |
| *** nchakrab has joined #zuul | 11:31 | |
| gtema | mordred: should I update https://review.openstack.org/#/c/572829/ with respect to https://review.openstack.org/#/c/577616/ and sdk==0.16.0? | 11:50 |
| *** D0han has joined #zuul | 12:08 | |
| D0han | hi, im looking for more materials, other presentations, videos etc about https://www.youtube.com/watch?v=zC1lY9_gfcE | 12:09 |
| D0han | 'ansible-based ci' | 12:09 |
| mordred | gtema: yes please - with 0.16 being cut, we should be able to get that patch landed once you do | 12:16 |
| mordred | (I'll bug Shrews once he's up) :) | 12:17 |
| gtema | ok, thanks | 12:17 |
| mordred | D0han: hi! https://zuul-ci.org/ is the main website and probably a good place to start | 12:18 |
| D0han | thanks! will look at that | 12:18 |
| openstackgerrit | Artem Goncharov proposed openstack-infra/nodepool master: retire shade in favor of openstacksdk https://review.openstack.org/572829 | 12:19 |
| mordred | gtema: lgtm | 12:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove OpenStack driver waitForImage call https://review.openstack.org/580487 | 12:22 |
| gtema | mordred: ack | 12:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Use openstacksdk instead of os-client-config https://review.openstack.org/566158 | 12:23 |
| mordred | gtema: ^^ I rebased mine on top of yours | 12:23 |
| gtema | ok | 12:23 |
| gtema | should be good now finally | 12:24 |
| mordred | yah. that was fun :) | 12:25 |
| gtema | quite silent today | 12:25 |
| gtema | in all channels | 12:25 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Add a dequeue command to zuul client https://review.openstack.org/95035 | 12:32 |
| *** rlandy has joined #zuul | 12:38 | |
| rcarrillocruz | long live shade! | 12:50 |
| *** TheJulia is now known as needssleep | 12:58 | |
| mordred | rcarrillocruz: :) | 13:01 |
| *** samccann has joined #zuul | 13:12 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Revert "Revert "Install build bindep profiles alongside doc and test"" https://review.openstack.org/580872 | 13:22 |
| *** nchakrab_ has joined #zuul | 13:26 | |
| *** nchakrab has quit IRC | 13:29 | |
| *** gtema has quit IRC | 13:31 | |
| *** quiquell is now known as quiquell|lunch | 13:38 | |
| *** samccann has quit IRC | 13:47 | |
| *** samccann has joined #zuul | 13:48 | |
| *** samccann_ has joined #zuul | 13:49 | |
| *** samccann has quit IRC | 13:50 | |
| *** samccann_ is now known as samccann | 13:50 | |
| Shrews | mordred: what's up with the job timeouts on 572829? | 14:03 |
| *** nchakrab_ has quit IRC | 14:04 | |
| *** nchakrab has joined #zuul | 14:05 | |
| *** nchakrab has quit IRC | 14:05 | |
| *** nchakrab has joined #zuul | 14:06 | |
| mordred | Shrews: ugh. jeez. who the heck knows | 14:12 |
| Shrews | mordred: AttributeError: 'TaskManager' object has no attribute 'submit_function' | 14:13 |
| mordred | Shrews: hrm | 14:13 |
| mordred | oh for the love of | 14:15 |
| Shrews | why was https://review.openstack.org/580487 rebased on the shade removal changes? | 14:15 |
| * Shrews blames Friday | 14:16 | |
| mordred | Shrews: I can fix that on my next push | 14:17 |
| *** nchakrab has quit IRC | 14:18 | |
| mordred | Shrews: I think we might want to get this: https://review.openstack.org/#/c/414759 fixed so that the TaskManager stuff isn't conflicting - then do the shade patches on top of that? | 14:18 |
| mordred | of course, that patch probably has the same issue | 14:19 |
| mordred | oh. that's the positional argument thing | 14:20 |
| *** quiquell|lunch is now known as quiquell | 14:21 | |
| mordred | Shrews: I really want to finish cleaning this mess up | 14:22 |
| mordred | Shrews: I hit recheck on 414759 - I feel like I fixed that issue already | 14:23 |
| *** nchakrab has joined #zuul | 14:24 | |
| mordred | Shrews, tobiash, corvus, SpamapS: https://review.openstack.org/#/q/topic:container-images is the stack of things related to building zuul container images | 14:26 |
| mordred | (ignore the tripleo patches from last year) | 14:27 |
| D0han | is zuul limited to gerrit and github only? | 14:29 |
| mordred | D0han: currently yes. it's pluggable, so support for other systems can certainly be added - but so far nobody has | 14:30 |
| mordred | D0han: at various times various people have expressed interest in both gitlab and bitbucket plugins | 14:30 |
| mordred | and I believe we'd also like to see those added | 14:30 |
| D0han | makes sense | 14:31 |
| D0han | what about nodepool, can i have mixed one with some static and cloud? | 14:32 |
| openstackgerrit | Merged openstack-infra/nodepool master: Change TaskManager from is-a to has-a thread https://review.openstack.org/580463 | 14:32 |
| rcarrillocruz | yeah, there's a driver for static nodes | 14:38 |
| rcarrillocruz | D0han: | 14:38 |
| mordred | D0han: absolutely! | 14:38 |
| rcarrillocruz | even some drivers in review for AWS | 14:38 |
| rcarrillocruz | not sure if OCI (containers) already merged | 14:38 |
| rcarrillocruz | tristanC is the drivers mastah | 14:38 |
| *** nchakrab has quit IRC | 14:39 | |
| D0han | i have pretty big embedded project to put through ci/cd, and usually ready solutions are hard to apply | 14:43 |
| D0han | zuul looks neat so far | 14:43 |
| mordred | D0han: excellent - and yes, pre-existing things are often hard to apply to complex projects - I hope we're providing enough flexibility to be useful to you, but not so much that we're useless again :) | 14:45 |
| D0han | ;D | 14:46 |
| rcarrillocruz | D0han: fwiw, we use it at ansible networking, to test network appliances modules and roles, works a treat | 14:47 |
| D0han | sounds good | 14:48 |
| D0han | i will probably need to research nodepool to make sure it can do stuff | 14:49 |
| mordred | if it can't - definltey chat with us about use case you have that it can't meet | 14:50 |
| D0han | if i understand correctly, triggering flows/jobs/pipelines/whatever is on zuul, and nodepool takes care of providing nodes to execute them - but is there some part that keeps track of history what has been running on specific nodes/resources? | 14:52 |
| *** quiquell is now known as quiquell|off | 15:08 | |
| rcarrillocruz | D0han: https://ansible.softwarefactory-project.io/zuul/builds.html | 15:08 |
| rcarrillocruz | it will be available from Zuul dhasboard, builds | 15:09 |
| rcarrillocruz | as for nodes | 15:09 |
| rcarrillocruz | https://ansible.softwarefactory-project.io/zuul/nodes.html | 15:09 |
| rcarrillocruz | what node is assigned to what job is set on the job definition itself | 15:09 |
| *** yolanda__ has joined #zuul | 15:16 | |
| D0han | i have a pool of devices v1 (A,B,C,D) and v2 (A,E,F); job is set to run on pool v1 - can i check execution history regarding A? | 15:18 |
| *** EmilienM is now known as EvilienM | 15:18 | |
| *** yolanda_ has quit IRC | 15:18 | |
| rcarrillocruz | i don't think there's a way from dashboard to reference what jobs ran recently on $static nodes, but i think you can pull that info if you have logstash, like infra have or sf can install | 15:20 |
| rcarrillocruz | http://logstash.openstack.org/ | 15:20 |
| rcarrillocruz | from there you can search for node | 15:20 |
| rcarrillocruz | then reference back | 15:20 |
| mordred | it might also be possible to add $something to be able to track such a thing | 15:20 |
| mordred | Shrews: ^^ fwiw | 15:20 |
| rcarrillocruz | look on the left | 15:21 |
| rcarrillocruz | 'build_node' | 15:21 |
| rcarrillocruz | if you click on it, it will show jobs per node type | 15:21 |
| rcarrillocruz | in oyur use case, you would see per static nodes | 15:21 |
| Shrews | nodepool knows nothing about the jobs that run on the nodes it delivers | 15:21 |
| Shrews | your zuul job would have to log that info somewhere | 15:22 |
| Shrews | we have $something that logs such info: http://logs.openstack.org/59/414759/14/check/tox-py35/626e2c6/zuul-info/ | 15:23 |
| Shrews | i don't know offhand where that comes from | 15:23 |
| D0han | i need such feature because of embedded device development process, that helps to recognize broken nodes | 15:24 |
| rcarrillocruz | yah, honestly the only thing i can think of as a central location to look at 'what jobs ran on node FOO' is logstash, i.e. by mining job logs | 15:24 |
| mordred | well, I mean - we could log node into the build log | 15:25 |
| D0han | that brings another question - can zuul handle in some smart way 'suddenly broken nodes'? | 15:25 |
| mordred | (in zuul) | 15:25 |
| mordred | and then it would be possible to expose a way to search/report on that in the dashboard | 15:25 |
| D0han | this would be similar to jenkins | 15:26 |
| *** nchakrab has joined #zuul | 15:27 | |
| mordred | D0han: as for suddenly broken nodes - no, I do not believe so, although Shrews would know better than I. there's a bunch of work ongoing related to the scheduler and static nodes though | 15:27 |
| mordred | I think the tricky thing would be defining "suddenly broken" in a way that nodepool could detect such a thing | 15:27 |
| D0han | yep | 15:27 |
| mordred | since test jobs could obviously legitimately fail - or else they're not very useful test jobs ;) | 15:28 |
| D0han | yep again | 15:28 |
| D0han | so, theres need for health check during putting node into the pool | 15:29 |
| corvus | i can think of a way we could have zuul tell nodepool that it thinks a node is broken. so if someone can write a playbook which determins whether a node is broken, we can have nodepool take it out of service. | 15:29 |
| rcarrillocruz | corvus: like a pre - pre - run playbook | 15:29 |
| rcarrillocruz | ? | 15:30 |
| corvus | (this feauter doesn't exist yet, but most of the pieces are there to do that, so wouldn't be hard to add) | 15:30 |
| mordred | ++ | 15:30 |
| corvus | rcarrillocruz: i was thinking a pre-playbook with an extra return code that tells nodepool to hold the node | 15:30 |
| corvus | probably we would only allow trusted playbooks to return that code | 15:30 |
| mordred | oh yeah - that would totally work | 15:30 |
| rcarrillocruz | ic | 15:30 |
| corvus | and zuul records the result as a pre_fail (and re-runs the job on a new node) | 15:31 |
| D0han | youre talking about pre, so this health check would run just before the test? | 15:31 |
| corvus | D0han: yes | 15:32 |
| D0han | i think it would be better to run it during assimilation into the pool, so its known and ready for tests | 15:32 |
| mordred | well, I'd imagine that the pre playbook return would cause zuul to reschedule that job on a different node - so I think it would wind up having a similar effect - but would also cover "something went wrong on this node since the last time it was used in a test" | 15:33 |
| * mordred just thinking out loud | 15:34 | |
| D0han | this may make more sense with this info - i need to be able to get node from the pool and give it directly to dev/other system for some time, and node then comes back in unknown state | 15:34 |
| corvus | mordred: yeah -- hopefully the automatic rescheduling helps avoid a user impact | 15:34 |
| *** nchakrab has quit IRC | 15:36 | |
| *** nchakrab has joined #zuul | 15:36 | |
| D0han | hm, or maybe i need to stack zuul nodepool on top of lava | 15:37 |
| D0han | kinda redundant | 15:37 |
| corvus | D0han: if the node failure doesn't cause the job to fail, does it still matter that it's detected later rather than earlier? | 15:39 |
| D0han | yes, because 'later' creates delay for tests | 15:39 |
| *** nchakrab has quit IRC | 15:41 | |
| D0han | maybe even smarter would be to _not_ run health check if last test on it passed | 15:41 |
| corvus | D0han: well, we're trying to keep nodepool as simple as possible; it doesn't have any facility to run workloads on nodes. to do so means re-implementing a lot of what zuul does. | 15:41 |
| D0han | mhm | 15:42 |
| *** pawelzny has quit IRC | 15:46 | |
| *** gtema has joined #zuul | 15:49 | |
| gtema | mordred: now for me, what is with timeouts? Should 414759 be repaired and merged first or what? I am wondered, since shade>sdk change was always passing | 16:06 |
| *** GonZo2000 has quit IRC | 16:08 | |
| mordred | gtema: it's related to the taskmanager stuff | 16:08 |
| *** nchakrab has joined #zuul | 16:08 | |
| gtema | what is the plan? | 16:08 |
| mordred | gtema: there is another patch that is about aligning those: https://review.openstack.org/#/c/414759/ | 16:08 |
| *** harlowja has joined #zuul | 16:08 | |
| mordred | I've rechecked it to see if recent changes fix it (I feel like I did something to fix this already) | 16:09 |
| gtema | ok | 16:09 |
| mordred | and if it still comes back failing, we can debug further | 16:09 |
| mordred | oh - speaking of | 16:09 |
| mordred | Shrews, corvus: I THINK the issue with 414759 the previous time it ran had to do with upper-constraints ... | 16:10 |
| mordred | which is obviously not what we want in our lives | 16:10 |
| *** GonZo2000 has joined #zuul | 16:10 | |
| *** GonZo2000 has quit IRC | 16:10 | |
| *** GonZo2000 has joined #zuul | 16:10 | |
| mordred | but I think IIRC when I dug in last time I discovered that our nodepool-functional jobs are picking up upper-constraints because they're built on top of devstack and as a devstack plugin | 16:10 |
| *** harlowja has quit IRC | 16:10 | |
| mordred | so we probably want to dig in, figure that out (If I'm right) and rectify it, since nodepool does not follow upper-constraints | 16:11 |
| Shrews | mordred: it just failed again | 16:25 |
| corvus | Shrews, mordred: http://logs.openstack.org/59/414759/14/check/nodepool-functional-py35/6974ad2/controller/logs/screen-nodepool-launcher.txt.gz | 16:31 |
| mordred | yeah. that makes no sense to me - since that is not a required argument in openstacksdk or in that patch | 16:36 |
| mordred | which makes me think the wrong version of something is getting installed | 16:36 |
| mordred | there it is: http://logs.openstack.org/59/414759/14/check/nodepool-functional-py35/6974ad2/job-output.txt.gz#_2018-07-13_14_56_16_810166 | 16:38 |
| mordred | wait - nevermind | 16:38 |
| mordred | still not it | 16:38 |
| corvus | did run ever take a client arg? | 16:39 |
| corvus | (also, i really wish pythons error there included the class name) | 16:40 |
| mordred | right? | 16:41 |
| mordred | and yes - it did | 16:41 |
| mordred | the shade version does | 16:41 |
| mordred | OH | 16:42 |
| mordred | *DUH* | 16:42 |
| corvus | the job/plugin installs shade | 16:42 |
| mordred | yah - this patch hasn't migrated to sdk yet | 16:42 |
| mordred | this is trying to migrate to openstacksdk's task_manager but still using shade for rest calls | 16:42 |
| mordred | I think we need to squish this patch with gtema's patch | 16:43 |
| gtema | should I do something? | 16:44 |
| mordred | gtema: possibly ... | 16:44 |
| *** fbo is now known as fbo|off | 16:46 | |
| mordred | gtema: I believe we want to squash https://review.openstack.org/#/c/414759 into your patch - mind if I take a stab real quick and push up a new rev? | 16:48 |
| gtema | shure - do it | 16:48 |
| *** electrofelix has quit IRC | 16:57 | |
| *** GonZo2000 has quit IRC | 16:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove OpenStack driver waitForImage call https://review.openstack.org/580487 | 17:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove Task class https://review.openstack.org/580466 | 17:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Replace shade and os-client-config with openstacksdk. https://review.openstack.org/572829 | 17:03 |
| mordred | Shrews, gtema: rebased/squashed -I think that stack should work now | 17:04 |
| *** gtema has quit IRC | 17:08 | |
| tobiash | mordred: just tried out mitogen (outside of zuul yet) and it speeds up a clean openshift cluster deployment from about 45min to 25min | 17:23 |
| rcarrillocruz | jimi|ansible: ^ | 17:24 |
| rcarrillocruz | :-) | 17:24 |
| tobiash | using openshift-ansible | 17:24 |
| tobiash | and the deployment was successful :) | 17:24 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: try out mitogen for zuul jobs https://review.openstack.org/582654 | 17:54 |
| *** acozine1 has joined #zuul | 17:55 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: break ansible strategy https://review.openstack.org/582656 | 18:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove Task class https://review.openstack.org/580466 | 18:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Replace shade and os-client-config with openstacksdk. https://review.openstack.org/572829 | 18:34 |
| mordred | sigh. pep8 | 18:34 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: try out mitogen for zuul jobs https://review.openstack.org/582654 | 18:40 |
| tobiash | mordred: it seems like when using mitogen within zuul a job freezes with http://paste.openstack.org/show/725849/ | 19:02 |
| tobiash | just tried that in my testenv | 19:02 |
| tobiash | the assertion is here: https://github.com/dw/mitogen/blob/d493a3d7ca9c9440848739704f9a1ab2d4118de5/mitogen/core.py#L1370 | 19:07 |
| tobiash | shell tasks loop forever with 'Waiting on logger' | 19:12 |
| tobiash | looks like mitogen interferes with log streaming | 19:12 |
| tobiash | mordred: am I right that 'Waiting on logger' means that zuul-console isn't started? | 19:21 |
| tobiash | mordred: mitogen re-uses processes on the other end so that might interfere with the daemonization of zuul-console | 19:21 |
| corvus | tobiash: yeah, or at least, it means the executor can't connect to the zuul-console | 19:26 |
| tobiash | corvus: that's unlikely as it works without the mitogen patch | 19:26 |
| corvus | tobiash: no i mean i'm just trying to clarify that it means the ansible process on the executor can't open a socket to the zuul console server on the worker for some reason. a very likely reason is the one you suggest, but there could be others. :) | 19:27 |
| tobiash | so I think that with mitogen zuul-console might have a problem with daemonizing or daemonizing kills mitogen on the remote end somehow | 19:27 |
| tobiash | corvus: yes, I guess in most reasons it's connectivity or a forgotten zuul-console :) | 19:28 |
| tobiash | ok, at least nothing is listening on the node | 19:31 |
| *** sshnaidm|off has quit IRC | 19:40 | |
| tobiash | got it working with a hack | 20:01 |
| tobiash | it's in fact the fork that breaks it | 20:01 |
| *** nchakrab has quit IRC | 20:08 | |
| tobiash | soo, just did some quick tests with mitogen and zuul | 20:49 |
| tobiash | regarding shell tasks we unfortunately we gain nothing (most likely because of our log streaming) | 20:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: WIP: Add a role to return file comments https://review.openstack.org/579033 | 20:51 |
| tobiash | regarding other tasks (i.g. file module) I tested a relatively simple job with just 100 file tasks the execution time went down from 1m:55s to 1m:25s | 20:52 |
| tobiash | so it is an improvement but not as huge as I expected | 20:55 |
| tobiash | but I think we can expect lower cpu usage which could improve the scalability of the executors | 20:56 |
| tobiash | in a non-zuul related ansible playbook I saw a reduction of cpu time by 2/3 | 20:57 |
| tobiash | I also have to note that this task took almost 20s regardless of the ansible plugin: http://git.zuul-ci.org/cgit/zuul-jobs/tree/roles/validate-host/tasks/main.yaml#n12 | 21:00 |
| *** samccann has quit IRC | 21:03 | |
| tobiash | and about further 20s were executor preparation time | 21:04 |
| *** acozine1 has quit IRC | 21:11 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack-infra/zuul-jobs master: Attempt to copy the coverage report even if job fails https://review.openstack.org/582690 | 21:26 |
| *** sshnaidm|off has joined #zuul | 21:27 | |
| -openstackstatus- NOTICE: logs.openstack.org is offline, causing POST_FAILURE results from Zuul. Cause and resolution timeframe currently unknown. | 21:53 | |
| *** ChanServ changes topic to "logs.openstack.org is offline, causing POST_FAILURE results from Zuul. Cause and resolution timeframe currently unknown." | 21:53 | |
| *** rlandy has quit IRC | 22:11 | |
| *** ChanServ changes topic to "Discussion of the project gating system Zuul | Website: https://zuul-ci.org/ | Docs: https://zuul-ci.org/docs/ | Source: https://git.zuul-ci.org/ | Channel logs: http://eavesdrop.openstack.org/irclogs/%23zuul/ | Weekly updates: https://etherpad.openstack.org/p/zuul-update-email" | 23:38 | |
| -openstackstatus- NOTICE: logs.openstack.org is back on-line. Changes with "POST_FAILURE" job results should be rechecked. | 23:38 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!