| fungi | correlating node ip addresses and timestamps back to zuul job builds is nontrivial... just did this to track down some offending jobs based on vulnerability scan reports which came back from one of our donors: https://etherpad.openstack.org/p/wpy5XvRYWu | 00:12 |
|---|---|---|
| fungi | i wonder if there's something we can/should do to make that easier to analyze, or whether it's just yet another case for a central logserver | 00:13 |
| SpamapS | corvus: sorry I haven't been making it back to 542518 .. it's on my todo, but probably falling through to tomorrow | 00:30 |
| clarkb | fungi: I think we could make it much easier if zuul set the name of the job making the nodeset request | 00:31 |
| clarkb | fungi: then we could log that on the nodepool side and have all the node related data next to the job name | 00:31 |
| fungi | ooh, that's a good point | 00:35 |
| *** elyezer has quit IRC | 00:42 | |
| *** elyezer has joined #zuul | 00:45 | |
| corvus | fungi, clarkb: job name isn't sufficient, the solution needs to be a bit more sophisticated than that. buildset+job is unique enough. | 00:56 |
| clarkb | oh right job names are used more now | 00:56 |
| fungi | excellent point | 01:02 |
| corvus | we could also add the build uudid to the node info when we have it (which is after the request is fulfilled). but it would allow nodepool to log something like "deleting node ### used for build ###" | 01:03 |
| tristanC | corvus: how about zuul_node, zuul_nodeset tables and a zuul_nodeset_id column added to zuul_build? | 01:35 |
| *** elyezer has quit IRC | 01:43 | |
| *** elyezer has joined #zuul | 01:44 | |
| corvus | tristanC: that could work | 02:12 |
| *** myoung|bbl is now known as myoung | 02:46 | |
| *** jimi|ansible has joined #zuul | 02:53 | |
| *** bhavik1 has joined #zuul | 02:58 | |
| *** bhavik1 has quit IRC | 03:01 | |
| *** rlandy has quit IRC | 03:15 | |
| *** harlowja has quit IRC | 03:30 | |
| *** jimi|ansible has quit IRC | 03:34 | |
| *** harlowja has joined #zuul | 04:12 | |
| *** harlowja has quit IRC | 04:30 | |
| *** elyezer has quit IRC | 05:03 | |
| *** elyezer has joined #zuul | 05:04 | |
| *** harlowja has joined #zuul | 05:14 | |
| *** jimi__|ansible has joined #zuul | 05:24 | |
| *** jimi__|ansible is now known as jimi|ansible | 05:24 | |
| *** jimi|ansible has quit IRC | 05:25 | |
| *** jimi|ansible has joined #zuul | 05:25 | |
| *** elyezer has quit IRC | 06:20 | |
| *** elyezer has joined #zuul | 06:21 | |
| *** harlowja has quit IRC | 06:24 | |
| yolanda | rcarrillocruz, of course i remember. Before that, we were spending the day with manual hold/unhold... | 06:49 |
| *** elyezer has quit IRC | 07:26 | |
| *** elyezer has joined #zuul | 07:28 | |
| dmellado | morning yolanda | 07:32 |
| yolanda | hi | 07:32 |
| *** elyezer has quit IRC | 07:43 | |
| *** elyezer has joined #zuul | 07:44 | |
| *** jpena|off is now known as jpena | 08:16 | |
| *** sshnaidm|off is now known as sshnaidm|ruck | 08:27 | |
| *** saop has joined #zuul | 08:33 | |
| swest | corvus: tobiash: I'm fine with using git remotes instead of what I proposed in https://review.openstack.org/#/c/539975/. I have one question regarding where this remote should point to. I sounds like from what you proposed that instead of deleting 'origin' we just set the remote URL to a more or less bogus value. In this case I might still not know which changes are part of a *single* PR when there are other in-flight PRs, right?! | 08:47 |
| *** elyezer has quit IRC | 08:49 | |
| *** elyezer has joined #zuul | 08:51 | |
| tobiash | swest: in case of other pr's in the queue the remote would point to the head of the previous PR and not to master | 08:53 |
| tobiash | so if you have master -> A -> B, A would have master as origin/master and B would have A as origin/master | 08:54 |
| swest | tobiash: ok, was about to check that. good to know | 08:54 |
| tobiash | swest: just double checked git an example repo | 09:06 |
| tobiash | if origin/master is the state of either the previous PR in the dependency queue or current master git log origin/master..master returns exactly what we want | 09:06 |
| tobiash | namely the commits of the pr | 09:07 |
| *** hashar has joined #zuul | 09:08 | |
| *** ankkumar has joined #zuul | 09:09 | |
| swest | tobiash: sounds good. already working on it ... | 09:12 |
| tobiash | :) | 09:12 |
| ankkumar | tristanC, Hi | 09:13 |
| ankkumar | tristanC, i am trying to setup zuul for CI and i am getting error FileNotFoundError: [Errno 2] No such file or directory: 'bwrap'.. But bwrap is installed in the image or vm created. | 09:13 |
| rcarrillocruz | you need to install separately | 09:14 |
| rcarrillocruz | which OS you use | 09:14 |
| rcarrillocruz | ubuntu or centos | 09:14 |
| ankkumar | tristanC, detailed log.. http://paste.openstack.org/show/671817/ | 09:14 |
| rcarrillocruz | if ubuntu, you can use this PPA https://launchpad.net/~openstack-ci-core/+archive/ubuntu/bubblewrap/+index | 09:15 |
| rcarrillocruz | centos has it on repos (can't remember if you need to enable EPEL) | 09:15 |
| ankkumar | tristanC, any idea ?? | 09:21 |
| tobiash | corvus: I replied on 537335 | 09:23 |
| *** ankkumar has left #zuul | 09:43 | |
| *** ankkumar has joined #zuul | 09:44 | |
| rcarrillocruz | ankkumar: paste if bubblewrap is really installed by using the package manager | 09:47 |
| rcarrillocruz | if it is, paste where the binary is located | 09:47 |
| ankkumar | i am trying this version bubblewrap 0.1.7-1 | 09:49 |
| ankkumar | rcarrillocruz, previously i tried with version 0.2 | 09:51 |
| ankkumar | dpkg -l | grep bubble | 09:51 |
| ankkumar | ii bubblewrap 0.1.7-1 amd64 setuid wrapper for unprivileged chroot and namespace manipulation | 09:51 |
| rcarrillocruz | i'd recommend you remove it | 09:51 |
| rcarrillocruz | and use the PPA i pasted | 09:51 |
| rcarrillocruz | that's the one openstack infra uses | 09:51 |
| rcarrillocruz | i've used it on my test envs, and it works fine | 09:52 |
| ankkumar | rcarrillocruz, ok thanks i will try that | 09:53 |
| ankkumar | rcarrillocruz, I tried with this but it failed at the same step | 10:20 |
| ankkumar | with same error | 10:20 |
| rcarrillocruz | run 'which bwrap' | 10:20 |
| rcarrillocruz | also, which user are you running zuul with | 10:21 |
| rcarrillocruz | it could be that user does not have bwrap on its path | 10:21 |
| ankkumar | which bwrap | 10:21 |
| ankkumar | /usr/bin/bwrap | 10:21 |
| ankkumar | it is not running with sudo | 10:22 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix self fulfilling empty node requests https://review.openstack.org/544236 | 10:22 |
| *** tosky has joined #zuul | 10:44 | |
| openstackgerrit | Merged openstack-infra/zuul master: Extend stackdump to display the daemonize status https://review.openstack.org/542158 | 11:20 |
| *** saop_ has joined #zuul | 11:31 | |
| *** saop has quit IRC | 11:34 | |
| *** ankkumar has quit IRC | 11:35 | |
| *** saop_ has quit IRC | 11:43 | |
| *** sshnaidm|ruck has quit IRC | 11:46 | |
| *** elyezer has quit IRC | 11:50 | |
| *** elyezer has joined #zuul | 12:03 | |
| *** sshnaidm|ruck has joined #zuul | 12:20 | |
| *** patriciadomin_ is now known as patriciadomin | 12:21 | |
| *** dkranz has joined #zuul | 12:31 | |
| swest | tobiash: I just tried keeping the remotes/origin around but when I have a PR (A) and another PR (B) with a dependency on A I get: master -> B -> A | 12:43 |
| swest | tobiash: and my origin/master is not HEAD of B | 12:44 |
| *** tosky has quit IRC | 12:47 | |
| *** jpena is now known as jpena|lunch | 12:51 | |
| tobiash | swest: you need to create a speculative origin instead of keeping the original origin | 12:53 |
| tobiash | swest: the best place to start looking is probably http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py#n640 | 12:57 |
| *** elyezer has quit IRC | 13:00 | |
| hughsaunders | Hey, I'm still working on a nodepool plugin for jenkins.. and my current problem is that nodepool uses kazoo.recipies.lock, which doesn't look to be compatible with the apache curator lock recipes. Anyone know if there is a Java zk lock implementation that is compatible with kazoo's lock? | 13:00 |
| *** weshay_PTO is now known as weshay | 13:01 | |
| swest | tobiash: ATM I'm just doing a setRemoteUrl(canonical_name) instead of http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py#n723 | 13:01 |
| *** elyezer has joined #zuul | 13:04 | |
| tobiash | swest: didn't look deeper into that but you maybe need to enhance http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py#n686 to create that remote (together with the speculative branches of the previous merge item) and after that just remove the remote deletion | 13:04 |
| *** tosky has joined #zuul | 13:06 | |
| tobiash | swest: or in fact, set the branches of origin to their speculative revs of the previous merge item and set the remote to a bogus url instead of deleting it | 13:07 |
| electrofelix | rcarrillocruz: sorry didn't get back to responding yesterday, yes I recall the script used to hold static jenkins slaves after a job completed | 13:08 |
| *** chrnils has joined #zuul | 13:08 | |
| swest | tobiash: ok. thanks for the hint. | 13:08 |
| *** sshnaidm has joined #zuul | 13:19 | |
| rcarrillocruz | Shrews, mordred : wwhen you get around, https://github.com/ansible/ansible/pull/20974 lgtm, but would like to get David +1 | 13:20 |
| rcarrillocruz | we need to merge that today | 13:21 |
| rcarrillocruz | so i can ping mattdavis to cherry pick that onto 2.5 | 13:21 |
| rcarrillocruz | (devel is now 2.6) | 13:21 |
| rcarrillocruz | erm...wrong channel | 13:21 |
| rcarrillocruz | :-) , sigh, sorry | 13:21 |
| *** sshnaidm|ruck has quit IRC | 13:22 | |
| *** tosky has quit IRC | 13:23 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 13:26 |
| *** rlandy has joined #zuul | 13:31 | |
| *** tosky has joined #zuul | 13:40 | |
| *** jpena|lunch is now known as jpena | 13:50 | |
| fungi | to recap a current discussion from #openstack-infra, we've discovered a new way to introduce zuul config inconsistencies | 14:41 |
| fungi | our release request and approval process is laggy, and as a result we created a new branch from a point in a current branch's history which was incompatible with the latest state of configuration | 14:42 |
| fungi | basically the team maintaining an application requested a new series branch on january 25th indicating the then-current master branch state, on the 27th they merged a zuul config change to master which removed use of a job which was subsequently removed from the system later that day, then on the 30th the series branch creation was approved which cut a new branch from the requested point in time prior to that | 14:45 |
| fungi | configuration change, reintroducing use of the now-nonexistent job | 14:45 |
| fungi | given that events like branch creation (or repository import) happen outside zuul's control, i don't have a lot of hope for zuul preventing them from introducing configuration issues | 14:47 |
| fungi | on a positive note, the syntax error reported on subsequent changes made it easy to track down the culprit | 14:47 |
| pabelanger | not a fix for today, but could zuul create the branch speculatively some how? And even push it into gerrit on approval? | 14:49 |
| corvus | fungi: yeah, i think the defense for this is going to be 'drop the offending project from configuration'. or maybe we could just drop the branch. that's still a significant penalty, but it should be more focused. | 14:56 |
| corvus | pabelanger: we could speculatively load a config from any arbitrary branch point, but we'd need to figure out a way to tell zuul to do that. that will require some brainstorming. | 14:57 |
| pabelanger | ack | 15:01 |
| *** sshnaidm is now known as sshnaidm|ruck | 15:04 | |
| -openstackstatus- NOTICE: Due to a race in stable/queens branch creation and some job removals, Zuul has reported syntax errors for the past hour; if you saw a syntax error reported for "Job tripleo-ci-centos-7-ovb-containers-oooq not defined" you can safely recheck now | 15:06 | |
| fungi | yay ^ we have statusbot reporting in here now (finally)! | 15:06 |
| fungi | corvus: yeah, if we figure out how to ask zuul to attempt to speculatively load configuration from a particular set of commit ids then we could presumably run that in the release repo's gate pipeline to check for potential configuration deadlocks before the branches get created, since it's a post (well, release-post) pipeline job ultimately doing the branch creation | 15:09 |
| fungi | that still doesn't protect us from projects performing branch creation manually, but i expect those cases are less likely to attempt branching from a "stale" point in history anyway | 15:10 |
| fungi | if zuul gave us a look-before-you-leap option, i expect that's plenty sufficient | 15:11 |
| corvus | fbo is working on a patch to let us start with a broken config: https://review.openstack.org/535511 | 15:20 |
| kklimonda | can I abort jobs for review somehow? | 16:05 |
| dmsimard | kklimonda: by abandoning the review ? | 16:06 |
| dmsimard | I wonder if the GitHub side of things handles that (closing a PR) | 16:06 |
| kklimonda | mmm, abandon and then immediately restore - restoring wont re-trigger zuul? | 16:07 |
| kklimonda | how about jobs triggered with timer, or other jobs not tied to reviews? | 16:07 |
| corvus | kklimonda: these behaviors are configurable, it depends on your pipeline configuration | 16:08 |
| pabelanger | there is an old patch up to dequeue a change: https://review.openstack.org/95035/ | 16:09 |
| pabelanger | needs to be updated still | 16:10 |
| corvus | pipeline requirements generally cover whether something continues to run after a state change: https://docs.openstack.org/infra/zuul/user/config.html#attr-pipeline.require | 16:10 |
| kklimonda | pabelanger: it is indeed old :) thanks, I'll take a look | 16:10 |
| pabelanger | there is an implementation issue we need to resolve, but still think we can use it | 16:11 |
| corvus | i'd love to merge a 5-digit change :) | 16:13 |
| *** electrofelix has quit IRC | 16:20 | |
| *** rlandy is now known as rlandy|brb | 16:32 | |
| *** tosky has quit IRC | 16:36 | |
| *** rlandy|brb is now known as rlandy | 16:51 | |
| openstackgerrit | Merged openstack-infra/zuul master: Do not call merger:cat when all config items are excluded https://review.openstack.org/535509 | 16:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Ensure only python3 is installed with bindep.txt https://review.openstack.org/544560 | 17:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP add host/group vars https://review.openstack.org/544562 | 17:30 |
| corvus | andreaf: ^ want to take an early look at that (especially the docs) and see if it looks right? | 17:31 |
| corvus | i'm only doing the job attributes for now, because i think they're more important, and i think we may end up not needing or wanting nodeset attributes for this. if we decide we do, we can add them later. | 17:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Ensure only python3 is installed with bindep.txt https://review.openstack.org/544569 | 17:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: WIP: Support ubuntu-beaver (18.04) with nodepool dsvm https://review.openstack.org/544574 | 17:49 |
| *** chrnils has quit IRC | 17:51 | |
| *** hashar is now known as hasharAway | 17:55 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP add host/group vars https://review.openstack.org/544562 | 17:56 |
| openstackgerrit | Merged openstack-infra/nodepool master: Ensure only python3 is installed with bindep.txt https://review.openstack.org/544560 | 18:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Clean up old images for testing https://review.openstack.org/544582 | 18:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Switch to fedora-27 for nodepool dsvm https://review.openstack.org/544583 | 18:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Clean up old images for testing https://review.openstack.org/544582 | 18:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Switch to fedora-27 for nodepool dsvm https://review.openstack.org/544583 | 18:13 |
| *** jpena is now known as jpena|off | 18:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: WIP: Support ubuntu-beaver (18.04) with nodepool dsvm https://review.openstack.org/544574 | 18:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Switch to fedora-27 for nodepool dsvm https://review.openstack.org/544583 | 18:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: WIP: Support ubuntu-beaver (18.04) with nodepool dsvm https://review.openstack.org/544574 | 18:19 |
| andreaf | corvus: it looks good (I only looked at docs) | 18:22 |
| andreaf | corvus: how would inheritance work there? Can a child job add a single service to the list of services defined for the controller node for instance? | 18:25 |
| andreaf | glancing at the code I think the answer to that question is yes | 18:29 |
| corvus | andreaf: i believe so | 18:29 |
| *** openstackgerrit has quit IRC | 18:33 | |
| *** jpena|off is now known as jpena | 18:50 | |
| *** jpena is now known as jpena|off | 18:59 | |
| *** sshnaidm|ruck is now known as sshnaidm|off | 19:03 | |
| *** openstackgerrit has joined #zuul | 19:05 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Unpause Xenial build for non-src functional test https://review.openstack.org/544125 | 19:05 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Improve file renaming in stage-output https://review.openstack.org/544606 | 19:20 |
| *** harlowja has joined #zuul | 19:21 | |
| *** yolanda has quit IRC | 19:44 | |
| *** yolanda has joined #zuul | 19:45 | |
| *** hasharAway is now known as hashar | 20:05 | |
| pabelanger | had a question about zuul dependent pipelines if project A and project B were in a shared change queue. But project A only had branch foo and project B branch bar, could those change not be run in parallel as independent? | 21:10 |
| clarkb | not necessarily, project A may install project B from the other branch | 21:16 |
| clarkb | and if they are in a shared queue you are instructing zuul that this may be the case right? | 21:16 |
| pabelanger | Yah, I guess that would make sense | 21:17 |
| pabelanger | you setup a queue per job, so it is possible the job does what you mentioned. | 21:17 |
| pabelanger | cool, that answers my question | 21:17 |
| clarkb | pabelanger: one potential real world example of this would be zuul and ansible | 21:19 |
| clarkb | with master and devel respectively | 21:19 |
| pabelanger | clarkb: yup, good example | 21:20 |
| clarkb | (we aren't sharing queues with them now though aiui because we can't really cogate them properly but just an idea of how that might actually look like with real projects) | 21:20 |
| pabelanger | yah | 21:20 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Improve file renaming in stage-output https://review.openstack.org/544606 | 21:23 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Clean up old images for testing https://review.openstack.org/544582 | 21:41 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Switch to fedora-27 for nodepool dsvm https://review.openstack.org/544583 | 21:41 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: WIP: Support ubuntu-beaver (18.04) with nodepool dsvm https://review.openstack.org/544574 | 21:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix branch deletion after failed reconfig https://review.openstack.org/544716 | 22:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix branch deletion after failed reconfig https://review.openstack.org/544716 | 22:26 |
| *** threestrands has joined #zuul | 22:31 | |
| *** dkranz has quit IRC | 23:15 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add host/group vars https://review.openstack.org/544562 | 23:20 |
| *** rlandy is now known as rlandy|bbl | 23:41 | |
| *** hashar has quit IRC | 23:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Invert executor ram statsd metric https://review.openstack.org/544742 | 23:41 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!