| pabelanger | k | 00:00 |
|---|---|---|
| jamielennox | jeblair: just while you're on a roll, i had two things i left in channel last night | 00:04 |
| jamielennox | i see you left a comment on https://review.openstack.org/#/c/461981/ about using the connection - i had a similar thought but wasn't sure if you wanted to do something a bit more along the reporter filter idea | 00:05 |
| jamielennox | or we should just do that anyway for at least the github/gerrit reporters | 00:05 |
| jamielennox | but if that fix works for you i can do that properly today | 00:05 |
| jeblair | jamielennox: i *think* that fix is ideal, because it's essentially "every reporter should try to report the change if it can, automatically". i'll reply to jlk on that. | 00:06 |
| jamielennox | jeblair: right, it just leaves open the question of only use the sql or smtp reporter for some cases, but i'm happy to defer that until it's actually a problem | 00:07 |
| jeblair | jamielennox: yeah, so far we haven't had a desire to limit those by connection | 00:07 |
| jamielennox | yea, it would mean putting something in the config and it should be something we can do in a backwards compatible way later if required | 00:08 |
| jamielennox | ok, will do a version of that today and remove the source param | 00:08 |
| jeblair | yeah i think so | 00:08 |
| jamielennox | second was passing {{ zuul.project_name }} as canonical to the executor | 00:09 |
| jamielennox | this makes sense to me and fixes the problem of finding the source dir | 00:09 |
| jamielennox | however i'm looking through the executor/client + server and it looks like all this information would be available to send from the client | 00:10 |
| jamielennox | but the server is doing it's own source lookup | 00:10 |
| jamielennox | is there a reason for this or just the fastest thing to do at the time? | 00:10 |
| jeblair | jamielennox: clarkb and i had some bikeshed comments on that at https://etherpad.openstack.org/p/vB1WjfL804 | 00:12 |
| jeblair | jamielennox: to that question -- i think we need to do a project lookup for some reason anyway, which is why we do the lookup from source. at any rate, given a project object, the full range of canonical name attributes are available. | 00:13 |
| jamielennox | but so the pattern here: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?h=feature/zuulv3#n573 | 00:15 |
| jamielennox | is repeated a few times, where it seems like all your trying to do is find the canonical name, which we could just pass through from the client | 00:16 |
| jeblair | jamielennox: it's source.getGitUrl that's really driving that one | 00:19 |
| jeblair | (and yes, we could pass that across as well, but the executor's merger needs to be able to derive most of this from just source+project anyway, so i tried to keep the information passing to a minimum) | 00:20 |
| jamielennox | jeblair: any reason not to send thta as an arg from client? | 00:20 |
| jamielennox | ok | 00:21 |
| jamielennox | so why does the executor's merger (something i'm still wrapping my head around) need to do a source+project lookup? | 00:24 |
| jeblair | jamielennox: because it uses the source interface directly, since it's the thing pulling stuff from the git remote | 00:26 |
| pabelanger | okay cool, have ssh-agent working with trusted / untrusted playbook | 00:28 |
| jamielennox | yep, but why are those items based on sources rather than a url+sha combo or something | 00:28 |
| pabelanger | this is actually straightforward | 00:28 |
| jamielennox | the whole concept of executor-server having access to source/connection objects is still odd to me i guess | 00:29 |
| jeblair | jamielennox: the executor has an embedded merger. mergers talk to sources. | 00:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 00:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 00:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 00:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 00:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 00:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 00:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 00:55 |
| pabelanger | Ha, default shell for zuul isn't bash | 00:57 |
| pabelanger | fixing | 00:57 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 01:00 |
| pabelanger | okay, maybe that was the issue | 01:00 |
| *** jkilpatr has quit IRC | 01:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 01:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 01:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 01:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 01:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 01:17 |
| pabelanger | okay, that too way too many patches | 01:18 |
| pabelanger | but it worked | 01:18 |
| pabelanger | jamielennox: http://zuulv3-dev.openstack.org/logs/0cd7e63b415b4188bdd389aac0b8bf82/console.log | 01:18 |
| pabelanger | high level, but we could turn that into a trusted playbook | 01:18 |
| jamielennox | pabelanger: but do those variables hang around such that they would be used by playbooks running between the pre and post? | 01:20 |
| pabelanger | yes, I piped them to a file | 01:20 |
| pabelanger | we just need to source it first | 01:21 |
| jamielennox | i would be surprised/possibly concerned if you could set and keep environment variables in the executor from within the playbook | 01:21 |
| pabelanger | initial testing show I couldn't do that | 01:21 |
| pabelanger | but, is late and just hacked for 30mins | 01:21 |
| jamielennox | oh, so the playbook drops an rc file, you source it before running other things | 01:22 |
| pabelanger | yes | 01:22 |
| jamielennox | ok, yea, so it creates an interesting dependency between playbooks or between zuul and something created in a playbook | 01:22 |
| pabelanger | the last step would be to have bwrap source that, to find out SSH_AGENT_SOCK | 01:22 |
| pabelanger | or, run untrusted job in bwrap, to source rc (delegate_localhost), then run normal task using new ssh-agent socket | 01:23 |
| pabelanger | either way, can play with that more tomorrow | 01:23 |
| pabelanger | brain is shutting down | 01:23 |
| jamielennox | understood, we can talk about it then | 01:24 |
| *** yolanda has joined #zuul | 02:26 | |
| *** smyers_ has joined #zuul | 03:10 | |
| *** yolanda has quit IRC | 03:11 | |
| *** nibalizer has quit IRC | 03:11 | |
| *** smyers has quit IRC | 03:11 | |
| *** smyers_ is now known as smyers | 03:11 | |
| *** yolanda has joined #zuul | 03:17 | |
| *** nibalizer has joined #zuul | 03:19 | |
| jeblair | pabelanger, jamielennox: er, i thought we had agreed zuul should start the ssh agent? see the etherpad: https://etherpad.openstack.org/p/EIrCy4mhuR | 04:22 |
| jeblair | i really don't think we should have zuul depend on a user-supplied playbook. | 04:23 |
| jamielennox | jeblair: was my understanding as well, i think he was just mapping it out to see if it is possible | 04:23 |
| jamielennox | jeblair: cause agreed i don't think zuul should depend on something defined in a playbook | 04:23 |
| jamielennox | jeblair: before you jump out again, does the connection compare work for multiple gerrits? | 04:24 |
| jamielennox | is there an individual reporter object made each for each connection if there are two of the same type? | 04:25 |
| jeblair | okay. i guess now we have confirmed what we suspected. :) we can use about half of that change, but the agent stuff needs to be in zuul. i think it makes a lot of sense for the agent to be part of the interface that zuul provides to the playbooks. | 04:26 |
| jeblair | jamielennox: there might be any number of reporters; it's more of a data-structure kind of object than a long-lived singleton. the connection, however, is long-lived and unique. | 04:27 |
| jeblair | jamielennox: reporters have a reference to their associated connection, so that's the best thing to compare | 04:27 |
| jamielennox | jeblair: yea, ok, in our project-config the driver name and the connection name is the same thing so i was just getting confused where it lived | 04:28 |
| jeblair | jamielennox: yeah, me too. :) basically, everything in zuul.yaml is a connection name; the driver only appears in zuul.conf. | 04:29 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Only report to gerrit if the action is from gerrit https://review.openstack.org/461981 | 04:31 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Remove source from reporter https://review.openstack.org/462362 | 04:31 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Remove source from reporter https://review.openstack.org/462362 | 04:53 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Only report to gerrit if the action is from gerrit https://review.openstack.org/461981 | 04:53 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Remove source from reporter https://review.openstack.org/462362 | 05:07 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Only report to gerrit if the action is from gerrit https://review.openstack.org/461981 | 05:07 |
| *** isaacb has joined #zuul | 05:24 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Turn of time stamps in dib log https://review.openstack.org/462321 | 05:31 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: devstack: nodepool log to separate file https://review.openstack.org/462380 | 05:31 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Turn off time stamps in dib log https://review.openstack.org/462321 | 05:32 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Turn off time stamps in dib log https://review.openstack.org/462321 | 06:02 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: devstack: nodepool log to separate file https://review.openstack.org/462380 | 06:02 |
| *** hashar has joined #zuul | 06:44 | |
| *** isaacb has quit IRC | 06:52 | |
| *** jamielennox is now known as jamielennox|away | 07:06 | |
| *** isaacb has joined #zuul | 08:16 | |
| *** Cibo_ has joined #zuul | 08:57 | |
| *** Cibo_ has quit IRC | 09:26 | |
| *** bhavik1 has joined #zuul | 09:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Turn off time stamps in dib log https://review.openstack.org/462321 | 10:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: devstack: nodepool log to separate file https://review.openstack.org/462380 | 10:07 |
| *** yolanda_ has joined #zuul | 10:10 | |
| *** yolanda_ has quit IRC | 10:12 | |
| *** bhavik1 has quit IRC | 10:12 | |
| *** jkilpatr has joined #zuul | 10:53 | |
| lennyb | Hi, I am using zuul+jenkins (ThirdPartyCommonCI Solution). If I ask to merge a patchset that depends on unmerged commit I see that zuul does not return failure to the gerrit. Is there something that can be configured? #link http://paste.openstack.org/show/608845/ | 11:02 |
| *** yolanda has quit IRC | 11:13 | |
| *** yolanda has joined #zuul | 12:29 | |
| pabelanger | morning | 13:32 |
| dmsimard | Curious, how will users upload secrets file into zuul v3 ? Say, the equivalent of the jenkins credentials feature. | 13:33 |
| pabelanger | dmsimard: we have a secret section in top level config, which takes data. It is encrypted with pkcs1, IIRC | 13:35 |
| pabelanger | https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html#secrets | 13:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Copy nodepool logs to devstack log dir https://review.openstack.org/462566 | 13:43 |
| pabelanger | jeblair: SpamapS: left a comment on 453851 about accessing SSH_AUTH_SOCK inside brwap. | 13:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 13:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 13:57 |
| *** jamielennox|away is now known as jamielennox | 13:58 | |
| *** dkranz_ has joined #zuul | 13:59 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: POC of ssh-agent playbook https://review.openstack.org/462326 | 14:00 |
| pabelanger | if we could get a few reviews on https://review.openstack.org/#/c/462268/ and https://review.openstack.org/#/c/462300/ this morning, it would allow us to continue with our openstack-zuul-jobs topic today. | 14:25 |
| jeblair | pabelanger: why would we want to write a .ssh/config file? | 15:05 |
| jlk | jeblair: thanks for that reply on the reporter change. Makes sense | 15:07 |
| pabelanger | jeblair: I made some comments on 453851 explaining some of the logic, have you seen them? | 15:08 |
| jeblair | pabelanger: yes, that's what i'm asking about | 15:08 |
| pabelanger | k | 15:08 |
| pabelanger | so | 15:08 |
| *** isaacb has quit IRC | 15:09 | |
| pabelanger | my thoughts would be, if we created a .ssh/config file inside brap, I think ansible-playbook SSH command would use them, and get information about our SSH_AUTH_SOCK. Otherwise, we'd need to some how set the SSH_AUTH_SOCK inside bwrap container | 15:09 |
| pabelanger | zuul could also setup the SSH_AUTH_SOCK variable before calling bwrap. | 15:10 |
| *** hashar has quit IRC | 15:10 | |
| pabelanger | but, this POC should work without code changes to zuul, if it ran on executor | 15:11 |
| jeblair | pabelanger: we want zuul to start and stop the ssh agent, so it can set SSH_AUTH_SOCK when it runs ansible-playook (whether that's with bwrap or not). | 15:12 |
| jeblair | pabelanger: i think that will be much simpler than ssh config files | 15:12 |
| pabelanger | sure, zuul can start / stop ssh-agent too. I'm trying to understand why that is preferred | 15:15 |
| *** rcarrillocruz has quit IRC | 15:23 | |
| *** rcarrillocruz has joined #zuul | 15:29 | |
| jeblair | pabelanger: sorry, was eating breakfast. | 15:42 |
| jeblair | pabelanger: a couple of reasons -- first, while possible, i don't think that having playbooks start daemons on the executor is a model we should encourage | 15:42 |
| jeblair | pabelanger: (we might even do something to prevent that in the future) | 15:43 |
| jeblair | pabelanger: second, since it is a daemon, it seems like it will be more reliable to ensure that it's stopped correctly if we do it inside of zuul | 15:44 |
| jeblair | pabelanger: third, as you have noted, getting the agent shared between the multiple ansible processes is more difficult in playbooks. it's not easy for me to wrap my head around where the .ssh/config file would be in the different execution contexts | 15:45 |
| jeblair | pabelanger: fourth, i like the idea of not having the key material available in the execution context at all; so having the job start out with the agent and no private key is nice | 15:46 |
| jeblair | pabelanger: we need to provide the ssh key(s) to the playbooks as part of the executor api -- it looks like the agent is a good way of doing that | 15:47 |
| pabelanger | Right, I think that is reasonable. But, one thing I am not sure if we can control, is how operator will run things on an executor. Sure we can discourage people from starting daemons on executors but I could we really stop them? | 15:54 |
| jeblair | pabelanger: eventually, yes. we could decide to use bubblewrap or something similar even for trusted playbooks. maybe we allow more access, but we still do things like kill process groups, etc. | 15:59 |
| mordred | yah - a bubblewrap with more wider permissions, perhaps | 16:01 |
| pabelanger | even got me thinking, there would also be nothing stopping somebody to write a trusted job to change the untrusted.cfg in a jobdir too. So, I suspect we could see some crazy things in coming day. | 16:03 |
| jeblair | pabelanger: yep. it's called 'trusted' for a reason. :) | 16:03 |
| pabelanger | okay, so POC a side. I think ssh-agent would work nice for keeping private key out of bwrap | 16:04 |
| pabelanger | either single SSH key, like today or 1time SSH keys, like we discussed last night | 16:04 |
| pabelanger | jeblair: next exception from zuul. Trying to get job roles working: http://paste.openstack.org/show/608890/ | 16:27 |
| pabelanger | jeblair: I believe the key should be 'target_name' now? | 16:27 |
| pabelanger | jeblair: 462202 is the review in question | 16:29 |
| jeblair | pabelanger: are we running different zuul/executor versions? maybe restart them both with branch tip? | 16:30 |
| pabelanger | I am un sure, but I can upgrade everything sure | 16:30 |
| pabelanger | okay, restarted | 16:33 |
| pabelanger | along with upgraded | 16:33 |
| pabelanger | Hmm, I still see the same exception | 16:33 |
| pabelanger | let me see if I can reproduce with unit test | 16:34 |
| jeblair | pabelanger: same line numbers? | 16:35 |
| jeblair | (they were off from my branch tip, so i'd expect them to be different after an upgrade/restart) | 16:35 |
| pabelanger | line 850 now: http://paste.openstack.org/show/608891/ | 16:36 |
| jeblair | pabelanger: ok. so it looks like we don't have test coverage of a role specified as an untrusted zuul repo where that repo appears in the speculative change stack. probably we just need a test that proposes a change to the role repo itself to touch that. | 16:40 |
| jeblair | pabelanger: do you want to write that test, or do you want me to? | 16:41 |
| pabelanger | jeblair: if you want to sure, I was about to grab some lunch. | 16:41 |
| pabelanger | but, happy to do it when I get back | 16:41 |
| jeblair | pabelanger: i'll take care of it | 16:42 |
| pabelanger | jeblair: great, thanks | 16:43 |
| *** yolanda has quit IRC | 16:47 | |
| *** jkilpatr has quit IRC | 16:48 | |
| SpamapS | honestly, the bubblewrap alreayd has pretty broad permissions. It's not a terrible idea to run trusted playbooks inside of it (just without the local restriction in ansible.. so we can still do interesting things, but only the interesting things we expose to the bwrap | 16:53 |
| SpamapS | it would also make things more consistent | 16:53 |
| *** yolanda has joined #zuul | 16:54 | |
| pabelanger | enabling bwrap by default for trusted too, seems reasonable. With ability to opt out, if wanted | 16:55 |
| *** jkilpatr has joined #zuul | 17:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add a test which exercises a speculative role checkout https://review.openstack.org/462677 | 17:05 |
| jeblair | yeah, main thing with trusted bubblewrap is i expect trusted playbooks to use some things that we (as operators) know are on the system, like afs. we can make that work though. | 17:06 |
| jeblair | pabelanger: https://review.openstack.org/461092 inadvertantly fixes the bug you found, so i've approved it. | 17:07 |
| jeblair | pabelanger: so https://review.openstack.org/462677 just adds the regression test (which fails on branch-tip, but passes with 461092). | 17:07 |
| pabelanger | Yay for easy fixes | 17:09 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use the security context of the playbook when checking out roles https://review.openstack.org/461092 | 17:15 |
| pabelanger | jeblair: okay, fresh coffee in hand. okay to upgrade zuulv3-dev to ^? | 17:17 |
| pabelanger | Hmm, i think our jobs are broken now | 17:20 |
| pabelanger | for zuulv3 | 17:21 |
| pabelanger | looking | 17:21 |
| pabelanger | okay, think I see the issue | 17:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add git.openstack.org to workspace directory https://review.openstack.org/462684 | 17:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Be less agressive with implied branch matchers https://review.openstack.org/462268 | 17:37 |
| pabelanger | jeblair: what are your thought about exposing connection name to vars.yaml? | 17:37 |
| pabelanger | currently zuul.project does not contain it | 17:38 |
| SpamapS | pabelanger: so have you started working on any of the ssh agent stuff? | 17:39 |
| SpamapS | pabelanger: if not I'm going to take a crack at it. | 17:40 |
| pabelanger | SpamapS: nothing in zuul yet, just the POC playbooks | 17:40 |
| pabelanger | SpamapS: please do | 17:40 |
| SpamapS | I'm mostly just going to work on generating key and starting ssh-agent in zuul native, and then handing that off to the playbook that distributes said key. | 17:40 |
| pabelanger | does zuul need to create the key? I thought that was in playbook | 17:41 |
| pabelanger | actually, ignore me | 17:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Report job shadow errors to users https://review.openstack.org/462300 | 17:42 |
| jeblair | pabelanger, SpamapS: i think key generation is in playbook? (see second section of https://etherpad.openstack.org/p/EIrCy4mhuR ) | 17:51 |
| jeblair | pabelanger: i don't think the connection name should show up in vars; but we may need to include the canonical hostname for zuul_project. or include the hostname in zuul_project. | 17:52 |
| jeblair | pabelanger: in response to a question from jamielennox, clarkb and i put ideas in this etherpad: https://etherpad.openstack.org/p/vB1WjfL804 | 17:53 |
| SpamapS | jeblair: yeah that makes sense. :) | 17:56 |
| SpamapS | jeblair: just now getting to where I start an SSH agent and pass the env in. | 17:58 |
| pabelanger | jeblair: today we set zuul.project to item.change.project.name. If we changed it to item.change.project, we'd get zuul.project.name and zuul.project.canonical_hostname I think | 18:00 |
| pabelanger | zuul.project becomes a dict for use to use in playbooks | 18:00 |
| jeblair | pabelanger: yeah, i think that approximates clarkb's suggestion. | 18:00 |
| jeblair | i'm cool with that. | 18:01 |
| pabelanger | k, let me see about getting a patch and test up | 18:01 |
| *** Cibo_ has joined #zuul | 18:11 | |
| *** harlowja has quit IRC | 18:19 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Update zuul.project to dictonary for vars.yaml https://review.openstack.org/462698 | 18:21 |
| pabelanger | clarkb: jeblair: ^ thoughts? | 18:22 |
| pabelanger | not sure if we want to complete project object or limited version of it | 18:22 |
| SpamapS | jeblair: referring back to the etherpad, we left out where we think the "real" nodepool key gets added to the SSH agent. I'm inclined to add it in the playbook, and them remove it there as well once we've copied the per-build one to the nodes. Thoughts? | 18:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Use canonical_name in workspace directory https://review.openstack.org/462684 | 18:43 |
| jeblair | SpamapS: i was thinking zuul would add it -- that way it's zuul's way of saying to the playbooks "here's your inventory and keys, go nuts". | 18:48 |
| jeblair | SpamapS: (since the decision of which key to use, and where to find it on disk is zuul's anyway; having the playbook do it means the playbook has to know the private key path and be able to access it (which is extra tricky in bubblewrap) and so negates some of the advantage of having zuul manage the agent) | 18:49 |
| jeblair | SpamapS: (i added missing entry in etherpad) | 18:52 |
| SpamapS | jeblair: salient points. :) Ok, adding that. | 18:54 |
| jeblair | pabelanger: left comments | 18:56 |
| jeblair | pabelanger: er, strike my second comment. so i'm down to "left one (salient) comment" :) | 18:57 |
| pabelanger | jeblair: sure, let me remove connection_name | 18:58 |
| *** Cibo_ has quit IRC | 18:58 | |
| *** Cibo_ has joined #zuul | 18:59 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Use canonical_name in workspace directory https://review.openstack.org/462684 | 19:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Update zuul.project to dictonary for vars.yaml https://review.openstack.org/462698 | 19:00 |
| pabelanger | jeblair: ^ | 19:00 |
| pabelanger | I just thought of something, as I was unlocking my SSH key for signed commits. Now that we have secrets in zuulv3, could we not have zuul sign merge commits? | 19:01 |
| jeblair | pabelanger: cool. considering the out-of-band discussion on that, i'm going to go ahead and +3 it. | 19:01 |
| pabelanger | jeblair: yay | 19:01 |
| jeblair | pabelanger: you've given me something to think about over lunch. :) | 19:01 |
| pabelanger | :) | 19:02 |
| *** Cibo has joined #zuul | 19:02 | |
| *** Cibo_ has quit IRC | 19:03 | |
| *** Cibo has quit IRC | 19:08 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add SSH Agent Primitives and usage https://review.openstack.org/462712 | 19:13 |
| SpamapS | this is pretty WIP'y, and it's stacked all wrong, but ^^ is fairly complete as far as managing an SSH agent... just doesn't manage keys iwth it | 19:13 |
| * SpamapS goes to lunch | 19:13 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Update zuul.project to dictonary for vars.yaml https://review.openstack.org/462698 | 19:19 |
| pabelanger | okay, I am going to update zuulv3-dev to latest master | 19:36 |
| *** Cibo_ has joined #zuul | 19:37 | |
| pabelanger | upgraded / started again | 19:38 |
| *** harlowja has joined #zuul | 19:48 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Move playbook out of zuul https://review.openstack.org/462210 | 19:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add jobs back to .zuul.yaml https://review.openstack.org/462212 | 19:59 |
| *** Cibo_ has quit IRC | 20:07 | |
| pabelanger | Okay, I this makes me giddy https://review.openstack.org/#/q/topic:openstack-zuul-jobs | 20:21 |
| pabelanger | all green | 20:21 |
| pabelanger | 5 patches, across 2 branches, over 3 repos. All running #ansible playbooks and roles for testing jobs, all before any job has been merged! | 20:22 |
| pabelanger | super awesome! | 20:22 |
| jeblair | pabelanger: neat :) | 20:22 |
| jeblair | the next thing we should do is start separating things out into the stdlib. copy-git-repos and install-ssh-keys are the first things that should go there. wherever 'there' is. :) | 20:23 |
| pabelanger | indeed | 20:23 |
| jeblair | pabelanger: so you did decide to split jobs and roles? | 20:23 |
| pabelanger | jeblair: yes, mostly to see if it worked | 20:24 |
| jeblair | *nod* | 20:24 |
| jamielennox | pabelanger: thanks for the zuul.project patch, i had a way hackier one that i didn't want to propose yet | 21:22 |
| jamielennox | pabelanger: the other thing i think could be useful in there is a src_dir or similar | 21:22 |
| jamielennox | i can construct the source dir from canonical_name so it's not a big deal but it would make the playbook start really easy | 21:23 |
| pabelanger | jamielennox: do you have an example playbook handy? I don't understand 'src_dir' in your example | 21:25 |
| jamielennox | pabelanger: src_dir just being the directory that zuul checked the code out to | 21:25 |
| pabelanger | src_root? | 21:25 |
| jamielennox | whatever | 21:25 |
| jeblair | i think that's there already? | 21:26 |
| pabelanger | I thought we expose it | 21:26 |
| jamielennox | oh? | 21:26 |
| pabelanger | zuul.executor.src_root | 21:27 |
| jamielennox | i don't have a dump in front of me, but isn't that /tmp/tmpXXXXX/ ? | 21:28 |
| jeblair | jamielennox: yeah; do you mean where we copy stuff to on the node? | 21:28 |
| jeblair | jamielennox: (well, zuul.executor.src_root will be /tmp/tmpXXXX/work/src/) | 21:29 |
| jeblair | or something like that | 21:30 |
| jamielennox | so at the moment i run a script with chdir: "{{ bonnyci_workspace_root }}/src/{{ zuul.project }}" | 21:30 |
| jamielennox | again, not a big problem i have all the info i need, but it would be useful if that was just {{ zuul.src_dir }} or something | 21:30 |
| jeblair | yeah, so that's why the next thing for us to do is work on the standard lib thing | 21:30 |
| pabelanger | you could make that a top-level ansible var today, in your playbook too | 21:31 |
| jamielennox | yep, that's true | 21:31 |
| jeblair | because the 'copy to git repos' role should be something we share | 21:31 |
| pabelanger | yes | 21:31 |
| jamielennox | ok, not pushing it, it was just something i had included when playing around | 21:31 |
| jamielennox | but yea, it could be done in the ansible | 21:31 |
| jeblair | jamielennox: no, it's time for us to get serious about actually setting up shared standard roles :) | 21:32 |
| *** yolanda has quit IRC | 21:34 | |
| jeblair | jamielennox: but at any rate, the destination directory is a construct of the 'copy-git-repos' role (zuul doesn't have to know about it), so i think what we want is a (shared) base job definition that sets the variable you want -- similar to how we have the zuul_workspace_root variable now. | 21:34 |
| jeblair | i think this is the thing to work on after we get back from the summit | 21:34 |
| pabelanger | I have some general fears of scoping of variables in ansible. Looking forward to seeing what our first stdlib looks like | 21:36 |
| jamielennox | yea, it's split. ansible copies over the entire src_root (at least for me), but executor-server is specifying the path within that | 21:36 |
| jamielennox | like a can't remember where that '/src/' comes from | 21:37 |
| pabelanger | https://github.com/ansible/ansible/issues/2796 | 21:38 |
| jamielennox | pabelanger: yea, there's a number of things like that ansible could have done better initially, for now namespace everything by role | 21:39 |
| pabelanger | agree, this is one place I like puppet better for. And something we should try to impose in zuul stdlib | 21:40 |
| pabelanger | I wonder the amount of effort to add something like that into ansible | 21:40 |
| jamielennox | pabelanger: i imagine mind boggling and backwards incompatible | 21:41 |
| jeblair | i think we had a conversation about this, and agreed to for the most part, with exceptions for things like 'zuul_workspace_root' which are essentially global variables for all zuul-specific roles -- judging the 'zuul_' prefix to be sufficiently namespaced for that. | 21:42 |
| jamielennox | yep, thats a loose ansible convention here as well, the roles have namespaces of the role names, and anything that is truly global is namespace bonnyci_ and those variables are passed to the roles | 21:44 |
| jamielennox | ideally means a user would be able to deal with bonnyci_ variables as an entity, not the combination of many roles though it's harder and doesn't always hold in practice | 21:45 |
| pabelanger | woah | 22:08 |
| pabelanger | https://docs.ansible.com/ansible/include_role_module.html | 22:08 |
| pabelanger | If True the variables from defaults/ and vars/ in a role will not be made available to the rest of the play. | 22:08 |
| pabelanger | So, that is pretty awesome actually | 22:08 |
| pabelanger | and could offer some protection for roles leaking variables | 22:08 |
| pabelanger | will have to test that tomorrow | 22:09 |
| jlk | l've been waiting for somebody to describe a use case for that. | 22:11 |
| jlk | jamielennox: what's your current thoughts on tenacity? Should we port that work over from v2.5, or do you think we should do the retries some other way? | 22:14 |
| jeblair | jlk: what's tenacity in this context? | 22:35 |
| jlk | jeblair: a thing to let us re-try various github API actions | 22:36 |
| jeblair | gotcha | 22:36 |
| jeblair | being familiar with their git error rate, i can imagine that their web api error rate may not be entirely dissimilar. | 22:37 |
| jlk | heh | 22:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor https://review.openstack.org/461177 | 22:38 |
| adam_g | jlk: i think our usage of that could use some work. its currently decorating everything in the github connection, and retrying everything. the idea was to retry the possibly transient things and not things like merge failures due to repo perms | 22:38 |
| jlk | well, and that gets weird too | 22:39 |
| jlk | like, a merge failure due to perms, that could be a race condition with the status. But we already have a retry baked into the code there | 22:39 |
| jeblair | (fwiw, while it's perhaps not as urgent, the concept applies about as well to gerrit -- we do occasionally get reporting errors there too) so if this is generalizable at all, that'd be neato. | 22:44 |
| pabelanger | jeblair: did you have a chance to think about signed commits over lunch? Not that I'm saying we should implement them | 22:48 |
| jlk | to what end? pushing the merged commit back up? | 22:51 |
| mordred | pabelanger: fwiw, if we get to the point where zuul is making the merge commit and pushing it rather than triggering the merge action via api, I thnk it would be awesome to have zuul sign those commits | 22:55 |
| pabelanger | Ya, it was just something that popped into my head, as I unlocked my key this afternoon | 22:56 |
| mordred | ++ | 22:56 |
| pabelanger | not sure if we'd want to do it or not | 22:56 |
| jlk | ah. yeah, I don't think we'd be doing that on Github any time soon | 22:57 |
| jlk | gets tricky with ssh keys and whatnot | 22:57 |
| pabelanger | mordred: not sure if you seen, but https://review.openstack.org/#/q/status:open+topic:openstack-zuul-jobs is a pretty cool stack. | 22:57 |
| jlk | and extending write access to zuul | 22:58 |
| pabelanger | and ready for reviews :D | 22:58 |
| pabelanger | jlk: ya, I am not too familiar how github is working. | 22:58 |
| pabelanger | jeblair: so, I think a found another issue in zuulv3. I think it is possible for a change to get lost by zuul, and untested. | 22:59 |
| pabelanger | jeblair: I think the sequence of events is, upload bad patch to gerrit for zuulv3 to test, job for patchset loops until RETRY_LIMIT is return. During the looping, upload patchset #2, it doesn't seem to abort patchset #1. And patchset #1 continues to run and patchset #2 will be untested | 23:01 |
| jlk | dequeue not firing off? | 23:03 |
| pabelanger | haven't look at logs yet | 23:09 |
| pabelanger | just something I noticed today | 23:09 |
| jamielennox | jlk: i'm not a fan of the way we're using tenacity, the problems it seems to be catching are real problems and it's muddling the results | 23:17 |
| jlk | kk | 23:18 |
| jamielennox | jlk: there is a retries= option to requests that should allow us to retry like a connection failure | 23:18 |
| jlk | can we plumb that through github3.py? | 23:18 |
| jamielennox | but at the moment it catches any error, so real things like auth failures are being retried where they won't work | 23:18 |
| jamielennox | jlk: i think you can default it on a requests.Session but i'd have to double check | 23:18 |
| jamielennox | pabelanger: where is base ending up in that stack? | 23:21 |
| jamielennox | also is there a rationale for splitting jobs from roles for openstack? | 23:22 |
| jeblair | pabelanger: yeah, my thinking was along the lines of mordred -- i think that becomes useful when we have zuul pushing commits for public consumption; it's probably not going to buy us a lot before then though (the internal-to-zuul channels that commits move around are all otherwise secured) | 23:24 |
| pabelanger | jamielennox: https://review.openstack.org/#/c/462210/ is the base | 23:24 |
| jeblair | jamielennox: it ends up in openstack-zuul-jobs, but that's temporary -- that's a thing that needs to move into the stdlib so we can all share it | 23:24 |
| pabelanger | I also think if we have openstack-zuul-roles, it make possible promotion into stdlib zuul a little easier too. Since openstack-zuul-jobs would already be consuming it from an external source | 23:26 |
| pabelanger | but, mostly I think we can experiment for now | 23:26 |
| pabelanger | see how it works | 23:26 |
| jamielennox | pabelanger: sorry, i meant the base role, i didn't see it added back in anywhere, but in openstack-zuul-jobs is good for now | 23:27 |
| pabelanger | ah, right. Yes, that is just openstack-zuul-jobs right now. But, that is only because we don't have zuul stdlib ATM | 23:27 |
| jamielennox | i wasn't sure if we came down on stdlib in zuul code base or seperate | 23:28 |
| pabelanger | I think that will be the next step, maybe move prepare-workspace into zuul specific | 23:28 |
| pabelanger | actually, will zuul stdlib be a config-project or untrusted? | 23:29 |
| jeblair | pabelanger: we have some options, including building some things into zuul itself (so you can use some jobs/roles without adding a repo). | 23:34 |
| jeblair | pabelanger: so we might put a repo out there with a bunch of stuff that you can add to your zuul system. we might make a pypi package with things that we separately release. we might distribute things with zuul itself. we might do all of those depending on the nature of the specific items. | 23:36 |
| *** jkilpatr has quit IRC | 23:38 | |
| *** jkilpatr has joined #zuul | 23:39 | |
| pabelanger | jeblair: ack | 23:42 |
| pabelanger | also, last question. Where did we land on python3 support for nodepool? Are we ready to start accepting some patches for that on feature/zuulv3 or keep holding off | 23:43 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!