| *** tobiash has quit IRC | 04:58 | |
| *** tobiash has joined #zuul | 04:58 | |
| *** bhavik1 has joined #zuul | 05:02 | |
| *** tobiash has quit IRC | 05:03 | |
| *** tobiash has joined #zuul | 05:03 | |
| *** tobiash has quit IRC | 05:04 | |
| *** tobiash has joined #zuul | 05:04 | |
| *** saneax-_-|AFK is now known as saneax | 05:29 | |
| *** bhavik1 has quit IRC | 05:52 | |
| *** bhavik1 has joined #zuul | 05:53 | |
| *** abregman_ has joined #zuul | 06:09 | |
| *** bhavik1 has quit IRC | 06:32 | |
| *** toabctl has joined #zuul | 06:54 | |
| *** Cibo has quit IRC | 08:20 | |
| *** Cibo has joined #zuul | 08:20 | |
| *** saneax is now known as saneax-_-|AFK | 08:27 | |
| *** saneax-_-|AFK is now known as saneax | 08:36 | |
| *** Cibo_ has quit IRC | 08:48 | |
| *** hashar has joined #zuul | 08:48 | |
| *** saneax is now known as saneax-_-|AFK | 09:01 | |
| *** saneax-_-|AFK is now known as saneax | 09:11 | |
| *** saneax is now known as saneax-_-|AFK | 09:15 | |
| *** saneax-_-|AFK is now known as saneax | 09:58 | |
| rcarrillocruz | clarkb: if it looks good to you https://review.openstack.org/#/c/403732/ i will rebase the last d-g change about network check off it | 12:06 |
|---|---|---|
| pabelanger | morning | 12:39 |
| *** hashar has quit IRC | 12:51 | |
| *** hashar has joined #zuul | 12:53 | |
| *** abregman_ is now known as abregman|mtg | 13:25 | |
| *** saneax is now known as saneax-_-|AFK | 13:29 | |
| *** saneax-_-|AFK is now known as saneax | 13:37 | |
| *** abregman|mtg is now known as abregman | 13:52 | |
| *** saneax is now known as saneax-_-|AFK | 15:30 | |
| *** yolanda has quit IRC | 15:41 | |
| *** yolanda has joined #zuul | 15:47 | |
| *** abregman has quit IRC | 16:43 | |
| *** hashar has quit IRC | 16:47 | |
| *** saneax-_-|AFK is now known as saneax | 17:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: [WIP] Add publisher for Federated Message Bus (fedmsg) https://review.openstack.org/426861 | 17:36 |
| pabelanger | jeblair: threebean: tflink: mordred: I wrote a thing!^ | 17:36 |
| pabelanger | if I did it correctly, that does publish things, but something that needs some discussion, is the message format we actually publish. I have no idea what that looks like right now | 17:38 |
| jeblair | pabelanger: cool :) (but we should call it a reporter) | 17:38 |
| pabelanger | jeblair: yes | 17:38 |
| jeblair | (i mean, it's probably a publisher from a fedmsg pov, but a reporter from zuul pov) | 17:39 |
| pabelanger | agreed | 17:39 |
| jeblair | pabelanger: maybe add docs? | 17:39 |
| pabelanger | jeblair: Yes, I'd like to actually ask threebean some questions about fedmsg configuration things first. You might have some input too, but fedmsg has a os-cloud-config like thing for configuration. I'm thinking we can just drop the fedmsg config files outside of zuul (/etc/fedmsg.d for example) and import fedmsg will do the right thing. Or, do you think we could add them into the connection section for | 17:42 |
| pabelanger | zuul.conf? | 17:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: [WIP] Add reporter for Federated Message Bus (fedmsg) https://review.openstack.org/426861 | 17:43 |
| jeblair | pabelanger: yeah -- i think if fedmsg can handle (most of) it in its own config files, that would be best. anything that can't be handled that way can go in the connection section of zuul.conf. | 17:43 |
| threebean | +1 | 17:43 |
| threebean | drop files in /etc/fedmsg.d/ :) | 17:43 |
| pabelanger | perfect, that is what I was thinking too | 17:44 |
| pabelanger | threebean: ohai | 17:44 |
| pabelanger | threebean: are you still in brno? | 17:44 |
| threebean | I am! | 17:44 |
| threebean | you? | 17:44 |
| pabelanger | threebean: yes | 17:44 |
| pabelanger | we should meet up for food and beer things tonight | 17:44 |
| pabelanger | and discuss some fedmsg things | 17:44 |
| mordred | pabelanger: oh wow. that's an AMAZING error above in nodepool-dsvm :( | 17:45 |
| pabelanger | mordred: sad face indeed. I haven't debugged it as of yet | 17:45 |
| mordred | pabelanger: I've got another weird dsvm bug I need to debug too - in both cases I _believe_ something has changed in devstack that we're not taking in to consideration | 17:46 |
| mordred | pabelanger: I'm SUPER excited about figuring it out | 17:46 |
| Shrews | jeblair: So, over the weekend, I was thinking about your idea to eliminate the NodeRequestWorker thread and do polling in the launch manager. I think that's a good idea to eliminate some thread contention. | 17:50 |
| Shrews | jeblair: downside, sets me back a bit to redo some things | 17:50 |
| jeblair | Shrews: cool -- i think we can proceed with the extra thread for now and remove it later if you want. i don't think it's going to back us into a corner. but if you'd rather rework it now, that works too. | 17:51 |
| Shrews | jeblair: i considered that, but i think it would be harder to do later. just going to jump in now | 17:51 |
| jeblair | Shrews: okay, i support what works best for you :) | 17:52 |
| *** harlowja has joined #zuul | 18:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: [WIP] Add reporter for Federated Message Bus (fedmsg) https://review.openstack.org/426861 | 18:11 |
| *** saneax is now known as saneax-_-|AFK | 18:33 | |
| SpamapS | pabelanger: ^^ neat! | 18:45 |
| jlk | jeblair: I have another design question (this is still 2.5). I'm curious about the split of code between zuul/connection/<thing> and zuul/source/<thing>, and if I need to add code for <thing> to gather more information about a change before deciding to enqueue it, should I call that in the event handling stuff in zuul/connection/<thing> making it part of the event data, or should I call it from zuul/source/<thing> making it part of the Change data? | 18:53 |
| *** tobiash has quit IRC | 19:09 | |
| *** tobiash has joined #zuul | 19:27 | |
| *** hashar has joined #zuul | 19:27 | |
| jeblair | jlk: a lot of that structure is different in v3, but generally speaking, if it's information about the change (that persists beyond the event), it should probably be in the source. i hesitate to get too specific since i'm focused on v3. | 21:26 |
| jlk | fair enough | 21:26 |
| jhesketh | Morning | 21:40 |
| mordred | morning jhesketh | 21:54 |
| jhesketh | o/ | 21:56 |
| SpamapS | meeting in 3! | 21:57 |
| SpamapS | 2 even | 21:58 |
| jeblair | indeed, and our agenda has actual topics today! | 21:58 |
| jeblair | meeting in 0 in #openstack-meeting-alt | 22:00 |
| *** hashar has quit IRC | 22:30 | |
| SpamapS | rcarrillocruz: jeblair I learned about "Tang and Clevis" at devconf.cz for secret management.. | 23:01 |
| mordred | Shrews, clarkb: rolling upper-constraints forward in openstack/requirements seems to fix the nodepool and shade dsvm jobs | 23:01 |
| SpamapS | https://blog-ftweedal.rhcloud.com/2016/02/introduction-to-tang-and-clevis/ | 23:01 |
| jeblair | pabelanger: to continue that thought -- i think we're close to being ready for that tool, but my feeling is we probably want a little more development in terms of zuul actually running ansible before we get started. | 23:01 |
| SpamapS | I'm wondering if it might be useful for secrets in Zuul. | 23:01 |
| mordred | SpamapS: scanning the blog post, it looks interesting | 23:02 |
| Shrews | mordred: ++ | 23:02 |
| pabelanger | jeblair: sure, I am okay with that. And happy to get some ansible playbooks going first. It was just a common question this week in devconf.cz, talking to people about zuulv3 things. A lot of interested in how JJB to ansible converter loks | 23:02 |
| mordred | SpamapS: I need to read it further | 23:02 |
| pabelanger | looks* | 23:02 |
| mordred | pabelanger, jeblair: fwiw - back on the 2.5 branch I started some work to try to decouple the ansible translation from the zuul execution context | 23:03 |
| jeblair | SpamapS: do you have time to review that and the v3 secrets spec (where i think we recorded our precepts and thinking) and see how relevant it is? maybe an email thread if so? | 23:03 |
| *** saneax-_-|AFK is now known as saneax | 23:03 | |
| SpamapS | The short version is: Users have a secret. They get a public key from Tang(server) do some crypto, send a crypto result to Tang in the clear, and then when they want to decrypt, they ask the server for a cryptographic answer back, and they can use that answer to re-discover the secret key and do decryption. | 23:03 |
| pabelanger | mordred: oh, nice | 23:03 |
| SpamapS | So you can only decrypt when the Tang server is reachable. | 23:04 |
| pabelanger | Ya, it was pretty neat | 23:04 |
| SpamapS | jeblair: yeah that's exactly what I've been doing. | 23:04 |
| mordred | pabelanger: I think we tossed most of it- but it may be an ok starting point (or may not) | 23:05 |
| jlk | Tang never stores in the clear? | 23:05 |
| SpamapS | jlk: correct | 23:05 |
| clarkb | mordred: so it was another bad release? | 23:05 |
| SpamapS | jlk: it only has one half of a DH key operation. | 23:05 |
| jlk | isn't that like a huge win over barbican? | 23:05 |
| pabelanger | https://latchset.github.io/ | 23:05 |
| SpamapS | jlk: well, it's a massive win over barbican's key exchange yes. | 23:05 |
| SpamapS | You could still use Barbican for the meta-ops around Tang | 23:05 |
| mordred | clarkb: actually - it was that osc made a release which works with the new openstacksdk but the upper-constraint on osc didn't get bumped - so devstack was using new sdk with old osc | 23:05 |
| jeblair | SpamapS: okay. it sounds interesting. much of what we have done with zuul secrets is about making sure they are encodable into git repos. at first glance, i'm not sure how those relate. | 23:05 |
| pabelanger | mordred: ack | 23:06 |
| SpamapS | jeblair: we would encode something encrypted by a key guarded by Tang/Clevis interaction, and users could safely put their encrypted keys in their image. | 23:06 |
| SpamapS | And then we only let their instances contact Zuul to get the missing piece to decrypt. | 23:07 |
| mordred | pabelanger: starts here ish: https://review.openstack.org/#/c/372284/ - you can see I was trying to get things dependent on runtime split from things not | 23:07 |
| pabelanger | The big thing about the tang presentation, was the lack of infra structure is needed. It was actually impressive, assuming the algorithm was a valid approach | 23:07 |
| SpamapS | and any snapshots or replays would fail because zuul wouldn't respond to them. | 23:07 |
| pabelanger | mordred: okay, I'll look at it on my flight tomorrow | 23:08 |
| jeblair | SpamapS: you mean having a nodepool node talk to zuul? | 23:08 |
| SpamapS | Right, Tang stores some integers, and obviously needs to be protected at the network layer so only authorized hosts can interact with it, but that's it. | 23:08 |
| jeblair | SpamapS: eliminating backward facing communication from workers to zuul is a deliberate design decision for v3 | 23:09 |
| SpamapS | vs. HSM's and X.509's and etc. | 23:09 |
| jeblair | SpamapS: we want nodes to be able to run in untrusted clouds outside of a corporate firewall | 23:09 |
| mordred | I think there may be multiple pieces here | 23:09 |
| jeblair | thus all the hullabaloo about putting mergers on launchers, etc | 23:10 |
| SpamapS | we could also spin up tangs in their network for them. | 23:10 |
| SpamapS | and manage the L2 separation. | 23:10 |
| SpamapS | but yeah | 23:10 |
| SpamapS | I see where because it is built on reachability, and the current system is a push system, it creates problems | 23:10 |
| jeblair | SpamapS: yeah, most of our work on secrets is not because we don't trust zuul. we do -- there's little to be gained by employing a third-party secret provider (and heck, if you wanted too, theres even nothing in the zuulv3 space stopping you from doing so) | 23:13 |
| jeblair | SpamapS: it's that we want to be able to have as much of this in public git repos as possible | 23:14 |
| SpamapS | jeblair: I could see users not wanting Zuul to have their secrets. | 23:16 |
| SpamapS | Especially in the Zuul-aaS case. | 23:16 |
| SpamapS | I may not want peoples' secrets. | 23:16 |
| SpamapS | So it might make sense to have one version that has Zuul having private keys and pushing decrypted secrets to nodes, and another that uses someting like Tang to allow the users to keep the control. | 23:17 |
| mordred | SpamapS: I don't think it's going to get you there | 23:17 |
| mordred | SpamapS: at some point zuul is going to have to be able to execute some ansible that's going to need access to the secret or the ability to ask tang for the secret which is effectively the same thing | 23:18 |
| SpamapS | Note that the exchange was designed to not be MITM sensitive. The only vulnerable spot is the TOFU for client->Tang | 23:18 |
| pabelanger | I'm actually interested in this topic, but it is way too late for me. I'll catch up on the backscroll. | 23:19 |
| pabelanger | later all | 23:19 |
| SpamapS | mordred: why would Zuul have to execute that and not the test node? | 23:19 |
| mordred | SpamapS: zuul owns the test node | 23:19 |
| SpamapS | yyyyeah.. sorta. ;) | 23:20 |
| jeblair | SpamapS: why does zuul need to be involved then? if a zuul user wants to use a third-party credential store, what's zuul's involvement? | 23:20 |
| mordred | yah - that was what I was going to say | 23:20 |
| SpamapS | Unless it just plops an encrypted blob on the disk which is the stuff the user wants to retain control of. | 23:20 |
| SpamapS | jeblair: Yes I'm starting to ask myself that question too. | 23:20 |
| SpamapS | Perhaps the right thing is just to make it so Zuul can attach and detach nodes to / from a particular network | 23:21 |
| SpamapS | which it can probably already do | 23:21 |
| SpamapS | Zuul or Nodepool I guess | 23:21 |
| jeblair | SpamapS: yes, that's nodepool. | 23:21 |
| SpamapS | really | 23:22 |
| mordred | SpamapS: yah, I think you're descibing a thing which is a different thing from the zuul secrets system and which sounds more like some potentially interesting user content - or maybe an interesting part of the zuul stdlib - a pre-playbook which is "fetch blah from tang" or osmething | 23:22 |
| SpamapS | you can just make users establish VPN connections to their private Tang server. | 23:22 |
| mordred | yah | 23:23 |
| SpamapS | the thing I forgot is that the secrets system is _secrets for zuul_ not _secrets for nodes_ | 23:23 |
| jeblair | SpamapS: i'd caution about assuming openstack network-per-tenant for a zuul-as-a-service -- but that's something i'm happy to leave in your hands. :) | 23:23 |
| SpamapS | also I got excited because McCallum was receptive to having Tang rewritten in Rust. ;) | 23:24 |
| mordred | :) | 23:24 |
| mordred | NOW the source of SpamapS excitement surfaces | 23:24 |
| jeblair | SpamapS: it's secrets for nodes too | 23:24 |
| SpamapS | yep | 23:24 |
| SpamapS | I'm so smitten.. it's terrible ;) | 23:24 |
| SpamapS | jeblair: s/not/and/ .. good point | 23:24 |
| jeblair | SpamapS: we could really use more attention to the secrets stuff, so if you have time to look that over in the spec, that would be great. and if you see a way this impacts it, that would be great too. | 23:25 |
| SpamapS | I've read it twice today. :-D | 23:26 |
| jeblair | SpamapS: well, let me know if we left out anything, like what we're trying to accomplish or underlying reasoning. | 23:26 |
| SpamapS | I may edit it a little today and submit a patch to help remind readers what the use is. | 23:26 |
| SpamapS | Because the only mention I remember from those two meedings regarding what is done with the cleartext is pushing to the node's environment. | 23:27 |
| jeblair | SpamapS: and keep in mind that suporting zuul-as-a-service *is* a use case it was designed to address (specifically, users being able to safely put secrets in their hosted git repos) | 23:27 |
| SpamapS | s/meedings/readings/ | 23:27 |
| SpamapS | jeblair: right, the use case it's not going to tackle, and which is definitely much harder, is supporting doing that without knowing said secrets. :) | 23:28 |
| SpamapS | I'm also just kind of excited about the prospect of having a _usable_ key-escrow-like service. | 23:29 |
| jeblair | SpamapS: at a quick glance, it does look like a great system for that. :) | 23:29 |
| SpamapS | It's so tiny.. much eaier to lock it way way down and control access to it. | 23:29 |
| SpamapS | I find myself wondering if the Cinder folks could use it. | 23:29 |
| jeblair | SpamapS: when i think about issues like this though, i also think that at the end of the day, zuul is deciding what content to run on this node. i agree that in some environments, it might be useful to use a system like this. i think in many environments, it may not actually improve security. | 23:30 |
| Shrews | speaking of Rust... SpamapS: your recent blog post was both beautiful and disturbing | 23:31 |
| SpamapS | jeblair: there's a lot more to it than just key management, that's for sure. | 23:31 |
| SpamapS | Shrews: mission accomplished. | 23:31 |
| SpamapS | jeblair: like, because it's fun to think about it.. I think I could figure it out leveraging the secrets system from the v3 spec pretty simply.. | 23:33 |
| SpamapS | run a tang server behind a VPN | 23:33 |
| SpamapS | give VPN access creds to zuul secrets | 23:33 |
| SpamapS | have encrypted blob of stuff in the image build | 23:33 |
| SpamapS | zuul fires up node with mostly-useless image and launches it with VPN access creds | 23:34 |
| Shrews | also... If anyone wants to get their hands dirty in some nodepool v3 code, I have a minor-ish coding task that I don't have time to get to for a while | 23:34 |
| jeblair | SpamapS: (maximum utility here is when you are in an environment where you trust zuul to run code on a node which can expose credentials, but yet don't trust zuul with those credentials. for me, that's a fine line. :) | 23:34 |
| SpamapS | access creds grant Tang interaction, node decrypts blob, mounts as root FS, grants Zuul's SSH user limited privileges. | 23:35 |
| jeblair | SpamapS: yeah, i want zuul to be flexible enough to handle that kind of thing if someone wants to do it | 23:35 |
| jeblair | SpamapS: from what i've seen (limited!) i don't think it would work for openstack, and i'm not sure it should be our default recommendation for how people handle secrets. but i like the idea of having a system we can fundamentally build that on. | 23:36 |
| SpamapS | jeblair: yeah, the idea is just to be able to audit when and where keys are passed out. So yes, Zuul operators could then decrypt that key, and abuse the VPN access creds themselves to get to Tang. So there is further work to make sure that access is somehow locked down too and pretty soon you go "oh hell, just put the key in zuul" | 23:36 |
| SpamapS | and since Zuul is already expecting to have a key store.. it damn well better be secured and locked down. | 23:37 |
| SpamapS | jeblair: yeah I wouldn't even consider it for OpenStack's use cases. | 23:38 |
| SpamapS | I am more thinking about deployment cases. | 23:39 |
| * SpamapS has done enough distracting for one day | 23:39 | |
| *** Cibo_ has joined #zuul | 23:50 | |
| *** jamielennox is now known as jamielennox|away | 23:58 | |
| *** jamielennox|away is now known as jamielennox | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!