| *** jamielennox is now known as jamielennox|away | 00:20 | |
| *** jamielennox|away is now known as jamielennox | 00:27 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Add a failure message to zookeeper https://review.openstack.org/444673 | 00:40 |
|---|---|---|
| mordred | Shrews: you're so close on all the ectomies! | 03:14 |
| mordred | Shrews: +2 on the stack | 03:31 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Configure regional mirrors for our jobs https://review.openstack.org/439152 | 06:05 |
| *** bhavik1 has joined #zuul | 06:07 | |
| *** isaacb has joined #zuul | 07:24 | |
| *** Cibo_ has joined #zuul | 07:31 | |
| *** Cibo_ has quit IRC | 08:00 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** hashar has joined #zuul | 08:38 | |
| *** hashar has quit IRC | 08:58 | |
| *** bhavik1 has quit IRC | 09:00 | |
| *** hashar has joined #zuul | 09:08 | |
| *** bhavik1 has joined #zuul | 09:32 | |
| *** isaacb has quit IRC | 09:38 | |
| *** isaacb has joined #zuul | 09:48 | |
| *** hashar has quit IRC | 09:58 | |
| *** hashar has joined #zuul | 10:01 | |
| *** bhavik1 has quit IRC | 10:04 | |
| *** isaacb has quit IRC | 11:36 | |
| *** isaacb has joined #zuul | 11:44 | |
| *** Cibo_ has joined #zuul | 12:50 | |
| Shrews | pabelanger: oops, looks like i've duplicated your 436027 change in 444647 during my hackfest yesterday | 13:05 |
| *** Cibo_ has quit IRC | 13:22 | |
| *** hashar is now known as hasharfood | 13:27 | |
| *** isaacb has quit IRC | 13:47 | |
| *** isaacb has joined #zuul | 14:03 | |
| *** openstackgerrit has joined #zuul | 14:06 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove MySQL https://review.openstack.org/444919 | 14:06 |
| Shrews | mordred: that ^^^ was the best ectomy | 14:06 |
| *** hasharfood is now known as hashar | 14:16 | |
| Shrews | SpamapS: Who is Cullen Taylor and have they started on the nodepool side of https://storyboard.openstack.org/#!/story/2000897 ? If not, I'd like to knock that one out today since it completes what is needed from nodepool. | 14:17 |
| Shrews | eggshell: hi! see above :) | 14:19 |
| *** isaacb has quit IRC | 14:22 | |
| pabelanger | Shrews: np, I can abandon | 14:35 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove MySQL https://review.openstack.org/444919 | 14:56 |
| *** bhavik1 has joined #zuul | 15:11 | |
| *** jianghuaw has quit IRC | 15:14 | |
| pabelanger | clarkb: jeblair: mordred: https://review.openstack.org/#/c/438281/ seems to be the preferred way of defining out generic tox playbooks. Is that something we want to land today? This will allow me to keep iterating forward on establishing out base jobs for zuul in zuulv3-dev.o.o | 15:32 |
| jeblair | yep | 15:33 |
| pabelanger | danke | 15:34 |
| pabelanger | jeblair: also, are we at a point we can restart nl01.o.o? We have a few jobs stuck in queue on zuulv3-dev.o.o because lack of nodes | 15:35 |
| pabelanger | this is related to the locked ready nodes from last week | 15:36 |
| jeblair | pabelanger: we want to land https://review.openstack.org/444462 first | 15:37 |
| jeblair | pabelanger: i'll high-priority review its parents | 15:38 |
| pabelanger | great, thanks | 15:38 |
| jeblair | Shrews, mordred, pabelanger: in my reading of https://review.openstack.org/410812 it looks like we don't "consume both forms" as the commit message states. i'm fine with that, but wanted to double check the discrepancy... | 15:41 |
| jeblair | i think maybe that's left over from the version of the change to master? | 15:42 |
| pabelanger | Shrews: left a comment on 444919, curious what could use secure.conf in the future | 15:42 |
| Shrews | pabelanger: zookeeper creds | 15:42 |
| pabelanger | ya, that was I suspected | 15:42 |
| pabelanger | k | 15:42 |
| Shrews | pabelanger: jeblair: we also want to land the leaked node fix before restarting nl01 | 15:43 |
| pabelanger | Shrews: you have a pep8 failure, why I didn't +2 | 15:43 |
| pabelanger | Shrews: ++ | 15:43 |
| Shrews | pabelanger: yeah, adding 2 reviews before that one to fix the pep8 | 15:43 |
| jeblair | mordred: can you take a look at 442114 and 442124 please? | 15:53 |
| *** Cibo_ has joined #zuul | 15:55 | |
| clarkb | jeblair: mordred re consume both forms, we may want that in v3 in order to transition cleanly without deleting all nodes and rebuilding them all? | 16:00 |
| jeblair | clarkb: well, everything else needs to change in v3, so i'm not too worried about that changing too. i *do* think it's required if we land the equivalent patch in v0. | 16:01 |
| jeblair | Shrews, mordred: ^ but i would still like to clarify the intent since the patch doesn't seem to do what the commit message says. :) | 16:04 |
| Shrews | we sort of HAVE to delete all nodes for v3 because otherwise we have no record of them in zk. so both forms not required for v3 | 16:07 |
| clarkb | Shrews: oh right db change anyways | 16:07 |
| Shrews | but for v0, yeah, probably need both | 16:08 |
| jeblair | clarkb, fungi: who should we get to weigh in on https://review.openstack.org/443985 ? | 16:09 |
| clarkb | I'm trying to remembre who really wanted that feature. I think amrith with trove jobs, but there were others I am not remembering right now | 16:10 |
| fungi | right, amrith is the main one coming to mind since he approached us at the ptg about it | 16:11 |
| jeblair | yes, i think most requestors have been external to openstack | 16:11 |
| fungi | and in his case, it was really more of a "oh, jobs support dependencies? well i have these three jobs and when they succeed i want to run these two other jobs..." | 16:12 |
| pabelanger | our wheel build jobs could use it | 16:13 |
| pabelanger | since we only have to afs release one time, but build multiple things | 16:14 |
| pabelanger | but, we've since removed those jobs | 16:14 |
| *** Cibo_ has quit IRC | 16:15 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove MySQL https://review.openstack.org/444919 | 16:20 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove test_*_cleanup_on_start tests https://review.openstack.org/444973 | 16:20 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable test_node_delete_failure https://review.openstack.org/444974 | 16:20 |
| Shrews | pabelanger: those two new reviews should fix the pep8 fail in the mysql one | 16:20 |
| Shrews | and that's ALL of the nodepool tests enabled \o/ | 16:20 |
| jeblair | SpamapS: 444495 security spec looks great! mordred, clarkb, fungi: <-- you may be interested in that. | 16:25 |
| fungi | thanks! | 16:26 |
| pabelanger | Shrews: awesome | 16:27 |
| jeblair | pabelanger, dmsimard: commented on 444088 | 16:43 |
| dmsimard | jeblair: will look ty | 16:44 |
| Shrews | ugh, just realized that the zuul meeting is now 6pm my time | 16:46 |
| jeblair | i'm open to rescheduling it. | 16:47 |
| SpamapS | jeblair: glad you like it. I'm hoping to add some more details on a back-of-napkin plan for how to add some abstraction so we don't get too married to whatever we choose to use. | 16:53 |
| *** bhavik1 has quit IRC | 16:55 | |
| clarkb | ok hav ereviewed security spec | 16:56 |
| jeblair | i set the topic of all the zuulv3-related specs to zuulv3 so they should show up in people's dashboards | 17:03 |
| clarkb | jeblair: I commented on https://review.openstack.org/#/c/443985/1 too | 17:03 |
| mordred | jeblair: the commit message on commit-both-forms is just out of date | 17:03 |
| mordred | jeblair: I can update it if you like? | 17:04 |
| jeblair | mordred: i'm okay just landing it; i think we've had the necessary productive discussion :) | 17:04 |
| * SpamapS is terrible at using gerrit topics | 17:04 | |
| jeblair | mordred: (it's at the bottom of a pile; i don't think it's worth the churn) | 17:05 |
| jeblair | clarkb: thanks, will propose that in a followup | 17:06 |
| * Shrews apologizes for creating the pile | 17:07 | |
| mordred | jeblair: yah - I didn't want to upset thepile | 17:07 |
| jeblair | Shrews: never apologize for writing patches! :) | 17:07 |
| jeblair | okay, maybe not 'never'.... :) | 17:08 |
| jeblair | clarkb, pabelanger: https://review.openstack.org/444974 is the last change in the nodepool stack without a +W | 17:10 |
| *** hashar has quit IRC | 17:14 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Stop json-encoding the nodepool metadata https://review.openstack.org/410812 | 17:14 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Use node ID for instance leak detection https://review.openstack.org/444508 | 17:15 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Fix failure of node assignment at quota https://review.openstack.org/444462 | 17:15 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Store a pointer to the paused node request handler https://review.openstack.org/444520 | 17:15 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Re-enable test_disabled_label https://review.openstack.org/444646 | 17:15 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Re-enable test_node_az https://review.openstack.org/444647 | 17:15 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Fix provider-label association https://review.openstack.org/444650 | 17:16 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Re-enable test_node_ipv6 https://review.openstack.org/444651 | 17:16 |
| mordred | so much patches | 17:16 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove test_nodepool.test_job_* tests https://review.openstack.org/444652 | 17:16 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove Jenkins https://review.openstack.org/444654 | 17:16 |
| mordred | that last patch is fun :) | 17:16 |
| *** openstackgerrit has quit IRC | 17:18 | |
| clarkb | jeblair: Shrews in https://review.openstack.org/#/c/444974/1/nodepool/nodepool.py why is the unlock happening before the delete? Shouldn't you hold the lock while "writing"? | 17:18 |
| Shrews | clarkb: the lock is part of the root node path, so if you delete first, you delete the lock | 17:19 |
| clarkb | Shrews: wouldn't that be more "correct" | 17:20 |
| Shrews | clarkb: the "unlock" would throw an exception | 17:20 |
| clarkb | Shrews: right don't unlock if the previous statement is effectively an unlock | 17:20 |
| clarkb | but let the unlock happen as part of the delete to reduce the race (and perhaps remove it?) | 17:21 |
| Shrews | clarkb: it's actually relatively safe, even with the race. | 17:23 |
| Shrews | it's a pattern we use in the builder, too, iirc. Might be worth investigating what happens in your scenario and handling that more gracefully throughout, as an improvement. | 17:25 |
| jeblair | Shrews: i forget, can we issue a recursive delete? or do we have to delete the children first? | 17:26 |
| Shrews | recursive=True | 17:26 |
| Shrews | to the delete call | 17:26 |
| jeblair | Shrews: hrm, then it seems like that should be safe in this case? that may not hold for the cases in the builder, but maybe this is worth a try? | 17:27 |
| clarkb | I left a comment for historical purposes | 17:27 |
| jeblair | (i say it's safe because we know there aren't going to be any other children showing up under this node) | 17:27 |
| jeblair | clarkb: i added https://review.openstack.org/445022 to implement your doc suggestion | 17:31 |
| jeblair | oh tab | 17:31 |
| Shrews | heh, looks like the unlock wouldn't actually throw an exception | 17:36 |
| clarkb | Shrews: new bug? | 17:40 |
| Shrews | clarkb: no. happy feature? | 17:40 |
| *** Cibo_ has joined #zuul | 17:44 | |
| *** openstackgerrit has joined #zuul | 17:46 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove MySQL https://review.openstack.org/444919 | 17:46 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Re-enable test_node_delete_failure https://review.openstack.org/444974 | 17:46 |
| harlowja | jeblair if u get a sec https://github.com/python-zk/kazoo/pull/420/ | 17:47 |
| harlowja | that should fix the gevent stuff | 17:47 |
| Shrews | clarkb: there ya go ^^^ | 17:47 |
| clarkb | hanks | 17:47 |
| clarkb | *thanks | 17:48 |
| jeblair | harlowja: yay! what should i do? | 17:49 |
| harlowja | merge it, lol | 17:49 |
| jeblair | harlowja: git-merge it into my pr? | 17:49 |
| harlowja | actually i'm not sure where the rest of the kazoo folks are :-P | 17:49 |
| harlowja | maybe its just me left, lol | 17:50 |
| harlowja | jeblair nah, i'll just merge the fix | 17:50 |
| harlowja | then recheck yours | 17:50 |
| harlowja | i think that should work | 17:50 |
| jeblair | harlowja: cool | 17:50 |
| jeblair | harlowja: how do you tell travis to recheck? | 17:51 |
| harlowja | magic recheck button that i think only i have | 17:51 |
| harlowja | lol | 17:52 |
| mordred | harlowja: I always say that decisions are made by those who show up - you exist, therefor you are the kazoo developers | 17:52 |
| harlowja | :( | 17:52 |
| harlowja | lol | 17:52 |
| harlowja | jeblair http://imgur.com/a/jP83z (i get a restart job button) | 17:52 |
| harlowja | afaik that's admin to the repo only or something | 17:53 |
| jeblair | harlowja: ah thanks | 17:53 |
| harlowja | u may have to rebase, not sure, let's see | 17:53 |
| *** Cibo_ has quit IRC | 17:53 | |
| *** jamielennox is now known as jamielennox|away | 17:53 | |
| clarkb | that makes me feel better about "recheck" in general :P | 17:54 |
| harlowja | :-P | 17:54 |
| harlowja | jenkins also has a simlar button | 17:54 |
| harlowja | lol | 17:54 |
| jeblair | yeah, if travis still just checks out my pr branch tip (vs merging it into the target branch), then i think it will still fail. in that case, i guess i would update my pr with a merge commit from master. | 17:55 |
| harlowja | let's see, ha | 17:55 |
| jeblair | but maybe they merge into the target branch for tests now. i dunno. this is a nice test case. :) | 17:56 |
| mordred | jeblair: you can also rebase and force push and the PR will update | 17:56 |
| jeblair | mordred: will github people yell at me for that? | 17:56 |
| harlowja | mordred is SpamapS now with u at redhat? | 17:56 |
| harlowja | i can't keep track anymore | 17:56 |
| harlowja | lol | 17:56 |
| mordred | nope. it's required for some projects | 17:57 |
| mordred | jeblair: ^^ | 17:57 |
| clarkb | mordred: jeblair the problem with that is it breaks reviews iirc | 17:58 |
| SpamapS | I'm still blue | 17:58 |
| clarkb | so if you have review history that is relevant and important going forward you want to be careful doing that | 17:58 |
| SpamapS | well I mean, I was light blue | 17:58 |
| harlowja | lol | 17:59 |
| SpamapS | and then for like, 5 minutes, green rectangle | 17:59 |
| SpamapS | but now deep blue | 17:59 |
| clarkb | SpamapS: golang-client is older than gophercloud fwiw | 17:59 |
| mordred | clarkb: yah - if there are inline reviews | 17:59 |
| *** Cibo_ has joined #zuul | 17:59 | |
| SpamapS | clarkb: older < adopted | 17:59 |
| clarkb | SpamapS: sure, but your criticism goes both ways | 17:59 |
| mordred | clarkb: older than the rax version? or the non-rax version? | 17:59 |
| clarkb | SpamapS: why didn't gophercloud devs work with existing community instead? | 17:59 |
| clarkb | etc etc | 17:59 |
| clarkb | mordred: whatever is in github | 18:00 |
| clarkb | initial commit is 2 weeks newer than golang-client | 18:00 |
| SpamapS | clarkb: I may be totally insane on this, but it's a REST API client lib, and we shouldn't expect people to come into OpenStack just to make convenience libs for our REST API's. Or maybe we should. #JustSayin | 18:00 |
| mordred | yah | 18:01 |
| clarkb | SpamapS: I don't think they need to, but they litereally did they thing you are complaining about | 18:01 |
| clarkb | and got ahead doing it and now you want people to use that | 18:01 |
| clarkb | tldr collaborating is hard | 18:01 |
| SpamapS | I don't know who the they's are | 18:02 |
| mordred | well - it was ORIGINALLY a rackspace cloud client not a general openstack client | 18:02 |
| mordred | it grew into a general openstack client | 18:02 |
| clarkb | SpamapS: gophercloud neglected an existing project | 18:02 |
| clarkb | SpamapS: so rathre than engage existing community and tooling they went and made their own thing (this apperas to be your complaint today) | 18:02 |
| mordred | when gophercloud started, their intent was not being an openstack client library | 18:02 |
| mordred | when it started, it was very specifically being a rackspace cloud client - completely with built-in support for the silly rackspace api-key thing | 18:02 |
| SpamapS | I don't want to punish people for not doing the things we want them to do.. I want us to look at the situation on the ground today and make good choices about how to help OpenStack adoption in that context. | 18:03 |
| mordred | so it would have been inappropriate, at that time, in openstack | 18:03 |
| clarkb | mordred: oh interesting that makes the whole thing even more fun | 18:03 |
| mordred | yah - today gophercloud has its own non-rax org and they stripped the rax specific stuff | 18:03 |
| clarkb | SpamapS: I'm not suggesting we punish people. I'm just pointing out collaborating is hard | 18:03 |
| clarkb | SpamapS: and its unfair today IMO to complain at dims when gopherlcoud did/does literally the same thing | 18:04 |
| clarkb | instead the focus should be how to collaborate better | 18:04 |
| mordred | it is now quite intentionally a general openstack lib - and I believe they would be happy accepting a patch to add clouds.yaml support and whatnot | 18:04 |
| clarkb | mordred: cool | 18:04 |
| SpamapS | clarkb: agreed! And IMO, we have failed at it again by resurrecting a library that competes with the library the small OpenStack-using Go community has adopted as the one to use. | 18:04 |
| harlowja | punish all the people | 18:04 |
| clarkb | SpamapS: well we haven't really resurrected it | 18:04 |
| clarkb | SpamapS: its always been there an dalways worked aiui | 18:04 |
| clarkb | SpamapS: its just had less attention | 18:04 |
| SpamapS | that's fair | 18:05 |
| mordred | yah - that _does_ highlight my frustration with rax behavior in this regard over the years - focusing on "Rackspace Cloud" and sending devs to go add support to ecosystem things for "Rackspace Cloud" and not for "OpenStack" with docs mentioning that rackspace is an OpenStack public cloud so works | 18:06 |
| harlowja | mordred deep breath | 18:06 |
| mordred | but -that's an annoyance at rax business leader folks, not the devs | 18:06 |
| harlowja | how is rackspace and openstack working out anyway, i didn't hear so well :-/ | 18:08 |
| clarkb | harlowja: oh while I am nitpicking email threads, there are pletnly of openstack public clouds out there today :P | 18:08 |
| harlowja | 1 billion clouds | 18:09 |
| harlowja | lol | 18:09 |
| SpamapS | berzillion is the technical term | 18:10 |
| mordred | clarkb: did harlowja say something about there not being? | 18:13 |
| clarkb | mordred: yes he said openstack is dead in public cloud sso we should give up | 18:13 |
| harlowja | lol | 18:13 |
| clarkb | and use cloudstack and euca? | 18:13 |
| harlowja | euca only | 18:13 |
| harlowja | lol | 18:13 |
| * harlowja didn't think rax was doing so well (at least from what i hear from the back channels with projects that move things to AWS and such) | 18:14 | |
| SpamapS | being acquired by private equity isn't usually a good isgn. | 18:15 |
| mordred | harlowja: jesus man | 18:15 |
| SpamapS | but sometimes those things get chopped up and some of it survives. We don't know really. | 18:16 |
| dmsimard | jeblair: so you know, I think that sort of makes sense -- maybe we should redraft the spec to be a bit more generic, something along the lines of providing users an easy way to configure callbacks ? | 18:16 |
| clarkb | harlowja: I was thinking of ovh and internap and citycloud and datacentred and vexxhost and entercloud and everyone else at https://docs.openstack.org/developer/os-client-config/vendor-support.html | 18:16 |
| Shrews | eggshell: around to discuss https://storyboard.openstack.org/#!/story/2000897 ? | 18:16 |
| mordred | harlowja: you're as bad at the corrupt silicon valley tech press - OpenStakc is doing very well in public cloud | 18:16 |
| mordred | harlowja: unless you have a completely US-centric world view driven only by VC exits | 18:16 |
| SpamapS | clarkb: https://www.openstack.org/marketplace/public-clouds/ is another good list. | 18:16 |
| dmsimard | jeblair: i.e, I'm definitely okay with ARA being sort of a "JJB builder" where it'd install ara and configure the callback | 18:16 |
| * mordred throws wet cat at harlowja | 18:16 | |
| harlowja | lol | 18:17 |
| SpamapS | Though the o-c-c one is more useful :) | 18:17 |
| eggshell | Shrews: sure. Haven't had the chance to dive to deep into it yet. | 18:17 |
| dmsimard | jeblair: versus what would end up being a tight coupling -- it's okay to keep zuul as lean and as simple as possible | 18:17 |
| SpamapS | harlowja: what's dead is speculation. :) | 18:17 |
| Shrews | eggshell: mind if i code the nodepool part of that then? I'd like to put that up today if possible. | 18:17 |
| Shrews | eggshell: i'll leave you the zuul part :) | 18:17 |
| SpamapS | the speculation was that if you just setup an API for running IaaS, the dumptrucks of money would divert into your datacenter instead of AWS/GCE/Azure's | 18:17 |
| * SpamapS will take this discussion to his inside-voice now though | 18:18 | |
| eggshell | Shrews: go ahead :) | 18:18 |
| harlowja | SpamapS ya, fair point | 18:18 |
| Shrews | eggshell: great, thx | 18:19 |
| harlowja | clarkb how many of those clouds are contributing to openstack (the codebase/s), any idea? | 18:20 |
| dmsimard | mordred: TBH (disclaimer, $oldjob=Internap) public cloud with real customers on OpenStack is kind of hard | 18:20 |
| clarkb | harlowja: a good chunk of them contribute the test resources we run our testing on | 18:21 |
| clarkb | harlowja: and ovh does a lot of work with swift aiui | 18:21 |
| mordred | dmsimard: internap made a bunch of left-field choices in its deployments | 18:21 |
| rbergeron | oh boy humans talking lots | 18:21 |
| harlowja | lol | 18:21 |
| dmsimard | The market is saturated -- if they're not in AWS/GCE/Azure, what they're looking for is probably just cheap VPSes and it's a race to the bottom for that | 18:21 |
| clarkb | harlowja: apparently they have been doing the EC work recently? something like that | 18:21 |
| mordred | dmsimard: again - that assumes the market is US | 18:22 |
| dmsimard | mordred: I know all about the deployment choices, I was a part of them :D | 18:22 |
| mordred | dmsimard: there are more countries than the US | 18:22 |
| mordred | dmsimard: :) | 18:22 |
| harlowja | nooooooo | 18:22 |
| harlowja | there is only US | 18:22 |
| harlowja | lol | 18:22 |
| mordred | there might even be places where they do not want to put their data into the control of a company based in Seattle | 18:22 |
| mordred | even if that company has sa datacenter in their country | 18:22 |
| harlowja | guess i should move to europe based on https://www.openstack.org/marketplace/public-clouds/ | 18:23 |
| dmsimard | right, we had some customers reaching out to us just because of the Canada datacenter locations | 18:23 |
| mordred | now - if literally ANY of the US OpenStack cloud providers had worked _together_ instead of competing with eachother | 18:23 |
| dmsimard | mordred: the inter-cloud story is bad | 18:23 |
| mordred | the US story also might be better | 18:23 |
| dmsimard | like federation or that cisco intercloud thing | 18:23 |
| mordred | dmsimard: you don't need those things | 18:24 |
| harlowja | mordred isn't tha a tough sell when u are racing to the bottom | 18:24 |
| harlowja | *that | 18:24 |
| mordred | harlowja: what, working with each other? | 18:24 |
| harlowja | ya | 18:24 |
| mordred | that's literally the ONLY WAY ANONE WAS EVER GOING TO BE SUCCESSFUL | 18:24 |
| dmsimard | Yeah .. interop is a must | 18:25 |
| harlowja | i get that point, what was missing in the equation to make that happen? | 18:25 |
| harlowja | (in your view) | 18:25 |
| mordred | lack of arrogance | 18:25 |
| jeblair | oh, hi | 18:25 |
| jeblair | can we take the openstack pontificating to another channel? | 18:25 |
| harlowja | lol | 18:25 |
| * clarkb apologizes for pushing the snowball down the hill | 18:26 | |
| harlowja | :) | 18:26 |
| * mordred blames clarkb | 18:26 | |
| clarkb | and points at #openstack-dev as good alternate venue | 18:26 |
| * mordred blames himself for being an easy troll target | 18:26 | |
| harlowja | :) | 18:28 |
| jeblair | dmsimard: cool -- i think it's okay to have the spec call out ara as a first-class use-case still. i think a goal here is "make it easy to use ara". | 18:29 |
| dmsimard | jeblair: I'm okay with that | 18:29 |
| pabelanger | are we ready to restart nl01.o.o? I see we've merged some nodepool changes | 18:29 |
| jeblair | pabelanger: yes | 18:30 |
| jeblair | harlowja: 419 checks passed | 18:31 |
| pabelanger | jeblair: great, did you want to do it, or should I? | 18:33 |
| jeblair | pabelanger: can you please? | 18:34 |
| pabelanger | restarted | 18:36 |
| pabelanger | jeblair: should I expect zuul start clearing out is pipeline or should I do something else? EG: manually delete ready nodes | 18:38 |
| *** bhavik1 has joined #zuul | 18:40 | |
| dmsimard | jeblair: fyi I'm a bit all over the place right now but I'll rework the draft sometime this week if I have a chance | 18:44 |
| *** bhavik1 has quit IRC | 18:50 | |
| jeblair | pabelanger: should be automatic; if not, we may have more bugs | 18:57 |
| jeblair | pabelanger, Shrews: i think we should add a 'nodepool request-list' command | 18:57 |
| pabelanger | jeblair: okay, I think we have more bugs | 18:58 |
| jeblair | it looks like the launcher is throwing some exceptions | 18:58 |
| jeblair | and it seems stuck again | 18:59 |
| SpamapS | clarkb: jeblair do you two think it might be a good idea to have a separate spec for disk space monitoring, or just flesh out the disk space bits a bit more in the launcher security spec? | 19:00 |
| SpamapS | I'm leaning toward the latter | 19:01 |
| clarkb | SpamapS: depending on the preferred security tooling they could be tightly coupled | 19:01 |
| clarkb | eg docker with its fs mounting | 19:02 |
| SpamapS | docker does overlayfs by default IIRC | 19:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Record SSH public keys for new nodes in ZK https://review.openstack.org/445055 | 19:03 |
| Shrews | pabelanger: zuul launcher or nodepool launcher? | 19:04 |
| SpamapS | ah no, aufs is default.. but the likely have mostly the same features | 19:04 |
| pabelanger | Shrews: both are not doing anything currently. I believe the issue is on the nodepool side, but jeblair might have a better handle on it | 19:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Record SSH public keys for new nodes in ZK https://review.openstack.org/445055 | 19:06 |
| clarkb | SpamapS: in new docker its btrfs I think because aufs is going by wayside in kernel? something like that | 19:07 |
| clarkb | SpamapS: I remember kolla wanting xenial test nodes for this | 19:07 |
| Shrews | pabelanger: jeblair: based on the np exceptions, i'd say the request is still locked somewhere | 19:09 |
| SpamapS | clarkb: looks like most people actually end up using overlay2 | 19:09 |
| SpamapS | because aufs has been gone from ubuntu kernels since 14.04 | 19:09 |
| SpamapS | or could be 16.04 | 19:09 |
| jeblair | SpamapS: re spec: let's keep them together; the du thread idea is so dirt-simple we might go ahead and implement it before the spec lands, but if we keep it around it should co-evolve with the rest of what's in there; and if it's not necessary, we can remove it. | 19:11 |
| Shrews | though it seems to have cleared up (the locking) | 19:13 |
| SpamapS | clarkb: I hadn't really looked closely at the docker storage drivers.. but honestly.. they have a lot of what we want built in. Hm. | 19:13 |
| SpamapS | especially the device-mapper one | 19:14 |
| pabelanger | Shrews: left a question on 445055 | 19:15 |
| Shrews | pabelanger: not sure how to answer it. i sort of copied what the zuul code was doing. is there an advantage of using paramiko over ssh-keyscan? | 19:16 |
| Shrews | and why the heck is https://review.openstack.org/444973 not gating? | 19:16 |
| Shrews | pabelanger: oh, for the dependency | 19:18 |
| clarkb | Shrews: because when you already have a verified +1 and zuul revotes verified +1 that second +1 isn't actually emmitted in the event | 19:18 |
| clarkb | Shrews: so zuul doesn't see the change as gateable. You can reapprove | 19:18 |
| pabelanger | Shrews: operating system dependency vs python dependency? About all I can think of | 19:18 |
| pabelanger | Shrews: either works for me honestly | 19:19 |
| Shrews | clarkb: neat | 19:19 |
| SpamapS | oh hm | 19:19 |
| Shrews | pabelanger: i'll go with whatever jeblair suggests :) | 19:19 |
| SpamapS | docker's device-mapper driver actually _does not_ do what we want. | 19:19 |
| SpamapS | It has one pool, for all the containers' write space | 19:19 |
| pabelanger | Shrews: WFM! | 19:19 |
| SpamapS | clarkb: ^ | 19:19 |
| Shrews | pabelanger: he originally suggested the zuul code, so ... :) | 19:20 |
| SpamapS | so one container can fill the pool | 19:20 |
| pabelanger | ++ | 19:20 |
| Shrews | pabelanger: can i get your eyes on https://review.openstack.org/444974 ? | 19:21 |
| SpamapS | and same for overlay | 19:21 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove test_*_cleanup_on_start tests https://review.openstack.org/444973 | 19:22 |
| pabelanger | Shrews: +3 | 19:25 |
| Shrews | \o/ | 19:25 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Re-enable test_node_delete_failure https://review.openstack.org/444974 | 19:31 |
| Shrews | wait for it... | 19:31 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove MySQL https://review.openstack.org/444919 | 19:31 |
| Shrews | mmmmmm.... that merge makes me tingley all over | 19:31 |
| pabelanger | \o/ | 19:31 |
| * Shrews takes a smoke break | 19:32 | |
| pabelanger | SpamapS: are you a DM or DD for debian? | 19:32 |
| SpamapS | pabelanger: DD, need something sponsored? | 19:34 |
| SpamapS | (We should ITP zuul ;) | 19:34 |
| pabelanger | SpamapS: actually! I have some old packaging for nodepool / zuul kicking around, now that we are close to zuulv3, was going to push that into debian. But ya, need sponsored :) | 19:35 |
| pabelanger | won't happen today, but might send some packages your way | 19:35 |
| clarkb | SpamapS: re docker that seems poorly designed | 19:35 |
| clarkb | SpamapS: but guessing much eaiser to implement | 19:35 |
| jeblair | clarkb, fungi, mordred, SpamapS: when you have a moment, thoughts on the question in https://review.openstack.org/445055 would be appreciated. | 19:36 |
| clarkb | wow I derped that response | 19:38 |
| clarkb | it quoted the top level comment and not any of hte inline comments, I fail | 19:38 |
| clarkb | but I like using paramiko there | 19:38 |
| fungi | jeblair: the question being whether to also use paramiko for keyscan instead of ssh subprocess? | 19:38 |
| jeblair | fungi: yes. i would strike 'also' from that though, since i expect us to remove all other uses of ssh from nodepool (ready scripts should move to zuul pre-playbooks) | 19:39 |
| jeblair | fungi: so basically, we have a dependency either on paramiko or ssh-keyscan for a pretty simple task. | 19:39 |
| fungi | oh, i see... | 19:39 |
| fungi | so i guess it boils down to whether it continues to carry a dependency on paramiko or on command-line ssh client | 19:40 |
| jeblair | ya | 19:40 |
| jeblair | i mean, the client is *probably* already there. almost certainly. | 19:40 |
| jeblair | but still | 19:41 |
| fungi | the latter is almost guaranteed to be present, so it's really a choice between a subprocess call or an additional python dep for one single function | 19:41 |
| clarkb | fungi: jeblair well zuul on windows or zuul in containers likely won't have an openssh toolchain | 19:41 |
| clarkb | (there are other barriers to zuul on windows, but zuul in containers is doable today) | 19:41 |
| pabelanger | paramiko means pip install nodepool would just work out of box, single command | 19:41 |
| fungi | yep | 19:42 |
| jeblair | i don't think we have made any promises regarding the first thing. the second, otoh, is good to keep in mind. | 19:42 |
| fungi | paramiko _does_ claim windows support too, for what that's worth https://github.com/paramiko/paramiko/blob/master/setup.py#L59 | 19:43 |
| fungi | hrm, though it does itself depend on cryptography and pyasn1 | 19:45 |
| Shrews | I'm about to put up the paramiko alternative, fwiw | 19:45 |
| clarkb | fungi: for reading key files I think | 19:46 |
| fungi | pyasn1 is less of a concern, cryptography is not pure-python (which makes paramiko impure as well), but if we're already using cryptography for something else then that's nbd | 19:46 |
| fungi | jeblair: rcarrillocruz: we were talking about using the pyca/cryptography library for encrypting secrets with zuul though, right? if so, we're eating that cost in part of the toolchain anyway | 19:49 |
| jeblair | fungi: yeah, i think that's still the plan there | 19:51 |
| Shrews | jeblair: pabelanger: paramiko seems to return only the "ssh-rsa" key, where as ssh-keyscan returns other types (at least when i test against localhost). Is that ok? | 19:55 |
| pabelanger | Oh, interesting | 19:55 |
| pabelanger | I don't think we have rsa keys on our workers | 19:55 |
| pabelanger | let me check | 19:55 |
| clarkb | we do | 19:55 |
| pabelanger | cool | 19:55 |
| Shrews | get_remote_server_key() doesn't take any options to control that http://docs.paramiko.org/en/2.1/api/transport.html#paramiko.transport.Transport.get_remote_server_key | 19:56 |
| Shrews | so, not sure what the difference is there | 19:56 |
| *** Cibo_ has quit IRC | 19:56 | |
| clarkb | does paramiko support ecdsa at all? | 19:57 |
| Shrews | but at least locally, ssh-keyscan will return ssh-ed25519 and ecdsa-sha2-nistp256 | 19:57 |
| Shrews | clarkb: *shrug* | 19:57 |
| pabelanger | https://github.com/paramiko/paramiko/issues/794 | 19:58 |
| pabelanger | I think we'd hit that | 19:58 |
| pabelanger | so, looks like ecdsa is in the pipeline. I see some open pull requests for it | 20:00 |
| fungi | rsa is "good enough" here in my opinion, and if we want to be able to default to other host key types we can try to pitch in on paramiko fixes i guess | 20:00 |
| *** Cibo_ has joined #zuul | 20:02 | |
| jeblair | but is it the case that if a nodepool user didn't use rsa keys at all, we would fail to get a fingerprint? | 20:03 |
| SpamapS | clarkb: I was wrong actually | 20:04 |
| clarkb | jeblair: I think openssh at least will request the one it knows and warn about the others but not force you to type yes for them? | 20:04 |
| SpamapS | clarkb: they added a size option https://github.com/docker/docker/issues/3804 | 20:04 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Record SSH public keys for new nodes in ZK https://review.openstack.org/445055 | 20:05 |
| clarkb | I'd have to test that behavior I want to say it will work fine with openssh as long as remote host has rsa host key | 20:05 |
| Shrews | ^^^ paramiko version | 20:05 |
| fungi | jeblair: yeah, if they configured sshd not to have an rsa host key, this would probably break (though our previous use of paramiko for ready scripts would similarly have run afoul of that, so doesn't seem like much of a regression to me if any?) | 20:06 |
| jeblair | good point. also https://github.com/paramiko/paramiko/pull/911 looks promising | 20:07 |
| pabelanger | ya, I think these are know issues upstream (paramiko). Which is good to see | 20:08 |
| pabelanger | 911 is track for the 2.2 release too | 20:08 |
| Shrews | funny that 911 doesn't adjust the API for get_remote_server_key | 20:12 |
| Shrews | or add a new one | 20:13 |
| *** hashar has joined #zuul | 20:32 | |
| SpamapS | clarkb: I'm playing with docker storage drivers now... looks like not all of them take the size= opt | 20:35 |
| SpamapS | "For the overlay2 storage driver, the size option is only available if the backing fs is xfs and mounted with the pquota mount option. Under these conditions, user can pass any size less then the backing fs size." | 20:35 |
| SpamapS | mmmmm XFS | 20:36 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add request-list nodepool command https://review.openstack.org/445169 | 20:36 |
| Shrews | jeblair: ask and ye shall receive ^^^^ | 20:36 |
| Shrews | sometimes, at least | 20:37 |
| *** jamielennox|away is now known as jamielennox | 20:37 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove AllocatorTestCase and RoundRobinTestCase https://review.openstack.org/445175 | 20:45 |
| *** Cibo_ has quit IRC | 20:52 | |
| jhesketh | Morning | 20:57 |
| jeblair | Shrews: nice! | 20:57 |
| Shrews | i think the paramiko host keys change is not working. the dsvm jobs seem to be taking quite a while | 20:59 |
| jeblair | unpossible | 21:01 |
| Shrews | if the paramiko call doesn't return anything, we'd be stuck repeatedly building nodes | 21:02 |
| Shrews | which is what i suspect is happening | 21:02 |
| SpamapS | meeting? | 21:02 |
| jeblair | SpamapS: in one hour | 21:02 |
| SpamapS | OH | 21:03 |
| SpamapS | DST yay | 21:03 |
| jeblair | yeah, that :| | 21:03 |
| SpamapS | it's actually good because I was double booked last week | 21:04 |
| Shrews | jeblair: should we consider it an error if we can't get any host keys? that's what the code does now | 21:05 |
| jeblair | Shrews: i think so; it is in our env. if that's inconvenient for other users, a config option makes sense. | 21:06 |
| * Shrews impatiently waits for logs | 21:07 | |
| jeblair | this is where the 'finger random log i ask for' thing is really going to come in handy | 21:10 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Record SSH public keys for new nodes in ZK https://review.openstack.org/445055 | 21:11 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add request-list nodepool command https://review.openstack.org/445169 | 21:12 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove AllocatorTestCase and RoundRobinTestCase https://review.openstack.org/445175 | 21:12 |
| Shrews | silly "%s" with an int | 21:13 |
| jeblair | Shrews: ? | 21:14 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Record SSH public keys for new nodes in ZK https://review.openstack.org/445055 | 21:14 |
| Shrews | nm. silly , instead of % :) | 21:14 |
| jeblair | ah, that makes more sense :) | 21:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add request-list nodepool command https://review.openstack.org/445169 | 21:15 |
| jeblair | Shrews: one more time though | 21:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove AllocatorTestCase and RoundRobinTestCase https://review.openstack.org/445175 | 21:15 |
| jeblair | Shrews: inline comment | 21:15 |
| Shrews | gah | 21:15 |
| jeblair | quotes optional | 21:16 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Record SSH public keys for new nodes in ZK https://review.openstack.org/445055 | 21:16 |
| * Shrews cries | 21:16 | |
| jeblair | Shrews: you're getting very fast at that | 21:16 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add request-list nodepool command https://review.openstack.org/445169 | 21:16 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove AllocatorTestCase and RoundRobinTestCase https://review.openstack.org/445175 | 21:17 |
| Shrews | *phew* seems to be passing now | 21:53 |
| pabelanger | nice | 21:54 |
| jeblair | it's zuul meeting time in #openstack-meeting-alt | 22:00 |
| pabelanger | clarkb: fungi: want to review / approve 445055 ? Since you commented on it | 22:02 |
| Shrews | b 17 | 22:03 |
| *** jamielennox is now known as jamielennox|away | 22:03 | |
| Shrews | gah | 22:03 |
| fungi | pabelanger: maybe post-meeting | 22:03 |
| *** jamielennox|away is now known as jamielennox | 22:07 | |
| *** jamielennox has left #zuul | 22:39 | |
| *** jamielennox has joined #zuul | 22:39 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add generic tox job (multiple playbooks) https://review.openstack.org/438281 | 23:02 |
| pabelanger | \o/ | 23:03 |
| pabelanger | I'll rebase tomorrow the rest of the stack | 23:03 |
| *** hashar has quit IRC | 23:05 | |
| jhesketh | pabelanger: did you abandon the other approach? | 23:06 |
| Shrews | pabelanger: so, those instances without "nodepool_provider_name" in infracloud-chocolate (from the launcher log) will need to manually deleted because we never put that info into those instances | 23:07 |
| Shrews | pabelanger: they're leaked nodes (no ZK entry) | 23:07 |
| pabelanger | jhesketh: not yet | 23:08 |
| pabelanger | Shrews: actually! Those are nodes from nodepool.o.o (production) | 23:08 |
| pabelanger | so, things are working as expected | 23:08 |
| pabelanger | aside from the log spam | 23:08 |
| Shrews | pabelanger: oh, good | 23:08 |
| pabelanger | will loop back, need to help with family | 23:09 |
| Shrews | pabelanger: i think there's a problem with the pause-at-quota algorithm now. after 7pm here so i really need to go make dinner. can take a look tomorrow if we can wait that long | 23:09 |
| Shrews | jeblair: fyi ^^^ | 23:10 |
| * Shrews aways for food | 23:11 | |
| jlk | SpamapS: reading through the security spec, what about rkt? | 23:11 |
| jlk | doesn't even appear to require a daemon | 23:11 |
| jamielennox | jlk: daemonless is good, but i don't think which container format changes any of the actual security principles | 23:14 |
| jlk | right, I meant to address the complexity of Docker | 23:14 |
| SpamapS | jlk: I've heard of it, but don't really know how it works. | 23:15 |
| jlk | I'm reading up on it | 23:15 |
| jlk | seems to approach security differently | 23:15 |
| jlk | https://coreos.com/rkt/docs/latest/devel/architecture.html | 23:15 |
| SpamapS | and yes, daemonless and simple would be nice. | 23:16 |
| jamielennox | complexitiy of docker generally refers to networking though right? | 23:16 |
| SpamapS | no the whole thing is getting big | 23:16 |
| SpamapS | storage drivers | 23:17 |
| SpamapS | networks | 23:17 |
| SpamapS | dockerhub | 23:17 |
| jlk | https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html might be of help too | 23:17 |
| SpamapS | just lots of moving pieces to exploit | 23:17 |
| jamielennox | oh, right, i understand they've jammed a whole bunch of crap in there | 23:17 |
| jlk | it's multiple daemons now | 23:17 |
| jlk | docker daemon to take API calls, and containerd to actually run the containers | 23:17 |
| jamielennox | i was just thinking of it from a usability and config perspective | 23:17 |
| jamielennox | Shrews: any chance there is an upgrade doc for nodepool -> v3? | 23:19 |
| SpamapS | jlk: heh, I should probably also just mention systemd-nspawn | 23:20 |
| jlk | If we want to go really low level, runC is an option too | 23:21 |
| jlk | since we basically just want shell in a box | 23:21 |
| jamielennox | SpamapS: i konw you meant that as a joke, but for OS level containment/cgroups etc, systemd-nspawn is probably a reasonable choice | 23:22 |
| jamielennox | bikeshed = orange | 23:22 |
| clarkb | SpamapS: I was actually going to look into it and then probably suggest it (the systemd thing) | 23:22 |
| clarkb | SpamapS: since it does userspace containering now too iirc | 23:22 |
| jamielennox | SpamapS: embrace the systemd | 23:23 |
| fungi | Shrews: on 445055 i guess get_remote_server_key() returns a binary blob rather than the typical ascii formats ssh-keyscan emits? | 23:24 |
| clarkb | and appraently can be used without having systemd proper on the system? | 23:25 |
| SpamapS | The thing is | 23:26 |
| SpamapS | we need mixed readonly/immutable/writable dirs | 23:27 |
| SpamapS | jamielennox: NEVER | 23:27 |
| SpamapS | that's impossible | 23:27 |
| SpamapS | ;) | 23:27 |
| SpamapS | so the reason docker/lxc/bubblewrap are nice is they have the facilities to help manage the dirs underneath the playbook | 23:27 |
| SpamapS | becase you need git trees (readonly but dynamic), ansible+dependencies (readonly, immutable and reusable), and scratch space for transfers/artifacts/etc (writable) | 23:28 |
| clarkb | do we need to enforce that though assuming the container itself is divested fro mthe host sufficiently? | 23:29 |
| jlk | I was going to ask the same | 23:29 |
| fungi | attention citizen: trust the systemd; the systemd is your friend; your current clearance level is high programmer | 23:29 |
| SpamapS | you going to build a whole chroot for every playbook execution? | 23:29 |
| jlk | the "immutable" part is that you restart each time from the base container | 23:29 |
| clarkb | SpamapS: why not? its incredibly cheap with eg btrfs | 23:29 |
| jlk | rather than continuing to use a scratch space | 23:29 |
| jamielennox | SpamapS: yes, the container build part is basically instant | 23:29 |
| SpamapS | clarkb: yes, but that's what docker does for you already. | 23:29 |
| clarkb | SpamapS: nspawn does it too | 23:30 |
| SpamapS | the programming time and maintenance of chroot builders is not free | 23:30 |
| jlk | build the userland once, start it again and again and again | 23:30 |
| clarkb | SpamapS: if paired with btrfs | 23:30 |
| clarkb | but yes its non zero overhead that has to be paid somewhere | 23:30 |
| clarkb | fwiw I thought that at PTG we decided that the git repos would be in the scratch space | 23:30 |
| clarkb | and we wouldn't put effort into multiple classes of data there | 23:31 |
| fungi | s/btrfs/lvm2/ for tat matter | 23:31 |
| SpamapS | systemd-nspawn --template interesting | 23:31 |
| SpamapS | clarkb: Oh, I was not present for that but it's fine. | 23:31 |
| SpamapS | I'd prefer that they stay readonly but that's only for clean implementation.. if somebody wants to trash the git trees in trying to break out I don't care. | 23:32 |
| clarkb | fungi: well nspawn specifically supports btrfs snapshots with the template flag SpamapS noted above | 23:32 |
| SpamapS | my main point was just that they're not there to be written to | 23:32 |
| clarkb | so its a really cheap way to hvae a thing | 23:32 |
| fungi | clarkb: oh, interesting that systemd would tie a feature to one specific fs | 23:32 |
| jlk | in my head, seems "easiest" to let them run amok inside the container, and not worry about read-onlying things inside the container, Unless we need to bind mount something from the host in | 23:32 |
| SpamapS | clarkb: --image= can make use of lvm's | 23:33 |
| clarkb | SpamapS: ah cool | 23:33 |
| jamielennox | if executors are used once only i don't care if people trash their own git repos | 23:33 |
| clarkb | jamielennox: ya I think that was what we ended up deciding in the room | 23:34 |
| clarkb | basically the only thing that will suffer if you od that is you so meh | 23:34 |
| SpamapS | So yeah that's the other option... make zuul-executor a single-job thing | 23:34 |
| clarkb | :) | 23:34 |
| jamielennox | clarkb: not tied, it says it can use a regular directory as a template, but in which case you involve a full copy of the directory, if btrfs it does COW | 23:34 |
| jlk | pop an executor, clone the code from within it, run the ansible? | 23:34 |
| SpamapS | but making zuul-executor a single-use thing just punts the setup/cleaning to something else | 23:34 |
| SpamapS | (like kubernetes) | 23:34 |
| clarkb | jlk: right snapshots are supported if using btrfs | 23:35 |
| clarkb | er jamielennox ^ | 23:35 |
| jlk | well.. | 23:35 |
| clarkb | btrfs is not required | 23:35 |
| SpamapS | I keep hearing btrfs gets slower and slower the longer you use it. | 23:35 |
| jamielennox | strongly recommended | 23:36 |
| jlk | I thought more that zuul-executord would run on the host, and it would in turn launch/remove the container processes for doing the ansible work | 23:36 |
| SpamapS | jlk: that's how I think it should work yes | 23:36 |
| fungi | so in theory you could leverage lvm2 copy-on-write snapshots with --image | 23:36 |
| SpamapS | fungi: correct | 23:36 |
| clarkb | SpamapS: that was an issue back with cephfs' implementation but I want to say that they have sorted htat out and is a big reason why cephfs is no longer beta | 23:36 |
| SpamapS | clarkb: Oh they're back on btrfs? | 23:37 |
| pabelanger | jlk: yes, thats how I last heard it | 23:37 |
| SpamapS | They were giving up on it 2 years ago. | 23:37 |
| jlk | doesn't look like systemd-nspawn can limit cpu/memory, etc.. is that true? | 23:37 |
| pabelanger | but lots of things being discussed now | 23:37 |
| jamielennox | I would agree with a executord model that you request executors from | 23:37 |
| SpamapS | jlk: that would be weird because cgroups are about limiting cpu/memory :-P | 23:37 |
| jamielennox | jlk: it starts a systemd process so i would expect it to go through that | 23:37 |
| fungi | jlk: apply cgroups liberally and lather? | 23:37 |
| clarkb | SpamapS: I think their future is bluestore which is write to disk directly, but pretty sure there is intermediate use btrfs but maybe its xfs and I am just behind the times) | 23:37 |
| jlk | I was just reading https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html#rkt-vs-systemd-nspawn | 23:38 |
| jeblair | SpamapS, clarkb: (late response for something like 10 minutes ago: yes, the git repos are in the scratch area, and i believe that's the current intent. we can change it but i don't think we have to; they are definitely per-job and there's no compelling reason to make them readonly afaik. if zuul pushes merges, we'll have something else do that.) | 23:38 |
| Shrews | jamielennox: no | 23:38 |
| Shrews | fungi: yes | 23:38 |
| clarkb | jlk: you can limit cpu/memory etc with nspawn | 23:39 |
| clarkb | jlk: but you apply the cgroup rules post container start looks like | 23:39 |
| clarkb | jlk: via systemctl | 23:39 |
| fungi | Shrews: okay, cool (if somewhat strange). thanks! | 23:39 |
| jlk | clarkb: ah, eww. | 23:39 |
| jlk | or whateve.r | 23:39 |
| SpamapS | clarkb: yeah I recall bluestore as their path to happiness. | 23:39 |
| SpamapS | wind out of sails | 23:40 |
| SpamapS | systemd-nspawn just not quite all the things | 23:41 |
| clarkb | jlk: I mean it makes senes, there is already a tool to edit cgroup rules just reuse it | 23:41 |
| jamielennox | Shrews: how would you feel about me splitting nodepool-webapp into an app rather than something you have to --no-webapp | 23:41 |
| * SpamapS needs a thing that is all the things | 23:41 | |
| clarkb | jlk: rewriting tools to rewrite them is silly and hard on users :) | 23:41 |
| jlk | clarkb: sure, it's the typical unix problem. "ALl these tools exist, just use them better", and then we get things like Docker, which use them better, but then gets seen as "too complex doing too many things". | 23:42 |
| jlk | rinse, repeat :) | 23:42 |
| Shrews | jamielennox: i don't have an issue with that, but should check with jeblair and pabelanger | 23:42 |
| * Shrews aways again | 23:42 | |
| pabelanger | Shrews: jeblair: jamielennox: no issue here | 23:42 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Record SSH public keys for new nodes in ZK https://review.openstack.org/445055 | 23:43 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Remove the --no-delete option from nodepool https://review.openstack.org/445241 | 23:43 |
| jeblair | jamielennox: if you're talking v3: yeah, i think that makes sense -- with the launchers fully distributed, it doesn't make sense for it to be combined with them anymore. | 23:46 |
| jamielennox | jeblair: yep, i'm talking v3 | 23:47 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Encapsulate determining the event purpose https://review.openstack.org/445242 | 23:47 |
| SpamapS | realistically this rkt stage1 with systemd-nspawn has all the cgroupy/namespacey things.. and has a really nice simple interface | 23:49 |
| SpamapS | I like that rkt's API is mostly writing text files to disk and then executing something | 23:49 |
| SpamapS | I remember now meeting the CoreOS engineers early on and noting that they were _very_ excited about systemd. | 23:52 |
| clarkb | rcarrillocruz: jeblair I have reviewed the ansiblification of overlay networking in d-g | 23:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP Add per-repo public and private keys https://review.openstack.org/406382 | 23:54 |
| clarkb | SpamapS: you can also do things like hvae a unit that runs in a container iirc | 23:54 |
| clarkb | SpamapS: so its possile that zuul-executor-container@.service is a thing that zuul-executor just starts with some args and done, check exit codes, and make barbeque | 23:56 |
| clarkb | (I'm hand waving) | 23:56 |
| SpamapS | clarkb: if that includes a prep statement that slaps together a chroot, and a post that cleans itup.. sure. :) | 23:57 |
| jlk | wait | 23:57 |
| jlk | wait wait | 23:57 |
| jlk | "slaps together a chroot" | 23:57 |
| jlk | why isn't that a pre-baked image? | 23:57 |
| SpamapS | it starts with a copy of that | 23:58 |
| jlk | that's baked once a day/week/executord-restart ? | 23:58 |
| SpamapS | and then adds scratch space and dumps git trees in | 23:58 |
| jlk | okay, so yeah, I guess I'm used to higher level tooling that does this | 23:58 |
| jlk | where you say "launch this image and this process inside of it" | 23:58 |
| jlk | and it makes does that | 23:58 |
| SpamapS | docker | 23:58 |
| SpamapS | honestly | 23:58 |
| SpamapS | It's a whole big hot mess | 23:58 |
| jlk | rkt can do that too | 23:58 |
| clarkb | I mean thats what nspawn does too | 23:58 |
| SpamapS | not really | 23:59 |
| SpamapS | nspawn is giving me the hand-waves on cgroups | 23:59 |
| clarkb | SpamapS: it doesn't build the image for you, but you can give it one | 23:59 |
| SpamapS | I get that there's a way | 23:59 |
| pabelanger | clarkb: ideally, nodepool-builder can build it, and some how pass it back to zuul-executor | 23:59 |
| clarkb | you basically say "use this thing over here that might be one of these specific 'image' types" | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!