| rbergeron | clarkb: ask mordred or shrews what happens when i eat seafood.... not sure tuna melts and gaggy are mutually exclusive | 00:13 |
|---|---|---|
| rbergeron | ;) | 00:13 |
| clarkb | oh right I recall something about fried fish bceause everything fried is awesome except not | 00:13 |
| mordred | clarkb: we thought "clearly fried calamari will be fine - because fried - and also calamari isn't really seafood" | 00:14 |
| mordred | clarkb: not so much | 00:14 |
| rbergeron | usually giant tentacle chunks arent quite even calamari, are they? | 00:30 |
| SpamapS | I'm sure giant squid use those tentacles to capture small animals on dry land so, obviously, tentacle != Seafood | 00:54 |
| *** jamielennox is now known as jamielennox|away | 01:54 | |
| *** adam_g has quit IRC | 03:07 | |
| *** adam_g has joined #zuul | 03:11 | |
| *** adam_g has quit IRC | 03:28 | |
| *** adam_g has joined #zuul | 03:31 | |
| *** adam_g has quit IRC | 03:43 | |
| *** adam_g has joined #zuul | 03:44 | |
| *** jamielennox|away is now known as jamielennox | 04:23 | |
| *** adam_g has quit IRC | 04:38 | |
| *** adam_g has joined #zuul | 04:41 | |
| *** jamielennox is now known as jamielennox|away | 05:11 | |
| *** jamielennox|away is now known as jamielennox | 05:51 | |
| *** isaacb has joined #zuul | 06:01 | |
| *** jamielennox is now known as jamielennox|away | 06:23 | |
| *** isaacb has quit IRC | 07:03 | |
| *** adam_g has quit IRC | 07:04 | |
| *** adam_g has joined #zuul | 07:04 | |
| *** hashar has joined #zuul | 07:19 | |
| *** DangerousDaren has joined #zuul | 07:21 | |
| *** clarkb has quit IRC | 07:30 | |
| *** clarkb has joined #zuul | 07:30 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** bhavik1 has joined #zuul | 08:46 | |
| *** jasondotstar has quit IRC | 08:48 | |
| *** jasondotstar has joined #zuul | 08:51 | |
| *** bhavik1 has quit IRC | 09:08 | |
| *** hashar is now known as hasharAway | 09:42 | |
| *** Cibo_ has joined #zuul | 09:55 | |
| *** Cibo_ has quit IRC | 10:04 | |
| *** adam_g has quit IRC | 10:17 | |
| *** adam_g has joined #zuul | 10:18 | |
| *** jkilpatr has quit IRC | 10:40 | |
| *** smyers has quit IRC | 10:40 | |
| *** smyers has joined #zuul | 10:41 | |
| *** jkilpatr has joined #zuul | 10:57 | |
| *** adam_g has quit IRC | 11:01 | |
| *** adam_g has joined #zuul | 11:03 | |
| *** adam_g has quit IRC | 11:11 | |
| *** adam_g has joined #zuul | 11:12 | |
| *** GeppyZ has joined #zuul | 11:17 | |
| *** GeppyZ has left #zuul | 11:18 | |
| *** hasharAway is now known as hashar | 12:06 | |
| *** dkranz has quit IRC | 12:31 | |
| *** jamielennox|away is now known as jamielennox | 12:49 | |
| *** openstackgerrit has joined #zuul | 13:10 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use ssh for git-upload-pack https://review.openstack.org/436802 | 13:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use ssh for git-upload-pack https://review.openstack.org/436802 | 13:36 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use ssh for git-upload-pack https://review.openstack.org/436802 | 13:43 |
| *** rcarrillocruz has quit IRC | 13:55 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/zuul feature/zuulv3: Switch from testrepository to stestr https://review.openstack.org/466071 | 13:58 |
| jeblair | pabelanger, mordred: let me know when you're around to chat about ansible_log | 14:00 |
| pabelanger | jeblair: I have some time now | 14:00 |
| *** dkranz has joined #zuul | 14:04 | |
| *** Cibo_ has joined #zuul | 14:13 | |
| *** rcarrillocruz has joined #zuul | 14:19 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/zuul feature/zuulv3: Switch from testrepository to stestr https://review.openstack.org/466071 | 14:26 |
| pabelanger | Shrews: next question :) Is there any reason we don't have kazoo.client keep retrying connection attempts for zk? http://paste.openstack.org/show/610461/ | 14:27 |
| mordred | jeblair: morning! happy to talk about logs | 14:28 |
| pabelanger | seems we could use kazoo.retry.KazooRetry to keep retrying until a connection was made | 14:28 |
| Shrews | pabelanger: it should retry automatically. | 14:29 |
| Shrews | pabelanger: there may be a max retry policy we could adjust | 14:29 |
| pabelanger | Shrews: yes, it retried x times and stopped | 14:29 |
| pabelanger | I'd be okay with it trying forever | 14:29 |
| mordred | Shrews: I didn't add a unit test because you can't use mock on datetime.datetime (I did write one) | 14:30 |
| jeblair | mordred, pabelanger: yay we're all here! | 14:30 |
| mordred | Shrews: oh - sorry, that was for other channel | 14:30 |
| Shrews | mordred: no | 14:31 |
| jeblair | mordred, pabelanger: so pabelanger was noting that ansible_log is unreadable | 14:32 |
| jeblair | example: http://zuulv3-dev.openstack.org/logs/473ba003bbb64297927e8b21641f5178/ansible_log.txt | 14:32 |
| mordred | oh - yah. that's not great! | 14:32 |
| jeblair | i thought we had a thing so that we didn't dump the stdout as json thing, but instead streamed the console log from the worker into ansible_log line-by-line? | 14:33 |
| Shrews | neat | 14:33 |
| mordred | ah - that's not where the garbage is coming from | 14:34 |
| mordred | the garbage is coming from us printing the "command" that was run | 14:34 |
| mordred | but the command that was run was a long script block | 14:34 |
| mordred | so we should NOT print the command that was run | 14:34 |
| jeblair | mordred: i see some "changed" lines... isn't that the stdout-in-json thing? | 14:35 |
| mordred | or we should at the very least not print it if it's a script block | 14:35 |
| mordred | jeblair: right- that's the "task" reporting its results. so what it's saying is "I ran something (changed=True is always true for cmd tasks) and here is what I ran (cmd=)" | 14:35 |
| mordred | jeblair: for our cases, I do not believe we find that valuable | 14:36 |
| jeblair | mordred: oh, i see. okay, though i think maybe we're printing both cmd and stdout? | 14:36 |
| mordred | jeblair: yes. we are | 14:37 |
| mordred | jeblair: so - two things to fix | 14:37 |
| mordred | in fact - I think maybe we should just not print result lines for cmd tasks | 14:37 |
| jeblair | http://paste.openstack.org/show/610464/ | 14:37 |
| mordred | since we log cmd tasks in our own way | 14:37 |
| jeblair | slightly reformattted json blob output ^ | 14:37 |
| jeblair | mordred: there's some other stuff in there -- start, end, rc... can we keep those but elide the cmd, stdout, stderr stuff? | 14:39 |
| mordred | I dont thinkn we need any of them because of the way we log all cmd/shell tasks | 14:39 |
| mordred | well - ok - maybe rc is useful | 14:39 |
| mordred | so - let's just log start, end and rc if it's a cmd/shell task | 14:39 |
| mordred | delta is time elapsed too | 14:40 |
| jeblair | ya, delta is really nice actually :) | 14:40 |
| jeblair | mordred: honestly, i'm not sure cmd is that bad? | 14:40 |
| *** Cibo_ has quit IRC | 14:41 | |
| mordred | jeblair: cmd will include the entire test of any script block passed | 14:41 |
| jeblair | lemme go find that playbook | 14:41 |
| pabelanger | Ya, having both will be help for for debugging purposes | 14:42 |
| jeblair | http://git.openstack.org/cgit/openstack-infra/openstack-zuul-roles/tree/roles/openstack-info/tasks/main.yaml | 14:42 |
| jeblair | so i guess if we do that sort of thing, we'll end up with cmd blobs | 14:43 |
| mordred | jeblair: right. that's the content that's in the cmd parameter of that return dict | 14:43 |
| jeblair | mordred: i think you've convinced me.... after all, if we want to log that, we can 'set -x' it? | 14:43 |
| mordred | yah | 14:43 |
| mordred | so we need to add a v2_playbook_on_task_end method to zuul/ansible/callback/zuul_stream.py | 14:44 |
| mordred | that strips things if it's a cmd output | 14:44 |
| pabelanger | ya, set -x works for me | 14:45 |
| jeblair | mordred: is that plugin responsible for writing that data to the logs? i thought there was a built-in callback plugin for that? | 14:46 |
| mordred | our zuul_stream callback plugin is a subclass of the default plugin | 14:46 |
| mordred | so the default plugin handles most of the things | 14:47 |
| jeblair | mordred: okay, so it is *the* output plugin, and it's performing all of the normal output plugin duties by inheritance, then also adding the streaming bits | 14:47 |
| mordred | yes | 14:47 |
| jeblair | got it | 14:47 |
| jeblair | pabelanger: does that sound good to you? | 14:48 |
| pabelanger | jeblair: yes, works for me | 14:48 |
| mordred | k. I'm pushing that on to my stack real quick | 14:49 |
| jeblair | mordred, pabelanger: thanks! | 14:49 |
| Shrews | pabelanger: looks like *maybe* setting connection_retry={'max_tries': -1} in the KazooClient() object might do what you want there? just took a quick glance at the kazoo code though | 15:05 |
| pabelanger | Shrews: yes, that is what I am going to test shortly | 15:05 |
| jlk | Hello? Yes, it is morning. | 15:06 |
| Shrews | pabelanger: though the default seems to be None but i don't see immediately how that translates to anything | 15:06 |
| Shrews | but... meeting | 15:06 |
| jlk | jeblair: mordred: can one of you click the +w on the driver specific pipeline requirements change? I think we have enough votes. | 15:07 |
| jeblair | jlk: done | 15:08 |
| jlk | woo. I'll be doing the rest of the rebase today (hopefully). | 15:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Bump diskimage-builder dependency to 2.0.0 https://review.openstack.org/467282 | 15:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Bump diskimage-builder dependency to 2.0.0 https://review.openstack.org/467282 | 15:15 |
| Shrews | smyers: hi! dkranz said you were interested in nodepool things. welcome and feel free to ask any questions you may have about it here. :) | 15:25 |
| smyers | o/ | 15:29 |
| smyers | I have a few questions, but right now I'm mainly interested in this "drivers" spec: https://review.openstack.org/#/c/461509/ | 15:33 |
| smyers | I'm curious about how that might actually be implemented, especially with the openstack-centric provisioning currently going on. I'm also curious if it's expected to get merged. :) | 15:34 |
| jeblair | smyers: yeah, i'll make it mergeable and put it on today's infra meeting agenda | 15:35 |
| smyers | It looks like a great proposal, but...daunting. | 15:35 |
| Shrews | smyers: I think the "how" is to be worked out by the folks implementing it. There are some others here (tristanC and tobiash maybe?) who I think are interested in that spec as well. Putting some heads together to work on a proposed API would be a good first step. | 15:37 |
| *** DangerousDaren has quit IRC | 15:41 | |
| smyers | One question I have is if the version of nodepool that is being developed on the zuulv3 feature branch has an expected version. I'm tired of saying "The version of nodepool being developed on the zuulv3 feature branch". | 15:49 |
| jeblair | smyers: i'd like to call it 3.0 to sync it with zuul. though 1.0 is also an option :) | 15:49 |
| Shrews | i vote for 42.0 | 15:50 |
| jeblair | smyers: at any rate, if you say 'nodepool v3' we will definitely know what you're talking about. (anything >=1 certainly means that branch) | 15:50 |
| jeblair | Shrews: we should release 42.0 followed by 3.0. | 15:50 |
| smyers | Yeah, I go back and forth between 1.0 and 3.0, and occasionally consider then disregard .5 because semver | 15:51 |
| clarkb | also I don't think there is a single version. we've already released 0.4.0 off of code merged back into master from the zuulv3 branch | 15:51 |
| pabelanger | remote: https://review.openstack.org/467300 Create zuul-base-jobs and zuul-jobs | 16:00 |
| *** jkilpatr has quit IRC | 16:04 | |
| mordred | jeblair, smyers, Shrews: in my brain part of the v3 effort is recognizing nodepool as a component of zuul ... if it wouldn't be even more disruptive, I'd suggest we rename the nodepool repo to 'zuul-nodepool' | 16:05 |
| jeblair | okay, nodepool drivers spec is green now and is on the infra agenda | 16:05 |
| mordred | jeblair: wot! | 16:05 |
| mordred | I mean "woot" - not "wot" as in "what???" | 16:05 |
| jeblair | mordred: wot you say??!? | 16:05 |
| pabelanger | Shrews: So, have we documented how to decommission a nodepool-builder? | 16:06 |
| pabelanger | I guess we'd need different nodepool.yaml files for each server running | 16:07 |
| pabelanger | and stage removing images from the server we want to shutdown | 16:07 |
| *** hashar is now known as hasharAway | 16:08 | |
| Shrews | pabelanger: not that i'm aware of | 16:08 |
| clarkb | pabelanger: I think you can just turn off the service, wait two days (so images rotate out) then delete | 16:09 |
| clarkb | no need for config or anything | 16:09 |
| pabelanger | clarkb: I don't think so, because nb01 need to delete the images that are uploaded | 16:10 |
| pabelanger | if it is off, they linger | 16:10 |
| Shrews | pabelanger: wow, 467282 has all sorts of py35 fail | 16:11 |
| pabelanger | Shrews: we still need to land topic:py3-nodepool patches | 16:11 |
| clarkb | pabelanger: yes would require a separate cleanup step | 16:11 |
| Shrews | pabelanger: ugh, i thought we had. guess i mixed it up with zuul | 16:12 |
| pabelanger | clarkb: ya, not sure I like the manual clean up | 16:13 |
| pabelanger | so | 16:13 |
| pabelanger | it would be cool if we could run nodepool image-build --builder=nb03.o.o fedora-25 and force the migration | 16:14 |
| pabelanger | that should allow us to move images to a new builder | 16:14 |
| clarkb | thats what turning off the service gets you right? the images will move | 16:14 |
| clarkb | because 02 and 03 will build the necessary images and the ones for 01 will be marked delete (but won't actually delete because 01 isn't running) | 16:15 |
| pabelanger | forcing a rebuild will delete the originals hower | 16:15 |
| pabelanger | right | 16:15 |
| pabelanger | Or add image-delete --force to have another builder delete them | 16:16 |
| pabelanger | I'll shutdown for now | 16:16 |
| clarkb | I think I like ^ since it more closely matches what you actually need | 16:16 |
| clarkb | (which is easy cleanup) | 16:16 |
| pabelanger | k, I'll look at --force option in a bit | 16:16 |
| clarkb | or even image-prune --force (delete all images marked delete regardless of "ownership") | 16:17 |
| pabelanger | ya | 16:18 |
| clarkb | re using configuration transitions to implement it, I think we should avoid that because we've seen how that causes confusion around cleanup in the past. Like removing a provider | 16:19 |
| clarkb | I expect that a "turn it off then clean it up" process will be more approachable? but maybe not | 16:19 |
| *** jkilpatr has joined #zuul | 16:20 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Strip unneeded cmd, changed, stdout and stderr from results https://review.openstack.org/467310 | 16:29 |
| mordred | jeblair, pabelanger: ^^ that should fix the logs. I also left a note for a future improvement - but I need to jump on a call now so I left it for later | 16:30 |
| jeblair | mordred: cool, left a couple comments | 16:37 |
| jeblair | i've managed to catch the py3 repo leak issue locally once. i don't think it's related to any github stuff -- i've seen it hit other tests too. it's probably just the order testr is running them in. | 16:38 |
| jlk | whew! | 16:39 |
| jlk | and boo | 16:39 |
| clarkb | jeblair: did you see my comemnt about trying to pry more debugging info out of gc on the change? | 16:41 |
| jeblair | clarkb: nope, i'll go look | 16:41 |
| jeblair | clarkb: ah good ideas | 16:42 |
| jeblair | clarkb: i had added a __del__ monkeypatch too, but i worry the log line i put in there is perhaps slowing things down enough for it to actually work; i haven't repro'd since then | 16:43 |
| * SpamapS shakes head at weird lockup caused by ssh-agent patches :-P | 16:43 | |
| jeblair | i'll undo that and try the gc ideas | 16:43 |
| pabelanger | clarkb: not sure if you seen: https://review.openstack.org/#/c/467282/ bumps DIB to 2.0.0 to get virtualenv logic out of nodepool | 16:43 |
| clarkb | pabelanger: I ahdn't, approved | 16:44 |
| *** bhavik1 has joined #zuul | 16:47 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Bump diskimage-builder dependency to 2.0.0 https://review.openstack.org/467282 | 16:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Fetch server console log if ssh connection fails https://review.openstack.org/452494 | 16:54 |
| *** harlowja has joined #zuul | 16:56 | |
| pabelanger | clarkb: we could port ^ to master also. I decided to do feature/zuulv3 because of recent config changes | 16:57 |
| *** Cibo_ has joined #zuul | 16:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Temporarily(?) add some debug statements around git repo creation https://review.openstack.org/466810 | 17:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467324 | 17:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467325 | 17:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467326 | 17:05 |
| * jeblair starts firing a zuul cannon at the problem | 17:06 | |
| *** bhavik1 has quit IRC | 17:16 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add default-branch property to projects https://review.openstack.org/467334 | 17:20 |
| *** adam_g has quit IRC | 17:21 | |
| *** adam_g has joined #zuul | 17:22 | |
| *** tobiash_ has joined #zuul | 17:30 | |
| *** adam_g has quit IRC | 17:30 | |
| *** adam_g has joined #zuul | 17:31 | |
| jeblair | 8 out of 8 builds with the extra debugging passed, so it seems like it's very sensitive to timing. | 17:36 |
| jeblair | i'll disable automatic gc before we start the check | 17:36 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Temporarily(?) add some debug statements around git repo creation https://review.openstack.org/466810 | 17:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467324 | 17:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467325 | 17:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: py3 race test change https://review.openstack.org/467326 | 17:38 |
| jlk | jeblair: in gerritmodel, what's the purpose of deepcopying the required_approvals in order to tidy them? | 17:40 |
| Shrews | hrm, daemon lib doesn't seem to play nice with asyncio servers | 17:41 |
| Shrews | that makes me sad | 17:41 |
| jlk | I'm sure it's a basic thing that I'm ignorant to | 17:41 |
| jlk | oh, you copy them to a holder for printing them out | 17:42 |
| jlk | and then you manipulate the existing ones for machine reasons | 17:42 |
| jeblair | jlk: the tidy method modifies them in place, so i don't want to modify the passed-in argument | 17:42 |
| jeblair | jlk: and yes to the rest of that | 17:43 |
| jlk | thanks. | 17:43 |
| jlk | and what's the basic idea behind tidying them? | 17:44 |
| jlk | making the values real regex things, or time_to_seconds things? | 17:44 |
| Shrews | mordred: i don't suppose in your autobahn explorations that you saw any examples of daemonizing a server, did you? | 17:46 |
| Shrews | guess i'll try manual fork/pid stuff | 17:53 |
| *** Cibo_ has quit IRC | 17:57 | |
| clarkb | Shrews: thats odd, daemon lib doesn't really touch threading? | 18:03 |
| Shrews | clarkb: i don't think it has to do with threading | 18:03 |
| clarkb | Shrews: is it closing some file descriptor it shouldnt? | 18:04 |
| Shrews | the asyncio event loop seems to not work correctly | 18:04 |
| *** jkilpatr has quit IRC | 18:06 | |
| tobiash_ | Shrews, smyers, tristanC: the drivers spec is definitely something I would like to work on, but I don't think I have time for this next week. The two weeks after that I'm on vacation. So I think I could start working on that in about 3-4 weeks if nobody started this until then. | 18:06 |
| *** jkilpatr has joined #zuul | 18:07 | |
| clarkb | Shrews: ya I'm guessing its closing some file that asyncio is opening to run the eventloop | 18:07 |
| clarkb | Shrews: since daemonization includes closing all the fds | 18:07 |
| clarkb | Shrews: you can whitelist fds with the daemon lib if you can find the fd | 18:08 |
| SpamapS | wow that was stupidity... | 18:10 |
| SpamapS | apparently if you save your SkyMiles # in your password manager with a space at the end.. delta.com started rejecting it yesterday | 18:10 |
| SpamapS | great coding folks! | 18:10 |
| jeblair | tobiash_: thanks, that helps with planning. | 18:12 |
| jeblair | jlk: yes to both of those -- basically compiling it ahead of time for quick evaluation later | 18:12 |
| tobiash_ | jeblair: I saw in the logs that there was some discussion about the start-reporting on unmanaged projects (https://review.openstack.org/#/c/455711/) | 18:31 |
| tobiash_ | jeblair: was there some consensus how to proceed with this topic? | 18:32 |
| jlk | jeblair: I spot something I see that may be logic counter to what the comment says, and I'd like you to check my thoughts on this. | 18:47 |
| jlk | nope, I think I get it. | 18:49 |
| *** adam_g has quit IRC | 18:57 | |
| jeblair | tobiash_: i don't have a great solution to all of the issues. i think your patch is an improvement and i'm leaning toward thinking we should merge it, and see if we can further improve things later. | 18:58 |
| tobiash_ | jeblair: sounds reasonable | 18:59 |
| *** adam_g has joined #zuul | 18:59 | |
| tobiash_ | did I hit the earliest point in time where zuul knows that there are jobs to run (at least that was my intention)? | 19:00 |
| *** tobiash_ has quit IRC | 19:19 | |
| jeblair | tobiash: i think so; i'll double check when i review more closely | 19:57 |
| jeblair | clarkb, jlk, SpamapS: with enough debugging to be useful, i can't get the py3 race to happen :( | 19:58 |
| jeblair | i'm starting to think we should just disable the leak check | 19:58 |
| clarkb | ugh the best kind of bug | 19:59 |
| jlk | ouch | 20:07 |
| SpamapS | doesn't py3 offer _better_ cycle detection and thus it's supposed to clean up more things? | 20:12 |
| SpamapS | ah unless the objects have __del__ methods | 20:14 |
| SpamapS | jeblair: it's entirely possible this is just a wild goose chase and there's a timing problem where the order of refcount decrementing of a git.Repo and its references in py3 means it simply can't be gc'd | 20:15 |
| jeblair | SpamapS: i think in py3 everything should be able to be gc'd, but yes, i think the order/timing here might be causing a delay. there *is* a __del__ method which i think triggers the new behavior in pep 442 | 20:19 |
| SpamapS | Changed in version 3.4: Following PEP 442, objects with a __del__() method don’t end up in gc.garbage anymore. | 20:19 |
| SpamapS | yerp | 20:19 |
| jeblair | (i'm genuinely surprised that disabling the gc before we perform our check somehow caused the bug to not appear though) | 20:20 |
| jeblair | i thought that would make it more reliably appear | 20:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add override-branch property to job repos https://review.openstack.org/467375 | 20:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename 'repos' job attribute to 'required-projects' https://review.openstack.org/467376 | 20:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename 'repos' job attribute to 'required-projects' https://review.openstack.org/467376 | 20:21 |
| *** jkilpatr has quit IRC | 20:22 | |
| SpamapS | jeblair: the mysteries of gc behavior have never sat well with me. *cough*rust*cough* | 20:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add override-branch property to job repos https://review.openstack.org/467375 | 20:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename 'repos' job attribute to 'required-projects' https://review.openstack.org/467376 | 20:29 |
| *** harlowja_ has joined #zuul | 20:32 | |
| *** harlowja has quit IRC | 20:33 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add driver-specific pipeline requirements https://review.openstack.org/466105 | 20:37 |
| SpamapS | jlk: just to repeat over here.. it looks like bwrap may not work, even with a USER_NS capable kernel, from inside a container. :( | 20:44 |
| SpamapS | which is what I thought was one of the main awesome features of bwrap | 20:45 |
| jeblair | mordred: fyi ^ | 20:46 |
| SpamapS | root@4c49ba0b0b9a:/# ssh-agent /home/zuul/zuul/.tox/py27/bin/zuul-bwrap ~/tmp/work_dir ~/tmp/ansible_dir /bin/bash | 20:46 |
| SpamapS | No permissions to creating new namespace, likely because the kernel does not allow non-privileged user namespaces. On e.g. debian this can be enabled with 'sysctl kernel.unprivileged_userns_clone=1'. | 20:46 |
| jeblair | SpamapS: do you think that's closer to "temporary bug" territory or "fundamental design problem"? | 20:46 |
| SpamapS | userns_clone is == 1 and that command works fine on the host OS without being inside a docker container. | 20:46 |
| SpamapS | jeblair: I'm hoping it's a temporary impedence mismatch between docker and the kernel that just grew these capabilities. | 20:47 |
| SpamapS | but I'm still reading. | 20:47 |
| jeblair | k | 20:47 |
| SpamapS | docker does some lockdown, and it may just be standing in the way because it's new. | 20:47 |
| jlk | SpamapS: what if that outer container is a privileged container? | 20:47 |
| SpamapS | # strace -f /home/zuul/zuul/.tox/py27/bin/zuul-bwrap ~/tmp/work_dir ~/tmp/ansible_dir /bin/bash | 20:48 |
| SpamapS | strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted | 20:48 |
| SpamapS | I can't strace to find out what call is even failing. :-/ | 20:48 |
| SpamapS | jlk: that's exactly what I'm testing now | 20:48 |
| jlk | yeah that's what I would think to start with, that the capabilities extended to the process (container) is too minimal | 20:48 |
| SpamapS | of course, I was dumb, and did this without a Dockerfile, so having to build a new one :-P | 20:48 |
| jlk | also sounds like we're on new and novel territory | 20:48 |
| SpamapS | jlk: question is, do people who like to run things in containers, think its ok to give zuul-executor a privileged container? | 20:48 |
| jlk | well | 20:49 |
| jlk | privileged is like a wide swath, it's possible we could discover and narrow down to the exact permission we need to give it | 20:49 |
| jlk | We'd still run zuul as non-root inside the container | 20:49 |
| jlk | the extra rights would matter if somebody breaks out of zuul and escalates to "root" inside the container | 20:50 |
| SpamapS | escalates out of ansible, and bubblewrap ;) | 20:51 |
| SpamapS | I'm more wondering like, if I'm using BlueMix's kubernetes.. are they going to be able to give me a privileged container? | 20:51 |
| jlk | well | 20:51 |
| jlk | yes | 20:51 |
| jlk | ah. | 20:51 |
| jlk | I have no idea | 20:51 |
| jlk | but we need a privileged container for disk-image-builder anyway | 20:52 |
| SpamapS | did you get that working? | 20:52 |
| jlk | yes | 20:53 |
| SpamapS | sweet | 20:53 |
| jlk | Not to the point of testing the booted image in openstack, but nodepool at least built the image and booted the VM | 20:53 |
| SpamapS | wow.. I just wrote a Dockerfile that worked on the second try :-P | 20:53 |
| * SpamapS thinks maybe this Docker thing has legs | 20:54 | |
| jlk | which was the blocking point for getting a noop job to "execute" | 20:54 |
| jlk | hahah | 20:54 |
| jlk | yeah, I've been really pleased with the utility of it | 20:54 |
| SpamapS | yay | 20:57 |
| SpamapS | zuul@74951e87609f:~$ ssh-agent /usr/local/bin/zuul-bwrap ~/tmp/work_dir ~/tmp/ansible_dir /bin/bash | 20:57 |
| SpamapS | bash-4.4$ | 20:57 |
| SpamapS | so yeah, --privileged fixes it | 20:57 |
| SpamapS | jeblair: ^ | 20:57 |
| * SpamapS breathes sigh | 20:59 | |
| jeblair | SpamapS: so the container version of "sudo make me a sandwich" | 20:59 |
| SpamapS | jeblair: indeed | 20:59 |
| SpamapS | can try every iteration of this https://docs.docker.com/engine/reference/run/#security-configuration | 21:00 |
| SpamapS | guessing seccomp is the culprit | 21:01 |
| SpamapS | and we just need to whitelist the ones bwrap needs to make | 21:01 |
| SpamapS | but this is a bit rabbit hole-y at the moment | 21:01 |
| * SpamapS has confirmed.. things work more or less the way they should | 21:01 | |
| SpamapS | on zesty :) | 21:02 |
| clarkb | being able to create nested namespaces is something that has to be allowed iirc | 21:02 |
| clarkb | I wonder if its as simple as that? | 21:02 |
| pabelanger | is ssh-agent running bubblewrap in that example above? | 21:02 |
| SpamapS | clarkb: oh is there maybe a specific flag for that? | 21:02 |
| SpamapS | pabelanger: it is | 21:02 |
| SpamapS | zuul-bwrap picks up SSH_AUTH_SOCK from the environment | 21:03 |
| SpamapS | which I thought would be the most operator friendly thing for testing | 21:03 |
| clarkb | SpamapS: https://success.docker.com/KBase/Introduction_to_User_Namespaces_in_Docker_Engine | 21:03 |
| clarkb | also I love that fqdn | 21:03 |
| SpamapS | since you can start your own ssh-agent and load keys to test | 21:03 |
| SpamapS | clarkb: what's at fail.docker.com ? ;-) | 21:04 |
| jlk | the whale | 21:05 |
| *** harlowja_ has quit IRC | 21:05 | |
| SpamapS | so, we can get the zesty kernel on xenial just by apt installing linux-image-extra-4.10.0-21-generic - Linux kernel extra modules for version 4.10.0 on 64 bit x86 SMP | 21:05 |
| SpamapS | ok, I've confirmed that it's possible. Back to debugging ssh-agent code. :-P | 21:06 |
| jlk | https://vangogh.teespring.com/og_pic/2251813/2393336/front.jpg?v=2015-04-28-04-07&background-image=wood&effects=inner-glow | 21:06 |
| SpamapS | yeah they have a stupid robot now when it breaks | 21:08 |
| *** harlowja has joined #zuul | 21:08 | |
| jlk | question for the bike shed aficionados | 21:09 |
| jlk | when declaring a pipeline requirement for a github review state | 21:09 |
| jlk | we've got, username, type (approved, comment, etc), newer-than, older-than, email, and "permission" or "permissions" (to indicate the review needs to be somebody with write access. | 21:10 |
| jlk | possible values for permission(s) are ('write', 'read') | 21:10 |
| jlk | any preference on "permission" vs "permissions" ? | 21:10 |
| SpamapS | jlk: do they logical AND/OR together? | 21:21 |
| SpamapS | or is it an enum? | 21:21 |
| SpamapS | if enum, then permission. if logical, permissions | 21:21 |
| jlk | They OR | 21:21 |
| SpamapS | they OR but there's two? so mentioning both is like not mentioning any? | 21:22 |
| *** dkranz has quit IRC | 21:23 | |
| jlk | well, if you have write, you implicitly have read | 21:23 |
| jlk | a user submitting a review can have either read or write perms. | 21:24 |
| jlk | so if you want to allow reviews from _anybody_ you'd have to list both read and write perms. | 21:24 |
| jlk | otherwise we'd have to do weird things inside zuul to say if you mark read, also allow write? | 21:24 |
| jlk | actually. hrm. | 21:24 |
| jlk | You _have_ to have read to leave a review | 21:25 |
| jlk | so maybe it only makes sense to have a single option for permission: write? | 21:25 |
| jlk | or some other way of expressing a boolean? | 21:25 |
| jlk | jeblair: what are your thoughts here? | 21:25 |
| SpamapS | jlk: if I say 'permission: read' wouldn't that imply writers too? | 21:35 |
| jlk | that's what makes it odd | 21:36 |
| jlk | What I thikn I'm trying to account for is whether or not you want to require write access to trigger the pipeline | 21:37 |
| SpamapS | I think it makes sense. In that.. I understand that writers automatically get read. Only Unix does this explicitly ;) | 21:37 |
| jlk | +2 vs +1 | 21:37 |
| jlk | so you think that permission: should be an enum, that would accept either 'read' or 'write'. | 21:38 |
| jlk | and we document that people with write count for 'read' | 21:39 |
| SpamapS | jlk: it makes logical sense to me. I'd definitely like to hear counter arguments. :) | 21:43 |
| *** jkilpatr has joined #zuul | 21:44 | |
| jeblair | (incidentally, 'read' and 'read-write' might be good enum names to make that clear; not sure how divergent from github terminology that is though) | 21:44 |
| jlk | github has 'admin', 'write', 'read', 'none' | 21:47 |
| jeblair | jlk: generally (and if this isn't consistent, it might be nice to clean it up before 2.0), if something can take more than one value, we make the key plural (even if the value can be supplied as a singleton). example: "branches: stable" "branches: [stable, master]" are both valid. so, yeah, if enum, 'permission', if scalar_or_list, 'permissions' | 21:47 |
| *** jroll has quit IRC | 21:47 | |
| jlk | we're mapping 'admin' and 'write' to just 'write' | 21:47 |
| *** jroll has joined #zuul | 21:47 | |
| *** jroll has quit IRC | 21:49 | |
| jlk | I suppose we could support read, write, and admin levels | 21:49 |
| jlk | the enum would be one of those three | 21:49 |
| jeblair | so far, they seem like a strict hierarchy | 21:50 |
| *** jroll has joined #zuul | 21:53 | |
| *** hasharAway has quit IRC | 22:11 | |
| *** Cibo_ has joined #zuul | 22:45 | |
| * SpamapS adds more debugs to try and find weird race in ssh agent code :-P | 22:53 | |
| *** tristanC has quit IRC | 23:17 | |
| *** adam_g has quit IRC | 23:32 | |
| *** adam_g has joined #zuul | 23:33 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
