Tuesday, 2017-09-19

*** lennyb has quit IRC02:04
*** lennyb has joined #zuul02:06
tobiashSpamapS: this was the fix for the stream did not terminate problem: https://review.openstack.org/#/c/491027/204:25
tobiashSpamapS: so maybe you don't have this yet or found an additional edge case where this still occurs04:26
SpamapStobiash: I have that commit.05:44
SpamapSDeployed up to master from Friday at this point.05:44
tobiashSpamapS: hm, then you probably found a new edge case where this fix didn't work05:45
tobiashSpamapS: wait, master or feature/zuulv3?05:45
SpamapSYeah I'm not seeing any errors in the logs around the same time.05:45
SpamapStobiash: feature/zuulv3 ;)05:45
tobiash;)05:45
SpamapSsoon to be master I Hope05:45
tobiashdon't know what the plans are for branching05:45
SpamapSYeah it's something to think about after cutover05:47
*** Shrews has quit IRC06:05
*** harlowja has quit IRC06:05
*** Shrews has joined #zuul06:05
*** mnaser has quit IRC06:12
*** mnaser has joined #zuul06:30
*** xinliang has quit IRC07:02
*** xinliang has joined #zuul07:15
*** hashar has joined #zuul07:18
*** xinliang has quit IRC07:42
*** xinliang has joined #zuul07:42
*** xinliang has quit IRC07:42
*** xinliang has joined #zuul07:42
*** hashar has quit IRC08:36
*** hashar has joined #zuul08:37
*** electrofelix has joined #zuul08:45
*** bhavik1 has joined #zuul10:30
*** bhavik1 has quit IRC10:33
*** jkilpatr has quit IRC10:42
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: WIP: Honor cloud quotas before launching nodes  https://review.openstack.org/50383811:00
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't fail on quota exceeded  https://review.openstack.org/50305111:00
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Make max-servers optional  https://review.openstack.org/50428211:00
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support cores limit per pool  https://review.openstack.org/50428311:00
openstackgerritTobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support ram limit per pool  https://review.openstack.org/50428411:00
*** jkilpatr has joined #zuul11:15
*** dkranz has joined #zuul12:56
*** hashar_ has joined #zuul13:06
*** hashar has quit IRC13:08
dmsimardmordred, jeblair: are we making sure that the executor can't connect to itself through it's public IP ? doesn't seem covered here: https://github.com/openstack-infra/zuul/blob/feature/zuulv3/zuul/ansible/action/normal.py#L3813:45
mordreddmsimard: nope, we are not14:14
mordreddmsimard: connecting to itself shouldn't be an issue though, as the executor should not have an ssh key that is authorized to any users on the executor14:14
dmsimardmordred: right, so context is we were looking for the right way of connecting to containers which might be hosted on localhost14:15
dmsimardmordred: there was this "workaround" but the proper fix is probably to have the driver handle the executor restrictions somehow14:15
mordreddmsimard: well, I do not think it's a problem to have them ssh to the container on localhost - the first thing we do in the base job is swap the ssh keys out14:18
mordreddmsimard: so it shouldn't be an issue to use containers on localhost (obviously you'll need ports :) )14:19
dmsimardmordred: right now it is because the check (linked from normal.py) is fairly naive in that it only checks for localhost or 127.014:22
dmsimardso if I try to connect on a localhost container on port 50000, it will still deny local executor execution14:23
mordredoh - right - sorry - still waking up ...14:23
mordreddmsimard: so - for that, I believe you'll want to pass either the public or private ip of the machine with a port if you want to use a container that's also hosted on the local machine14:24
mordredusing localhost as the ip of that will not work14:24
mordredthe localhost entry in a catalog is 'special'14:24
dmsimardmordred: right, that's what I meant by "working around" with the public ip14:24
dmsimardfeels kind of dirty though14:25
mordredwell - overall I think it's more correct. in the general case, there's no way for nodepool to know that a given container will be co-located on the same host as an executor14:26
mordredwhile it might work as an optimization in the case where it's an AIO setup - from a plumbing perspective I think it would be extra complexity to have nodepool try to sometimes provide localhost as the ip and sometimes the actual IP14:26
mordredwhereas if nodepool just always provides the non-localhost IP, it should work with both AIO and multi-node setups14:27
dmsimardmakes sense14:32
mordreddmsimard: now, in addition to that, at SOME point we may want to figure out connection=docker for using docker connection rather than ssh ... and I don't think our action plugins support doing that currently14:38
dmsimardmordred: right, I discussed that with tristanC -- there are different connection plugins (not just 'docker')14:38
mordredyah14:38
dmsimardhttps://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/connection14:39
mordreddmsimard: we're likely to get to connection=winrm before connection=docker since windows is a thing tobiash is working towards right now14:39
dmsimardtechnically, is there anything preventing that from working already ?14:39
dmsimardcould I just set a 'ansible_connection: winrm' var ?14:39
mordreddmsimard: we explicitly disallow doing that14:40
dmsimardoh14:40
mordreddmsimard: if you try any connection other than ssh right now it will break14:40
tobiashdmsimard: nodepool tells zuul the connection in the windows patch stack14:40
dmsimardcool14:40
tobiashdmsimard: what's missing yet is a patch in zuul to honor that14:40
tobiashbut that should be easy14:40
dmsimardmordred: so re: last night's trusted thing14:40
tobiashjust hadn't time for this yet14:41
dmsimardmordred: you're saying not everyone will be able to, say, add a job to the keystone project ?14:41
dmsimardmordred: where's the boundary ? only trusted projects may add a job to a project that isn't their own ?14:42
mordredtobiash: yah - as part of that we'll need to whitelist the winrm connection plugin I think14:47
mordreddmsimard: yes. only trusted projects14:47
mordreddmsimard: this goes nicely with the thing jeblair was talking about with making the 'name' field in the project pipeline definition optional14:48
mordreddmsimard: since for non-trusted projects, there is only one valid value for it14:48
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add strip_branch_ref compat option  https://review.openstack.org/50529014:48
dmsimardmordred: ok, and the trusted projects are defined where ? zuul.conf on the nodes ?14:50
mordreddmsimard: main.yaml14:50
mordreddmsimard: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/main.yaml14:50
mordreddmsimard: sorry - 'config-projects'14:50
dmsimardmordred: TIL about zuul/main.yaml14:50
mordreddmsimard: it's the place we tell zuul about tenants and what projects exist14:51
openstackgerritTristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: configloader: don't use path in SourceContext comparaison  https://review.openstack.org/50529314:53
tobiashmordred: thanks for this hint, I would have been run into this for sure :)14:55
openstackgerritMerged openstack-infra/zuul master: Add strip_branch_ref compat option  https://review.openstack.org/50529014:59
jlko/15:35
SpamapStobiash: I have some interest here to run against Windows test nodes too, so I may be up for helping you in a month or so.15:37
tobiashSpamapS: I'll probably have this running this or next week. Already have nodepool booting windows nodes into ready state. The zuul connection should be (hopefully) the last missing peace :)15:49
tobiashS15:51
tobiashs/peace/piece/15:52
jlkThis week I'm poking at docker being in swarm mode, so that you can ask a pile of docker hosts to "make me a container" and it'll go find a place to put one. Then get details about it to use the docker connection plugin along with docker_host info to do ansible things in the remote container.15:52
jlkwith the idea that "making the docker swarm" is outside of nodepools duties, but you just feed it a swarm endpoint (much like a cloud endpoint).15:52
SpamapStobiash: cool15:53
SpamapSjlk: I know k8s adds another level of complexity... but.. if you were to start there... I would have more reason to help. :)15:53
jlkjamielennox is thinking down that path15:54
jlkmulti-tenancy is going to be weird there, according to him15:54
SpamapSsobeit15:54
SpamapSmulti-tenancy is weird in openstack too.. since zuul has multiple tenants but nodepool still just has one project in openstack.15:55
jlksure, but consumers don't get an openstack API to play with15:55
SpamapSAFAICT, the tenants in Zuul are for visibility grouping, not access control.15:55
jlkI guess maybe the better question is, what should consumers be exposed when using nodepool with k8s?15:56
SpamapSSince all tenants can request all node labels.15:56
jlkshould they just get direct access to a container running in k8s?15:56
jlkshould they get some sort of k8s interface to launch their service?15:56
tobiashLabel restrictions for tenants is also part of my roadmap some time in future15:57
SpamapSjlk: I think the playbooks should get access to a k8s with the desired images if the job has a k8s node type.15:59
jlkI think that's the problem, if you have access to a k8s, you have access to other job's k8s containers too15:59
jlkbecause afaik k8s doesn't do multi-tenancy in that way16:00
jlkbut I'm just echoing what jamielennox was finding, so I'm likely getting it wrong16:00
SpamapSk8s has RBAC's, but you may be right, the role that can spin things up may be able to list and mess with others16:00
SpamapSjlk: actually, if the secrets for the k8s are only exposed to trusted playbooks, this shouldn't be a problem.16:03
SpamapSlet the untrusted pre playbook write out the desired pods, and trusted playbooks do the actual k8s work.16:04
*** isaacb has joined #zuul16:04
jlkthat would be ... hrm.16:05
SpamapSWell first we have to ask ourselves why we want containers.16:05
SpamapSThe answer, I think, is density.16:05
jlkresource packing.16:05
jlkspeed16:05
SpamapSSo I think it's entirely fine to suggest that if users want faster tests and more resources to run them, they give up some control.16:06
SpamapSThis isn't for testing how your thing works with k8s.16:06
SpamapSFor that, spin up a k8s in vms.16:06
jlknod16:06
SpamapSBut having it be able to consume a k8s means I can use my fairly significant sized CaaS to run a lot more tests in a Zuul here at GoDaddy.16:06
SpamapS(CaaS == Containers aaS)16:07
jlkyeah16:07
jlkso k8s is a means to a container, not a means to test k8s or k8s like deployments16:07
jlkyou won't do k8s things, you'll just consume containers out of it.16:07
*** hashar_ is now known as hashar16:08
SpamapSMostly yes.16:08
SpamapSI think we can do some k8s things, but they're mostly just docker-swarmy things running through k8s because k8s is going to be a lot more common than swarm IMO.16:08
jlkyeah, that's reasonable. That's a shortcut to using something like gcp or azure too16:08
SpamapSPrecisely.16:09
SpamapSThat said, swarm makes a lot more sense for one-off short lives things.16:09
SpamapSbecause zomg simple.16:09
jlkyeah... :)16:09
SpamapSAnd there are likely others that are somewhere in between.16:09
jlkalthough, first struggle point, get the daemon listening on the tcp port and let me remotely connect to it.16:09
jlkbahaha.  69.whatever is not the same as 169.whatever16:20
dmsimardjeblair: once the gerrit dust has settled, I wanted to ask about logstash for v3 jobs. It doesn't look like logs are sent to logstash yet so I wanted to see if it was an issue or the logstash work was not yet finished.16:24
*** isaacb has quit IRC16:45
jlklimited success!16:47
jlkansible -i hosts docker -m raw -a "executable=ash ls" -vvvv16:48
jlk<5e4f9204f794> EXEC ['/usr/local/bin/docker', u'-H', u'tcp://169.44.161.54:2375', 'exec', '-i', u'5e4f9204f794', u'/bin/sh', '-c', u'ash -c ls']16:48
jlk5e4f9204f794 | SUCCESS | rc=0 >>16:48
jlkI seem to recall that we have some static nodes in our zuul v3 configuration. Where was that at?17:06
jlkblah, I think I remembered wrong.17:27
*** olaph has quit IRC17:30
jeblairdmsimard: i left it at "maybe it works now" but haven't followed up to see if it does, or if not, what the problems are.  will probably involve checking logstash processer server logs (root) so i'll have to poke at it in a little while.17:30
jlkjeblair: at this time, we do not have support for "static" nodes, do we?17:31
mordredjeblair, dmsimard, jlk, SpamapS, Shrews, jamielennox: today is fix-gerrit-upgrade day for some of us - but it's worth noting that ansible 2.4 was released today. if anyone has time, it might be worthwhile to start figuring out what in 2.4 may have broken things for us17:31
mordredjlk: not today - there are patches proposed in nodepool, but we've been deferring dealing with them til post rollout17:32
*** olaph has joined #zuul17:32
mordredjlk, SpamapS: I'd love to chat about the swarm/docker/k8s stuff once gerrit is less on fire17:33
jlksounds good.17:34
jeblairalso, swarm/docker/k8s stuff is on our post-transition roadmap, but it's not exactly at the top of the list.  i'd love to talk about it too, but i'd also like to get v3 out the door first.17:34
jlkagreed.17:35
jlkthis is "scratching a personal interest" not a "lets shove this in"17:35
jeblairwhew, cool :)17:35
jeblair(and i realize the status of the post-transition roadmap is "jeblair is to write it up with more words and send it out for discussion"; that's probably not going to happen today though :)17:36
jlkalso it's kind of "spinning my wheels until gerrit fire is out and we have a roadmap" thing ;)17:37
jeblair++17:37
*** electrofelix has quit IRC17:48
*** harlowja has joined #zuul17:49
Shrewsmordred: yeah, i was going through the 2.4 change list earlier. didn't see anything obvious.18:01
Shrewsnot sure how complete that list was (internal release announcement)18:02
* Shrews will poke further18:02
ShrewsCHANGELOG.md is likely more complete18:04
mordredShrews: I was also thinking that proposing a patch to bump the ansible requirement might break some tests18:08
mordredShrews: AND - we should probably audit the plugins dirs in the new release to make sure that a) we have specific action/filter plugin things for any new action/filter plugins that we want to block - and b) that we add the new plugin types to the list of directories to look for in blocking people from adding plugins to their in-repo content18:10
Shrewsmordred: yep. already on #118:14
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul feature/zuulv3: DNM: Test Ansible 2.4  https://review.openstack.org/50535418:15
Shrewsthere is a new stderr callback18:16
SpamapSjeblair: for me it may be "the only way I get to play with zuul" so it may raise in priority for me. ;)18:21
jeblairSpamapS: i understand, though i hope that as a team we can find a balance so that we get fundamental work done.  if we aren't able to actually put together a basic functioning v3 that other folks can use, then no one will get to work on it :(18:26
jeblairwe need to be in a place where this isn't an either/or choice -- that we have a roadmap and we all agree on it, and are all working in parallel18:27
jeblairi just don't think we're there yet, nor will we be until we have things like new-user documentation, or first-class github support18:28
openstackgerritTimo Tijhof proposed openstack-infra/zuul master: Status: Remove use of deprecated jQuery jqXHR `complete` method  https://review.openstack.org/50536618:42
openstackgerritTimo Tijhof proposed openstack-infra/zuul master: Status: Don't toggle panel when clicking patch link  https://review.openstack.org/50536818:46
jeblairthese ^ would be good to port to v3 if anyone has a moment18:46
Shrewsjeblair: i'll grab 'em18:49
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Status: Remove use of deprecated jQuery jqXHR `complete` method  https://review.openstack.org/50536918:53
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Status: Don't toggle panel when clicking patch link  https://review.openstack.org/50537018:53
dmsimardjeblair: re: logs -- I think we'll need to troubleshoot things, yes. Searching a build_change after a review that is only zuul v3 does not pull up anything.19:00
dmsimardmordred: 2.4 -- I had to cut a dot release for ara to pin Ansible<2.4. I was optimistic that I could get it fixed in time but nope.19:01
dmsimardI know what the first problem is and know how to fix it, just haven't got around to it yet. I don't know yet if there are any problems hidden underneath.19:02
mordreddmsimard: cool - sounds like something we should be able to fix fairly easily once we get a second19:02
dmsimardI suppose we might want to rebase our forks of the different modules/plugins we have in zuul v3 at the very least19:03
openstackgerritMerged openstack-infra/zuul master: Status: Remove use of deprecated jQuery jqXHR `complete` method  https://review.openstack.org/50536619:04
openstackgerritMerged openstack-infra/zuul master: Status: Don't toggle panel when clicking patch link  https://review.openstack.org/50536819:04
jlkYeah I kind of wonder if wouldn't make sense to maintain a patch file (git format-patch) of our changes to the plugins/modules/etc we patch from Ansible, so that we can more easily port those changes to newer versions.19:06
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Use publish-docs-draft base job for docs-draft publishers  https://review.openstack.org/50462419:12
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Removed unused 'status: ' string from log line  https://review.openstack.org/50537819:12
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Emit shell instead of script tasks  https://review.openstack.org/50537919:12
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Omit some jobs from shared queue calculation  https://review.openstack.org/50538019:12
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Status: Remove use of deprecated jQuery jqXHR `complete` method  https://review.openstack.org/50536919:14
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Status: Don't toggle panel when clicking patch link  https://review.openstack.org/50537019:14
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Deal with link-logs macro  https://review.openstack.org/50538719:38
mordredjlk: yah - although I'm _hoping_ I can get time to get stuff upstreamed during 2.5 and maybe stop having a local fork20:00
jlkIf nobody is working on porting the patches forward, I can take a peek.20:01
jlkShrews: are you forward patching things?20:01
mordredjlk: I may take you up on that - step one is I need to write a spec20:01
mordredthat summarizes the stuff we talked about in london and proposes a plan for it20:02
Shrewsjlk: just the two above20:02
jlkyeah I wasn't talking about taking them upstream, just keeping our patched versions updated with ansible release20:02
Shrewsjlk: i was going to look at the ansible module diffs next, but getting close to EOD for me if you want to do that20:02
jlkokay, I've got some work hours yet, so I'll take a few on.20:03
*** jesusaur has quit IRC20:08
*** jesusaur has joined #zuul20:11
*** jkilpatr has quit IRC20:18
mordredShrews, jlk: have fun with those! that's our lovely log-streaming hacks20:21
jlkShrews: dmsimard: mordred: et al, lets coordinate https://etherpad.openstack.org/p/zuul-ansible-2.420:23
pabelanger+120:33
*** jkilpatr has joined #zuul20:42
jlkwhich version of Ansible have we been using?20:55
jlkv2.3.2.0-1 ?20:55
mordredyah. whatever the latest 2.3 is20:56
mordredthere's a version cap in zuul's requirements.txt20:56
openstackgerritMonty Taylor proposed openstack-infra/zuul-sphinx master: Update exception message to include directories  https://review.openstack.org/50540020:58
jlkyeah a cap, but I wasn't sure if there was a upper-limits coming into play somewhere20:59
jlkhrm.20:59
*** dkranz has quit IRC21:01
jlkoh, I get it now21:01
jlksome of our plugins are doing magic to import the upstream plugin, then we do some extra stuff and eventually super call the upstream plugin.21:01
mordredjlk: yah. that should be _most_ of them21:31
jlkdone digging through lookups/plugins/modules in our tree. A few could use some updates, and a few need a a fair amount of inspection.21:31
jlkgoing afk for a bit21:31
mordredcool21:31
mordredjlk: thanks!21:31
jlkde nada21:32
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Add disabled network action plugins for 2.4  https://review.openstack.org/50541921:47
*** hashar has quit IRC21:59
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Disable action and lookup plugins from 2.4  https://review.openstack.org/50541922:03
mordredjlk, Shrews: ^^ that should be all the new stuff22:04
mordred(the new added plugins, not the stuff jlk already has listed)22:05
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Port in changes from ansible 2.4 command module  https://review.openstack.org/50543022:29
mordredShrews, jlk: ^^ I stole command - there's no need to assault anybody else with ugly mordred hacks22:30
*** openstackgerrit has quit IRC22:47
jlkmordred: LOL, no worries.23:14
jlkmordred: did you mean to add .pyi files?23:20
jlkoh hrm, I guess so.23:24
jlkreading when one was added, that's... gross23:24
clarkbthats how you do type identification righ?23:31
jlkyeah, but these are blank files23:32
pabelangerokay, I have stopped nl02.o.o in preperation for landing our new nodepool-launcher configuration in the morning23:49
pabelangerdmsimard: do you still need access to the held nodes in rax-ord?23:51
dmsimardNo, told clarkb to release them a while back. They were helpful in understanding the multi node issues.23:51
pabelangerk23:51
*** openstackgerrit has joined #zuul23:54
openstackgerritTristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /tenants route  https://review.openstack.org/50326823:54
openstackgerritTristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/status route  https://review.openstack.org/50326923:55
openstackgerritTristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs route  https://review.openstack.org/50327023:55
openstackgerritTristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/builds route  https://review.openstack.org/46656123:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!