Thursday, 2018-08-30

*** jimi|ansible has quit IRC00:48
openstackgerritDavid Moreau Simard proposed openstack-infra/nodepool master: WIP: Implement an Amazon EC2 driver  https://review.openstack.org/53555801:18
SpamapSso, before I solve it in an ugly way.. has anybody solved /usr/bin/python missing in an elegant way?01:18
dmsimardSpamapS: on a remote node ?01:18
SpamapSYes.. I want to use a stock image that doesn't have python2 installed01:19
dmsimardSpamapS: https://docs.ansible.com/ansible/2.6/modules/raw_module.html01:19
SpamapSand Ansible has really screwed this up IMO (they really should try /usr/bin/python3 first and then fall back to /usr/bin/python)01:19
SpamapSdmsimard: that won't work for zuul01:19
SpamapSzuul starts by doing something with -m setup01:19
SpamapSwhich 'splodes because of missing /usr/bin/python01:19
dmsimardSpamapS: oh no01:19
dmsimardWe can fix that01:20
SpamapSWell, I was just going to go ahead and set ansible_python_interpreter somewhere and make sure it gets set on every invocation01:21
SpamapSOne way to solve this is to make sure the job variables and site variables get plumbed through to that call to setup01:21
dmsimardSpamapS: I wonder if you could boot with user-data and make something install before opening up ssh01:21
dmsimardSpamapS: s/install/run/ .. like ln -s /usr/bin/python3 /usr/bin/python or something01:22
dmsimardSpamapS: dunno, there's probably a lot of ways to work around it01:22
clarkbspamaps ya ansible should do more imo01:23
*** jiapei has joined #zuul01:23
SpamapSdmsimard: yeah that's not a bad idea01:23
SpamapSbut ... ansible.. totally blowing it IMO01:23
SpamapSand ... I'd submit a patch.. but... wow.. they seem busy over there01:24
dmsimardSpamapS: not disagreeing, trying to look through their issues if they have something01:24
dmsimardSpamapS: is that a stock AMI on ec2 ?01:27
SpamapSI mean I don't even need userdata, I can just snapshot an instance after doing that and use it. But... yeah.01:27
SpamapSdmsimard: yeah, ubuntu 16.04 and 18.04 both ship without /usr/bin/python01:27
dmsimardSpamapS: ok, let me ping around01:27
SpamapSI'm just making my own AMI right now01:27
SpamapSby snapping one that's already booted with the symlink01:28
SpamapSbut.. seems like we should make zuul figure this out on its own.01:28
SpamapSAnd ultimately, ansible.01:28
dmsimardSpamapS: I'm not sure what's the big deal in making Ansible attempt to use /usr/bin/python3 (or /usr/bin/python2) if /usr/bin/python is not present...01:29
clarkbfwiw chances are you want to build your own images anyway? but ya ansible gives up really quivk imo01:29
SpamapSEven if I do build my own01:30
SpamapSI want to use python301:30
SpamapSalways01:30
SpamapSand /usr/bin/python != python301:30
clarkbya zuul uses python2 for some reason that may not be necessary anymore01:30
SpamapSThat's already pretty much been stipulated by the python world. :-P01:30
clarkbI forget the reason though01:30
SpamapSI don't see zuul setting it at all01:30
SpamapSIt's just the default value01:31
SpamapS/usr/bin/python01:31
SpamapSand that's fine.. I kind of feel like zuul should make it easy to set ansible_python_interpreter per job or label.01:31
SpamapSIn fact I had hoped vars: { ansible_python_interpeter: /usr/bin/python3 }  would work01:32
SpamapSbut it doesn't, because we don't set job vars on that -m setup run01:32
clarkbhttps://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py#n150901:32
clarkbit sets it there at least01:32
SpamapSAh that's very interesting01:32
clarkbgit log/blame on that line will likely say why01:32
clarkbI know there was a reason though01:32
SpamapSwell then I'll just install python2 and deal with it01:33
dmsimardSpamapS: added a comment in https://github.com/ansible/ansible/issues/19605#issuecomment-41716081101:35
dmsimardworth a shot :)01:36
*** rlandy|bbl is now known as rlandy01:37
*** rlandy has quit IRC01:42
SpamapSdmsimard: awsome thanks01:50
*** bhavikdbavishi has joined #zuul02:54
*** jiapei has quit IRC03:33
*** bhavikdbavishi has quit IRC03:34
*** snapiri has joined #zuul05:59
*** openstackgerrit has quit IRC06:07
*** pcaruana has joined #zuul07:14
*** goern has joined #zuul07:19
*** hashar has joined #zuul07:35
*** hwoarang has joined #zuul07:49
*** jpena|off is now known as jpena07:54
*** electrofelix has joined #zuul08:21
*** panda is now known as panda|rover08:52
*** jesusaur has quit IRC09:27
*** jesusaur has joined #zuul09:34
*** hashar is now known as hasharLunch10:11
*** hwoarang has quit IRC11:20
*** hwoarang has joined #zuul11:20
*** hasharLunch is now known as hashar11:21
*** jpena is now known as jpena|lunch11:27
*** sshnaidm is now known as sshnaidm|afk11:28
*** threestrands has quit IRC11:38
*** TheAceOfHearts has joined #zuul11:41
*** hwoarang has quit IRC11:46
*** hwoarang has joined #zuul11:46
*** gouthamr has quit IRC11:49
*** dmellado has quit IRC11:50
*** openstackgerrit has joined #zuul12:09
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add config-errors notifications drawer  https://review.openstack.org/59714712:09
*** jpena|lunch is now known as jpena|off12:31
*** jpena|off is now known as jpena12:32
*** rlandy has joined #zuul12:37
*** myoung|pto is now known as moyung13:03
*** moyung is now known as myoung13:03
openstackgerritSimon Westphahl proposed openstack-infra/nodepool master: Update static nodes in Zookeeper on config change  https://review.openstack.org/59815613:11
dmsimardtristanC, SpamapS: re: ec2 driver, I sent PS11/12 which adds support for profile selection (i.e, something other than [default] in .aws/credentials), works around assumptions around block device mapping and otherwise tries to bring the driver more in line with what the openstack driver looks like13:13
dmsimardthe required config looks a bit different now and I've updated the fixture to reflect that https://review.openstack.org/#/c/535558/10..12/nodepool/tests/fixtures/config_validate/good.yaml13:14
openstackgerritSimon Westphahl proposed openstack-infra/nodepool master: Update static nodes in Zookeeper on config change  https://review.openstack.org/59815613:18
swestShrews: tristanC: ^ I made a proposal for updating static nodes in Zookeeper if the config changed. Would be great if you could give some feedback13:21
Shrewsswest: left a comment13:27
tristanCShrews: swest: left a comment too, i wonder why static config change aren't handled by the config object __eq__ checks instead?13:28
tristanCdmsimard: works for me. I guess we now need to add tests for getting approved in nodepool?13:30
dmsimardtristanC: there's still a lot of work, I'll add a list of todo in the commit message13:31
dmsimardtristanC: quota management is one, yes -- but also diskimage, processes like deleting leaked nodes, etc13:32
tristanCdmsimard: couldn't these be added as follow-up?13:33
*** samccann has joined #zuul13:35
dmsimardtristanC: I think quota management and diskimage management might be appropriate for a second iteration but things like deleting leaked nodes aren't13:37
dmsimardtristanC: I also suck at writing tests so I hope someone can help with that13:37
dmsimardSpamapS, clarkb: #ansible-devel confirmed that they would also like Ansible to attempt to use something other than /usr/bin/python if it's not there. They've already given a few shots at it but were not happy with the implementations, it's likely to be in the 2.8 roadmap13:41
*** toabctl has joined #zuul13:44
swesttristanC: not quite sure what you mean with "... handled by the config object __eq__ checks instead", since the node config is a dict.13:51
tristanCswest: for openstack, the provider is removed if the config eq failed, e.g. https://git.zuul-ci.org/cgit/nodepool/tree/nodepool/provider_manager.py#n5513:52
tristanCand there: https://git.zuul-ci.org/cgit/nodepool/tree/nodepool/driver/openstack/config.py#n9213:53
tristanCthat's how labels setting update get applied during reconfiguration13:54
swesttristanC: I think the provider manager and __eq__ check is fine. Adding a new pool also works. The problem is that if the config changes and a new provider is created the existing static nodes are not updated.14:03
*** jimi|ansible has joined #zuul14:08
*** sshnaidm|afk has quit IRC14:10
*** sshnaidm|afk has joined #zuul14:19
*** sshnaidm|afk is now known as sshnaidm14:20
*** sdoran has joined #zuul15:03
*** pcaruana has quit IRC15:12
openstackgerritSorin Sbarnea proposed openstack-infra/zuul master: Display correct cursor when hovering over patchset header  https://review.openstack.org/59822216:12
*** shachar has joined #zuul16:12
*** snapiri has quit IRC16:14
*** jimi|ansible has quit IRC16:16
*** rlandy is now known as rlandy|afk16:19
*** gouthamr has joined #zuul16:49
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Provide per-project ssh key to executor  https://review.openstack.org/59772716:51
SpamapSdmsimard: thanks for the follow-up!17:23
SpamapSdmsimard: I'll pull in ,12 today17:24
*** jpena is now known as jpena|away17:27
dmsimardSpamapS: neat, let me know if you run into anything -- it's nowhere finished yet17:30
SpamapSdmsimard: would be nice to get something merged soon17:41
dmsimardagreed but I don't think it matches the quality of something that can be released17:41
dmsimard(yet)17:41
*** dmellado has joined #zuul17:44
SpamapSdmsimard: It's beta-or-so17:45
SpamapS;)17:45
SpamapSI'd really also like to figure out how we can leverage nodepool-builder in AWS17:45
SpamapSI was looking at the Denver PTG schedule17:47
SpamapSand I didn't see zuul explicitly mentioned17:47
dmsimardbelieve fungi or clarkb were looking at putting a pad together17:48
clarkbSpamapS: zuul decided to not have an official on calendar presence at the PTG aiui17:48
SpamapSmmk17:48
clarkbbut I'm sure there will be carvable corners for such things17:49
clarkbas for AWS support by nodepool builder I Think dib can already build amis (but may need better docs around that), then its just a matter of having the builder speak the upload api?17:49
*** electrofelix has quit IRC17:50
dmsimardSpamapS: I don't have time to spend on nodepool-builder/diskimage support in the short term unfortunately (really tight time constraints)17:50
SpamapSdmsimard: yeah I was looking into it yesterday17:50
SpamapSthen gave up and just snapshotted an instance and used that ami17:50
dmsimardhaha17:50
dmsimardyeah it's a non-negligible amount of work that I don't have the bandwidth -- our image builds are just done out of band right now (or we pick available AMIs).. it works for the time being so I'll circle around back to it later if I can17:51
dmsimardI don't even know what's special about AMIs vs regular qcow2/raw images we're building today17:52
clarkbdmsimard: the disk and kenrel are separate iirc, otherwise not much17:52
*** hashar is now known as hasharDinner18:00
dmsimardI am tagging ARA 0.16.0 for release, no problems spotted in automated and manual testing18:09
SpamapSYeah there's a few scripts out there that will slurp the boot bits out of a qcow2 and make an ami18:35
*** toabctl has quit IRC18:51
*** rlandy|afk is now known as rlandy19:05
*** toabctl has joined #zuul19:07
SpamapSsilly question: how do I tie a template to every single project?19:11
dmsimardSpamapS: there was a regex feature that Tobias was working on, not sure if it landed19:15
dmsimardSpamapS: looks like it has https://review.openstack.org/#/c/535713/19:15
SpamapSdmsimard: oh that's perfect19:17
clarkbSpamapS: https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n4 an example of it in use19:20
SpamapSclarkb: yeah that's exactly what I need19:21
SpamapSI have a silly github-only pipeline that turns approved reviews into labels so that people can use labels for self-approve... but up until now I was just having to hang it off each project.19:21
*** jpena|away is now known as jpena|off19:38
*** hasharDinner is now known as hashar19:56
*** mugsie has quit IRC20:25
*** TheAceOfHearts has quit IRC20:35
*** mugsie has joined #zuul20:39
*** harlowja has joined #zuul20:43
*** samccann has quit IRC21:11
openstackgerritSorin Sbarnea proposed openstack-infra/zuul master: Display correct cursor when hovering over patchset header  https://review.openstack.org/59822221:13
ssbarneaif anyone can approve that one liner ^^ css fix it would be great. I tested using styler and works well.21:16
ssbarnearlandy:if you can review https://review.openstack.org/#/c/571176/ it would be great, is one line, and zero risk.21:20
rlandyvoted21:22
ssbarneathanks21:22
*** dmellado has quit IRC21:31
*** dmellado has joined #zuul21:31
dkehnWe have and instance of zuul running. The zuul-web client is serving on port 9000. We can see the tenant page in a browser. When we point a github webhook to http://<hostname>:9000/api/connection/github we don anything on the server side logs for zuul nor does the payload deliver correctly from github, any advise?'t see21:38
pabelangerhttps://zuul-ci.org/docs/zuul/admin/github_setup.html says it is http://<IP ADDRESS>:9000/api/connection/github/payload21:40
pabelangerassuming github is the connection name21:40
dkehnsorry I forgot the word payload21:42
dkehnand we have DNS, I'm assuming that will work or must it be IP_ADDRESS21:43
pabelangerdkehn: and zuul-web is running?21:44
dkehnyes21:44
*** dkehn is now known as dkehn_21:46
*** myoung has quit IRC21:47
*** dkehn has joined #zuul21:49
pabelangerdkehn_: what connection name did you use for github.com21:50
dkehnpabelanger: yes21:51
dkehnpabelanger: sorry had to switch to different system21:51
pabelangernp, I was asking what was the name use setup in zuul.conf for github.com connection21:52
pabelanger[connection github]21:52
dkehn[connection github]21:54
dkehnsshkey = /var/lib/zuul/.ssh/id_rsa21:54
dkehndriver = github21:54
dkehnapi_token = <secret>21:54
dkehnwebhook_token = <secret>21:54
dkehngit_host = github.com21:54
pabelangerand 9000 is open in on firewall?21:55
dkehnyes its open21:56
pabelangeryou should be able to use curl --request POST with some dummy json to the URL and see the request in zuul debug logs21:56
dkehniptables confirmed21:56
dkehntrying21:57
*** TheAceOfHearts has joined #zuul21:59
dkehnwe seem to be getting a 400 Bad Request21:59
pabelangerfor example21:59
pabelangerhttp://zuul.openstack.org/api/connection/github/payload22:00
pabelangeris what we have in openstack-infra I believe22:00
pabelangerbut, because it doesn't respond to GET, get 50022:00
pabelangerdkehn: check debug logs in zuul-web22:00
pabelangerI have to run now for some food, will catch up on backscroll22:00
pabelangergood luck22:01
*** TheAceOfHearts has quit IRC22:01
dkehnwe can't seem to get zuul-web logging22:01
*** TheAceOf_ has joined #zuul22:01
*** TheAceOf_ is now known as TheAceOfHearts22:04
corvusdkehn: another useful resource is in github, you can see the response github receives to its payload requests22:07
dkehncorvus: thanks22:12
*** hashar has quit IRC22:16
*** sshnaidm is now known as sshnaidm|off22:35
SpamapShm22:35
SpamapSI think something changed in the github driver subtly (probably in release notes I know). I have a pipeline that tags stuff on approved reviews, but it won't even admit a change into its queue until the change satisfies Github's "can this merge" criteria (it's missing required checks). I'd like to disable that.. anybody know how?22:36
corvusSpamapS: i take it that pipeline doesn't run the checks (ie, this isn't gate, this is something else?)22:39
SpamapScorvus: http://paste.openstack.org/show/729189/22:40
SpamapSthat's the entirety of the pipeline22:40
SpamapSAnd then I have a template that attaches a noop in that pipeline to every project22:40
SpamapS(It has an unlabel-on-push counterpart too)22:41
corvusSpamapS: is it 'dependent' for a reason?22:41
SpamapScorvus: no reason I can think of22:41
SpamapSI probably just thought "seems legit"22:41
corvusSpamapS: i think independent doesn't perform the mergability check, so if you switch that you should be good22:41
SpamapSyeah that makes sense22:41
openstackgerritMerged openstack-infra/zuul master: Add missing parentheses to title invocation  https://review.openstack.org/59536122:48
*** TheAceOfHearts is now known as TheAceOfHearts_23:22
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: Add guidelines on variable naming  https://review.openstack.org/59592023:28
dmsimardtristanC: you know what ?23:32
tristanCSpamapS: if adding ec2 to nodepool tree is possible, how are we going to manage its requirements? shouldn't we use "boto3 [aws]" extra flag?23:32
clarkbtristanC: is there a reason to not just alwaysinstall boto if nodepool supports ec2?23:34
dmsimardtristanC: writing all this python code with boto and the nodepool business logic kind of sucks tbh. I wonder if we could just use Ansible as the "driver" ? Ansible has modules to create VMs in AWS idempotently and manage them. It's complicated and they got it all figured out.23:35
tristanCclarkb: to reduce complexicity. aws usere shouldn't have to pull openstack sdk and vice-versa23:35
clarkbtristanC: I mean I pip install and don't think about it typically23:35
clarkb(even less thinking if using the pbrx images)23:36
dmsimardtristanC: it's kind of the same question that comes up in packaging... Like for example for Glance/Cinder/Neutron, we don't install all the backend packages23:37
dmsimardDistributions don't typically install everything but they might have a sane default, though23:37
tristanCclarkb: works for me too :)23:38
clarkbdmsimard: there has been talk of potentially supporting a generic ansible provisioner then if ansible supports it so would nodepool23:38
tristanCdmsimard: actually, the nodepool package in fedora has a sub-package for openstack that requires openstacksdk23:38
clarkbdmsimard: the trouble comes when you need to cleanup leaked resources as with openstack (unsure if you run into similar problems with aws) and need more business logic in nodepool itself23:39
dmsimardclarkb: exactly, that's the point, then we don't need to write all the provisioning logic23:39
clarkbI know gce suffers the same issues23:39
dmsimardYeah.. it's not perfect but it's fixable23:39
tristanCdmsimard: the issue with using ansible is speed, using a sdk seems more efficient for nodepool23:39
dmsimardtristanC: I'm not sure the tradeoff in performance vs maintability/feature is worth the hassle23:40
clarkbI'm personally a fan of the proper sdk implementations myself just based on the amount of work it takes to make openstack clouds happy and hearing similar problems from users of other clouds23:40
clarkbbasically ansible will cut it for when you boot 2 VMs a day, not 25k23:40
dmsimardReally? Why?23:40
clarkbbecause you leak all the things, ports, volumes, floating IPs23:41
clarkband you need to have software that manages that23:41
SpamapSIt's a neat idea to make a generic ansible driver.....23:41
dmsimardThat's the purpose of nodepool, no ?23:41
dmsimardI'm not saying let's get rid of nodepool23:41
SpamapSbut as we go more toward "thing pool" and less "node pool", I think native drivers are going to make more sense.23:42
dmsimardI'm saying nodepool should use ansible to provision things instead of writing code for that23:42
clarkbdmsimard: yup, I just don't know how you'd effectively do that with ansible in the middle? ansible list all ports, ok now show all of them and figure out which ones are active and attached to valid instance, Now go back through and delete the rest. Expressing that in python is far easier23:42
dmsimardThe aws modules are like 1k lines long each and there's dozens DOZENS23:42
SpamapSdmsimard: yeah I don't agree and I think that's going to end up being just as hard as writing good drivers.23:42
clarkbbasically its easier to express the lifecycle and associated exceptions in a proper programming language23:42
SpamapS^^ that23:43
dmsimardclarkb: the leaking cleanup doesn't have to be ansible23:43
SpamapSand aws's resource based API is pretty close to shade.23:43
tristanCsdk also improve connection pool and keep a single authenticated session23:43
SpamapS(the low level one is just as awful as low level openstack)23:43
dmsimardThe tradeoff would be to be able to support all the things Ansible supports23:43
dmsimardPros and cons :)23:44
clarkbthe associated tradeoff is the support is only as good as ansibles which ime is honestly not always that great (for aws its prbably fine)23:44
SpamapSI think the struggle right now is unwinding the assumptions we've been able to build up with just two very different drivers (static and openstack)23:44
SpamapSdmsimard: but that' a falacy, because we will still have to write ansible playbooks that work the way your playbook driver expects, and some ansible modules are nothing like others.23:45
clarkbin particular backward incompatible changes show up every release and a lot of the provisioning modules do not have stability guaruntees like the "core" of ansible last I checked23:45
SpamapSIn fact frankly, I don't think Ansible is very good as a provisioner.23:45
SpamapSI'd choose terraform before I'd choose Ansible, for that.23:45
SpamapSbut then I look at that and I think.. for this narrow use case.. 1000 lines of python is probably better than 1000 lines of ansible.23:46
dmsimardSpamapS: sure23:46
clarkb(looks like ec2 is makred stableinterface at least)23:46
dmsimardRight tool for the right job23:46
dmsimardIf terraform is better, let's use it :D23:46
dmsimard(I've been meaning to try it)23:46
dmsimardI know there's an Ansible module for terraform but that's about it23:47
clarkb(azure is preview)23:47
SpamapSIt's called command: ;-)23:48
*** TheAceOfHearts_ has quit IRC23:48
* SpamapS isn't a fan of ansible modules that just wrap CLI tools that have perfectly reasonable CLI's23:49
SpamapSIf your command sucks at idempotency, ok, wrap it.23:49
clarkbas a side note, the openstack modules are basically nodepool's grandchildren (nodepool -> shade -> os_* mdules in ansible). At least in that case we had to do the work in nodepool first and there isn't much incentive to add more layers to it in nodepool23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!