Tuesday, 2017-06-27

rbergeronjlk: ay. "words" but i have to go take a package to ups before i get charged 800 dollars for not taking the package00:10
rbergeronlike, i have a bunch of words -- but also they are related to v3 and not v4 of the api and ... that whole thing is a separate00:10
rbergeronum00:10
rbergeronevent00:10
rbergeronbbiaw00:11
rbergeronlol00:11
jlkshoo00:11
rbergeroni love shoes!00:11
jlkSQUIRREL!00:11
*** xinliang_ has joined #zuul03:40
xinliang_hi guys, I am running zuul 2.5.2 on arm64 ubuntu 16.04. I found that the zuul can read the gerrit event but it can't trigger the job. Any clue or how to debug this?03:44
xinliang_I can't find any useful log in /var/log/zuul/03:44
tristanCxinliang_: could you share the layout.yaml ?03:45
xinliang_the job status is : build:noop-check-communication:debian-jessie    0       0       103:45
tristanCxinliang_: oh, and is zuul-merger working too?03:46
xinliang_tristanC: sure.  here is the layout.yml: https://github.com/xin3liang/project-config-linaro-ci/blob/master/zuul/layout.yaml03:47
tristanCxinliang_: does the noop-check-communication job exists?03:48
xinliang_tristanC: yes,  service zuul-merger status says it is running.03:49
xinliang_The job noop-check-communication, I can see it the jenkins web page and I can run it manually03:49
xinliang_it can run success03:50
tristanCxinliang_: hum... is the job stuck in "queued" ?03:52
xinliang_jenkins's queue? no it can run success03:52
tristanCxinliang_: i mean in zuul status03:53
xinliang_i add a patch to sandbox project and add recheck comment it has none response. https://review.openstack.org/#/c/477442/103:53
xinliang_tristanC: how to check the zuul queue status?03:54
tristanCxinliang_: zuul-server has a webapp, listening on port 8001 by default, you can query its status by running curl http://localhost:8001/status.json | python -mjson.tool03:56
xinliang_tristanC: it stuck in fetch the jason file. But 8001 port is listen. http://paste.openstack.org/show/613768/04:00
xinliang_here is the output of echo 'status' | nc 127.0.0.1 4730 -w 1 :http://paste.openstack.org/show/613766/04:00
xinliang_one strange thing is that i can telnet to gearman server it is  stucked when running 'telnet  127.0.0.1 4730'04:02
tristanCxinliang_: to quit telnet, you may ^], then type quit04:09
tristanCxinliang_: without the status it's hard to tell what's going on, perhaps the zuul.log may help04:10
xinliang_tristanC: I mean I can't telnet to  the gearman server04:10
tristanCthat's odd, well using nc is fine too04:11
xinliang_tristanC: here is some of  zuul.log: http://paste.openstack.org/show/613772/04:14
tristanCxinliang_: the relevant part starts with "zuul.GerritSource: Updating <Change 0x..., review-number>", a complete run looks like: http://paste.openstack.org/show/613774/04:17
xinliang_So none of "GerritSource" in my zuul.log04:21
xinliang_zuul is not running properly, I will check the config again04:23
xinliang_tristanC: thanks :-)04:23
tristanCxinliang_: you're welcome :-)04:27
*** jlk` has joined #zuul05:12
*** jlk has quit IRC05:13
*** EmilienM has quit IRC05:13
*** EmilienM has joined #zuul05:22
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Support custom playbook locations  https://review.openstack.org/47767205:34
*** ianw_ has joined #zuul05:36
*** ianw has quit IRC05:39
*** ianw_ is now known as ianw05:39
*** isaacb has joined #zuul05:39
*** isaacb has quit IRC06:14
*** isaacb has joined #zuul06:27
*** jeblair has quit IRC07:08
*** jeblair has joined #zuul07:09
xinliang_tristanC: now I can get the status of zuul: http://paste.openstack.org/show/613785/07:20
tristanCxinliang_: nice, then you should try to recheck the test change and observe zuul.log and the status07:22
xinliang_still no response to the gerrit event, it seems that zuul-merger is not working07:23
tristanCxinliang_: though, iirc, zuul-server should queue the change before asking zuul-merger to stage the change07:26
tristanCwhen a change is stuck in queued, it either mean it's waiting for the merger, or because there are no executor ready to run the job07:27
xinliang_i can see something like: zuul.GerritSource: Updating <Change 0xffff30f24790 470823,3>  but no other more07:27
xinliang_tristanC: how to check executor then07:28
tristanCxinliang_: for jenkins, the job node label needs to be available07:29
xinliang_it is idle now. it should be available07:30
tristanCon the jenkins job configuration page, at the "Label Expression" section it should say something like "07:31
tristanCLabel is serviced by 1 node"07:31
xinliang_tristanC: can't find "Label Expression" section07:57
*** isaacb has quit IRC07:59
tristanCxinliang_: did you use a "node" parameter for the job?08:00
xinliang_yes, i can see08:01
*** isaacb has joined #zuul08:02
xinliang_tristanC: I find the string: "Label debian-jessie is serviced by 1 node"08:03
xinliang_in the job config08:03
xinliang_i see the that merger is wait here:  job = self.worker.getJob()08:07
*** isaacb has quit IRC08:09
*** isaacb has joined #zuul08:15
*** hashar has joined #zuul08:20
*** smyers has quit IRC09:34
*** mgagne has quit IRC09:35
*** smyers has joined #zuul09:35
*** mgagne has joined #zuul09:37
*** mgagne is now known as Guest2879609:37
xinliang_tristanC: there might be problem with my executor, i found that the jenkins service status is exited.09:58
xinliang_with warning:  http://paste.openstack.org/show/613814/09:59
xinliang_there are execptions on /var/log/jenkins/jenkins.log: http://paste.openstack.org/show/613815/10:00
xinliang_I should fix these problems10:01
*** jkilpatr has joined #zuul10:57
*** dkranz has joined #zuul12:59
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: DNM crappy followup patch  https://review.openstack.org/47814413:12
*** maxamillion has joined #zuul13:26
maxamillionrbergeron: o/13:26
Shrewsjeblair: mordred's patch there in 478144 ^^^ demonstrates that the websocket problem is likely autobahn related, as I also had suspected.13:39
Shrewsmy vote is to move away from autobahn13:39
*** isaacb has quit IRC13:55
*** isaacb has joined #zuul14:13
*** hashar has quit IRC14:31
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Replace autobahn with aiohttp for websockets  https://review.openstack.org/47814414:38
mordredShrews, jeblair: there is a new version of that patch that's more complete and correct - I definitely agree with Shrews about moving away from autobahn - there is too much magic it's doing that we do not need - the code with aiohttp is much more direct and to the point and it allows us to do the things we want to do14:40
dmsimardautobahn suddenly made me remember https://github.com/auchenberg/volkswagen14:41
* dmsimard goes back to his cave14:41
mordreddmsimard: hahahahahaa14:41
jeblairmordred: what was the reason you picked autobahn over aiohttp earlier?14:41
dmsimardmordred: oh man, zuul is totally not implemented in volkswagen that's a selling point.14:42
mordredjeblair: I'm not sure I picked it over aiohttp per-se as much as I picked it because it seemed to be a thing that had a mode that would work without eventlet or twisted14:43
mordredjeblair: it was more "this library works and is maintained" - I do not remember if I considered aiohttp at that time14:43
jeblairok.  aiohttp sounds good to me.14:44
mordredjeblair: yah - also - turns out this also answers the question "can you run aiohttp in its own thread"14:44
mordredjeblair: since, you know, that's now the reason that aiohttp is better than autobahn :)14:44
clarkbautobahn works with python2 and 3, and is actively maintained and I think is considered the golden standard for websockets in python land14:47
mordredyah - that's also likely why I started there - the python2 and python3 support14:48
mordredShrews: want me to just squash the two patches?14:49
jeblairautobahn supports python2?  what about the websocket stuff requires python3?14:49
Shrewsmordred: yes plz14:50
mordredjeblair: eventlet14:50
jeblairah14:50
mordredjeblair: it supports python2 via eventlet or twisted14:50
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming  https://review.openstack.org/46335314:52
mordredShrews: done14:52
Shrewsmordred: awesome. now i can rework the test. thx14:52
*** isaacb has quit IRC15:12
*** hashar has joined #zuul15:22
clarkbShrews: jeblair https://review.openstack.org/#/c/476677/1 is the parent of https://review.openstack.org/#/c/476678/1 which you ahve reviewed and approved but can't merge until the parent is reviewed and approved too. Would be great if you can take a look at the parent when you get a chance too17:11
clarkbjeblair: similar situation with https://review.openstack.org/#/c/476266/1 being the parent of the cleanup failed upload records change17:12
jeblairclarkb: done!17:27
clarkbtyty17:28
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Use private attribute lookup interanlly in builder.py  https://review.openstack.org/47667717:30
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Don't join image upload workers on stop()  https://review.openstack.org/47667817:30
openstackgerritMerged openstack-infra/nodepool master: Disable CleanupWorker thread for test_image_upload_fail  https://review.openstack.org/47626617:31
openstackgerritMerged openstack-infra/nodepool master: Cleanup failed upload records  https://review.openstack.org/47622317:31
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming  https://review.openstack.org/46335317:34
Shrewsjeblair: https://review.openstack.org/475505 is a follow up to your comments on the parent (which could use a +A)17:36
*** jlk` is now known as jlk17:43
*** jlk has quit IRC17:44
*** jlk has joined #zuul17:44
* SpamapS didn't know what autobahn was until today17:48
SpamapSmordred: I think you meant the opposite: "websocket streaming will be switching to autobahn"18:11
mordredno, I do not18:11
mordredSpamapS: autobahn does not work for us18:12
mordredSpamapS: aiohttp does18:12
*** Shuo has joined #zuul18:12
mordredSpamapS: autobahn layers on too much extra goo that we do not need that actually gets in the way of us being able to do the things we need to do - this is understandable, since they support asyncio, eventlet and twisted backends so they obviously need to provide a compat layer18:13
*** Shuo has quit IRC18:14
SpamapSmordred: so wouldn't that mean that we want to switch to *aiohttp*18:14
*** Shuo has joined #zuul18:14
mordredSpamapS: yes.18:14
SpamapSyou wrote the other thing18:14
mordredSpamapS: wait - did I say what you quoted?18:14
mordreddangit18:14
mordredthank you18:14
SpamapSnp18:14
SpamapStwo words, same start letter, almost same # of chars.18:15
mordred++18:15
*** Shuo_ has joined #zuul18:15
jlko/18:16
mordredSpamapS: fwiw, https://review.openstack.org/#/c/463353/19/zuul/streamer.py is the autobahn version and https://review.openstack.org/#/c/463353/21/zuul/web.py is the aiohttp version if you wanna see the differences18:16
SpamapSmordred: I like the aiohttp version. Reminds me of the tokio futures stuff I"ve been poking at in Rust.18:18
clarkbmight help reviews if changes went into streamer.py then file was renamed at end?18:18
*** Shuo has quit IRC18:19
mordredclarkb: the patch originally created zuul/streamer.py and doens't have any previous reviews on streamer.py - is there a reviewing use-case I'm missing?18:21
clarkbmordred: oh for some reason I thought it had merged already but tahts the same change just in different patchsets. That said you could do inter patchset diffs more easily that way18:22
clarkbI don't know that it matters since its not merged yet18:22
mordredyah - I think if there had been reviews on the old versions that would be more important18:22
mordredSpamapS: I agree - it felt much nicer to write/hack on too18:23
Shrewsclarkb: i would highly recommend NOT reviewing it yet. lots of changes needed still18:23
mordredSpamapS: the flow was very understanable, as opposed to the callback methods on the service class from the autobahn approach where you didn't really know what was going on between callbacks18:24
Shuo_pabelanger: what is the status of ansible-role-zuul? could I use it as zuul demo environment setup?18:33
mordredShuo_: pabelanger is on vacation - I'm not sure ansible-role-zuul is up to date as of today though18:39
clarkbShuo_: mordred left some comments18:40
clarkber shrews ^18:41
Shuo_mordred: and clarkb: thanks for providing the info. Right now, I am wondering that to build a demo-ish cluster, how easy/hard it would be to a perosn not familiar with Zuul?18:41
jeblairShuo_: i think at this point it would still be hard, but possible.18:42
jeblairShuo_: we're just working on the documentation now, it should be merging soon.18:43
jeblairShuo_: i think it will be a lot better in a few weeks when that's done and pabelanger is back and able to update ansible-role-zuul18:43
*** pbrobinson has joined #zuul18:49
*** Shuo_ has quit IRC18:55
*** Shuo has joined #zuul19:05
Shuojeblair: Regarding the last message I saw from James "it would be hard, but not impossible", my goal is not to be too hand-off from the community (we could work under guidance)19:07
Shuojeblair: "i think it will be a lot better in a few weeks when that's done and pabelanger is back and able to update ansible-role-zuul" -- we are totally fine for waiting several weeks.19:10
*** mmedvede has quit IRC19:25
*** mmedvede has joined #zuul19:26
*** mmedvede has quit IRC19:28
*** mmedvede has joined #zuul19:31
*** Shuo has quit IRC19:31
*** mmedvede has quit IRC19:42
*** mmedvede has joined #zuul19:42
*** rfolco_ has joined #zuul19:55
*** jkilpatr has quit IRC20:12
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming  https://review.openstack.org/46335320:14
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming  https://review.openstack.org/46335320:19
*** rfolco_ is now known as rfolco20:19
Shrewsmordred: ok, just need to implement the gearman/finger portion now. that new test properly fetches the "Test test" artificial response.20:21
mordredShrews: woot!20:26
*** jkilpatr has joined #zuul20:31
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Add run-bindep role and add it to unittests pre  https://review.openstack.org/47826420:32
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Port in tox jobs from openstack-zuul-jobs  https://review.openstack.org/47826520:32
*** rcarrill1 has joined #zuul20:33
*** rcarrillocruz has quit IRC20:34
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Port in tox jobs from openstack-zuul-jobs  https://review.openstack.org/47826521:12
*** jkilpatr has quit IRC21:18
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Port in tox jobs from openstack-zuul-jobs  https://review.openstack.org/47826521:24
*** hashar has quit IRC21:25
*** jkilpatr has joined #zuul21:31
*** maxamillion has left #zuul21:48
*** Shuo has joined #zuul21:50
*** jkilpatr has quit IRC21:53
*** jkilpatr has joined #zuul21:53
mordredjeblair: I think we want to land https://review.openstack.org/#/c/477672 before we get too much further with jobs22:10
jeblairmordred: fair enough22:11
jeblairmordred: +322:12
jlkGOOD NEWS22:15
jlkgithub search lets me search for:   "Depends-On" z8s-sandbox/pull/2 type:pr in:body22:15
jlkand it's only returning the PR with that in the body, not in a comment.22:15
jlknow, time for an office party.22:16
mordredjlk: woot!22:16
jeblair\o/22:17
jeblairdid someone (pabelanger?) push up a change with an 'add host to inventory' task?22:17
jeblairi could have sworn i saw that but can't find it22:18
mordredjeblair: I do not knkow - do we need one of those?22:18
mordredor do you need an example?22:18
mordredoh - this is for publishing the logs22:18
jeblairmordred: yeah, i think we need to do that to add the logserver to the inventory to use synchronize22:18
mordredjeblair: http://docs.ansible.com/ansible/add_host_module.html is what you want22:19
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Support custom playbook locations  https://review.openstack.org/47767222:19
jeblairi think that should be safe in the base post playbook, since nothing will run after it22:19
mordredjeblair: it's only valid for the given playbook invocation22:19
mordredjeblair: so it's safe regardless22:19
jeblairmordred: oh neat22:19
mordredit only edits the in-memory copy of the inventory22:19
jeblaircause i had ideas swirling in my head about accidentally running something on logs.o.o22:20
jeblairmordred: a role that copies stuff from all hosts to executor seems straightforward...22:34
jeblairmordred: a role that copies from executor to log hosts seems less so, since i want the remote host to be logs.o.o.  that sounds like a second play in the post playbook.  does that sound right to you?22:34
jeblairmordred: i guess it can be a second play with a role22:35
jeblairhttps://etherpad.openstack.org/p/WWUw5wHlEY22:36
jeblairsomething like that?22:36
jeblair22:44 < openstackgerrit> James E. Blair proposed openstack-infra/openstack-zuul-roles master: Add collect and upload log roles  https://review.openstack.org/47831122:45
jeblair22:45 < openstackgerrit> James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Upload logs to logs.openstack.org  https://review.openstack.org/47831222:45
jeblairi'm going to need to move that into project-config to make it trusted, but i wanted to put it out there22:46
jeblair22:58 < openstackgerrit> James E. Blair proposed openstack-infra/project-config master: Move base job to project-config  https://review.openstack.org/47831322:59
jeblair22:59 < openstackgerrit> James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Remove base job  https://review.openstack.org/47831522:59
jeblairSpamapS: would you please see my comments on PS1 of https://review.openstack.org/477168 for background on the py2/py3 aspect of that script?23:05
SpamapSjeblair: ACK, looking23:07
SpamapSI need like, a little urwid pong game to play when my gertty gets slow syncing.23:09
jeblairwe could at least add a spinning cursor23:09
jeblair/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|23:10
SpamapSjeblair: hah23:11
SpamapSjeblair: so, py2.. IMO we should just abandon it. We can likely make encrypt_secret work with python 3.4, which gets us RHEL6/CentOS6/Ubuntu 14.04.23:13
SpamapSanything older and I start to wonder about openssl versions ;)23:14
SpamapSOtherwise we need to devise a way to test encrypt_secret. :-P23:15
clarkbSpamapS: the aiohttp code is already python3.5 only (it uses await)23:15
openstackgerritMerged openstack-infra/zuul feature/zuulv3: requirements: update ansible version  https://review.openstack.org/47522823:16
SpamapSclarkb: right, but encrypt_secret is a CLI tool for zuul end-users.23:16
SpamapSSo it makes sense for it to have a little bit wider support matrix.23:17
clarkbmaybe? it is pretty trivial to run the encyption on a modern install then deploy to eg rhel6 if you are still stuck on it23:17
jeblairSpamapS: it does already work with py2.  i can test the result after the py3 update and make sure it still does.  after that, i don't expect it to change (at least, not in a way that would trouble me about py2/3 compatibility)23:18
clarkbobviously if it just works then great. It just seems like a lot of effort for rhel6 which is really old now23:18
jeblairSpamapS: so it seems like py2 for this is pretty low cost.  i agree, we're pretty darn close to the point where it's not needed though.23:18
SpamapSThe cost to us is only in maintaining some kind of tests.23:19
SpamapSI don't feel that strongly about dropping py2 for it.23:19
SpamapSBut I feel a little more strongly that it should grow tests so we don't break it.23:19
jeblairi'm happy to have py3 but not py2 tests for it.23:20
SpamapSand I was kind of enjoying being py3 only for a minute ;)23:20
SpamapSthat's an interesting idea. Make a best effort to be py2, but only really support py3.23:21
SpamapS(sounds like the way py3 has been treated for the last 5 years ;)23:21
jeblairturnabout is fair play23:21
SpamapSpy3 has a secret: py3 is not left handed23:21
jeblairrust is not left handed either?23:22
SpamapSrust is the immune to iocaine.23:22
SpamapSs/the //23:22
jeblairSpamapS, mordred: i note that you both write ansible task names without trailing periods23:30
jeblair(i know, i know, bear with me)23:30
jeblairSpamapS, mordred: i think pabelanger had training periods.23:31
jeblairshould we adopt the non-training period style?23:31
jeblair(as i think about it, these are most similar to long entries, which usually don't have periods)23:32
jeblairer log entries23:32
jeblairmordred: 478311 478313 478315 (zuul error is expected) are the job config changes needed to get log publishing.23:37
jeblairafter we land those, i'll try restarting the executor and see what's broke.  :)23:38
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add log streaming test  https://review.openstack.org/47107923:46
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove py2 SocketServer import  https://review.openstack.org/47539923:47
*** Shuo has quit IRC23:48
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Use Event in streaming test  https://review.openstack.org/47550523:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!