| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Serve keys from canonical project name https://review.openstack.org/504807 | 00:03 |
|---|---|---|
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add support for result data in child jobs https://review.openstack.org/504808 | 00:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: add abstract job attribute https://review.openstack.org/504809 | 00:03 |
| dmsimard | pabelanger, tristanC: bah https://github.com/Logan2211/ci-stack | 00:03 |
| dmsimard | so many people working on their own zuul/nodepool deployment thingies | 00:04 |
| dmsimard | FYI multinode things are ready to review: https://review.openstack.org/#/q/topic:zuulv3-multinode | 00:22 |
| dmsimard | The implementation is in zuul-jobs and it is integration tested in openstack-zuul-jobs | 00:22 |
| dmsimard | clarkb ^ | 00:22 |
| SpamapS | now to figure out what's going on with my status urls | 02:59 |
| *** jesusaur has quit IRC | 03:08 | |
| *** jesusaur has joined #zuul | 03:27 | |
| openstackgerrit | zhangyangyang proposed openstack-infra/nodepool master: Remove py26 support https://review.openstack.org/504836 | 03:37 |
| * SpamapS is very confused how to get web-based log streaming to work | 04:14 | |
| tobiash | SpamapS: how far have you got? Maybe I can give you a hint. | 05:20 |
| tobiash | SpamapS: zuul-web must be running and needs to be able to connect to the executors on the finger port | 05:22 |
| tobiash | SpamapS: if you use apache as a proxy in front of zuul-web you need to handle the websocket path differently as apache doesn't seem to be able to handle websocket streaming and normal web content with one proxy | 05:23 |
| tobiash | SpamapS: I have something like this in the apache settings for zuul-web: http://paste.openstack.org/show/621303/ | 05:24 |
| SpamapS | tobiash: I can manually paste the uuid into the url to the static/stream.html just now :) | 05:24 |
| SpamapS | (and I"m using nginx, so I just learned how to upgrade the ws:// 's | 05:24 |
| tobiash | SpamapS: so the streaming part itself works? | 05:24 |
| SpamapS | tobiash: yes, streaming works | 05:25 |
| SpamapS | but a) I get no statuses set at all on my github PR | 05:25 |
| tobiash | SpamapS: I think ngingx is better in this regard | 05:25 |
| SpamapS | b) comments are finger:// | 05:25 |
| SpamapS | https://www.nginx.com/blog/websocket-nginx/ | 05:25 |
| SpamapS | ^^ that one got me working | 05:25 |
| SpamapS | pretty simple, seems like it would be easy in Apache too | 05:25 |
| SpamapS | but yeah maybe nginx handles better internally | 05:26 |
| SpamapS | anyway, right now i think I just have to figure out how to get zuul to comment with the right URL | 05:26 |
| tobiash | SpamapS: difference about nginx and apache is that this nginx settings work for mixed websocket and non-websocket stuff and for apache you need to have seperate settings | 05:26 |
| SpamapS | and I can do my demo tomorrow without feeling embarrassed that i have to copy/paste the UUID out of the logs into my streaming URL | 05:27 |
| tobiash | SpamapS: for the correct comments your base job needs to set a special var | 05:27 |
| tobiash | SpamapS: look here: https://github.com/openstack-infra/project-config/blob/master/playbooks/base/post-logs.yaml | 05:28 |
| tobiash | SpamapS: https://github.com/openstack-infra/zuul-jobs/blob/master/roles/upload-logs/tasks/main.yaml#L38 | 05:28 |
| SpamapS | Yeah that's a good point | 05:29 |
| tobiash | SpamapS: this is the task that tells zuul about the log url which will replace the finger url when reporting to github | 05:29 |
| SpamapS | I think the bonnyci jobs I cribbed from need some secrets setup which I didn't setup | 05:29 |
| SpamapS | since they copy logs from executor -> logs host | 05:29 |
| SpamapS | also my status.json doesn't get any urls in it which seems odd | 05:30 |
| SpamapS | I'd at least expect finger:// urls while the job runs | 05:30 |
| SpamapS | guessing I'm missing something in my pipeline config that will set the pattern | 05:31 |
| tobiash | that looks strange | 05:33 |
| SpamapS | yea it's kinda driving me crazy :-P | 05:36 |
| SpamapS | because tracing through this there are a ton of 'status urls' | 05:37 |
| *** yolanda has joined #zuul | 06:11 | |
| *** bhavik1 has joined #zuul | 06:19 | |
| *** hashar has joined #zuul | 06:46 | |
| *** bhavik1 has quit IRC | 07:17 | |
| qwc | hi! quick question, we're using zuul 2.5.1 and i am trying to use templates with multiple parameters, but it doesn't allow it, did i something wrong with the syntax ? | 07:57 |
| qwc | - name: mytemplate | 07:58 |
| qwc | parameter1: blah | 07:58 |
| qwc | parameter2: blahblah | 07:58 |
| qwc | (when i want to use it in a project of course. ;) | 07:58 |
| qwc | mordred: maybe ? | 08:01 |
| *** electrofelix has joined #zuul | 08:15 | |
| qwc | well nevermind, found a workaround for now. but still would like to know how it is possible to use multiple parameters in a template definition of a project, because i get exceptions there everytime when i have more than one... just highlight me, quassel will save it and notify me. :) | 08:55 |
| *** _ari_ has quit IRC | 09:07 | |
| *** weshay has quit IRC | 09:07 | |
| *** _ari_ has joined #zuul | 09:07 | |
| *** weshay has joined #zuul | 09:10 | |
| *** pabelanger has quit IRC | 09:48 | |
| *** _ari_ has quit IRC | 09:49 | |
| *** _ari_ has joined #zuul | 09:50 | |
| *** pabelanger has joined #zuul | 09:51 | |
| *** rcarrillocruz has quit IRC | 10:04 | |
| *** rcarrillocruz has joined #zuul | 10:10 | |
| *** jkilpatr has quit IRC | 10:49 | |
| *** jkilpatr has joined #zuul | 11:09 | |
| *** dkranz has joined #zuul | 12:04 | |
| Shrews | jeblair: was the nodepool problem you were hunting on Friday due to the config issue? | 12:40 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Only uninstall rdo-release if it was installed in the first place https://review.openstack.org/504936 | 13:02 |
| fbo_ | hi, I have a zuulv3 deployment and I'm trying to define a job in a repo declared in "untrusted-projects" in main.yaml but either for "shell" or "command" ansible module I'm getting "ERROR: Executing local code is prohibited". Is it a normal behavior ? or should I define my project under config-projects ? | 13:53 |
| dmsimard | fbo_: untrusted jobs can't execute code on the executor | 13:57 |
| dmsimard | i.e, targetting or delegating to localhost | 13:57 |
| fbo_ | dmsimard: well the exec of the "shell" or "command" command will happen on the slave not on the executor in my case. The task is not maked delegated. But well ok I'm trying to declare the repo in config-projects then. | 14:02 |
| dmsimard | fbo_: okay, if you can't figure it out, I would need to see an example job/playbook that's not working so I can understand better | 14:03 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: WIP: Honor cloud quotas before launching nodes https://review.openstack.org/503838 | 14:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Make max-servers optional https://review.openstack.org/504282 | 14:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support cores limit per pool https://review.openstack.org/504283 | 14:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't fail on quota exceeded https://review.openstack.org/503051 | 14:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support ram limit per pool https://review.openstack.org/504284 | 14:06 |
| *** hashar is now known as hasharAway | 15:28 | |
| SpamapS | /var/log/zuul/zuul-scheduler-debug.log:2017-09-17 22:35:08,898 DEBUG zuul.GithubReporter: Reporting change <Change 0x7f8d940c1da0 2,e463c389bf3bff6473a6aa2ddda05986d714d0ed>, params {'comment': True, 'status': 'failure', 'status-url': 'http://zuul1.cloud.phx3.gdg/{tenant.name}/{pipeline.name}/{change.project.canonical_name}/{change.number}/{buildset.uuid}/'}, status: | 15:41 |
| SpamapS | blank status | 15:41 |
| SpamapS | I wonder if I missed something in my pipeline config | 15:42 |
| Diabelko | hello | 15:45 |
| pabelanger | SpamapS: status is failure in your paste | 15:46 |
| pabelanger | oh | 15:46 |
| pabelanger | at the end | 15:46 |
| SpamapS | pabelanger: right | 15:47 |
| pabelanger | odd, that you have 2 status fields | 15:47 |
| mordred | SpamapS: you have a misplaced ' | 15:47 |
| mordred | no? oh - nevermind | 15:47 |
| mordred | I can't read either | 15:47 |
| SpamapS | actually | 15:47 |
| jeblair | SpamapS: i see http://paste.openstack.org/show/621341/ in the github test fixtures | 15:47 |
| SpamapS | that log message is borken | 15:47 |
| *** maxamillion has quit IRC | 15:48 | |
| Diabelko | Do you guys have a date in mind for Zuulv3 release? | 15:48 |
| SpamapS | https://github.com/openstack-infra/zuul/blob/feature/zuulv3/zuul/driver/github/githubreporter.py#L114-L118 | 15:48 |
| SpamapS | Diabelko: hi! Not really. We'll think about locking down 3.0 once OpenStack is fully migrated onto zuulv3 | 15:49 |
| mordred | Diabelko: we're planning openstack rollout next wednesday (sept 25) - after that we expect it to be a couple of months before we cut a 3.0 release and point other people at using/running it | 15:49 |
| Diabelko | mhm, got it | 15:49 |
| mordred | Diabelko: (we want up update docs and some other things like the puppet-openstackci repo 3rd party operators use before hand) | 15:49 |
| Diabelko | is there any difference in job definitions and layouts I should be aware of before migrating? | 15:49 |
| jeblair | Diabelko: we'll have a roadmap for the v3 release after we do the openstack cutover | 15:50 |
| mordred | Diabelko: yes. a lot - lemme get you some links | 15:50 |
| Diabelko | we're in the process of migrating from Jenkins to Zuulv2.5 | 15:50 |
| Diabelko | and I need some info to make sure I'm not wasting a lof of effort in rewriting those jobs ;) | 15:50 |
| SpamapS | so many answers. :) | 15:50 |
| Diabelko | mordred: thanks | 15:50 |
| mordred | Diabelko: yah - you're at a 'fun' time, as v3 is runnable but not yet documented, and 2.5 is definitely on its way out the door ... | 15:51 |
| mordred | Diabelko: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html is the reference docs for job content | 15:51 |
| jeblair | Diabelko: the best thing you can do is avoid using jenkins plugins -- if you can make your jobs just use shell snippets, it will be easier to migrate. | 15:51 |
| mordred | Diabelko: we added this: https://docs.openstack.org/infra/manual/zuulv3.html for openstack users which might also be helpful | 15:51 |
| Diabelko | awesome | 15:52 |
| mordred | Diabelko: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.yaml http://git.openstack.org/cgit/openstack-infra/zuul-jobs and http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs all have v3 content in them if you prefer looking at examples | 15:52 |
| Diabelko | and yes, I am aware 2.5 is going to be obsolete pretty soon | 15:52 |
| Diabelko | all right, thanks | 15:53 |
| Diabelko | I'll take a look at that | 15:53 |
| mordred | Diabelko: we have a migration script that will convert JJB to zuulv3 - it's not NEARLY perfect and contains openstack-specific transformations in a few places | 15:53 |
| mordred | Diabelko: but once we use it next week for openstack, should be a decent starting point for other folks who also want to do the transition | 15:53 |
| SpamapS | question about that log line I pasted earlier | 15:53 |
| SpamapS | shouldn't the {}'s be resolved? | 15:54 |
| Diabelko | mhm, sure | 15:54 |
| Diabelko | it seems like I better convince people to hold off a little bit | 15:54 |
| SpamapS | url is the result of item.formatUrlPattern | 15:54 |
| mordred | Diabelko: yah - a few more weeks and you'll have a much better path straight to v3 | 15:55 |
| Diabelko | especially that I know there's dependency graph coming | 15:55 |
| mordred | ++ | 15:55 |
| mordred | SpamapS: yes - formatUrlPattern seems like it should be producing expanded urls | 15:56 |
| SpamapS | yeah so I'm confused now | 15:56 |
| mordred | SpamapS: so - the status: at the end of the line is a red herring | 15:58 |
| SpamapS | mordred: yeah figured that out | 15:59 |
| SpamapS | I think that's a little borken | 15:59 |
| mordred | k. good | 15:59 |
| mordred | yah - easy fix for that | 15:59 |
| SpamapS | (though it does help me narrow down the Reporting log messages) | 15:59 |
| mordred | SpamapS: just so I'm fully up to speed on where you're at and what you're looking at | 16:01 |
| SpamapS | http://paste.openstack.org/show/621342/ | 16:01 |
| SpamapS | there's my pipeline configs.. any red flags? | 16:01 |
| mordred | SpamapS: the issue you're running in to is that while the job is running there is no update to the PR status with a link to the per-change zuul status page right? | 16:02 |
| SpamapS | mordred: right | 16:02 |
| SpamapS | mordred: also secondary issue: the link in comments is finger:// | 16:02 |
| mordred | SpamapS: yah | 16:02 |
| SpamapS | However, web streaming _does work_ if I copy the build id out of logs into my url bar ;) | 16:02 |
| mordred | SpamapS: if you just open up the full status page while a job works, does *that* show the stream urls? | 16:03 |
| SpamapS | oh wait hah | 16:03 |
| mordred | SpamapS: or are the links in the status page themselves broken? | 16:03 |
| SpamapS | no status-url in start: | 16:03 |
| SpamapS | let's try that.. just for fun | 16:03 |
| SpamapS | I have no "status page" | 16:03 |
| SpamapS | I have status.json | 16:03 |
| SpamapS | (I may have a status page, but I don't know where BonnyCI is stuffing it) | 16:04 |
| mordred | gotcha. you're wanting the stream urls to go directly into the status reporting then | 16:04 |
| mordred | SpamapS: so - at the moment, the logic for emitting finger:// vs. stream.html urls is in formatJSON | 16:06 |
| SpamapS | 2017-09-18 09:06:04,786 DEBUG zuul.ExecutorClient: Build <gear.Job 0x7f8d940de320 handle: b'H:10.36.156.42:20' name: executor:execute unique: df4ce646ce914da097c43b827d7ddabe> update {'worker_log_port': 9099, 'worker_hostname': 'zuul1.cloud.phx3.gdg', 'url': 'finger://zuul1.cloud.phx3.gdg:9099/df4ce646ce914da097c43b827d7ddabe', 'worker_name': 'zuul1.cloud.phx3.gdg'} | 16:06 |
| mordred | SpamapS: if you want zuul to report static/stream.html instead of finger:// - we may need to do a little work | 16:07 |
| *** maxamillion has joined #zuul | 16:07 | |
| SpamapS | 2017-09-18 09:05:58,655 DEBUG zuul.GithubReporter: Reporting change <Change 0x7f8d940c1da0 2,e463c389bf3bff6473a6aa2ddda05986d714d0ed>, params {'comment': False, 'status': 'pending', 'status-url': 'http://zuul1.cl | 16:07 |
| SpamapS | oud.phx3.gdg/zweb/static/stream.html?uuid={buildset.uuid}&logfile=console.log'}, status: | 16:08 |
| SpamapS | Ok, so _that_ looks right except the unresolved URL | 16:08 |
| SpamapS | That was fixed by adding a status-url to the pipeline | 16:08 |
| SpamapS | mordred: I honestly don't care about status.json right now | 16:08 |
| SpamapS | I just want the PR status to get updated right | 16:08 |
| mordred | SpamapS: right.I know - what I'm saying is that the url you want is produced currently in formatJSON so I was thinking it was possible we might need to write a patch to expose the right thing for yo - but it seems you're much closer, so that's good | 16:09 |
| jlk | SpamapS: did you figure out your reporting URL thing? | 16:09 |
| SpamapS | jlk: nope, still working through it | 16:09 |
| mordred | jlk: he's working on that right now | 16:09 |
| jlk | ah okay | 16:09 |
| SpamapS | jlk: I am guessing Bonny's zuulv3 isn't quite functional on this level either. | 16:10 |
| jlk | I apologize in advance for any boneheadedness you run into. | 16:10 |
| jeblair | SpamapS: s/buildset.uuid/build.uuid/ | 16:10 |
| jlk | SpamapS: it might not be. I know that jamielennox had some things up for reporting URL type things, we used to define it in the pipeline, and I don't know if it's the same still | 16:10 |
| SpamapS | Ohh this is annoying and a github problem I think | 16:10 |
| SpamapS | jeblair: THANKS! | 16:10 |
| SpamapS | so.. I'm using a personal repo and I have the zuul user as a collaborator | 16:11 |
| SpamapS | and I THINK there are issues with collaborators clearing statuses | 16:11 |
| SpamapS | jlk: do you remember anything like that? | 16:11 |
| SpamapS | I'm actually getting the right URL(wrong uuid until jeblair fixed me) on new PR's... just not on rechecks. | 16:11 |
| jlk | clearing statuses? | 16:12 |
| SpamapS | jlk: yeah, the PR statuses of pending/failure/success ... | 16:12 |
| jlk | We don't "clear" status so much as set a new one. | 16:13 |
| jlk | It _should_ set a "pending" status on the start of the pipeline job | 16:13 |
| SpamapS | oh I think I know what happened | 16:13 |
| SpamapS | this is weird | 16:13 |
| SpamapS | so because my pending status had no url | 16:13 |
| SpamapS | new statuses couldn't | 16:13 |
| SpamapS | now the urls are working | 16:13 |
| jlk | huh, weird | 16:14 |
| jlk | that's an odd scenario | 16:14 |
| SpamapS | Ok I think I have status urls now | 16:15 |
| SpamapS | still finger in comments tho | 16:15 |
| SpamapS | tox-linters finger://zuul1.cloud.phx3.gdg:9099/57a374b77ae0470c81895b541803e827 : RETRY_LIMIT in 1m 13s | 16:15 |
| jlk | that finger stuff is relatively new, I wonder if that's an option toggle. | 16:16 |
| jeblair | well, you get a finger url if the log copying playbook doesn't get run or return the log url | 16:16 |
| SpamapS | oh good point | 16:17 |
| SpamapS | I think somebody already pointed that out to me actually | 16:17 |
| SpamapS | need a zuul_result | 16:17 |
| jeblair | yep | 16:17 |
| SpamapS | and in this case I'm standalone so I just have to copy the logs into somewhere on the same box that has a webroot | 16:17 |
| SpamapS | this is Unix | 16:18 |
| SpamapS | I *know* this. | 16:18 |
| jeblair | you *can* override it, but the point of zuul_result was so that you don't have to have the url pattern in two places (zuul config and log playbook) | 16:18 |
| jlk | You make a zuul_result or you keep getting the finger. | 16:19 |
| SpamapS | it puts the url pattern in the results or it gets the finger | 16:19 |
| jlk | basically | 16:21 |
| mordred | SpamapS: last week there was a lurking joke that we should add an emoji to the status page next to the stream.html with the finger url - and it should be a finger emoji | 16:21 |
| mordred | SpamapS: we figure there would be no negative unintended implications of small finger emojis | 16:22 |
| SpamapS | where is the example for zuul_result? | 16:22 |
| * SpamapS needs to crib it :-P | 16:22 | |
| SpamapS | mordred: of course. fingers are universal. | 16:23 |
| SpamapS | nearly 100% inclusive | 16:23 |
| SpamapS | It's not called zuul_result is it? | 16:25 |
| * SpamapS thinks we said the wrong word | 16:25 | |
| SpamapS | zuul_return | 16:26 |
| * SpamapS goes back to hacking | 16:26 | |
| jlk | alright, time to figure out where I was before my brain got all melty at the end of last week. | 16:29 |
| SpamapS | ghhrraaahhh github! (statuses seem to be a best-effort.. sometimes works, sometimes not) | 16:33 |
| jlk | Is there high priority things to review today? | 16:35 |
| jlk | SpamapS: the API is just saying "LOL NOPE" ? | 16:35 |
| SpamapS | jlk: actually I never see what the API ever says | 16:35 |
| SpamapS | but it's just that status urls were working | 16:35 |
| SpamapS | and now they're not | 16:35 |
| SpamapS | and it's not clear why :-P | 16:35 |
| jlk | taking the call but not actually doing the thing? | 16:35 |
| jlk | huh... | 16:35 |
| jlk | like, our code is not sending it right, or they're code is not accepting it right? | 16:35 |
| SpamapS | 2017-09-18 09:34:54,974 ERROR zuul.QueueItem: Error while formatting url for job None: unknown attribute 'dict' object has no attribute 'uuid' in pattern http://zuul1.cloud.phx3.gdg/zweb/static/stream.html?uuid={b | 16:36 |
| SpamapS | jeblair: build.uuid? | 16:36 |
| jlk | ugh python. Tell me the name of the variable, that'd be so much more helpful | 16:36 |
| SpamapS | jlk: I'm also just in 1 hour before demo panic mode.. so you can probably ignore most things I put a ! on ;) | 16:36 |
| SpamapS | need to quit touching it | 16:37 |
| * jlk deletes crass response | 16:37 | |
| SpamapS | jlk: oh I know it's build | 16:37 |
| SpamapS | I had buildset.uuid and it was the wrong uuid apparently ;) | 16:37 |
| SpamapS | but build doesn't seem to have a .uuid | 16:38 |
| jlk | I wonder if we're not setting that in the github driver? | 16:38 |
| SpamapS | build.uuid? that's not really in the github driver | 16:39 |
| SpamapS | that's an internal zuul object that the formatter gets | 16:39 |
| jeblair | SpamapS: oh weird, buildset and build should both be valid. the github tests use buildset.uuid. but that doesn't make sense to me; i think build.uuid is what you'd want | 16:40 |
| jeblair | but then, i don't actually know what this url is for... so. | 16:40 |
| SpamapS | jeblair: is it possible this is before build.uuid is set for some reason? | 16:40 |
| SpamapS | jeblair: I need the uuid to pass to stream.html | 16:40 |
| SpamapS | which I thought would be build.uuid | 16:40 |
| jeblair | SpamapS: yeah, i think if there's no build object this can happen | 16:41 |
| SpamapS | buildset.uuid is the thing above that | 16:41 |
| jeblair | SpamapS: oh of course, this is for overall item status, so build doesn't make sense anyway | 16:41 |
| SpamapS | so buildset.uuid was the thing? | 16:41 |
| jeblair | SpamapS: yeah, that will get you a value, but build.uuid will not in this case. | 16:41 |
| SpamapS | cause it didn't look right when I tried it | 16:42 |
| jeblair | SpamapS: having said that -- what is this url for? :) | 16:42 |
| SpamapS | jeblair: github has triggered a start in a check pipeline, I want to set the "Details" link on the 'pending' status | 16:42 |
| SpamapS | I wonder if there's a model mismatch at play here | 16:42 |
| jeblair | SpamapS: right, but what is at this url? | 16:42 |
| SpamapS | jeblair: static/stream.html | 16:42 |
| SpamapS | so people can watch | 16:42 |
| SpamapS | so the problem .. now I see it | 16:43 |
| SpamapS | there's multiple streams possible | 16:43 |
| SpamapS | but this is one trigger | 16:43 |
| jeblair | SpamapS: oh, i see that now, sorry; was confusing it with an earlier url. | 16:43 |
| SpamapS | so I opened a pr on fooorg/barrepo and it may fire off 5 jobs.. but we only set one status | 16:43 |
| jlk | yeah I don't know if Zuul itself has a url that's suitable to look at all the potential jobs for a given pipeline + build | 16:43 |
| jlk | er pipeline + change | 16:43 |
| jeblair | SpamapS: right. so i think the idea there was to link to the status page, but filtered for the one item. then you can follow that to the 5 jobs. | 16:43 |
| SpamapS | right and I don't know if I have a status page | 16:44 |
| SpamapS | jlk: do you know if bonny's status page works for v3 with the tenant prefixed status.json? | 16:44 |
| SpamapS | I can't even find the status page | 16:44 |
| jlk | I don't, I think that's a jamielennox question. | 16:44 |
| SpamapS | yeah me too ok | 16:44 |
| jlk | also there's work going on currently to move all this over to webapp | 16:44 |
| jlk | er zuul-web | 16:45 |
| SpamapS | yeah | 16:46 |
| mordred | I believe the curent SpamapS thing is a stop-gap to just get a URL there that doesn't then involve copy/edit/paste - with the plan being the one-item-status-page thing | 16:46 |
| mordred | but getting him all the way to that for today's demo is highly unlikely :) | 16:46 |
| SpamapS | I am just going to try and jam in the usual status html and see if that works | 16:46 |
| mordred | jlk: in answer to your question from earlier - gerrit is down for upgrade today, so no, no pressing zuul reviews (unless you have them locally in a gertty, in which case "all of them") | 16:47 |
| jlk | I don't see much in gertty | 16:47 |
| jlk | I'll go back to banging on webooks via zuul-web | 16:48 |
| mordred | jlk: cool. | 16:49 |
| SpamapS | weird, the status app is in the zuul-scheduler role, but for some reason it didn't get put onto my server | 16:52 |
| SpamapS | woot status page working | 16:55 |
| mordred | SpamapS: woot! | 16:56 |
| mordred | SpamapS: and with the correct links? | 16:56 |
| * SpamapS fist pumps | 17:00 | |
| SpamapS | PR -> status page works -> log stream works | 17:00 |
| SpamapS | now result logs aren't working | 17:00 |
| SpamapS | but I can like.. watch that | 17:00 |
| SpamapS | and see why | 17:00 |
| jlk | huzzah! | 17:01 |
| SpamapS | 2017-09-18 17:01:14.500235 | localhost | the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'zuul_log_path' is undefined | 17:01 |
| SpamapS | that'll do it | 17:01 |
| jlk | jeblair: for gearman jobs, did you want them all to start with "zuul:" ? I'm registering a job for each github based connection, and so it'll look like: "zuul:github:<connection_name>:payload". Seems long, but does that make sense? | 17:30 |
| *** electrofelix has quit IRC | 17:35 | |
| SpamapS | 2017-09-18 17:36:16.648727 | [Zuul] Log Stream did not terminate | 17:36 |
| SpamapS | getting these all over. what's that? | 17:36 |
| SpamapS | mordred: Shrews ^ ? | 17:36 |
| Shrews | i have not seen that before | 17:36 |
| pabelanger | we did get that a while back, not sure it is still an issue. zuul_stream was still running I think? | 17:37 |
| * SpamapS is going to blame centos | 17:38 | |
| jlk | I thought mordred was chasing something like that | 17:39 |
| Shrews | looks like the thread the callback plugin creates to do the streaming does not terminate as it should | 17:40 |
| SpamapS | I won't worry about it | 17:40 |
| pabelanger | ya, I don't think it impacted zuulv3.o.o when I saw it | 17:40 |
| SpamapS | It's annoying in the logs | 17:41 |
| pabelanger | more like a warning | 17:41 |
| pabelanger | +1 | 17:41 |
| pabelanger | SpamapS: do you get any streaming? or just the spamming of it? | 17:41 |
| pabelanger | if no other streaming, maybe check firewall ports | 17:42 |
| SpamapS | I get streaming | 17:43 |
| SpamapS | it's great | 17:43 |
| Shrews | pabelanger: did the nodepool issues get resolved? | 17:46 |
| jeblair | jlk: i think we can drop zuul from that; i think '<driver>:' is probably sufficiently namespaced... | 17:46 |
| Shrews | and was it just the config? | 17:46 |
| jlk | jeblair: alrighty | 17:47 |
| SpamapS | I see you guys talking about 'keep' all the time | 17:47 |
| SpamapS | how do I get my executor to 'keep' ? | 17:47 |
| * SpamapS wants logs to stick around | 17:47 | |
| jeblair | SpamapS: "zuul-executor keep" | 17:47 |
| SpamapS | oh | 17:47 |
| jlk | jeblair: also, how do we feel about the github driver exclusively owning the URL path of /connection/{connection}/payload ? | 17:47 |
| SpamapS | like | 17:47 |
| SpamapS | in the systemd job? | 17:47 |
| jeblair | jlk: you might consider dropping 'connection_name' from that to make it more robust against name changes | 17:48 |
| SpamapS | or does it poke something into a command socket? | 17:48 |
| SpamapS | and then if the latter, is there an unkeep? | 17:48 |
| jeblair | jlk: heh, my last comment was about gearman job names | 17:48 |
| jeblair | jlk: you might consider dropping 'connection_name' from *the gearman function name* to make it more robust against name changes | 17:48 |
| jlk | jeblair: hrm, but it's a different handler for each connection | 17:49 |
| pabelanger | SpamapS: almost, I have a change up to no longer share providers across launchers | 17:49 |
| pabelanger | SpamapS: sorry | 17:49 |
| pabelanger | Shrews: ^ | 17:49 |
| pabelanger | Shrews: once gerrit is back online, I hope to finish it up | 17:49 |
| jlk | each connection object will get their own handler. If you have two github connections, each are separate connections | 17:49 |
| jlk | er separate job handlers | 17:49 |
| jeblair | jlk: hrm | 17:49 |
| jlk | just like they each would register their own http handler in the old method | 17:50 |
| Shrews | pabelanger: i sort of feel like we should build some protection against that (or just plain ol' fully support it, which is much more work) | 17:51 |
| Shrews | not quite sure how to do the former though (or the latter for that matter) | 17:51 |
| jeblair | jlk: yeah. if we wanted to do my suggestion, we'd have to add *another* layer of indirection in there (a dispatcher in the github driver to hand gearman jobs to the different connection threads. it's possible, but maybe not worth it?) | 17:54 |
| jeblair | jlk: so let's do your thing for now. :) | 17:54 |
| jlk | seems complicated :) | 17:54 |
| jlk | how do you feel about github owning the "payload" final target though? so that zuul_url/connection/<name>/payload is exclusively github driver's playground? | 17:55 |
| pabelanger | Shrews: Ya, I didn't realize it was incorrect until jeblair linked the zuulv3 spec again. | 17:55 |
| tobiash | Shrews: or document that this can lead to races about max servers (which could be toleratable in some deployments) | 17:55 |
| jlk | "payload" seems awfully generic, and other drivers may need to do the webhook like ingest. | 17:55 |
| pabelanger | we can run multiple nodepool-launcher, just not with same provider info | 17:56 |
| jeblair | Shrews, pabelanger: yeah, the short version is that you can wedge a provider if you ask for a number of nodes sufficiently close to the quota of the provider *and* there is insufficient turnover (eg because min-ready is too high and causing a node to sit forever). in tracking this down, we found that we had multiple launchers running for the same providers which exacerbated the effects (basically, caused the problem to affect 2 requests instead ... | 17:59 |
| jeblair | ... of 1). | 17:59 |
| jeblair | Shrews, pabelanger: we also found that, in this specific case, that some of the nodes in the stuck request had errored. if nodepool had been able to clear the errored nodes it would have been able to un-wedge itself sooner, but doing that is also a substantial change (you don't want it to end up in an infinite loop if the cloud is erroring) | 18:00 |
| jeblair | Shrews, pabelanger: i think the thing that would have helped this situation the *most* would be a way to cycle out ready nodes when a provider is 'paused' waiting for quota. | 18:01 |
| jeblair | that's probably the weakest point in the algorithm, the easiest to fix, and will make the most difference. it certainly would have avoided the problem this time. and in fact, "delete the ready node" is what i did to unwedge it. | 18:02 |
| pabelanger | ah cool | 18:03 |
| jeblair | jlk: i think we need to namespace the url *either* with /github/ (ie, /<driver>/), or declare that /connection/<name>/ is owned by the driver. i don't think anything else depends on connection/name (except keys, and they are moving), so i think either is okay? | 18:04 |
| * mordred likes driver | 18:04 | |
| mordred | although ... | 18:04 |
| jlk | oh I hadn't thought about it, that /connection/<name>/ is inherently owned by the connection, which is mapped to the driver. | 18:05 |
| mordred | jlk: if we had a 'global' or a 'driver' version, how do we map from a payload to the correct connection? | 18:05 |
| mordred | ah right | 18:06 |
| jeblair | mordred: yah, i think it's /github/payload/<connection> or /connection/<connection>/payload | 18:06 |
| jlk | but the way zuul-web is written, we have to sit on all of the potential connection names. It registers a path of /connection/{connection}/payload, and sends them all to the handle github function. We could put a different function there that figures out which driver gets it, but | 18:06 |
| mordred | jlk: I think we should write a new function - so that the github driver registers a per-connection endpoint | 18:07 |
| jlk | /github/payload/{connection} seems reasonable to me. It'll look pretty okay in the github app configuration too | 18:07 |
| mordred | meaning after the call, we'd wind up with /connection/github/payload and only that | 18:07 |
| jlk | mordred: it kind of dues | 18:07 |
| jlk | well sortof | 18:07 |
| jlk | zuul-web works differently | 18:07 |
| jlk | I don't think there's a way to live-add a new endpoint | 18:07 |
| jlk | I think that's what you're suggesting, is that we add one | 18:08 |
| jlk | right now we register a new gear job | 18:08 |
| mordred | yah - I've got some half-finished code from a while back adding that - but pointing you at it will need to wait until gerrit is back :) | 18:08 |
| jlk | we've got half that code from registering the path on webapp | 18:09 |
| jlk | I had hoped to delete all that though :D | 18:09 |
| jlk | If we sufficiently namespace by the driver, I think we're okay | 18:09 |
| jlk | let a driver own a top level /<path> | 18:09 |
| jlk | which matches the gear job of <driver>:something | 18:10 |
| mordred | ah - that's fine by me | 18:10 |
| mordred | jlk: I'd love to keep a container in the url if we can -- like /driver/{driver}/payload or whatnot - so that potentially in the future we could add a GET /driver and get a list of available drivers | 18:13 |
| mordred | jlk: but I don't feel _super_ stringly about that - just a thing to think about as you poke at it | 18:13 |
| jlk | I'd have to be /driver/{driver}/payload/{connection} or something like that | 18:13 |
| jlk | does that seem reasonable? | 18:14 |
| *** hasharAway is now known as hashar | 18:21 | |
| mordred | jlk: seems reasonable to me - can you tell me again just so I'm caught up why /connection/{connection}/payload isn't working? | 18:53 |
| mordred | jlk: (I'm not sure I've pieced togetherthe entire state properly in my head) | 18:53 |
| * rbergeron waves at the humans | 18:56 | |
| * mordred waves at the rbergeron | 19:00 | |
| * SpamapS finishes demo and passes out | 19:00 | |
| mordred | SpamapS: how did it go? | 19:00 |
| SpamapS | mordred: ++ | 19:00 |
| mordred | SpamapS: woot! | 19:00 |
| SpamapS | mordred: There's a lot of knowledge of Jenkins-Pipeline (the Groovy thing) so it was a new and interesting audience to show Zuul to. | 19:01 |
| clarkb | did zuul coin pipeline before jenkins? I've been noticing the term get used a lot but not sure if zuul inherited it from somehwere | 19:02 |
| SpamapS | I wonder if we could implement the Blair algorithm in Groovy | 19:02 |
| SpamapS | clarkb: I don't think so | 19:02 |
| SpamapS | pipeline is a 10x overloaded term | 19:02 |
| SpamapS | FYI, fixed python3.5 has hit *-updates, so infra can remove it from the ppa | 19:05 |
| mordred | woot | 19:05 |
| pabelanger | yay | 19:27 |
| dmsimard | SpamapS: what was the purpose ? you're trying to replace pure jenkins by nodepool+zuul ? | 19:27 |
| SpamapS | dmsimard: sort of... | 19:58 |
| SpamapS | dmsimard: it's more to show that there is another way :) | 19:58 |
| jeblair | well, we *were* using the word before the jenkins "workflow-aggregator" plugin started to rebrand itself as 'pipeline'. ;) | 20:02 |
| jeblair | but i borrowed it from the vfx industry. | 20:02 |
| *** hashar has quit IRC | 20:03 | |
| pabelanger | ILM :D | 20:04 |
| jlk | mordred: I'm wary of using /connection/{connection}/payload, because in zuul-web we have to register that route as a function, and the function is github specific. If I add a second connection, say gitlab, that also needs to deliver a payload, we'd have to come up with another verb than "payload" for it to make use of. | 20:51 |
| mordred | jlk: AH. gotcha. thank you | 20:51 |
| jlk | mordred: alternatively, we make a "handle payload" function that determines what driver the connection uses and then routes to a sub-function specific to that driver. | 20:51 |
| mordred | jlk: that makes it very clear to me where the issue is | 20:51 |
| mordred | jlk: yup. I like that a lot | 20:52 |
| jlk | define "that", which you like | 20:53 |
| *** dkranz has quit IRC | 20:53 | |
| jlk | mordred: option A) /driver/{driver}/payload/{connection} or option B) a generic handle_payload that calls out to appropriate driver level handlers after introspection? | 20:56 |
| jlk | (I'm not even sure how to make option B) work) | 20:56 |
| jlk | I guess github could just be hardcoded to know about "/payload" as a final path, or /payload/connection or.. | 20:57 |
| mordred | jlk: both sound good - I think a) solves the immediate problem ... | 20:57 |
| Shrews | Are we having a zuul meeting today? | 20:58 |
| jlk | I'll run with A and see where it gets us for reviews and such. | 20:58 |
| mordred | jlk: for b) if we had a /connection/{connection}/payload - and then had that call a gearman function called payload:<connection name> - then it should work? | 20:59 |
| jlk | Shrews: I'll likely miss the meeting, have to play a spot of family taxi. | 20:59 |
| clarkb | its probably worth having a quick sync up meeting? I'm guessing gerrit will still be reindexing at meeting time and so we shouldn't be too distracted by gerrit upgrade | 21:00 |
| mordred | jlk: or - add a method to the connection plugin interface and have /connection/{connection}/payload call connection.handleWebhookPayload for the correct connection - but have thedefault impl of that plugin method be "submit gearman function payload:<connection>" | 21:00 |
| mordred | clarkb: wfm | 21:00 |
| jlk | mordred: not exactly. We are doing github signature validation at the zuul-web level, before passing along to gearman. I couldn't figure out a way to pass things through gearman that didn't break the body signature. | 21:00 |
| jlk | mordred: so we'd have to have some logic in zuul-web to deal with "what driver is this" if we have to do anything driver specific. | 21:01 |
| jlk | alternatively, if I could figure out how to take an aiohttpd Request object and shove it through gearman and still have a valid signature, that'd be neat! | 21:01 |
| mordred | jlk: nod. so a) is the right choice for now - since that allows us to register a function with zuul-web on a per-driver basis | 21:06 |
| Shrews | jlk: maybe it could be pickled? never used that, but i understand that library is useful for such things | 21:06 |
| jlk | the object has to be able to be serialized, and for whatever reason the aiohttp object can't. | 21:07 |
| Shrews | i wouldn't recommend such complexity, anyway :) | 21:07 |
| jlk | I can validate the signature of the binary blob, but I can't transport that binary blob. If I encode it as utf-8 I can transport it, but it breaks the signature | 21:07 |
| jlk | hrm. | 21:09 |
| jlk | thinking about this more | 21:09 |
| jlk | I think A) is still problematic, sortof. Maybe not. | 21:10 |
| jlk | /driver/{driver}/payload/{connection} -> _handlePayloadPost -> _handle{Driver}Post (where we code one for Github now, and send 404s for anything else, but allows for new drivers to write a handler here). | 21:11 |
| *** jkilpatr has quit IRC | 21:39 | |
| SpamapS | pickle is almost never the right answer. :-/ | 21:41 |
| SpamapS | jlk: you can likely serialize the parts that will matter later.. the headers and the body, individually. | 21:42 |
| jlk | Unless the question is "pickle backs?" | 21:42 |
| SpamapS | we can pickle backs? | 21:42 |
| jlk | https://en.wikipedia.org/wiki/Pickleback | 21:43 |
| SpamapS | jlk: you can base64 encode it. | 21:43 |
| jlk | hrm. | 21:44 |
| SpamapS | Is it expected to be a purely binary blob though? | 21:44 |
| SpamapS | I guess if you don't know what to expect.. you can't assume until you get to your logic engine on the other side of gearman. | 21:44 |
| jlk | it comes in as binary, but you can decode it into json | 21:44 |
| jlk | right | 21:45 |
| jlk | we want zuul-web to do as little with it as possible | 21:45 |
| SpamapS | could just have a cascade there. | 21:45 |
| SpamapS | try: json_decode except ValueError: try: base64encode | 21:45 |
| SpamapS | then do that in reverse on the other side | 21:46 |
| SpamapS | actually you can wrap the base64 or json in an envelope with the type | 21:46 |
| jlk | ugh. so it's bytes object | 21:47 |
| jlk | can't "decode('base64')" it | 21:47 |
| SpamapS | you can't really do that in py3k anymore anyway IIRC | 21:48 |
| SpamapS | you have to use the base64 module directly | 21:48 |
| clarkb | ya | 21:48 |
| SpamapS | welcome... TO THE FUUUUUTURE | 21:48 |
| jlk | huh | 21:48 |
| SpamapS | base64 isn't really an encoding | 21:48 |
| SpamapS | because it doesn't encode characters. | 21:49 |
| SpamapS | that was a bug in py27 that they allowed it | 21:49 |
| SpamapS | py2k I guess | 21:49 |
| jlk | nope | 21:51 |
| jlk | damn | 21:51 |
| jlk | if I use base64.decodebytes(body) (where body is the bytes object) it alters it enough to break the signature. | 21:51 |
| jlk | although. | 21:51 |
| SpamapS | jlk: that shouldn't be. :-/ | 21:51 |
| jlk | ¯\_(ツ)_/¯ | 21:52 |
| jlk | we do a secret.encode('utf-8'), body, hashlib.sha1).hexdigest() | 21:53 |
| jlk | body if passed in as bytes works, body if passed in as base64.decodebytes(bytes) it does not | 21:53 |
| SpamapS | yeah.. hrm | 21:54 |
| SpamapS | so body is something like b"a b c d\0" and it works, but when you decode, you get like, b"a b c d"? | 21:54 |
| jlk | maybe? | 21:54 |
| dmsimard | Are we doing a Zuul meeting in 5 ? | 21:55 |
| dmsimard | jeblair, mordred, clarkb, Shrews, SpamapS, jlk ^ | 21:55 |
| jlk | I'm out. | 21:55 |
| clarkb | I'll be around though will want to focus on the gerrit upgrade once reindexing is done | 21:56 |
| clarkb | maybe we can do a quick sync up and then get back to gerrit things? | 21:56 |
| clarkb | we can also use infra meeting tomorrow for zuul things if it helps | 21:56 |
| dmsimard | Although the ptg just ended, it's probably worth pointing out what's on our plate if we want to flip the switch next monda | 21:56 |
| dmsimard | monday* | 21:56 |
| jeblair | yeah, i think the better thing might be to (mostly?) skip this one and sync up at infra meeting | 21:56 |
| SpamapS | jlk: I typically use base64.b64encode and base64.b64decode ... | 21:56 |
| dmsimard | +1 for infra sync | 21:56 |
| SpamapS | I wouldn't mind hearing a zuul-specific recap of zuuly things discussed at ptg | 21:57 |
| SpamapS | Sounds like there were some designish things around zuul-web discussed. | 21:57 |
| clarkb | lets plan for doing ptg recap tomorrow in infra meeting (its mostly zuul things) | 22:00 |
| *** jkilpatr has joined #zuul | 22:17 | |
| SpamapS | hmmm... I thought we got rid of the 30s lag after playbooks finished when we added --die-with-parent ? | 22:18 |
| SpamapS | 2017-09-18 22:18:13.528128 | TASK [fetch-stestr-output : Register stestr directory] | 22:19 |
| SpamapS | 2017-09-18 22:18:43.674119 | [Zuul] Log Stream did not terminate | 22:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Rename tox_upper_constraints_file to tox_constraints_file https://review.openstack.org/502348 | 22:47 |
| mordred | SpamapS: I also thought we had that fixed | 22:48 |
| jeblair | it was a 60s lag i think | 22:49 |
| jeblair | and i believe it is fixed | 22:49 |
| mordred | SpamapS: you may want to poke a little bit more on that Log Stream did not terminate ... check logs for tracebacks - it might be something different/evil | 22:51 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Move github webhook from webapp to zuul-web https://review.openstack.org/504267 | 23:09 |
| jlk | mordred: jeblair: ^^ that does the thing we talked about, leaving space for other drivers to hook into their own /payload path so that GitHub doesn't squat on all of it. | 23:09 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!