| *** sshnaidm_ has quit IRC | 00:25 | |
| *** sshnaidm has joined #zuul | 00:29 | |
| *** jamesmcarthur has joined #zuul | 00:34 | |
| *** jamesmcarthur has quit IRC | 00:58 | |
| *** jamesmcarthur has joined #zuul | 01:23 | |
| *** jamesmcarthur has quit IRC | 01:31 | |
| *** swest has quit IRC | 01:41 | |
| *** swest has joined #zuul | 01:57 | |
| *** bhavikdbavishi has joined #zuul | 02:06 | |
| *** bhavikdbavishi has quit IRC | 02:13 | |
| *** jamesmcarthur has joined #zuul | 02:29 | |
| *** bhavikdbavishi has joined #zuul | 02:45 | |
| *** bhavikdbavishi has quit IRC | 02:50 | |
| *** bhavikdbavishi has joined #zuul | 02:54 | |
| *** jamesmcarthur has quit IRC | 02:57 | |
| *** rlandy|bbl is now known as rlandy | 02:59 | |
| *** jamesmcarthur has joined #zuul | 03:17 | |
| *** jamesmcarthur has quit IRC | 03:22 | |
| *** rlandy has quit IRC | 03:47 | |
| *** jamesmcarthur has joined #zuul | 03:54 | |
| *** jamesmcarthur has quit IRC | 03:58 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: config: add tenant.toDict() method and REST endpoint https://review.openstack.org/621344 | 04:21 |
|---|---|---|
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add config page https://review.openstack.org/633667 | 04:21 |
| *** bjackman has joined #zuul | 04:21 | |
| *** jamesmcarthur has joined #zuul | 04:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: configure-pypi-mirrors: add initial role https://review.openstack.org/581165 | 04:35 |
| *** pcaruana has joined #zuul | 04:58 | |
| *** quiquell|off is now known as quiquell | 05:46 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add some quoting in 'gerrit query' commands https://review.openstack.org/649879 | 05:49 |
| *** quiquell is now known as quiquell|rover | 05:57 | |
| bjackman | So, re my issue on the Gerrit event-stream format, seems gerrit has reported the approval values on every comment-added event since v2.13 | 05:58 |
| bjackman | See https://gerrit.googlesource.com/gerrit/+/71b58990e9278e0d1128c396dd382a9a7cd5c6c9 | 05:59 |
| bjackman | mordred, If you or anyone else are running Gerrit v2.14, would be great if you could double check that the comment-added gerrit trigger filtered on approvals is working as expected | 06:01 |
| bjackman | i.e. in my case on v2.17 if a change got Workflow +1 but failed gate tests, every subsequent comment in gerrit re-triggers gate tests because the comment-added event includes the pre-existing Workflow +1 approval | 06:02 |
| SpamapS | Including the failure comment zuul leaves? like, does it create a never ending test loop? | 06:20 |
| *** quiquell|rover is now known as quiquell|brb | 06:23 | |
| bjackman | SpamapS, No it doesn't go into a loop | 06:28 |
| bjackman | Maybe Zuul has a filter somewhere for comments written by itself? | 06:28 |
| SpamapS | Possibly | 06:29 |
| * SpamapS barely knows Gerrit<->Zuul so it's hard to help much :-/ | 06:29 | |
| *** gtema has joined #zuul | 06:48 | |
| *** jamesmcarthur has quit IRC | 06:52 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Cleanup executor specific requirements https://review.openstack.org/649428 | 06:58 |
| *** hashar has joined #zuul | 07:04 | |
| *** quiquell|brb is now known as quiquell|rover | 07:08 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add support for 'oldValue' comment-added field https://review.openstack.org/649900 | 07:14 |
| *** themroc has joined #zuul | 07:30 | |
| *** jpena|off is now known as jpena | 07:43 | |
| *** spsurya has joined #zuul | 07:55 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add some quoting in 'gerrit query' commands https://review.openstack.org/649879 | 08:28 |
| *** jamesmcarthur has joined #zuul | 08:49 | |
| *** jamesmcarthur has quit IRC | 08:54 | |
| *** electrofelix has joined #zuul | 08:58 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Add proper __repr__ to merger repo https://review.openstack.org/649949 | 09:05 |
| *** shachar has quit IRC | 10:09 | |
| *** shachar has joined #zuul | 10:09 | |
| *** bhavikdbavishi has quit IRC | 10:22 | |
| *** bjackman has quit IRC | 10:42 | |
| *** bjackman has joined #zuul | 10:43 | |
| *** jpena is now known as jpena|lunch | 10:58 | |
| *** zbr has quit IRC | 11:14 | |
| *** bhavikdbavishi has joined #zuul | 11:25 | |
| *** jamesmcarthur has joined #zuul | 11:30 | |
| *** zbr has joined #zuul | 11:31 | |
| *** jamesmcarthur has quit IRC | 11:34 | |
| *** quiquell|rover is now known as quiquell|rover|e | 11:40 | |
| *** quiquell|rover|e is now known as quique|rover|eat | 11:40 | |
| mordred | bjackman: oh - so - in our gate config for openstack, we have our gate pipeline configured to leave a -2 | 11:41 |
| *** shachar has quit IRC | 11:46 | |
| *** snapiri has joined #zuul | 11:46 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add support for 'oldValue' comment-added field https://review.openstack.org/649900 | 11:46 |
| *** hashar has quit IRC | 11:52 | |
| *** rlandy has joined #zuul | 11:58 | |
| *** hashar has joined #zuul | 12:02 | |
| bjackman | mordred, can I see your config? | 12:10 |
| mordred | bjackman: https://opendev.org/zuul/project-config/src/branch/master/zuul.d/pipelines.yaml#L28-L62 | 12:12 |
| mordred | there's the one for the zuul tenant, which is a little cleaner | 12:12 |
| *** quique|rover|eat is now known as quiquell|rover | 12:13 | |
| *** jamesmcarthur has joined #zuul | 12:18 | |
| bjackman | mordred, so as I understand the bug, I would expect every comment on a patch with Workflow +1 to trigger a re-gate | 12:18 |
| bjackman | The Verified -2 won't prevent it | 12:18 |
| mordred | I think it will - because a patch with verified -2 doesn't meet gerrit's approval criteria - so you shouldn't get approval payloads | 12:19 |
| mordred | although maybe that doesn't make any sense | 12:20 |
| mordred | bjackman: I'm still on my first coffee | 12:21 |
| bjackman | :D | 12:21 |
| bjackman | mordred, yeah the approval fields in the event stream are irrespective of the submittability criteria | 12:21 |
| mordred | oh - also - there was something you asked like 2 days ago that I was going to tell you something, but then you were gone, then I was gone ... lemme see if I can find what it was | 12:21 |
| *** jpena|lunch is now known as jpena | 12:22 | |
| mordred | ah! I found it: "bjackman | Or is there perhaps a way to run 'docker build' from the root of the Zuul ..." | 12:22 |
| mordred | bjackman: have you seen the image build jobs and/or the docs about getting them set up - we've got some really sexy docker image build jobs that do all the things you want them to do | 12:23 |
| mordred | bjackman: https://zuul-ci.org/docs/zuul-jobs/docker-jobs.html and https://zuul-ci.org/docs/zuul-jobs/docker-image.html | 12:23 |
| bjackman | Ah yeah I saw that, I was more thinking of something that I can use for a local development loop | 12:24 |
| mordred | nod | 12:24 |
| bjackman | Unfortunately it seems like as soon as you touch any of the source files you have to totally rebuild the base docker image | 12:24 |
| bjackman | Which is not a very fun thing to do through the Great Firewall of Vietnam's bandwidth haze | 12:25 |
| bjackman | So I have settled again on taking the upstream image then pasting my modified Zuul code over the top :S | 12:26 |
| bjackman | mordred, As for the trigger bug, I am starting to suspect that you are actually suffering from it but nobody ever notices. The only time you would notice would be if a patch failed gate checks and then some discussion took place in Gerrit before anyone uploaded a new patchset | 12:31 |
| mordred | bjackman: well that's what you get for not living in silicon valley with a fiber optic internet connection :) | 12:31 |
| bjackman | mordred, of course :D | 12:31 |
| mordred | bjackman: that should be easy enough to verify ... | 12:32 |
| bjackman | mordred, is there a sandbox repo or something?\ | 12:32 |
| *** jamesmcarthur has quit IRC | 12:32 | |
| bjackman | (Or we could just do a leap of faith and Code-Review +2 Workflow +1 a known broken patch) | 12:32 |
| mordred | bjackman: https://review.openstack.org/#/c/649466/ failed in the gate | 12:33 |
| mordred | bjackman: I have just left a comment on it | 12:34 |
| bjackman | Ah awesome | 12:34 |
| mordred | bjackman: now - it's also possible that this won't show the issue you're talking about since I wasn't the one to have originally voted +A | 12:34 |
| bjackman | I had a look for such a change but i guess my gerrit search-fu was lacking | 12:34 |
| mordred | so my comment may not have the +A vote in it | 12:34 |
| mordred | but let's see | 12:34 |
| bjackman | mordred, Ah yes, that is true | 12:34 |
| bjackman | I think it needs to be corvus | 12:35 |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: Break stuff https://review.openstack.org/649976 | 12:38 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: A reporter for Elasticsearch with the capability to index build and buildset results in an index. https://review.openstack.org/644927 | 12:38 |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: Break stuff https://review.openstack.org/649976 | 12:39 |
| bjackman | mordred, if you are brave you could try +1ing this: https://review.openstack.org/#/c/649976/1 | 12:39 |
| mordred | bjackman: nah - I think let's not do that in the zuul repo - we do have a sandbox repo we can use | 12:40 |
| bjackman | OK | 12:40 |
| mordred | bjackman: https://opendev.org/openstack-dev/sandbox | 12:40 |
| *** bhavikdbavishi has quit IRC | 12:41 | |
| bjackman | mordred, awesome will test there | 12:42 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul CLI: allow access via REST https://review.openstack.org/636315 | 12:58 |
| *** bjackman has quit IRC | 12:59 | |
| *** bjackman has joined #zuul | 13:02 | |
| bjackman | mordred, OK.. I didn't get an email, but when I load the change page at https://review.openstack.org/#/c/649980/1 I can briefly see a "Build failed" comment from Zuul, before the UI plugin kicks in and replaces it with the table view | 13:03 |
| bjackman | So I think the bug is there! | 13:03 |
| bjackman | Hmm actually, that is in the check pipeline | 13:03 |
| *** themroc has quit IRC | 13:05 | |
| bjackman | Ah yes now I have two emails | 13:05 |
| bjackman | mordred, Not sure how to find the pipeline config that's relevant to the sandbox project. But I commented again, and the check tests ran again. | 13:12 |
| *** quiquell|rover is now known as quiquell|off | 13:13 | |
| *** bhavikdbavishi has joined #zuul | 13:14 | |
| mordred | bjackman: https://opendev.org/openstack-infra/project-config/src/branch/master/zuul.d/pipelines.yaml#L58-L102 | 13:18 |
| mordred | bjackman: however - yes - I do believe that is demonstrating the behavior you've been talking about | 13:19 |
| *** bjackman has quit IRC | 13:19 | |
| mordred | bjackman: the gate pipeline for openstack has a "Clean check" requiement - so if check isn't positive, if zuul wants to put it into the gate, it'll re-run check first | 13:19 |
| mordred | which is why you are seeing it run the check twice | 13:20 |
| mordred | so we've got it narrowed down to "if gate jobs fail and the person who approvied it leaves a follow up comment and keeps the +A vote in their followup comment, it will retrigger the gate jobs to run, but not on comments from other people" | 13:21 |
| *** bjackman has joined #zuul | 13:37 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul CLI: allow access via REST https://review.openstack.org/636315 | 13:38 |
| *** bjackman has quit IRC | 13:45 | |
| *** bjackman has joined #zuul | 13:45 | |
| *** spsurya has quit IRC | 13:51 | |
| *** bhavikdbavishi has quit IRC | 13:51 | |
| *** altlogbot_2 has quit IRC | 13:54 | |
| *** bjackman has quit IRC | 13:56 | |
| *** bjackman has joined #zuul | 13:58 | |
| *** bjackman has quit IRC | 13:58 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul CLI: allow access via REST https://review.openstack.org/636315 | 13:58 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Retrieve full list of jobs with details per tenant via API https://review.openstack.org/635714 | 13:58 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add new merger job to get role definitions from a repository https://review.openstack.org/637181 | 13:58 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Provide roles per tenant via Zuul API https://review.openstack.org/650012 | 13:58 |
| *** bjackman has joined #zuul | 13:58 | |
| *** bjackman has quit IRC | 14:02 | |
| *** bjackman has joined #zuul | 14:02 | |
| *** bjackman has quit IRC | 14:09 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul CLI: allow access via REST https://review.openstack.org/636315 | 14:30 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add new merger job to get role definitions from a repository https://review.openstack.org/637181 | 14:45 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Provide roles per tenant via Zuul API https://review.openstack.org/650012 | 14:45 |
| *** hashar has quit IRC | 14:53 | |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Retrieve full list of jobs with details per tenant via API https://review.openstack.org/635714 | 15:01 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add new merger job to get role definitions from a repository https://review.openstack.org/637181 | 15:01 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Provide roles per tenant via Zuul API https://review.openstack.org/650012 | 15:01 |
| pabelanger | Hmm, I'm unsure why 2 master branches show up on the following zuul dashboard page: https://dashboard.zuul.ansible.com/t/ansible/project/ansible-network/windmill-config | 15:37 |
| pabelanger | looks like the project config isn't merged properly | 15:38 |
| clarkb | pabelanger: I think they are from two different projects | 15:40 |
| clarkb | so its projectA/master projectB/master ? | 15:40 |
| corvus | there's information missing from that page; we need to add the source context there | 15:41 |
| corvus | pabelanger: but it means that somewhere you have a project stanza that adds the 'ansible-python-jobs' template, and somewhere else you have a project stanza which adds the 'system-required' template | 15:41 |
| pabelanger | Yah, that is correct. they are both defined in different projects | 15:44 |
| pabelanger | okay, that better explains it | 15:44 |
| Shrews | ianw: https://review.openstack.org/638533 has apparently had two +2's for over a month. :/ you ready for that to merge now? | 15:47 |
| Shrews | also, the zk auth review for nodepool (https://review.openstack.org/619155) has had my +2 for a couple of weeks. corvus, mordred, tobiash... interested in reviewing that? | 15:50 |
| corvus | yes! | 15:50 |
| mordred | Shrews, corvus: https://review.openstack.org/#/c/640643/ also has 2 +2s now - shall we merge it too? | 15:51 |
| Shrews | mordred: i'd like to review that one first... doing so soon | 15:51 |
| *** sshnaidm_ has joined #zuul | 15:52 | |
| Shrews | tristanC's openshift pod driver could also use a review | 15:53 |
| Shrews | https://review.openstack.org/590335 | 15:53 |
| *** sshnaidm has quit IRC | 15:54 | |
| clarkb | re 638533 that is an improvement on some relatively new stats reporting so should be fairly safe to get in (we don't have to worry about people having used a stable metric name for 5 years) | 15:54 |
| *** bhavikdbavishi has joined #zuul | 15:55 | |
| Shrews | clarkb: k. feel free to approve that then, if you wish | 15:55 |
| clarkb | ya reviewing now to make sure I've not missed anything but can approve when done | 15:56 |
| clarkb | ya its a slight change if you managed to find the new feature but I think this is a necessary update to reduce confusion around the metrics all being the same for the different formats | 15:57 |
| clarkb | considering it hasn't existed in that form for very long I think we can get it in now | 15:57 |
| * clarkb approves | 15:57 | |
| *** jamesmcarthur has joined #zuul | 15:58 | |
| Shrews | mordred: approved | 15:59 |
| Shrews | mordred: and commented about the start/stop methods | 16:00 |
| *** yolanda has quit IRC | 16:33 | |
| *** electrofelix has quit IRC | 16:58 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Remove TaskManager and just use keystoneauth https://review.openstack.org/640643 | 17:03 |
| *** jpena is now known as jpena|off | 17:08 | |
| *** gtema has quit IRC | 17:14 | |
| *** bjackman has joined #zuul | 17:18 | |
| *** jamesmcarthur has quit IRC | 17:19 | |
| *** bjackman has quit IRC | 17:23 | |
| tobiash | corvus: I think I have an issue with an sql issue in provides/requires with postgres: http://paste.openstack.org/show/748883/ | 17:25 |
| tobiash | corvus: this exception bubbled up to the runhandler which effectively made zuul stop doing anything | 17:25 |
| *** rlandy is now known as rlandy|ruck | 17:27 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add some quoting in 'gerrit query' commands https://review.openstack.org/649879 | 17:34 |
| *** shachar has joined #zuul | 17:34 | |
| corvus | tobiash: i don't understand that error; does the table definition look correct? | 17:35 |
| tobiash | corvus: I'm currently digging up the actual sql query so I can check that | 17:36 |
| tobiash | corvus: but we should at least catch that exception so this doesn't block all of zuul | 17:36 |
| corvus | tobiash: i wonder if we have pg tests for that, or are they only mysql | 17:37 |
| corvus | yeah | 17:37 |
| *** snapiri has quit IRC | 17:37 | |
| corvus | should probably fail the whole buildset in that case | 17:37 |
| *** spsurya has joined #zuul | 17:37 | |
| tobiash | corvus: the where clause on zuul_provides.name, should that be a string or a list of strings? | 17:38 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Proposed spec: tenant-scoped admin web API https://review.openstack.org/562321 | 17:40 |
| tobiash | corvus: this is the query with a mark where I expect the error: http://paste.openstack.org/show/748887/ | 17:40 |
| tobiash | corvus: yeah, sql expects a string there and zuul put a tuple or list in there | 17:42 |
| Shrews | i think you want to use IN there | 17:43 |
| Shrews | zuul_provides.name IN ('name1', 'name2') ... or some such | 17:44 |
| corvus | tobiash: i think i see the bug | 17:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Fix SQL error when querying for tuples https://review.openstack.org/650086 | 17:46 |
| corvus | tobiash, Shrews: ^ i think that's the bug -- my guess is we don't have a test case with two provides/requires so we're not hitting that | 17:46 |
| tobiash | corvus: sounds reasonable, thanks a lot! | 17:46 |
| Shrews | that seems about right | 17:47 |
| corvus | i left a comment on that change pointing to the invocation | 17:47 |
| corvus | i'm unable to finish that change right now, so if anyone wants to continue working on it, feel free; otherwise i'll get back to it maybe this afternoon or tomorrow | 17:48 |
| tobiash | corvus: thanks for the starter, I'll finish it | 17:49 |
| corvus | tobiash: or perhaps in the mean time you may want to try that as an emergency patch :) | 17:49 |
| tobiash | my emergency action was to dequeue the change and add a comment to not recheck :) | 17:50 |
| corvus | i think test_provides_requires_check_old_success may be a good candidate to expand the test for a second artifact | 17:50 |
| *** sshnaidm_ is now known as sshnaidm|off | 17:50 | |
| corvus | tobiash: ^ | 17:50 |
| tobiash | thanks, will do | 17:50 |
| corvus | ok, really afk now | 17:50 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix debug log message https://review.openstack.org/650088 | 18:05 |
| Shrews | super easy review ^^^ | 18:06 |
| openstackgerrit | Merged openstack-infra/nodepool master: Update dib stats https://review.openstack.org/638533 | 18:14 |
| *** bhavikdbavishi has quit IRC | 18:22 | |
| corvus | Shrews: +2 on 619155, did not +3 because i'm about to go on vacation; feel free to do so of course | 18:35 |
| corvus | i guess you can read that 2 ways. so... er, feel free to +3. or feel free to go on vacation. :) | 18:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix SQL error when querying for tuples https://review.openstack.org/650086 | 18:36 |
| clarkb | but that or is an xor? | 18:36 |
| corvus | *absolutely* | 18:36 |
| *** ChrisShort has joined #zuul | 18:37 | |
| tobiash | corvus: have a nice vacation :) | 18:38 |
| corvus | thanks! i'm still around today and tomorrow; i just don't want to take on any ops duties that might extend past that | 18:38 |
| tobiash | ah ok | 18:38 |
| tobiash | clarkb, corvus, Shrews: I added tests for 650086 so this should be ready for review | 18:40 |
| *** hashar has joined #zuul | 18:42 | |
| openstackgerrit | Merged openstack-infra/nodepool master: doc: add provider driver development instructions https://review.openstack.org/639595 | 18:47 |
| daniel2 | is there a way to make nodepool ignore SSL errors when connecting to a jenkins target? | 19:04 |
| Shrews | corvus: what if i approve it, restart nodepool, and THEN IMMEDIATELY go on vacation too??? | 19:05 |
| Shrews | :) | 19:05 |
| tobiash | Shrews: that is a great idea :) | 19:06 |
| Shrews | the place i used to work for back when dinosaurs walked the earth had a guy that would religiously implement untested things in production every friday right before he left | 19:06 |
| Shrews | he was a founder, so we couldn't say anything about it. just deal with the fallout | 19:07 |
| tobiash | oh, that's a bad combination | 19:07 |
| Shrews | he also coded as root | 19:07 |
| Shrews | corvus: nodepool's zuul-quick-start job seems to be failing a lot today | 19:12 |
| Shrews | http://logs.openstack.org/64/642964/1/gate/zuul-quick-start/31afd7b/job-output.txt.gz#_2019-04-04_18_35_46_753527 | 19:13 |
| pabelanger | so, I am seeing something odd in my zuul scheduler debug logs, and unsure why I am seeing it. | 19:21 |
| pabelanger | for example: http://paste.openstack.org/ | 19:21 |
| pabelanger | sorry | 19:21 |
| pabelanger | http://paste.openstack.org/show/748896/ | 19:21 |
| pabelanger | I am unsure why those projects are showing up | 19:21 |
| pabelanger | as far as I can tell, I didn't add our github app to those repos | 19:22 |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix debug log message https://review.openstack.org/650088 | 19:29 |
| mordred | pabelanger: I was thinking that maybe something in a reverse search | 19:31 |
| Shrews | pabelanger: i'm suspected it's the "in:body" part of the query that's finding those. see: https://github.com/project-koku/korekuta/pull/57 as an example | 19:31 |
| mordred | pabelanger: but like, https://github.com/project-koku/korekuta/pull/57 | 19:31 |
| mordred | Shrews: jinx | 19:31 |
| Shrews | i wish we had the github query logged so we could see change.uris values it's using there | 19:34 |
| Shrews | pattern = ' OR '.join(change.uris) | 19:34 |
| Shrews | query = '%s type:pr is:open in:body' % pattern | 19:34 |
| mordred | Shrews: that's not a terrible idea | 19:36 |
| pabelanger | Hmm, k, my brain isn't making the click right now. But in https://github.com/project-koku/korekuta/pull/57 I don't see any reference to project-config | 19:36 |
| Shrews | mordred: pabelanger: oh it's there! self.log.debug("Ran search issues: %s", query) | 19:38 |
| Shrews | pabelanger: can you extract that? | 19:38 |
| Shrews | should be shortly after the "Found PR..." message | 19:39 |
| pabelanger | sure, 1 sec | 19:39 |
| pabelanger | Shrews: so, I see something like: http://paste.openstack.org/show/748899/ | 19:42 |
| pabelanger | then a large number of dynamically created queue | 19:43 |
| Shrews | i put that in github.com search box and project-koku/korekuta is the first hit | 19:43 |
| Shrews | for SOME reason | 19:43 |
| Shrews | that being: https://github.com/search?q=github.com%2Fansible%2Fproject-config%2Fpull%2F30+type%3Apr+is%3Aopen+in%3Abody | 19:43 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Whitelist pageup and down keys in xterm https://review.openstack.org/649694 | 19:44 |
| tobiash | pabelanger: ^ | 19:44 |
| mordred | Shrews: maybe it's only looking for each of the tokens individually | 19:44 |
| pabelanger | tobiash: thanks! will review shortly | 19:45 |
| Shrews | mordred: maybe, but that would be super lame | 19:45 |
| Shrews | or maybe the / character is being ignored/removed for the search | 19:47 |
| Shrews | so that it becomes: github.com ansible project-config pull 30 | 19:48 |
| corvus | if so, we probably need to perform our own filter on the returned results | 19:48 |
| corvus | "ansible" "project" and "config" are not very distinctive :) | 19:49 |
| corvus | tobiash: i think the config in that provides/requires test is shared by another, so the other test needs updating too; are you fixing that or do you want me to take it over again? | 19:55 |
| tobiash | corvus: just noticed that and already on it | 19:56 |
| corvus | k | 19:56 |
| corvus | Shrews: have a link to a quickstart failure? | 19:56 |
| Shrews | corvus: posted after my mention | 19:56 |
| corvus | derp | 19:56 |
| corvus | heh, i just found that change independently :) | 19:56 |
| tobiash | I'll unshare the 'both-user' so we have buildsets with one and buildsets with multiple requirements | 19:57 |
| corvus | k | 19:57 |
| corvus | Shrews: the line you linked isn't actually the failure -- it's an ignored error. it looks like that job failed because it hung here: http://logs.openstack.org/64/642964/1/gate/zuul-quick-start/31afd7b/job-output.txt.gz#_2019-04-04_18_35_46_793624 | 19:59 |
| corvus | note the two timestamps at the top there | 19:59 |
| Shrews | yeah, wasn't sure if that error was an artifact of the timeout, or the cause of it, or even expected | 20:00 |
| corvus | Shrews: tobiash noted that the job had been taking longer (mostly because the upstream ubuntu mirrors have been having problems). the failures i have seen previously had more output though, so i'm not sure if that case is related to what tobiash observed, or something new. | 20:01 |
| corvus | regardless, he proposed https://review.openstack.org/649384 to deal with it temporarily | 20:01 |
| corvus | and started looking at https://review.openstack.org/649448 for a more long-term solution | 20:01 |
| tobiash | corvus: didn't you propose a better fix using provides/requires? | 20:01 |
| tobiash | ah that :) | 20:01 |
| Shrews | i +3'd that | 20:02 |
| corvus | meanwhile, i also have some changes which restructure that job to use the buildset registry, so it shouldn't actually build images anymore | 20:02 |
| corvus | they hit some merge -2s, i'll kick them | 20:02 |
| corvus | mordred left a cryptic message on https://review.openstack.org/649466 : "test trigger issue" | 20:03 |
| corvus | i'm not sure if that was an analysis of the failure or.... | 20:03 |
| corvus | it doesn't make sense that zuul-tox-remote would fail on that change, and it looks like it's just a busy system timeout, so i'll recheck it | 20:04 |
| tobiash | corvus: I had the same thought | 20:05 |
| corvus | there are 2 changes after that, then we'll have the new zuul-quick-start job | 20:05 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix SQL error when querying for tuples https://review.openstack.org/650086 | 20:05 |
| mordred | corvus: that has nothing to do with anything | 20:06 |
| corvus | mordred: surrealist review comments? | 20:06 |
| pabelanger | I kinda got side tracked by the unknown projects in scheduler logs, but I was really trying to figure out why I'm getting to following error from zuul: Unable to freeze job graph: Project ansible/project-config is not allowed to run job windmill-config-deploy best I can see, I've defined allowed-projects correctly: https://github.com/ansible-network/windmill-config/blob/master/.zuul.d/jobs.yaml#L2 | 20:07 |
| pabelanger | I am starting to wonder if it is because the job has a secret | 20:07 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix SQL error when querying for tuples https://review.openstack.org/650086 | 20:08 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix SQL error when querying for tuples https://review.openstack.org/650086 | 20:08 |
| * tobiash hates that .keep file... | 20:08 | |
| tobiash | mordred: do you know if there's a way to tell yarn not to delete the .keep file? | 20:09 |
| fungi | openstack feedback on zuul's oip confirmation was just sent to the foundation ml: http://lists.openstack.org/pipermail/foundation/2019-April/002747.html | 20:09 |
| tobiash | every time I run yarn it deletes it and I accidentally add the removal of it to the next commit I do... | 20:10 |
| corvus | pabelanger: see paragraph 2 of https://zuul-ci.org/docs/zuul/user/config.html#attr-job.allowed-projects | 20:11 |
| corvus | and the big red box for the reason :) | 20:12 |
| pabelanger | Ha, okay that is what I figured. | 20:12 |
| pabelanger | I can actually refactor that secret out, with the usuage of project ssh keys | 20:13 |
| pabelanger | usage* | 20:13 |
| corvus | pabelanger: i also discussed a potential change to that with logan- that might apply (where we could override that in a config project) but haven't implemented it yet | 20:14 |
| pabelanger | okay, cool. For now, I'll drop the secret usage and try again. | 20:20 |
| *** jamesmcarthur has joined #zuul | 20:21 | |
| *** jamesmcarthur has quit IRC | 20:25 | |
| *** pcaruana has quit IRC | 20:29 | |
| *** pcaruana has joined #zuul | 20:30 | |
| pabelanger | tobiash: +2 works great, thanks | 20:33 |
| pabelanger | https://review.openstack.org/649694/ is anybody else would like to review for pgup / pgdn support in xterm.js | 20:33 |
| openstackgerrit | Merged openstack-infra/zuul master: Increase timeout of zuul-quick-start https://review.openstack.org/649384 | 20:35 |
| pabelanger | Hmm, I have a feeling out log streaming might only work if ansible_host is an IP address. Trying to confirm now. | 20:45 |
| pabelanger | s/out/our | 20:46 |
| SpamapS | corvus: FYI, I moved us from a post pipeline (pushes and git hashes) to a promote pipeline (PRs close merged) last night, whith gate built images going live. It works great. Thanks for blazing that trail! | 20:46 |
| corvus | SpamapS: presentation time! :) | 20:47 |
| mordred | SpamapS: \o/ | 20:47 |
| pabelanger | mordred: am I reading http://git.zuul-ci.org/cgit/zuul/tree/zuul/ansible/base/callback/zuul_stream.py#n123 right, if ip is None, we'd raise an exception, and don't seem to use host, if able to resolve IP from it | 20:49 |
| pabelanger | think that is what I am seeing currently | 20:50 |
| pabelanger | Yah, that is the issue. Reverting back to ansible_host=<ip> fixes console streaming. | 20:52 |
| pabelanger | I'll try and work on a patch | 20:52 |
| *** spsurya has quit IRC | 20:54 | |
| *** sshnaidm|off has quit IRC | 20:55 | |
| *** sshnaidm|off has joined #zuul | 20:58 | |
| *** mhu has quit IRC | 20:58 | |
| *** mhu has joined #zuul | 21:04 | |
| *** rlandy|ruck is now known as rlandy|ruck|brb | 21:21 | |
| daniel2 | I keep getting this with nodepool, cause we are using a self signed cert: SSLError: HTTPSConnectionPool(host='jenkins.*****', port=443): Max retries exceeded with url: /crumbIssuer/api/json (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)) | 21:22 |
| daniel2 | Not sure how to bypass this | 21:22 |
| clarkb | daniel2: hrm when we used jenkins we used self signed certs too but it may have just worked due to old python's inability to check validity | 21:25 |
| daniel2 | I added the cert to ubuntu's trust. | 21:25 |
| daniel2 | But didn't help. | 21:25 |
| clarkb | did you restart nodepool after that? | 21:25 |
| clarkb | and I think you have to run a ommand to rebuild the trust list too | 21:26 |
| daniel2 | clarkb: did update-ca-certificates, restarted nodepool and builder, still fails. | 21:27 |
| clarkb | does openssl s_client work after that? | 21:28 |
| daniel2 | clarkb: yeah, returns verified. | 21:29 |
| mordred | if you installed nodepool via pip, then the ubuntu ca update doesn't matter since it uses its own cert store | 21:30 |
| clarkb | oh right python requests uses its own cert list | 21:30 |
| mordred | yeah | 21:30 |
| mordred | for that I've lost track of the situation with how one might globally add one - I know that feeding a CA into the requests constructor works - but that might be too hidden to be useful | 21:31 |
| openstackgerrit | Merged openstack-infra/zuul master: Use opendev docker jobs https://review.openstack.org/649466 | 21:31 |
| daniel2 | I dont really want to directly modify nodepool's code. | 21:32 |
| daniel2 | I did try it already, using verify=False to the Jenkins class | 21:33 |
| daniel2 | But it didn't work. | 21:33 |
| clarkb | REQUESTS_CA_BUNDLE is the env var to set to an alternate path | 21:34 |
| clarkb | can you try running nodepool with that set to the system store /etc/ssl/certs/ca-certificate | 21:34 |
| clarkb | er ends with .crt suffix iirc | 21:35 |
| daniel2 | clarkb: it was just /etc/ssl/certs, but that worked, thanks! | 21:36 |
| clarkb | yay | 21:37 |
| *** rlandy|ruck|brb is now known as rlandy|ruck | 21:39 | |
| *** pcaruana has quit IRC | 21:51 | |
| *** hashar has quit IRC | 22:16 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Use the buildset registry in zuul-quick-start https://review.openstack.org/649467 | 22:27 |
| corvus | Shrews, mordred, tobiash: ^ there's the rebase of that due to the just-merged timeout increase; i removed the timeout increase, since the job no longer builds images | 22:28 |
| ianw | corvus: are you around to help debug why the arm64 node isn't getting assigned for https://review.openstack.org/552700 right now? | 22:35 |
| corvus | ianw: sure i can take a look | 22:37 |
| ianw | we had an active xenail arm64 node, i even logged into it, so trying to figure out why it didn't take that | 22:38 |
| *** mrhillsman_bbiab is now known as mrhillsman | 22:40 | |
| corvus | mordred, Shrews: that's failing with: openstack.exceptions.SDKException: Error in creating the server | 22:53 |
| corvus | mordred, Shrews: full traceback: http://paste.openstack.org/show/748911/ | 22:53 |
| corvus | (but of course, the installed version is different; that launcher was started on march 04) | 22:53 |
| corvus | mordred, Shrews: is it possible for openstacksdk to provide any more information in that case? | 22:54 |
| corvus | ianw: ^ | 22:54 |
| mordred | looking | 22:54 |
| corvus | maybe that's just sdk passing back what it got from nova... but it's unclear | 22:55 |
| mordred | yeah, this is what is happening. there are two possibilities | 22:55 |
| mordred | one is that we get ERROR back from nova, and also a json body with {'status': {'fault': 'some message'}} | 22:56 |
| mordred | if we get that, we include some message | 22:56 |
| mordred | the error message will say "Error in creating the server. Compute service reports fault: {reason}" | 22:57 |
| corvus | (based on line numbers, i think we must be running 0.23.0) | 22:57 |
| corvus | mordred: so it looks like the other case -- server['status'] == ERROR but with no 'fault' | 22:58 |
| mordred | yeah | 22:58 |
| mordred | in that case, to the best of my current knowledge, there is nowhere else to look for information | 22:58 |
| corvus | mordred: any idea where this code moved to in master? | 22:58 |
| corvus | oh, maybe _compute.py? | 22:59 |
| mordred | yes. openstack/cloud/_compute.py | 22:59 |
| mordred | yeah | 22:59 |
| mordred | should be the same code | 22:59 |
| corvus | mordred: remote: https://review.openstack.org/650161 Expand on a brief error message | 23:01 |
| mordred | corvus: ++ | 23:01 |
| corvus | mordred: feel free to take that or leave it; that would have just assured me that we really did hit the case where the remote server said "nope go away" | 23:01 |
| mordred | corvus: yeah - it's a fair thing to double check (sadly) given that there are at least a billion different places errors can come from - I'd also probably start debugging by looking to see if that was a masked exception :) | 23:02 |
| *** rlandy|ruck is now known as rlandy | 23:03 | |
| corvus | mordred: there are masked exceptions? | 23:03 |
| corvus | mordred: oh, or were you just saying you agree with my concern which lead to asking about the error | 23:03 |
| mordred | corvus: no. but that doesn't mean I wouldn't check first to make sure there weren't | 23:03 |
| mordred | yeah | 23:04 |
| corvus | gotcha. ++ | 23:04 |
| corvus | (and, ftr, from what i can see of having just looked at that code, that's not the case here) | 23:04 |
| mordred | yeah. I'd be more concerned if it were an API call - because there are a gajillion different ways error messages come back from API calls | 23:05 |
| mordred | but in this case the API is working and we have the server object - it's just an unhappy object | 23:05 |
| ianw | corvus / modred: ok thanks, i'll update https://storyboard.openstack.org/#!/story/2005353 and already ahve a ticket open with linaro about it ... it looks like gema fell into the unregistered channel so i'll try to reach out | 23:40 |
| openstackgerrit | Merged openstack-infra/zuul master: Use the buildset registry in zuul-quick-start https://review.openstack.org/649467 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!