| SpamapS | aaaand EOD strikes again | 00:00 |
|---|---|---|
| * SpamapS vows to return | 00:00 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Query changes only via relevant sources https://review.openstack.org/448257 | 00:45 |
| jlk | ^^ That was a 31 line (outside of tests) patch for 2.5, down to a 5 line patch for v3. | 00:52 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks https://review.openstack.org/446275 | 01:33 |
| jesusaur | jlk: wow, awesome, that's so much simpler | 01:34 |
| SpamapS | oh how embarassing | 05:24 |
| * SpamapS accidentally started refactoring on master.. that makes a lot of sense why it made no damn sense at all | 05:24 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Refactor out Changeish https://review.openstack.org/448913 | 06:08 |
| SpamapS | jeblair: please disregard my master-brain-fart ... ^^ is the real slim shady | 06:08 |
| SpamapS | though now I have 65 fails instead of 26 | 06:08 |
| *** isaacb has joined #zuul | 06:17 | |
| *** jasondotstar has quit IRC | 07:03 | |
| *** tristanC has quit IRC | 07:03 | |
| *** dmellado has quit IRC | 07:03 | |
| *** rattboi has quit IRC | 07:04 | |
| *** tristanC has joined #zuul | 07:05 | |
| *** rattboi has joined #zuul | 07:05 | |
| *** jasondotstar has joined #zuul | 07:08 | |
| *** dmellado has joined #zuul | 07:08 | |
| *** isaacb has quit IRC | 07:13 | |
| *** isaacb has joined #zuul | 07:23 | |
| lennyb | Hi, is there a way to skip files in zuul, but still be able to merge the code? I have a gating job #link http://paste.openstack.org/show/603873/ but I want to skip checking some files. currently I see a note of 'Staring job' in gerrit, but since no job was triggered, there is no SUCCESS/FAIL message and no code merge. | 08:40 |
| tobiash_ | lennyb: if you filter out every job but still want zuul to gate/merge stuff you can add additionally the "noop" job | 08:49 |
| *** openstackgerrit has quit IRC | 09:03 | |
| *** hashar has joined #zuul | 09:03 | |
| lennyb | tobiash_, thanks. | 09:25 |
| lennyb | tobiash_, you mean adding 2 jobs, one that will listen only to files that I want to skip, run noop and vote/merge, and another one to listen to all the rest? | 09:51 |
| tobiash_ | lennyb: no, just add noop to your project | 10:07 |
| tobiash_ | lennyb: noop is a reserved 'do nothing' job zuul provides and doesn't consume any resources like slaves | 10:08 |
| tobiash_ | lennyb: adding this then just assures that at least the noop job is run | 10:08 |
| tobiash_ | lennyb: in case your job is not filtered then your job plus noop 'run' | 10:09 |
| tobiash_ | lennyb: in case your job is filtered just the 'noop' job 'runs' which means zuul just checks if the patch can merge | 10:09 |
| * lennyb went to implement tobiash_ 's instructions. | 10:15 | |
| lennyb | I guess there is still no way to reload layout.yaml with restarting zuul | 10:31 |
| *** bhavik1 has joined #zuul | 10:47 | |
| lennyb | tobiash_, thanks. noop works. | 10:55 |
| lennyb | is there a way to configure zuul, that 'gate' job will abort/cancel ordinary 'check' pipeline/job that was already started? | 10:56 |
| tobiash_ | lennyb: how does that happen? normally gate only starts when check is done | 10:58 |
| lennyb | tobiash_, if I understood you correctly, noop still fails on merge conflict. I am trying to prevent bad code merges. | 10:59 |
| lennyb | I triggered gate pipeline without waiting for check to be finished | 11:00 |
| lennyb | and added precedence: high to gate pipeline | 11:00 |
| tobiash_ | lennyb: yes, noop still fails on merge conflict | 11:01 |
| tobiash_ | lennyb: zuul cannot cancel check jobs when gate starts | 11:01 |
| lennyb | tobiash_, thanks. your assistance and help highly appreciated | 11:04 |
| tobiash_ | yw | 11:05 |
| *** bhavik1 has quit IRC | 11:08 | |
| pabelanger | lennyb: SIGHUP on zuul will reconfigure jobs. No need to stop the process | 11:27 |
| lennyb | pabelanger thanks | 11:27 |
| *** openstackgerrit has joined #zuul | 12:26 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add check for valid zk attribute before disconnect https://review.openstack.org/449110 | 12:26 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Do not require secure file for nodepoold https://review.openstack.org/449118 | 12:37 |
| pabelanger | Shrews: jeblair: re: 446683. If I try / except for closing the socket, is it safe to assume if the socket fails to close, not to try closing Transport(), since it uses the socket? Or just be safe and try / except both individually | 12:51 |
| Shrews | pabelanger: not sure. we should probably try to close it anyway. i think clarkb (i may be mistaken) mentioned something somewhere about leaking paramiko connections causing issues... maybe i read that in #infra | 12:53 |
| Shrews | i can't remember much about it | 12:53 |
| pabelanger | safe is better | 12:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Remove SSH support from nodepool https://review.openstack.org/446683 | 12:56 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Docs: Remove "jenkins" reference https://review.openstack.org/449129 | 13:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Docs: Clarify secure file usage https://review.openstack.org/449130 | 13:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Docs: Remove cron references https://review.openstack.org/449140 | 13:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Docs: Correct availability-zones documentation. https://review.openstack.org/449147 | 13:30 |
| Shrews | hrm, our nodepool dsvm jobs seem to be timing out a lot now on ssh access | 13:33 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Docs: Remove refs to removed nodepool commands https://review.openstack.org/449152 | 13:39 |
| mordred | Shrews: I was just about to ask you if that was a thing you were seeing too | 13:42 |
| Shrews | mordred: are you seeing this outside of nodepool? | 13:42 |
| Shrews | well, i guess not | 13:42 |
| mordred | Shrews: not outside of nodepool -but I noticed it on a shade patch | 13:43 |
| Shrews | fun | 13:43 |
| mordred | yah. I'm super excited about that | 13:43 |
| Shrews | mordred: guess it's time to start digging into devstack changes | 13:44 |
| tobiash_ | how does nodepool check if the node has fully booted in the future if ssh support gets removed? | 13:45 |
| mordred | Shrews: I just pinged in qa channel | 13:45 |
| tobiash_ | via keyscan? | 13:46 |
| pabelanger | tobiash_: currently yes | 13:46 |
| mordred | tobiash_: I think the ssh-keyscan will still get us that - the keyscan shouldn't work unless ssh daemon is running and booted | 13:46 |
| pabelanger | right, a little trade off, as that doesn't really show us the inside of the VM | 13:47 |
| tobiash_ | ok, I think that will work as long as the user is preexisting in the image | 13:48 |
| pabelanger | I'm kinda on the fense of the removal of it, cause there is some value on doing SSH things in nodepool today. But, I understand why we want to remove it | 13:48 |
| tobiash_ | less code is better code :) | 13:49 |
| pabelanger | ya | 13:49 |
| pabelanger | Happy to try it out too | 13:49 |
| tobiash_ | just curious, is ansible capable of retrying ssh connections? | 13:50 |
| pabelanger | I think so | 13:50 |
| pabelanger | check ansible.cfg | 13:50 |
| mordred | I actually do not see a setting for that | 13:51 |
| mordred | although ssh -o arguments can be passed through | 13:51 |
| pabelanger | https://github.com/ansible/ansible/blob/devel/lib/ansible/constants.py#L374 | 13:52 |
| pabelanger | seems to be undocumented | 13:52 |
| Shrews | jeblair: rbergeron: I have made a pass through the current nodepool docs and put up reviews for inconsistencies I found. | 13:54 |
| Shrews | rbergeron: if you're interested, https://review.openstack.org/#/q/status:open+project:openstack-infra/nodepool+branch:feature/zuulv3+topic:docs | 13:55 |
| tobiash_ | would have saved me some time if this cool feature would have been documented :-/ | 13:56 |
| Shrews | mordred: wow. every review i have up is failing dsvm jobs | 14:00 |
| mordred | Shrews: yah - I have this feeling something has recently broken in openstack | 14:01 |
| Shrews | pabelanger: "Exception closing paramiko: local variable 't' referenced before assignment" | 14:02 |
| Shrews | pabelanger: http://logs.openstack.org/83/446683/9/check/gate-dsvm-nodepool/0dd848b/logs/screen-nodepool.txt.gz | 14:02 |
| pabelanger | Shrews: k | 14:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Remove SSH support from nodepool https://review.openstack.org/446683 | 14:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Add novaclient and keystoneauth debug logging https://review.openstack.org/449173 | 14:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Fetch server console log if ready_script fails https://review.openstack.org/358730 | 14:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Fetch server console log if ready_script fails https://review.openstack.org/358730 | 14:49 |
| clarkb | Shrews: pabelanger ya you need explicit closes on paramiko clients now or they leak connections | 14:54 |
| rbergeron | shrews: aye, will peek -- also, i am in RDU on monday / tuesday next week if you want to get together at some point in there (though my schedule is still in flux for things being done while there) | 15:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Fetch server console log if ssh connection files https://review.openstack.org/358730 | 15:34 |
| *** isaacb has quit IRC | 15:49 | |
| jlk | o/ | 16:23 |
| *** isaacb has joined #zuul | 17:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Fetch server console log if ssh connection fails https://review.openstack.org/358730 | 17:14 |
| * mordred o/ at jlk | 17:54 | |
| *** hashar has quit IRC | 17:58 | |
| *** isaacb has quit IRC | 18:05 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add support for github enterprise https://review.openstack.org/449258 | 18:06 |
| jeblair | clarkb, mordred, Shrews: regarding http://lists.openstack.org/pipermail/openstack-infra/2017-January/005018.html should ipv6-preferred be a provider or pool attribute? | 19:58 |
| mordred | jeblair: neither - we should just leave that in clouds.yaml | 20:00 |
| clarkb | jeblair: thats a hard one. | 20:00 |
| clarkb | mordred: but its nodepool config as nodepool also needs to be able to speak ipv6 | 20:00 |
| clarkb | so I think clouds.yaml is the wrong place for it | 20:00 |
| mordred | I disagree | 20:00 |
| mordred | the current logic is that shade detects whether the calling host can speak ipv6 | 20:00 |
| clarkb | if a cloud can speak ipv6 that doesn't reflect on whether or not nodepool can too | 20:00 |
| mordred | and if so, it defaults to ipv6 | 20:01 |
| mordred | ipv6-preferred is only used to turn off ipv6 communication | 20:01 |
| jeblair | oh i was assuming it was still a thing since https://review.openstack.org/410802 is still around, but a confess i don't know the current status of that | 20:01 |
| mordred | jeblair: yah - I want to land that for v3 | 20:01 |
| mordred | jeblair: I stopped pushing it for v2 because v3 was coming | 20:01 |
| jeblair | mordred: but that still has the option? | 20:01 |
| mordred | that's because I was trying to not break backwards compat too badly | 20:02 |
| mordred | but we're already breaking in v3 - so I think cleaning out the things where nodepool and shade duplicate each other would be stellar | 20:02 |
| mordred | maybe I should respond to your email with a write up of the bits I think shoudl all no longer be in nodepool.yaml so we can discuss | 20:02 |
| clarkb | mordred: so you'd proviude different clouds.yamls based on local host networking? (that seems wrong but I guess it would work so maybe there isn't anything wrong with it) | 20:02 |
| jeblair | mordred: might be easier to review https://review.openstack.org/447647 | 20:03 |
| mordred | clarkb: not really - you should normally never need the setting - it's only a release valve in case something is broken in your setup | 20:03 |
| mordred | jeblair: kk | 20:03 |
| clarkb | mordred: which is super common with ipv6 (which is why I am concerned) | 20:03 |
| clarkb | mordred: I had to turn off ipv6 globally at home when comcasting more often than I care to remember | 20:03 |
| mordred | clarkb: right - but so far none of our clouds with ipv6 have been comcast | 20:04 |
| mordred | but if they were, we'd just add "force-ipv4: true" into the clouds.yaml for our comcast cloud | 20:04 |
| mordred | THAT SAID - I still may be wrong here | 20:04 |
| mordred | since the host we're using to create the VM and the host we're actually using it from are, in fact, different hosts | 20:05 |
| mordred | so the shade-level detection of ipv6 capabilities is not going to work for us | 20:05 |
| clarkb | ya I think that is functional, just weird we'd use different clouds.yamls for the same cloud user/tenant depending on where we use the clients | 20:05 |
| jeblair | mordred: but still, in that case, we could set the ipv4 thing in clouds.yaml, right? | 20:05 |
| mordred | clarkb: I do not think that's what I was suggesting | 20:05 |
| mordred | jeblair: yes | 20:05 |
| mordred | clarkb: on our servers, if one of our servers we're using to launch things had broken ipv6, I'd expect us to deconfigure it fro that server's network config | 20:06 |
| mordred | and not have broken ipv6 configured live on a production server | 20:06 |
| clarkb | gotcha | 20:06 |
| clarkb | good pint | 20:06 |
| jeblair | clarkb: good pint? cheers! ;) | 20:07 |
| clarkb | prost | 20:07 |
| mordred | jeblair: so - I think there are a few different concerns | 20:07 |
| jeblair | so i think what i will do is leave it as a provider attribute in the patch i'm writing, then remove it as a followup, so it doesn't get buried. but now i will wait to hear concerns. :) | 20:08 |
| mordred | 1) servers launching and using clouds having broken ipv6 - can be dealt with as mentioned above, don't think we need an option - if you have broken ipv6, stop configuring broken | 20:08 |
| mordred | 2) clouds giving broken ipv6 - clouds.yaml has force-ipv4, we're fine | 20:08 |
| mordred | 3) an installation of zuul v3 which is not ours in which the nodepool launchers are on ipv6 enabled hosts but the zuul executors are not | 20:09 |
| mordred | I think I'd rather let people who try to run in that configuration be broken adn complain to us and then justify why they should not fix their environment | 20:09 |
| jeblair | mordred: 3 can still set force-ipv4, right? | 20:10 |
| mordred | jeblair: yes - 3 could also still set force-ipv4 - good point | 20:10 |
| mordred | so there is a release valve for those people | 20:10 |
| mordred | cool | 20:10 |
| jeblair | we can even put that in the manual with a lecture. :) | 20:10 |
| mordred | +100 | 20:10 |
| mordred | jeblair: I will respond to email and patch with thoughts on config things to remove | 20:10 |
| jeblair | okay. will go with plan above. back to the yaml mines. | 20:10 |
| jeblair | mordred: okay. check the patch first -- that may already remove most of the things. | 20:11 |
| mordred | jeblair: I did, and it does, but I left a few more comments | 20:17 |
| jeblair | mordred: i'm building an *immensely* complicated change on that; can we do those in a followup? | 20:18 |
| mordred | jeblair: oh, absolutely!!! | 20:19 |
| mordred | jeblair: I mostly listed them there for reference - maybe I shoudl go say that | 20:19 |
| jeblair | mordred: cool, i'd like to just fix the bug in that change, then land it and mine, then continue cleanup. i think it's great to reference them there. but maybe with a +2 :) | 20:19 |
| mordred | jeblair: ++ | 20:20 |
| *** hashar has joined #zuul | 20:30 | |
| * SpamapS opens up the box of 65 broken tests and starts digging in | 21:18 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks https://review.openstack.org/446275 | 21:19 |
| jesusaur | ^that^ isn't quite working yet, but i would appreciate a review to know if i'm way off base | 21:22 |
| jeblair | jesusaur: i don't think you should need to change the executor? this should all happen before launching a job. | 21:37 |
| jesusaur | jeblair: i think i may need to for the second test case i'm adding, but you're right that it was the wrong avenue for the original test case | 21:44 |
| jhesketh | Morning | 21:44 |
| jesusaur | i suppose i can split it into two changes, if that would make reviewing easier | 21:44 |
| jesusaur | jeblair: but either way, shouldn't the executor handle the case of a merge failure? | 21:45 |
| jeblair | jesusaur: gotcha. yes -- though we may want to send the specific commits that the merger used to the executor. then the executor can create exactly what the merger did. this has two advantages: it solves the problem you pointed it, and it also ensures that all the jobs for a given change will run with the same code. that's something we have today, and i think it would be a regression if we let that slip for v3. | 21:51 |
| jesusaur | jeblair: good point | 21:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Update nodepool config syntax https://review.openstack.org/448814 | 21:52 |
| jeblair | Shrews, mordred, clarkb: https://review.openstack.org/448814 passes tests locally | 21:54 |
| mordred | jeblair: woot | 21:54 |
| mordred | BWAHAHAHAHA | 21:54 |
| mordred | https://review.openstack.org/#/c/448814/2/nodepool/tests/fixtures/clouds.yaml | 21:54 |
| mordred | that makes me happy | 21:54 |
| jeblair | heh | 21:55 |
| mordred | jeblair: out of curiosity ... (I;m sure we've both talked about this and it's likely also written down somewhere) ... why "diskimages" instead of just "images"? | 21:57 |
| jeblair | mordred: in provider or at root? | 21:58 |
| mordred | in provider. in root makes total sense - the builder builds diskimages | 21:59 |
| jeblair | mordred: it's the section that says what diskimages to upload. it's a direct reference to the object specified in the root 'diskimages' section, so this better ties the two together. label then references the provider diskimage object. | 21:59 |
| mordred | yah. ok | 21:59 |
| mordred | makes sense | 21:59 |
| jeblair | mordred: if we wanted to launch nodes from non-diskimage based labels, i would probably set an 'image:' attribute on the provider label. | 21:59 |
| jeblair | mordred: (that doesn't really preclude calling provider.diskimages provider.images, however, i think if we did add non-diskimage image support to labels, having the diskimage portion of provider called diskimage makes that more clear) | 22:00 |
| mordred | ++ | 22:00 |
| mordred | jeblair: I like this patch a lot so far - the only things that stand out to my brain are the things we were talking about earlier that will be good follow ups | 22:01 |
| jeblair | ya, i'm going to do a doc shuffling patch, then those | 22:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Update docs for configuration syntax change https://review.openstack.org/449339 | 22:18 |
| mordred | jeblair: +2 on both - comments on both +2s | 22:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Remove api-timeout and provider.image-type https://review.openstack.org/449342 | 22:29 |
| jeblair | mordred: cool, will check. there's 2 out of 3 removals ^ | 22:29 |
| jeblair | mordred: for your network question -- are you saying you can say "network: NETWORK_ID" as well as "network: NETWORK_NAME"? or are you referring to the "network: net-id" and "network: net-label" forms? | 22:34 |
| jeblair | mordred: (the second is a deprecated form) | 22:34 |
| mordred | jeblair: well - it sort of depends I guess | 22:35 |
| mordred | jeblair: we can totally just do network: NETWORK_NAME_OR_ID and pass that in to the create_server call in shade and shade will figure it out. we can also collect network.name or network.id if we wanted to ... | 22:36 |
| jeblair | mordred: i vote #1 | 22:36 |
| * mordred hangs head for not having cleaned up the network bits in the create_server call yet | 22:36 | |
| mordred | awesome | 22:36 |
| mordred | that's what I vote too | 22:36 |
| jeblair | i'm writing it right now | 22:36 |
| jeblair | though, sorry, it distracted me from the ipv6 change | 22:36 |
| mordred | and if we hit a point in the future where someone needs to do more complex things with network connections and constructors, we can cross that bridge when we get there | 22:36 |
| jeblair | mordred: should that be a list or a single string? | 22:37 |
| jeblair | (it's a list right now) | 22:37 |
| Shrews | mordred: did you find anything on that node you held today for the nodepool dsvm failures? | 22:38 |
| Shrews | jeblair: did you want to -A https://review.openstack.org/447647 until we get the dsvm fails figured out? | 22:39 |
| Shrews | eh, the parent is still up for review | 22:40 |
| jeblair | Shrews: not especially -- the current patchset passed them. | 22:41 |
| Shrews | jeblair: umm, no? | 22:41 |
| jeblair | Shrews: yep. Patch set 5 passed them on march 20 at 21:19 | 22:42 |
| Shrews | oh, yeah. i was still on the parent | 22:42 |
| Shrews | duh | 22:43 |
| jeblair | there was a recheck since then which has failed along with everything else, but i'm happy than 647 passes | 22:43 |
| mordred | jeblair: I think a list | 22:48 |
| mordred | jeblair: one sec ... | 22:48 |
| SpamapS | does anything ever remove zuul refs from mergers? | 22:55 |
| jeblair | SpamapS: no. it's a problem. in v3 we should stop setting refs in mergers, then it won't be. | 22:56 |
| SpamapS | wellllllllll | 22:57 |
| jeblair | SpamapS: (we haven't done that yet) | 22:57 |
| SpamapS | so there's a problem with Github | 22:57 |
| SpamapS | If you push -f over the source branch for a PR | 22:57 |
| SpamapS | GitHub GC's aggressively | 22:57 |
| SpamapS | so having commits on the merger is nice for a specific case which is tracking down abusers. | 22:57 |
| mordred | jeblair: it can be either a name or a list | 22:58 |
| mordred | :param network: (optional) Network dict or name or ID to attach the | 22:58 |
| mordred | server to. Mutually exclusive with the nics parameter. | 22:58 |
| mordred | Can also be be a list of network names or IDs or | 22:59 |
| SpamapS | jeblair: basically as long as we can archive-on-gc we'll be fine. | 22:59 |
| mordred | network dicts. | 22:59 |
| mordred | jeblair: so it can be a name_or_id, a dict containing either an id or name key, or a list of any of those things | 22:59 |
| jeblair | mordred: cool, thanks | 22:59 |
| mordred | jeblair: basically - you pick - we'll handle it :) | 22:59 |
| jeblair | SpamapS: we might want to see about finding another way to handle that. i think even with the current behavior, the intent is that mergers aren't reliable long-term data stores. | 23:00 |
| jeblair | SpamapS: (part of the problem we're solving with removing refs from the mergers is that repos with many refs get *very* *very* slow) | 23:01 |
| SpamapS | jeblair: right, just as long as we can archive-on-removal that should be fine. | 23:02 |
| jeblair | SpamapS: that sounds great, but i don't know what that means. :) | 23:02 |
| SpamapS | however we identify refs that we're going to remove.. | 23:02 |
| jeblair | SpamapS: oh, i'm saying that in v3 we should not create them in the first place | 23:02 |
| SpamapS | at that point, I want a hook to be able to run an archive command with access to the commit | 23:02 |
| SpamapS | yeah, if we don't make refs, that' fine too, just give me an archive-all-the-merges option. :) | 23:03 |
| SpamapS | I am signing up to write it. | 23:03 |
| SpamapS | when the time comes to get rid of the refs | 23:03 |
| mordred | I bet you could do it as a site-specific pre-playbook | 23:03 |
| mordred | inyour base job | 23:03 |
| SpamapS | but basically, since Github is mutable, unlike Gerrit, we have to do extra work (thanks github!) | 23:04 |
| SpamapS | mordred: that's what I'm hoping actually | 23:04 |
| SpamapS | if there's a way to do a final playbook that runs before all others, that's where I'd do it. | 23:04 |
| mordred | just have a place you want to push refs to - and you could, before you do your rsync of the repos to the remote hosts, just also push to your archival location | 23:04 |
| SpamapS | bah | 23:04 |
| SpamapS | that did not parse well | 23:04 |
| mordred | I get what you're saying :) | 23:04 |
| SpamapS | a playbook, that is marked in the job config as final .. | 23:04 |
| SpamapS | because yeah, ideally I can just do a 'git: src={{local_repo}} dest={{archive_remote}}/{{ item }}' or whatever the right incantation is. | 23:06 |
| SpamapS | actually ideally | 23:06 |
| jeblair | at the moment at least, i think that could still be bypassed by not inheriting from the base job, so we might have to add something to enforce that. | 23:06 |
| SpamapS | I can convince GitHub to make refs on PRs | 23:06 |
| SpamapS | and end the madness ;) | 23:06 |
| jeblair | however, we may still want to consider doing this at the merger | 23:06 |
| jeblair | because that's 1:1 vs 1:N where N=number of jobs | 23:07 |
| *** hashar has quit IRC | 23:07 | |
| SpamapS | jeblair: you can't create jobs that don't exist in a config repo, .zuul.yml, right? | 23:07 |
| SpamapS | jeblair: you can't create jobs that don't exist in a config repo, **in** .zuul.yml, right? | 23:07 |
| mordred | jeblair: good point | 23:07 |
| jeblair | SpamapS: you can create jobs in any (.?)zuul.yaml in any kind of repo. | 23:07 |
| SpamapS | oh ok | 23:07 |
| SpamapS | so then we make base do something that enables nodes to work at all | 23:08 |
| mordred | or - put in a merger hook | 23:08 |
| SpamapS | yeah | 23:08 |
| jeblair | yeah, i think for this problem at least, we should focus on the merger. | 23:08 |
| mordred | cause like jeblair says, we make the merging once, one merge may have multiple jobs | 23:08 |
| SpamapS | we'll cross that bridge when we come to it. | 23:08 |
| SpamapS | Think of me when you remove refs. :) | 23:08 |
| SpamapS | weeeeee down to 14 failed tests | 23:09 |
| jeblair | and having the merger push the archive to a non-merger git repo lets us keep the merger lean and fast, so we don't have the problem we have today. | 23:09 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Refactor out Changeish https://review.openstack.org/448913 | 23:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Remove deprecated networks syntax https://review.openstack.org/449354 | 23:10 |
| jlk | only 14? sounds like progress! | 23:10 |
| jeblair | mordred: ^ that actually doesn't switch to relying on shade for that, but it does make the config change. | 23:10 |
| jeblair | mordred: i think the shade change should be an easy followup though | 23:11 |
| mordred | jeblair: cool- I can do the switch-to-shade patch | 23:11 |
| mordred | yah | 23:11 |
| jeblair | mordred: cool, thx | 23:11 |
| mordred | jeblair: oh - speaking of | 23:11 |
| mordred | the server_console on failure patch did not work | 23:11 |
| SpamapS | Ran 202 (-26) tests in 883.258s (+120.385s) | 23:11 |
| SpamapS | damn | 23:11 |
| SpamapS | 26 less probably means a lot timed out :-P | 23:11 |
| jeblair | mordred: even after the fix of the fix? | 23:11 |
| SpamapS | +120s too ohwell | 23:12 |
| mordred | because the shade get_server_console call relies on a shade extension to the task_manager api ... which I will fix soon | 23:12 |
| SpamapS | jlk: s/14/40/ ... but still, better than 65 | 23:12 |
| mordred | jeblair: the reasons are all bong, they aren't good, but they're trivial to fix, so I'm just going to do that rather than bore you with why any of the problems exist in the first place | 23:12 |
| jeblair | mordred: can i convince you to redo your ipv6-preferred change on the end of my stack? it needs updating since we've decided to break backwards compat, and i think you'd have an easier time with the port than i. | 23:14 |
| mordred | jeblair: yup | 23:15 |
| mordred | jeblair: I will do both ipv6-preferred and network | 23:16 |
| mordred | and also server-console | 23:16 |
| SpamapS | hrm.. most problems are AttributeError: 'FakeGerritConnection' object has no attribute 'connection' | 23:17 |
| mordred | SpamapS: have you checked to see if FakeGerritConnection has an attribute called connection ? | 23:18 |
| mordred | SpamapS: also, have you tried turning it off and turning it back on again? | 23:18 |
| * mordred lobs a marmot at SpamapS | 23:18 | |
| SpamapS | No this is just leftover master stupidity from porting my patch | 23:18 |
| pabelanger | mordred: did you see this: http://logs.openstack.org/30/358730/5/check/gate-dsvm-nodepool/af54c12/logs/screen-nodepool.txt.gz#_2017-03-23_20_08_20_043 | 23:20 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Refactor out Changeish https://review.openstack.org/448913 | 23:20 |
| pabelanger | for console-log | 23:20 |
| SpamapS | woooooot | 23:20 |
| * SpamapS thinks this might work now | 23:20 | |
| * SpamapS notes two pep8 problems argh. | 23:20 | |
| mordred | pabelanger: yup | 23:21 |
| mordred | pabelanger: working on fix now | 23:21 |
| pabelanger | ack | 23:21 |
| mordred | oh for the love of - I'm going to have no way to land it | 23:21 |
| mordred | sigh | 23:21 |
| mordred | this is going to be fun | 23:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Fetch server console log if ssh connection fails https://review.openstack.org/358730 | 23:26 |
| mordred | pabelanger: that may workaround that issue | 23:26 |
| pabelanger | ah | 23:27 |
| jeblair | mordred: how do i specify an image format in clouds.yaml? | 23:28 |
| mordred | jeblair: image_format: qcow2 | 23:29 |
| mordred | jeblair: it defaults to qcow2, and each vendor profile sets it | 23:30 |
| mordred | so it's rare you actually need to set it | 23:30 |
| jeblair | mordred: i don't see that in the docs? | 23:30 |
| mordred | jeblair: that's becaues i'm a bad person | 23:30 |
| jeblair | mordred: yeah, i'm trying to fix up the nodepool tests which use image format | 23:30 |
| mordred | jeblair: http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/defaults.json | 23:30 |
| mordred | jeblair: those are the default values every cloud object starts with | 23:31 |
| jeblair | mordred: oh, i found it | 23:31 |
| mordred | woot! | 23:31 |
| jeblair | mordred: it was the - to _ issue | 23:31 |
| jeblair | i thought i wrote a patch to fix that | 23:31 |
| mordred | \o/ | 23:31 |
| SpamapS | oh neat... I have native ipv6 so I was able to telnet into the OSIC node that ran my tests | 23:32 |
| mordred | SpamapS: isn't that awesome? | 23:32 |
| jeblair | mordred: i guess it doesn't work | 23:32 |
| SpamapS | We're livin in the future! | 23:32 |
| SpamapS | mordred: I'm always shocked when I run a 'ss -tn' and I see that there are more ipv6 than not. | 23:32 |
| SpamapS | mostly from my browser | 23:32 |
| jeblair | or maybe it was just constructor args? | 23:33 |
| SpamapS | Ran 184 (-18) tests in 752.955s (-130.301s) | 23:33 |
| SpamapS | FAILED (id=19, failures=4 (-10), skips=22) | 23:33 |
| SpamapS | two Alarm Clocks | 23:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Remove api-timeout and provider.image-type https://review.openstack.org/449342 | 23:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Remove deprecated networks syntax https://review.openstack.org/449354 | 23:34 |
| jeblair | mordred: ^ updated | 23:35 |
| mordred | woot | 23:35 |
| mordred | jeblair: it occurs to me that I should make an api call for that so that we're not just reading out of the config dict | 23:35 |
| SpamapS | I forget now how to switch to soft timeouts | 23:35 |
| jeblair | mordred: what is raw? | 23:36 |
| mordred | jeblair: one of the image formats? | 23:37 |
| jeblair | mordred: taskmanager | 23:37 |
| mordred | oh - it's a flag inside of shade that tells it not to interpret the result but instead return the result object unaltered ... it showing up here is _clearly_ a layer violation and I will kill it with fire | 23:38 |
| mordred | but I can't land patches to shade until we fix the dsvm job | 23:38 |
| mordred | so that in that nodepool patch is purely so we might be able to get debugging to figure out why the nodepool dsvm jobs are broken | 23:39 |
| jeblair | mordred: ok. i -1d with a request to add a TODO, but maybe we wait until we get the results before we push up a fix for that. :) | 23:40 |
| mordred | yah. | 23:41 |
| jeblair | (just so we don't forget to remove it) | 23:41 |
| mordred | jeblair: I'm ... so super happy that we're as deep on sorting out that issue as we are | 23:41 |
| clarkb | we should be able to push a self testing change with hacks around the raw issue to get the output right? | 23:44 |
| clarkb | also has anyone just tried running reproduce.sh to reproduce? | 23:44 |
| jeblair | clarkb: yeah, mordred's latest PS is that. i guess we could start a depends-on stack, but this is probably simpler for now. | 23:46 |
| clarkb | oh I didn't see there was a new patchset | 23:47 |
| mordred | there is also a held node | 23:48 |
| mordred | I put in a nodepool auto-hold | 23:48 |
| mordred | but have not been able to dive in to that level yet | 23:48 |
| clarkb | if I'm betting and guessing something related to dib 2.0 that makes the image not boot right | 23:50 |
| clarkb | 198.72.124.55 is the held server, I am going to take a quick look at it | 23:51 |
| mordred | clarkb: oh. I didn't even think about dib 2.0 | 23:52 |
| mordred | clarkb: aren't we using that in prod already? | 23:52 |
| pabelanger | right, but we've been using 2.0 for a while now. Maybe a week | 23:53 |
| clarkb | mordred: http://paste.openstack.org/show/603999/ there you go | 23:53 |
| clarkb | glean appears to be at fault | 23:54 |
| clarkb | I will look quickly at patching it | 23:54 |
| pabelanger | oh, where is that? | 23:54 |
| clarkb | pabelanger: in console log of nested VM off of auto held nodepool vm that ran the job | 23:55 |
| clarkb | IP is up a couple lines | 23:55 |
| clarkb | huh glean release is 6 months old | 23:55 |
| pabelanger | ah, I see issue in code | 23:55 |
| clarkb | oh wait its the src job | 23:55 |
| pabelanger | ya | 23:56 |
| clarkb | so its src glean? | 23:56 |
| clarkb | doesn't the regular job fail too though? | 23:56 |
| pabelanger | I think we need to see config-drive | 23:56 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add trigger capability on github pr review https://review.openstack.org/449365 | 23:56 |
| pabelanger | clarkb: it looks like we have an interface with no type | 23:57 |
| pabelanger | eg: ipv4 / ipv6 | 23:57 |
| clarkb | ya | 23:57 |
| clarkb | I think that may be normal and why there is a continue there | 23:58 |
| clarkb | we just don't handle it properly | 23:58 |
| pabelanger | ya | 23:58 |
| clarkb | https://review.openstack.org/449366 lets see if that does better (it will run the nodepool test too ya?) | 23:59 |
| pabelanger | ya | 23:59 |
| pabelanger | so, we should see something different | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!