| clarkb | pabelanger: will also be interesting to dig through the logs and see what our ratio between failed uploads and successful uploads is | 00:02 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Activate virtualenv before running dib https://review.openstack.org/404487 | 00:40 |
| adam_g | jeblair: i dont have handy ATM, but it was item.change.refspec and item.change.branch that causes them at https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/client.py?h=feature/zuulv3#n47 | 01:20 |
| adam_g | jeblair: item.change there is either a zuul.model.Change or zuul.model.Ref, depending on event. Only a Change has branch and refspec attributes. | 01:22 |
| adam_g | im kinda working backward from assumptions that are encoded in the test, so im not sure if the variance in type of item.change is a bug, or if its just something that needs to be fixed up with the Changish/Change/Ref work you mentioned | 01:23 |
| mordred | adam_g: the tl;dr on the Changeish/Change/Ref thing (which I can't remember if it's written down yet) | 01:36 |
| mordred | adam_g: is to rework them so that Ref is the base class and Change is a type of ref, rather than Changeish being the base class and Change and Ref each being types of that | 01:36 |
| jeblair | mordred: https://storyboard.openstack.org/#!/story/2000781 | 01:37 |
| mordred | adam_g: it's not an essential thing, and is pretty invasive, so I think it was a thing we weren't prioritizing - but when we did a data model walkthrough a couple of months ago - it jumped out as a thing that could be more understandable | 01:37 |
| mordred | jeblair: zomg | 01:38 |
| mordred | adam_g: so - ignore me and just read the actual story in storyboard | 01:38 |
| jeblair | (or add text to it if it should be more clear :) | 01:38 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool: Accept user-home in config validator https://review.openstack.org/404519 | 03:17 |
| *** bhavik1 has joined #zuul | 03:58 | |
| adam_g | mordred: ok, yah. makes sense. if its non-priority atm, maybe the hax in https://review.openstack.org/#/c/404466/ are ok for now? as it is i dont see post jobs workin | 05:07 |
| *** saneax-_-|AFK is now known as saneax | 05:19 | |
| *** _ari_ has quit IRC | 05:42 | |
| *** _ari_ has joined #zuul | 05:44 | |
| *** abregman has joined #zuul | 05:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul: Add hostname to ZUUL_URL when running with append_hostname https://review.openstack.org/404574 | 06:42 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/400536 | 07:45 |
| *** abregman is now known as abregman|mtg | 08:52 | |
| *** hashar has joined #zuul | 09:13 | |
| *** hashar has quit IRC | 09:36 | |
| *** hashar has joined #zuul | 09:57 | |
| *** abregman|mtg is now known as abregman | 09:59 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/400536 | 10:00 |
| *** gundalow_ has joined #zuul | 10:18 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/400536 | 10:19 |
| *** anteaya has quit IRC | 10:19 | |
| *** gundalow has quit IRC | 10:19 | |
| *** olaph has quit IRC | 10:19 | |
| *** hashar has quit IRC | 10:24 | |
| *** hashar has joined #zuul | 10:30 | |
| *** Cibo_ has quit IRC | 10:31 | |
| *** anteaya has joined #zuul | 10:31 | |
| *** olaph has joined #zuul | 10:36 | |
| *** Cibo_ has joined #zuul | 10:37 | |
| *** bhavik1 has quit IRC | 11:23 | |
| *** jlk has quit IRC | 11:31 | |
| *** jlk has joined #zuul | 11:31 | |
| *** rmoe has quit IRC | 11:31 | |
| *** jlk has quit IRC | 11:31 | |
| *** jlk has joined #zuul | 11:31 | |
| *** rmoe has joined #zuul | 11:31 | |
| *** willthames has quit IRC | 12:21 | |
| *** willthames has joined #zuul | 12:52 | |
| mordred | jhesketh: woot! the merge patch seems to be passing | 13:18 |
| jhesketh | mordred: yep, sorry I didn't get to it earlier in the day | 13:18 |
| jhesketh | the sooner we can merge it the better because the worlds keep shifting | 13:19 |
| jhesketh | ditto the zuul merge | 13:19 |
| openstackgerrit | Merged openstack-infra/zuul: Re-model the job auth https://review.openstack.org/399645 | 13:20 |
| mordred | jhesketh: yup | 13:42 |
| mordred | jhesketh: just +A'd the zuul merge | 13:48 |
| openstackgerrit | Merged openstack-infra/zuul: Merge branch 'master' into workingv3 https://review.openstack.org/389470 | 13:52 |
| mordred | \o/ | 13:53 |
| *** abregman is now known as abregman|mtg | 14:02 | |
| openstackgerrit | Merged openstack-infra/zuul: Re-enable requirement-newer-than tests https://review.openstack.org/401019 | 14:12 |
| openstackgerrit | Merged openstack-infra/zuul: Re-enable requirement-older-than tests https://review.openstack.org/401027 | 14:12 |
| openstackgerrit | Merged openstack-infra/zuul: Re-enable requirement-username tests https://review.openstack.org/401028 | 14:12 |
| openstackgerrit | Merged openstack-infra/zuul: Re-enable requirement-email tests https://review.openstack.org/401029 | 14:12 |
| Shrews | anyone know how nb01 is looking this morning? | 14:17 |
| pabelanger | o/ | 14:33 |
| pabelanger | Shrews: all images built | 14:33 |
| pabelanger | and almost all were uploaded | 14:34 |
| pabelanger | http://paste.openstack.org/show/590964/ | 14:34 |
| pabelanger | current output | 14:34 |
| pabelanger | going to review: https://review.openstack.org/#/c/400536/ | 14:36 |
| pabelanger | see if we can land that first thing this morning | 14:36 |
| mordred | pabelanger: that's good news! | 14:41 |
| Shrews | pabelanger: those that are "uploading" are left over from the older version of the builder? | 14:42 |
| pabelanger | mordred: indeed | 14:42 |
| pabelanger | Shrews: yes, believe so | 14:42 |
| pabelanger | the first 4 build attempts for all images had problems | 14:43 |
| pabelanger | DIB issues | 14:43 |
| Shrews | pabelanger: let's also try to land https://review.openstack.org/404438 today | 14:44 |
| *** willthames has quit IRC | 14:47 | |
| Shrews | maybe we should also try to clean up the ordering of that output to include provider and build and upload IDs | 14:48 |
| pabelanger | ++ | 14:51 |
| openstackgerrit | Merged openstack-infra/nodepool: Make build/upload worker sleep time configurable https://review.openstack.org/404438 | 14:57 |
| *** saneax is now known as saneax-_-|AFK | 14:58 | |
| pabelanger | Shrews: clarkb: jeblair: This is likely a good time to fix nodepool-builder having to be stopped / started for each nodepool.yaml change. | 15:31 |
| pabelanger | checking to see if this is still the case | 15:31 |
| Shrews | pabelanger: it picks up changes | 15:32 |
| pabelanger | ah neat | 15:32 |
| pabelanger | thanks, makes me happy again | 15:33 |
| *** abregman|mtg is now known as abregman | 15:36 | |
| Shrews | pabelanger: mind reviewing 400836? | 15:47 |
| *** abregman has quit IRC | 15:53 | |
| *** abregman has joined #zuul | 15:55 | |
| pabelanger | Shrews: +2, so jeblair mordred can review | 16:06 |
| openstackgerrit | Merged openstack-infra/nodepool: Re-enable alien-image-list command and tests https://review.openstack.org/400836 | 16:25 |
| *** gundalow_ is now known as gundalow | 16:26 | |
| *** gundalow is now known as gundalow_ | 16:37 | |
| *** gundalow_ is now known as gundalow | 16:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Correct logic problem with job trees https://review.openstack.org/400456 | 16:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Fix variants not picking up negative matches. https://review.openstack.org/399871 | 16:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Define the internal noop job https://review.openstack.org/404864 | 16:40 |
| openstackgerrit | Merged openstack-infra/nodepool: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/400536 | 16:49 |
| pabelanger | \o/ | 16:49 |
| pabelanger | grabbing a shower, then relocating to coffee shop for today | 16:50 |
| pabelanger | nb01.o.o looks to be idle now | 16:50 |
| *** abregman has quit IRC | 16:54 | |
| *** hashar has quit IRC | 17:13 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/zuul: Re-enable test_success_pattern as test_success_url https://review.openstack.org/400455 | 17:15 |
| jeblair | adam_g, pabelanger: i'm looking into the post test / merger items thing. i think i have my head wrapped around it; should have something to show soon. | 17:17 |
| pabelanger | great | 17:25 |
| clarkb | the images that failed are likely real failures right? | 17:29 |
| adam_g | jeblair: oh, cool | 17:29 |
| clarkb | that will give us a good idea of how unreliable this is over time(and maybe we feed that back to $clouds and glance) | 17:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add image_name to UploadWorker INFO message https://review.openstack.org/404891 | 17:35 |
| pabelanger | clarkb: yes, I can see some shade exceptions | 17:35 |
| pabelanger | I haven't looked into it more | 17:35 |
| pabelanger | I should restart nodepool-builder to pick up the latest commits | 17:36 |
| mordred | 2016-11-30 07:02:46,498 ERROR shade.exc: Image creation failed: Error finding address for https://image.compute.bhs1.cloud.ovh.net/v2/images/a19664de-ee96-4c4e-b651-2f47594239d1/file: Unable to establish connection to https://image.compute.bhs1.cloud.ovh.net/v2/images/a19664de-ee96-4c4e-b651-2f47594239d1/file: The write operation timed out | 17:37 |
| mordred | :) | 17:37 |
| pabelanger | okay, restarted | 17:38 |
| mordred | that seems to be the error | 17:38 |
| pabelanger | ya | 17:39 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Use diskimage-builder checksum files https://review.openstack.org/404432 | 17:41 |
| pabelanger | mordred: clarkb: ^ would be great to get that landed today too | 17:41 |
| pabelanger | https://review.openstack.org/#/c/400311/ landed too, which will do checksum in parallel now | 17:42 |
| pabelanger | will ask openstack-dib for a release | 17:42 |
| mordred | pabelanger: I thnk you're going to hit a pep8 barg on that ... | 17:43 |
| pabelanger | boo | 17:43 |
| pabelanger | local looks good | 17:43 |
| mordred | oh - ok. ignore me then :) | 17:43 |
| mordred | the patch looks good | 17:43 |
| pabelanger | Oh | 17:44 |
| pabelanger | I think I see | 17:44 |
| pabelanger | missing comma | 17:44 |
| mordred | pabelanger: https://review.openstack.org/#/c/404432/2/nodepool/provider_manager.py line 287/288 | 17:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Use diskimage-builder checksum files https://review.openstack.org/404432 | 17:44 |
| mordred | pabelanger: also, jeblair had a comment on ps2 that I agree with | 17:45 |
| pabelanger | yup | 17:45 |
| pabelanger | updating now | 17:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Use diskimage-builder checksum files https://review.openstack.org/404432 | 17:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Use diskimage-builder checksum files https://review.openstack.org/404432 | 17:48 |
| pabelanger | okay, should be ready now | 17:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Don't merge post-merge items https://review.openstack.org/404903 | 18:06 |
| jeblair | adam_g, pabelanger: this is the source of my confusion: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/scheduler.py#n1491 | 18:07 |
| jeblair | (that's in master) | 18:07 |
| jeblair | we never called _makeMergerItem (the master equivalent of the zuulv3 branch's make_merger_item) for post-merge changes, which is why we never saw those errors | 18:08 |
| jeblair | the fact that it kinda-sorta halfway attempted to handle post-merge changes must be leftover from earlier usage | 18:08 |
| jeblair | i considered removing support for them completely from zuulv3 and sending an empty 'items' list to the launcher to say "don't merge anything when you run this job" | 18:09 |
| jeblair | but i thought that might unduly restrict us later -- we may want the launcher to know about the item being run, and if we did that, we'd just end up adding the data back | 18:09 |
| jeblair | so i think going with the solution of making the item list robust as adam_g did, but then having the launcher-merger filter out items that don't actually need to be merged (since they are not pre-merge changes) is probably the cleaner approach | 18:10 |
| jeblair | so that's what i did in https://review.openstack.org/404903 and adam_g's change is good as-is i think | 18:11 |
| openstackgerrit | Merged openstack-infra/zuul: Enable test_post* https://review.openstack.org/404466 | 18:16 |
| mordred | jeblair: I love when your changes have an order of magnitude more in the commit message than the patch itself https://review.openstack.org/#/c/400456/ | 18:16 |
| pabelanger | jeblair: cool, thanks for the info | 18:18 |
| jeblair | mordred: i really hate code. it's full of bugs. we need less of it, not more! :) | 18:20 |
| mordred | jeblair: ++ | 18:20 |
| mordred | jeblair: code is the worst | 18:20 |
| *** Shuo has joined #zuul | 18:22 | |
| rcarrillocruz | folks, i rebased the d-g changes on top of mordred nice output change | 18:23 |
| mordred | rcarrillocruz: woot | 18:23 |
| rcarrillocruz | mind reviewing the chain starting on https://review.openstack.org/401975 ? | 18:23 |
| * mordred goes to review | 18:23 | |
| rcarrillocruz | they all got +1 | 18:23 |
| rcarrillocruz | thanks | 18:23 |
| * rcarrillocruz goes afk a bit, picking up wife at work | 18:23 | |
| mordred | rcarrillocruz: this doens't look right: https://review.openstack.org/#/c/402208/9/playbooks/roles/start_fresh_logging/defaults/main.yaml,unified | 18:26 |
| mordred | rcarrillocruz: hrm - also - http://logs.openstack.org/75/401975/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/eb0291a/logs/devstack-gate-setup-host.txt.gz#_2016-11-30_12_05_37_912 isn't printing the name you added to that task - I think we need to fix the callback plugin | 18:29 |
| openstackgerrit | Merged openstack-infra/nodepool: Use diskimage-builder checksum files https://review.openstack.org/404432 | 18:30 |
| pabelanger | So, just thinking out loud here. How do we default our image build to start at 10:00 UTC now for nb01.o.o? | 19:07 |
| clarkb | pabelanger: aiui we don't | 19:08 |
| clarkb | pabelanger: its going to queue things up once they timeout past the last build time | 19:08 |
| pabelanger | right, which is good but our image build times are going to shift over time | 19:10 |
| pabelanger | especially if a build fails for what ever reason | 19:10 |
| pabelanger | lets see what happens today | 19:11 |
| timrc | Okay, I think I found my calling: to be a truck driver - https://aws.amazon.com/snowmobile | 19:12 |
| pabelanger | http://paste.openstack.org/show/591003/ | 19:12 |
| pabelanger | that is our current age of images | 19:12 |
| clarkb | pabelanger: right that was all intentional | 19:12 |
| *** saneax-_-|AFK is now known as saneax | 19:12 | |
| clarkb | pabelanger: so in ~2hours fedora23 wil start | 19:13 |
| pabelanger | clarkb: ya, just trying to see how that is going to come into play when we need to land a change to all images for exmaple | 19:13 |
| clarkb | then jessie after it and so on | 19:13 |
| pabelanger | ya | 19:13 |
| clarkb | pabelanger: you would manually trigger the builds yourself in that case | 19:13 |
| clarkb | whcih is still supported aiui | 19:13 |
| clarkb | (I read code that seemed to support it at least) | 19:13 |
| pabelanger | right | 19:13 |
| pabelanger | okay, I'm happy to wait and see what happens today | 19:14 |
| pabelanger | also excited to to get nb02.o.o online too when we are ready :) | 19:14 |
| jeblair | clarkb, pabelanger: yes. :) we decided to not bother trying to tie things to a schedule because that hasn't really worked for us for a long time anyway (it takes a long time to build and upload the images, so the idea of "images come on line around 10 o'clock" just doesn't work past one or two providers and images) | 19:22 |
| jeblair | clarkb, pabelanger: manual build triggers, or if you're feeling daring, deleting one of the old images will serve to speed up the process of getting something in place. if the issue is correcting an erroneous build, deleting the most recent build is actually the right answer there anyway. | 19:23 |
| pabelanger | Yup, happy to adapt. | 19:24 |
| pabelanger | think I'm just going to wait until the normal schedule builds happen | 19:25 |
| pabelanger | make sure it works as expected | 19:25 |
| *** SpamapS has quit IRC | 19:26 | |
| rcarrillocruz | mordred: hey, thx, that should default to /opt/stack, def a bug | 19:39 |
| rcarrillocruz | as for the callback, yeah, noticed it | 19:39 |
| rcarrillocruz | i'll take a stab at it | 19:39 |
| *** SpamapS has joined #zuul | 19:42 | |
| *** jeblair is now known as inserted_human | 19:45 | |
| *** inserted_human is now known as jeblair | 19:45 | |
| *** hashar has joined #zuul | 19:55 | |
| *** jesusaur has joined #zuul | 20:21 | |
| *** saneax is now known as saneax-_-|AFK | 20:30 | |
| pabelanger | next build should start in 20mins | 20:59 |
| pabelanger | using latest version of code too | 20:59 |
| mordred | \o/ | 21:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Correct logic problem with job trees https://review.openstack.org/400456 | 21:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Fix variants not picking up negative matches. https://review.openstack.org/399871 | 21:03 |
| adam_g | jeblair: i think that all makes sense (the make_merge_items comments), thanks for clarifying | 21:04 |
| jeblair | whew! | 21:04 |
| *** hashar has quit IRC | 21:09 | |
| pabelanger | and fedora-23 build started | 21:25 |
| pabelanger | but, first build today failed | 21:25 |
| pabelanger | checking | 21:25 |
| pabelanger | 2016-11-30 21:22:11,451 INFO nodepool.image.build.fedora-23: Error: Failed to synchronize cache for repo 'updates' | 21:27 |
| pabelanger | looks like a repo failure | 21:27 |
| pabelanger | next build looks better | 21:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Revert "Revert "Store ssh_host_key of remote node"" https://review.openstack.org/360674 | 21:46 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Make diskimage-builder command configurable for testing https://review.openstack.org/404976 | 21:53 |
| pabelanger | want to start writing some tests when our dib-command fails, figures ^ will give me an easy way to start using /usr/bin/false as the command | 21:54 |
| greghaynes | we had a deal for injecting failure where we'd just set DIB_SHOULD_FAIL in the config | 21:55 |
| pabelanger | ya, that too | 21:56 |
| pabelanger | should check it again | 21:56 |
| pabelanger | SHOULD_FAIL as image meta field | 21:59 |
| pabelanger | ah, seen an issue | 21:59 |
| pabelanger | if images: [] in providers, we cannot use it | 22:00 |
| pabelanger | let me play with it today | 22:00 |
| pabelanger | but now, I relocate home for food | 22:00 |
| jeblair | pabelanger: we should be able to use the DIB_SHOULD_FAIL thing anywhere we need to. and while the configurable dib command is a way to configure things for testing, we actually get more flexibility with DIB_SHOULD_FAIL which lets us specify that some images should fail and others should not. | 22:03 |
| jeblair | i actually kind of like the configurable dib command because it lets us remove the fake check from the actual production code -- but i just want to make sure we don't lose sight of setting DIB_SHOULD_FAIL as the way to actually trigger failures. | 22:04 |
| greghaynes | WFM, just making sure we didnt forget ;) | 22:41 |
| clarkb | jeblair: not only that but the configurable dib command doesn't actually solve the intended issue | 22:45 |
| clarkb | (it feels dirty to expose it to users as being for that) | 22:45 |
| clarkb | but maybe thats a documentation thing "if dib is installed to a virtuaelnv it is not sufficient to run the disk-image-create binary from that venv, you must also source the env as well" | 22:46 |
| clarkb | or something | 22:46 |
| jeblair | right, well, *today* we're talking about it in the context of tests, so i'm just focusing on that. :) | 22:46 |
| pabelanger | jeblair: Ya, I'll look into DIB_SHOULD_FAIL too | 23:02 |
| pabelanger | I liked swapping out the dib_cmd to /usr/bin/false, as a easy way to reproduce when disk-image-create fails for unknown reason | 23:03 |
| jeblair | pabelanger: sure, but that's exactly what DIB_SHOULD_FAIL does -- should be even easier :) | 23:07 |
| pabelanger | jeblair: SHOULD_FAIL? I only see that | 23:10 |
| jeblair | yeah i'm just repeating what greghaynes said :) whatever looks vaguely like that is the thing | 23:10 |
| pabelanger | ack | 23:11 |
| pabelanger | I'll give that a go | 23:11 |
| *** willthames has joined #zuul | 23:15 | |
| pabelanger | jeblair: so, if we set images: [] in providers section, we loose the ability to determine the image-type to pass to disk-image-create. Was thinking if img_types was empty, maybe just default it to qcow2 (this is the default if we remove -t from disk-image-create) | 23:23 |
| pabelanger | in reference too: https://etherpad.openstack.org/p/um87VzudKq | 23:24 |
| jeblair | pabelanger: hrm, maybe we should modify the etherpad then and say that in that case we don't build an image (but we also don't delete it) | 23:25 |
| pabelanger | jeblair: if we do that, I see no way to only build an image | 23:29 |
| pabelanger | actually | 23:29 |
| pabelanger | I guess we'd add a provider but have it be pause | 23:29 |
| pabelanger | I'll look at it again in the morning | 23:29 |
| jeblair | pabelanger: yeah, i don't think it's that important. | 23:30 |
| jeblair | the idea of building without uploading sounds nice, but if we don't know what formats to build, then maybe we just shouldn't do it. | 23:30 |
| pabelanger | Ya, don't have a real world example of build only today | 23:34 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!