| jeblair | mordred: it's possible (likely?) your git source path idea may end up being implemented by an ansible role that may be able to be customized to some extent... probably per-site... possibly per-tenant, maybe even per-job. if it's the last one, then we can actually have classes of jobs that have 'go-style' checkouts, and classes that have flat checkouts. (and maybe even classes that have puppet-style checkouts?) | 00:00 |
|---|---|---|
| mordred | jeblair: neat | 00:22 |
| mordred | I'm a fan of all of those things | 00:22 |
| *** harlowja has joined #zuul | 00:30 | |
| jamielennox | so any chance there is a doc on the upgrade from nodepool 2.5 to 3 ? | 01:20 |
| jeblair | jamielennox: you mean the builder thing we just did? that'd be from nodepool 0.3 to 0.4(unreleased). no; if you or someone else wants to write one, that'd be great. :) | 01:29 |
| jeblair | jamielennox: we have my one sentence doc "run a zookeeper and point nodepool at it". we have the puppet-nodepool and puppet-openstackci modules, and http://docs.openstack.org/infra/system-config/nodepool.html | 01:30 |
| jamielennox | jeblair: yea, that's what i meant, getting over the new zuulv3 merge | 01:30 |
| jamielennox | i know to turn on zookeeper, but can i turn off mysql? | 01:30 |
| jeblair | jamielennox: no, mysql+gearman are still used by the launcher portion (that's up next) | 01:31 |
| jamielennox | ok, so just enable zookeeper, point it at it and leave everything else the same for now? | 01:32 |
| jeblair | jamielennox: yep | 01:32 |
| jamielennox | jeblair: sweet, super easy | 01:32 |
| jeblair | jamielennox: hope so! /me runs away before jamielennox reconsiders | 01:33 |
| jeblair | jamielennox: note, there's no migration of the existing build info | 01:34 |
| jeblair | jamielennox: so nodepool won't be able to launch nodes until the new builder creates and uploads new builds | 01:34 |
| jeblair | jamielennox: and the old ones won't be deleted or anything, so you'd have to delete those by hand | 01:34 |
| jeblair | jamielennox: we ran the zk builder in parallel for a while to build up a quorum of uploaded images before we switched the launcher to use it so we had no downtime. | 01:35 |
| jamielennox | jeblair: yea, i'm ok rebuilding the state | 01:35 |
| *** saneax is now known as saneax-_-|AFK | 01:35 | |
| jamielennox | so long as the new launcher still operates | 01:35 |
| jeblair | jamielennox: it should just wait for a build to show up in zk. | 01:37 |
| *** Cibo_ has quit IRC | 01:46 | |
| *** rcarrillocruz has quit IRC | 02:06 | |
| *** rcarrillocruz has joined #zuul | 02:07 | |
| *** Cibo_ has joined #zuul | 02:38 | |
| *** EmilienM has quit IRC | 03:19 | |
| *** EmilienM has joined #zuul | 03:21 | |
| *** rcarrillocruz has quit IRC | 04:01 | |
| *** bhavik1 has joined #zuul | 04:23 | |
| *** rcarrillocruz has joined #zuul | 04:59 | |
| *** abregman has joined #zuul | 06:13 | |
| *** willthames has quit IRC | 06:29 | |
| *** saneax-_-|AFK is now known as saneax | 06:55 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul: Remove TestScheduler.test_nonexistent_job https://review.openstack.org/410571 | 07:34 |
| *** bhavik1 has quit IRC | 07:59 | |
| *** Cibo_ has quit IRC | 08:24 | |
| *** Cibo_ has joined #zuul | 08:25 | |
| *** hashar has joined #zuul | 08:46 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** yolanda has quit IRC | 09:01 | |
| *** yolanda has joined #zuul | 09:05 | |
| *** jesusaur has quit IRC | 09:08 | |
| *** rmoe has quit IRC | 09:16 | |
| *** jesusaur has joined #zuul | 09:21 | |
| *** rmoe has joined #zuul | 09:30 | |
| *** Cibo_ has quit IRC | 09:49 | |
| *** rmoe has quit IRC | 09:52 | |
| *** Cibo_ has joined #zuul | 09:53 | |
| *** rmoe has joined #zuul | 09:55 | |
| *** jesusaur has quit IRC | 10:09 | |
| abregman | hi. question on nodepool - does it take into account the available resources when spawning up VMs? | 10:20 |
| *** jesusaur has joined #zuul | 10:22 | |
| *** rmoe has quit IRC | 10:24 | |
| *** jesusaur has quit IRC | 10:30 | |
| *** jesusaur has joined #zuul | 11:19 | |
| *** rmoe has joined #zuul | 11:23 | |
| *** hashar is now known as hasharAway | 11:31 | |
| *** jamielennox is now known as jamielennox|away | 12:43 | |
| *** jamielennox|away is now known as jamielennox | 12:50 | |
| *** hasharAway is now known as hashar | 13:07 | |
| *** abregman is now known as abregman|mtg | 13:18 | |
| *** abregman|mtg is now known as abregman | 13:44 | |
| pabelanger | abregman: yes, there is some docs in allocation.py about how it currently works. But nodepool will try to use the resources evenly across cloud providers | 13:56 |
| *** abregman is now known as abregman|mtg | 14:02 | |
| *** jlk has quit IRC | 15:01 | |
| *** openstackgerrit has joined #zuul | 15:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic https://review.openstack.org/360446 | 15:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Plumb ipv6-preferred into force-ipv4 setting https://review.openstack.org/360445 | 15:02 |
| *** jlk has joined #zuul | 15:03 | |
| *** jlk has quit IRC | 15:03 | |
| *** jlk has joined #zuul | 15:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Plumb ipv6-preferred into force-ipv4 setting https://review.openstack.org/410801 | 15:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic https://review.openstack.org/410802 | 15:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Stop json-encoding the nodepool metadata https://review.openstack.org/297950 | 15:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory https://review.openstack.org/107597 | 15:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Stop json-encoding the nodepool metadata https://review.openstack.org/410812 | 15:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory https://review.openstack.org/410813 | 15:25 |
| *** saneax is now known as saneax-_-|AFK | 15:41 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Stop json-encoding the nodepool metadata https://review.openstack.org/297950 | 16:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory https://review.openstack.org/107597 | 16:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory https://review.openstack.org/410813 | 16:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Stop json-encoding the nodepool metadata https://review.openstack.org/410812 | 16:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory https://review.openstack.org/410813 | 16:32 |
| pabelanger | mordred: the ansible fact is awesome! | 16:35 |
| pabelanger | very cool | 16:35 |
| mordred | pabelanger: yay! that also may be my oldest outstanding change :) | 16:36 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: WIP: Use thread local data in ZK API https://review.openstack.org/410853 | 16:41 |
| Shrews | jeblair: ^^^ is the first thing that must change in order to share a single ZK client. Will build on that, but had implications for how we test some things. | 16:43 |
| *** abregman|mtg is now known as abregman | 16:44 | |
| *** abregman has quit IRC | 16:44 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: WIP: Use thread local data in ZK API https://review.openstack.org/410853 | 16:45 |
| pabelanger | well, isn't this neat | 16:47 |
| pabelanger | | ubuntu-xenial-0000000005 | ubuntu-xenial | nb02 | | building | 01:12:57:24 | | 16:47 |
| pabelanger | and disk-image-create has been running for that long | 16:47 |
| pabelanger | it is stuck | 16:47 |
| pabelanger | 2016-12-13 04:00:45,857 INFO nodepool.image.build.ubuntu-xenial: Cloning into '/etc/puppet/modules/openafs'... | 16:48 |
| pabelanger | stuck cloning | 16:48 |
| pabelanger | guess we need to add a timeout around that logic | 16:48 |
| mordred | pabelanger: wow. how lovely | 16:48 |
| pabelanger | will do that today | 16:48 |
| mordred | yah - I think "the disk image has been building for one day" is always an error | 16:49 |
| Shrews | pabelanger: awesome. thx for doing that | 16:49 |
| pabelanger | mordred: ya, feature request, ability to abort builds after x mins | 16:49 |
| pabelanger | might hack on that | 16:50 |
| Shrews | I am not confused as to what things should go to nodepool's master branch vs the feature/zuulv3 branch | 16:51 |
| Shrews | s/not/now/ | 16:51 |
| jeblair | Shrews: i will clarify that shortly | 16:51 |
| Shrews | i assumed the feature branch was for the newly approved spec | 16:52 |
| jeblair | Shrews: re 410853 -- what about, instead of using thread-local data, simply stop storing the lock data on the zk class at all? the actual use of the lock is with a context manager, which gets returned, so release of the correct thing is handled. the only thing local storage of the lock gets us is the re-entrant detection... | 16:55 |
| jeblair | pabelanger: have you restarted? | 16:58 |
| pabelanger | jeblair: I have, just nb01 / nb02 | 16:58 |
| pabelanger | not sure if we want to do nodepool.o.o too | 16:58 |
| jeblair | pabelanger: i guess we could probably just double check that /opt/nodepool is correct and call that good enough | 16:58 |
| pabelanger | jeblair: indeed it is | 16:59 |
| jeblair | yeah, it's ad531ae5bdb8e5eef820b2946fa2d09cb3a5703e which is the master version of the merge commit | 16:59 |
| jeblair | it's now safe to tear apart the nodepool daemon itself on the feature/zuulv3 branch to start implementing the nodepool part of the zuulv3 spec | 17:00 |
| jeblair | the master branch of nodepool is where further updates to the builder, doc improvements to prepare for release, and other changes we want to go into production immediately should go. | 17:01 |
| jeblair | Shrews: ^ | 17:01 |
| pabelanger | Rip out all the things | 17:05 |
| jeblair | maybe we could use the zuulv3 topic on the nodepool master branch for things related to the new builder? | 17:07 |
| jeblair | so they show up on our zuulv3 dashboards | 17:08 |
| pabelanger | ack | 17:08 |
| *** hashar has quit IRC | 17:10 | |
| Shrews | jeblair: hrm, that's crazy enough it just might work. not sure why i had those vars to begin with now. maybe a hold over from some other thing i tried way back when | 17:11 |
| jeblair | Shrews: yeah, it'll be nice if this ends up being simpler :) | 17:13 |
| clarkb | pabelanger: mordred if you do add a timeout be sure its form when dib started running and not from when build was queued. We had timeouts but they were from when things werequeued and it turned out that you could wait queuing for longer than your timeout and never get a chance to build | 17:15 |
| mordred | clarkb: ++ | 17:16 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: WIP: Remove ZK lock variables https://review.openstack.org/410853 | 17:17 |
| mordred | jeblair, Shrews: re branches - I submitted the ansible changes above to both master and feature/zuulv3 because they're simple enough that we can land them in master if we want, but certainly would cause merge conflicts with launcher rework in v3 branch | 17:17 |
| mordred | jeblair, Shrews: so I wanted to make sure we had the option to land them in either if we decide to land them at all | 17:18 |
| mordred | (also, landing the v3 merge caused them to merge-conflict so they showed up near the top of my queue) | 17:18 |
| Shrews | mordred: it's looking like i probably won't start on the zuulv3 spec stuff until after the holidays, so conflicts until then should not be expected | 17:19 |
| mordred | kk | 17:20 |
| Shrews | jeblair: 410853 is indeed simpler now | 17:20 |
| Shrews | except for the tests | 17:20 |
| mordred | pesky tests | 17:20 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: WIP: Remove ZK lock variables https://review.openstack.org/410853 | 17:21 |
| jeblair | mordred: we can always merge master -> v3 as needed too | 17:22 |
| jeblair | mordred: in 360445, there are 2 references i don't understad: "devstack nodepool tests with the new support for dual stack networks" and "ipv6 is fixed in rackspace" | 17:24 |
| jeblair | mordred: we could try to make me understand these, or i'm happy to defer to you and clarkb if the two of you agree :) | 17:24 |
| Shrews | hrm, i guess, in theory, we don't need a separate thread to test the lock stuff since attempting to lock from the same thread should now fail in the same code path. | 17:25 |
| clarkb | jeblair: glean did not have support for v6 in raxit may not yet for all distros not sure | 17:25 |
| jeblair | Shrews: i would agree with that assessment | 17:26 |
| mordred | jeblair: I'm not sure I understand them either - especially since nodepool tests work with current devstack now | 17:26 |
| clarkb | jeblair: since rax doesnt use RAs like everyone else glean must statically assign v6 stuff | 17:26 |
| clarkb | which means you cant tell shade to use v6 in raxuntil gleab works | 17:26 |
| mordred | yah | 17:26 |
| jeblair | oh, are we not ipv6ing in rax right now? | 17:26 |
| mordred | we're not - because nodepool always uses public_v4 | 17:27 |
| jeblair | mordred: yeah, but does the node itself have ipv6? | 17:27 |
| mordred | or, rather, it uses public_v4|public_v6 based on the value of ipv6-preferred | 17:27 |
| clarkb | jeblair: on ubuntu I think yes now | 17:27 |
| clarkb | on not ubuntu I dont know | 17:27 |
| jeblair | like, it used to be the node was dual stack, but nodepool ignored the ipv6 (but ipv6 would still get used in tests) | 17:27 |
| jeblair | now it's ipv4 in nodepool because it prefers v4, and v4 only in the tests since glean doesn't ipv6? | 17:28 |
| pabelanger | we should have ipv6 in rax, let me double check | 17:28 |
| mordred | 360446 shows the thing I was actually trying to get to - which is using interface_ip instead of logic inside of nodepool. but to do that, we have to actually tell shade when we have a config that overrides ipv6 ability detection | 17:29 |
| mordred | I think the commit message in 360445 is making it more complex than it needs to be | 17:29 |
| pabelanger | inet6 2001:4801:7828:104:be76:4eff:fe10:684a/64 scope global | 17:30 |
| pabelanger | that is from rax-ord | 17:30 |
| mordred | woot | 17:30 |
| mordred | pabelanger: is that ubuntu or centos? | 17:30 |
| pabelanger | xenial | 17:30 |
| mordred | mind checking centos too? | 17:30 |
| pabelanger | it doesn't have it | 17:30 |
| pabelanger | I didn't update glean yet | 17:30 |
| pabelanger | but checking | 17:30 |
| mordred | k. so we only have v6 on ubuntu - which means our config for rackspace should be force-ipv4 | 17:30 |
| mordred | until we have otherwise have it | 17:31 |
| jeblair | with a todo: remove when glean updated for centos | 17:31 |
| mordred | yah | 17:31 |
| pabelanger | inet6 fe80::be76:4eff:fe10:ad02/64 scope link | 17:31 |
| pabelanger | is centos | 17:31 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: WIP: Remove ZK lock variables https://review.openstack.org/410853 | 17:32 |
| clarkb | its a repeat of the dont dhcpproblem | 17:32 |
| clarkb | extended to ipv6 | 17:32 |
| mordred | remote: https://review.openstack.org/410879 Be explicit about not using ipv6 on rackspace | 17:33 |
| mordred | there's a patch to make sure that we're explicit about this | 17:34 |
| jeblair | mordred: that's safe to land now, right? | 17:34 |
| mordred | jeblair: yah. it's a no-op currently | 17:35 |
| jeblair | mordred: okay, i think i understand all 3 patches, except for the part about "devstack nodepool tests" but it sounds like no one else does either and it's safe to ignore that. :) | 17:36 |
| mordred | jeblair: but will be important if we move to ipv6-by-default-if-possible-unless-turned-off | 17:36 |
| mordred | jeblair: yah | 17:36 |
| mordred | jeblair: I can update the commit message if you like? | 17:36 |
| jeblair | mordred: meh, it'll be fun for future archeologists | 17:36 |
| jeblair | mordred: i'm +2 on all 3 in the master branch | 17:37 |
| mordred | woot | 17:37 |
| mordred | jeblair: oh - there's two others - one of which I imagine we may want to -2 because it doesn't make sense in v3 - lemme change the topics real quick | 17:39 |
| jeblair | mordred: the metadata and facts ones? | 17:40 |
| mordred | (the 'record info as ansible inventory' is the one I tihnk we may just want to abandon/-2) | 17:40 |
| mordred | yah | 17:40 |
| jeblair | and yes, let's go with -2 on that -- it feels like adding something we're going to remove in 1 month | 17:40 |
| mordred | the only reason I could see keeping the facts/inventory one is in case doing it would make rcarrillocruz's devstack-gate ansiblying more similar to what it wants to look like in v3 | 17:41 |
| mordred | but I don't think it does | 17:41 |
| jeblair | yeah, i don't think so either; if we're wrong, we can pull it back out | 17:41 |
| * mordred goes to abandon | 17:41 | |
| jeblair | (and even so, i think if we did use it, it might actually make the devstack-gate -> v3 transition *harder*) | 17:41 |
| mordred | yah | 17:42 |
| mordred | k. both abandoned | 17:42 |
| jeblair | the metadata patch is still good -- part of me thinks we're trading one limit (255 chars) for another (20 fields). but we're probably better off with the fields approach. we may want to keep that in mind when we think about how we encode image information in there -- like maybe we should combine all 4 attributes of the image used into one field | 17:44 |
| clarkb | fwiw rcarrillocruz's changes could use another core reviewer at this point | 17:44 |
| jeblair | mordred: 297950 +2 on master | 17:44 |
| mordred | woot | 17:45 |
| clarkb | I can approve 410879 right? | 17:50 |
| clarkb | its noop | 17:50 |
| clarkb | mordred: ^ | 17:51 |
| mordred | clarkb: yah | 17:52 |
| jlk | I don't suppose anybody knows off-hand what infra does with zuul 2.5 to get console.html to properly render as html (with newlines and such)? | 17:57 |
| jlk | is that a zuul side thing or an apache config thing? | 17:57 |
| pabelanger | apache setting IIRC | 17:59 |
| pabelanger | jlk: we also use http://git.openstack.org/cgit/openstack-infra/os-loganalyze | 18:00 |
| jlk | hrm, I can't tel lif that's what's hitting http://logs.openstack.org/18/406118/4/gate/gate-nova-python27-db-ubuntu-xenial/52b969d/console.html | 18:02 |
| pabelanger | Yup, it is | 18:02 |
| pabelanger | we have a whole log processing backend that does that | 18:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Add a log message when ansible times out https://review.openstack.org/410888 | 18:03 |
| jlk | alrighty | 18:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Fix watchdog timeout fix https://review.openstack.org/410902 | 18:21 |
| clarkb | jlk: pabelanger however I want to say the logs should be wrapped in <pre></pre> and just work | 18:31 |
| clarkb | if hosted as html | 18:31 |
| jlk | yeah that'd be a simple improvement. | 18:32 |
| jesusaur | clarkb: dont they also go through os_loganalyze? | 18:32 |
| jesusaur | oh, you're saying they should Just Work without that, too | 18:32 |
| jlk | i'td be nice | 18:33 |
| clarkb | ya if they don't then its likely a regression and we should consider adding the pre tags back in | 18:33 |
| jlk | they dont | 18:36 |
| jlk | e.g. http://logs.bonnyci.com/logs/1/1/d4ff2139f57a78bd924fdfa33103739e8783ee1b/check_github/tox_py27/5739c46/console.html | 18:36 |
| pabelanger | clarkb: the file isn't actually HTML however, we just dump text into it | 18:36 |
| pabelanger | I suspect console.html is just to keep our log processing happy | 18:37 |
| clarkb | pabelanger: yes thats why we need to wrap in <pre> | 18:37 |
| jlk | Looking at the plugin it'd be pretty simple to toss the <pre> and </pre> on it | 18:37 |
| * jlk tosses up a patch | 18:37 | |
| pabelanger | clarkb: but we use it for telnet too | 18:37 |
| pabelanger | unless we add something into post_playbook | 18:37 |
| clarkb | pabelanger: thats fine you'll just see <pre> at beginning and end of telnet | 18:37 |
| clarkb | won't hurt anything | 18:37 |
| mordred | clarkb: my eyes. it'll hurt my eyes :) | 18:38 |
| pabelanger | clarkb: true | 18:38 |
| jlk | oh hrm. | 18:39 |
| jlk | maybe not so easy, because we open that file every time a command is ran | 18:39 |
| jlk | It's a new open every time. I mean, I guess it's okay to have a bunch of <pre> ines in there right? | 18:39 |
| clarkb | just write it the first time? | 18:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Wrap zuul_log in pre HTML tag https://review.openstack.org/410911 | 18:41 |
| jlk | hard to tell what "the first time" is | 18:41 |
| pabelanger | ^ | 18:41 |
| clarkb | jlk: not really if file not empty then pre | 18:41 |
| pabelanger | un tested | 18:41 |
| clarkb | er if file empty | 18:41 |
| jlk | oh fair. | 18:41 |
| jlk | but looks like pabelanger beat me to it | 18:41 |
| jeblair | wait what are we doing? | 18:41 |
| pabelanger | however, still not a fan of it | 18:41 |
| clarkb | making the html console file html so that browsers are happy | 18:41 |
| jlk | except his doesn't do it only the first time, it'd be every time. | 18:42 |
| mordred | maybe instead of changes to 2.5 - how about just runnign os_loganalyze on the web server serving the logs? it also gets all of those nice intra log links and whatnot | 18:45 |
| pabelanger | Ya, actually like that better | 18:45 |
| pabelanger | not sure I want to add HTML things into ansible | 18:46 |
| clarkb | one big reason is that loganaylze is suepr specific about what it will convert and add tags to | 18:46 |
| jlk | Well, I'd need to dig into os-logalize and see if it's appropriate for non-infra sites | 18:46 |
| clarkb | and I don't want to have to keep up with a bunch of external CI systems trying to match all tehri stuff | 18:46 |
| clarkb | if we claim the file is html then I think we should just wrap in <pre> which is super simple way to make it just work everywhere | 18:46 |
| mordred | right. | 18:47 |
| jlk | this ^^ | 18:47 |
| jeblair | how relevant is this to v3? | 18:47 |
| mordred | but 2.5 is explicitly not intended to work anywhere other than infra | 18:47 |
| pabelanger | could also ForceType text/plain for console.html today too | 18:47 |
| mordred | so it's the same problem | 18:47 |
| jlk | depends, what does v3 do for log exposure? | 18:47 |
| mordred | we need to rework log exposure in v3 to provide for the websockets stuff - and the fact that the tasks will all be ansible themselves so we really need to get a callback plugin to interleave any captured log output into ansible task success/fail statements | 18:48 |
| mordred | (could use a spec write up on that too, tbh) | 18:48 |
| clarkb | with timestmaps! | 18:52 |
| clarkb | sorry thats like peeve #1 using ansible | 18:52 |
| mordred | yah. I think it's going to be a fun callback plugin to write | 18:52 |
| jlk | heh, we've done that in Ursula, timestamps for the things | 18:52 |
| jlk | interlaced with the nice thing that came in v2 which was displaying which file the task came from | 18:53 |
| clarkb | yes its a thing everyone has to do themselves because ansible procalimed a while back that they would not add it to ansible itself | 18:53 |
| clarkb | though that was pre red hat and I think a way to try and get people to buy tower | 18:53 |
| clarkb | maybe thats no longer the attitude | 18:53 |
| jlk | They ship an example timestamps callback I think. | 18:57 |
| jlk | https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/timer.py | 18:58 |
| jlk | but it's very minimalistic | 18:58 |
| jlk | really not much better than calling "time ansible-playbook blah blah blah" | 18:58 |
| rcarrillocruz | o/ | 18:59 |
| * rcarrillocruz rejoices as d-g ansibly changes are going thru the gate | 19:02 | |
| * rcarrillocruz checks mentions | 19:02 | |
| pabelanger | rcarrillocruz: ya, I haven't had time to poke at it. But is cool | 19:20 |
| openstackgerrit | Merged openstack-infra/zuul: Add a log message when ansible times out https://review.openstack.org/410888 | 19:30 |
| openstackgerrit | Merged openstack-infra/zuul: Fix watchdog timeout fix https://review.openstack.org/410902 | 19:31 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul: Remove now-unused ZuulTestCase.resetGearmanServer https://review.openstack.org/410935 | 19:41 |
| pabelanger | jeblair: Shrews: should we remove the ability for nodepoold.py to launch a builder? And expect everybody to just use nodepool-builder command? | 19:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Remove the ability for nodepoold to launch a builder https://review.openstack.org/410939 | 19:49 |
| clarkb | pabelanger: what is the benefit of ^? | 19:50 |
| pabelanger | jeblair: Shrews: clarkb: mordred: ^ let me know what you think | 19:50 |
| clarkb | having an all in one daemon is nice and simple. I don't think we should require everyone to have a large deployment like we do | 19:50 |
| pabelanger | true, but starting another process on the same machine will work out of box. | 19:51 |
| clarkb | no it requires you to have more init scripts and such | 19:51 |
| pabelanger | I like the idea of removing it, as it reduces the things we need to test and support | 19:51 |
| clarkb | pabelanger: but thats actually how much of the testing works | 19:52 |
| pabelanger | okay, an init script then | 19:52 |
| clarkb | so its actually harder to test the otehr thing (which we do in the integration job but not unit/functional) | 19:52 |
| pabelanger | I thought we just called the builder directly for testing | 19:53 |
| pabelanger | not nodepoold | 19:53 |
| clarkb | we run it all in the same process | 19:53 |
| clarkb | which is what you are proposing we stop doing | 19:53 |
| pabelanger | shrugs, if we want to keep it, that is fine. was trying to get away from doing nodepoold --no-builder dance in our init.d scripts | 19:54 |
| clarkb | if we want to invert the default I think thats fine. But having the option is nice for simpler setups and it is how much of the testing runs too | 19:55 |
| Shrews | pabelanger: i don't really have an answer for that question. sorry | 20:02 |
| pabelanger | I'll poke at it again, see what is needed to invert the logic | 20:04 |
| rcarrillocruz | hmm | 20:41 |
| rcarrillocruz | voluptous question | 20:41 |
| rcarrillocruz | is it possible to define that an attribute is optional but required if another one is specified? | 20:41 |
| clarkb | rcarrillocruz: I think with a validation funcion | 20:45 |
| pabelanger | yup | 20:45 |
| pabelanger | I think we do something like that in grafyaml | 20:45 |
| rcarrillocruz | cool, let me read, thx | 20:45 |
| jeblair | clarkb, pabelanger: i think i lean toward removing support from nodepoold to launch a builder, mostly because even on a single host, it's better to launch them as separate processes (we rarely want to stop both of them at the same time). i think that any package or installation of them will need the init scripts for both anyway, so i think the main argument for being able to start them together is perceived ease of use for a new user. but i ... | 20:55 |
| jeblair | ... think that the confusion arising around whether to decide to start them together or not is worse than the confusion of having two processes. and things like puppet-openstackci and ansible deployments can make that easy. | 20:55 |
| clarkb | I think my biggest concern is just keeping number of things that change to a minimum to help users transition (this is why the builder can run in the main daemon too fwiw) | 21:00 |
| dmsimard | https://review.openstack.org/#/c/405613/ (ARA implementation patch) is based on top of the large series of ansiblezation patches and I just got the first (probably out of many) merge failure notifications. Should I just base it on current master ? | 21:14 |
| mordred | dmsimard: yah- I thnk so | 21:14 |
| mordred | dmsimard: there is enough ansible in current master that we should be able to see the output | 21:15 |
| dmsimard | Okay. | 21:15 |
| rcarrillocruz | dmsimard: yah, a bunch just merged, so you better pull and rebase | 21:26 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/zuul: Re-enable test_nonvoting_job https://review.openstack.org/410961 | 21:29 |
| dmsimard | some of these views would be a great fit for ARA to reduce the amount of navigation I think http://www.patternfly.org/pattern-library/content-views/list-view/#/_overview | 21:29 |
| dmsimard | maybe in the next version :) | 21:30 |
| clarkb | rcarrillocruz: dmsimard you don't need to pull and rebase | 21:32 |
| clarkb | just recheck | 21:32 |
| dmsimard | clarkb: I just rebased on top of master, it was a clean rebase so no effort. Took 5 seconds :) | 21:32 |
| dmsimard | mordred ninja +W'd the rebase | 21:33 |
| clarkb | ok doesn't hurt to just pointing out that zuul tests you against merged code | 21:33 |
| clarkb | so you don't need to rebase unless there are conflicts | 21:33 |
| dmsimard | but there was | 21:33 |
| clarkb | ah ok | 21:33 |
| dmsimard | at least as far as the jenkins comment says there was a merge conflict | 21:33 |
| dmsimard | it asks nicely to rebase :) | 21:33 |
| mordred | clarkb: it was based on one of the not-yet-ready ansibly patches - dmsimard rebased against master instead | 21:35 |
| clarkb | gotcha | 21:35 |
| dmsimard | ansibly ? that's the term ? okay, I'll stop using ansiblization :P | 21:35 |
| rcarrillocruz | hah, blame me for that 'term' | 21:35 |
| mordred | love ansibly as a term :) | 21:36 |
| jhesketh | Morning | 22:01 |
| SpamapS | adam_g: looks like we crossed streams on test_nonvoting_job | 22:17 |
| adam_g | SpamapS: yay | 22:17 |
| SpamapS | adam_g: I claimed it in task 3428, you in 3431, so probably an HTTP refresh problem :-P | 22:17 |
| SpamapS | adam_g: actually no, you are 2 days late. ;) | 22:17 |
| adam_g | SpamapS: no, that was my fault. no biggie, didnt burn | 22:19 |
| adam_g | ... much time on that | 22:19 |
| SpamapS | yeah it was an easy one | 22:19 |
| adam_g | SpamapS: abandoning mine | 22:20 |
| SpamapS | I think we're down to about 40 @skip's in feature/zuulv3 now. | 22:21 |
| adam_g | what is the plan wrt merge failure reporting? im looking at the tests now and see that thigns are no longer architected to support reporting it in the way it used to be done. | 22:21 |
| adam_g | im not seeing anything in the board about that, is there some notes anywhere about that? | 22:22 |
| adam_g | ... or is it deferred till post-ZK, since it'll probably involve some work in the gearman area | 22:22 |
| SpamapS | I'd expect merge reports to work more or less the same | 22:24 |
| clarkb | though the question of whether or not we needed that anymore did come up I think. Since gerrit checks as does github for you | 22:24 |
| clarkb | however I suppose $otherthing may not so keeping it around would still be good | 22:24 |
| clarkb | jeblair: is that where we left off ^ | 22:24 |
| jeblair | yeah, it came up in the zuultrigger discussion | 22:25 |
| jeblair | just limiting the discussion to real jobs (not merge-check noop jobs).... | 22:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Remove the ability for nodepoold to launch a builder https://review.openstack.org/410939 | 22:28 |
| jeblair | there's a tradeoff here -- we don't really need to merge changes until we launch them, so the most straightforward path is to have the launcher report a merge failure. however, that would happen after node allocation, so it could be a bit inefficient. if you have a source mergability check (eg, gerrit/github) as a pipeline requirement, that won't be an issue. otherwise, the only way to prevent that would be to invoke the merger on every ... | 22:28 |
| jeblair | ... change before sending it to the launcher to be merged again. | 22:28 |
| jeblair | that's also inefficient | 22:29 |
| jeblair | though it's only one extra merge for the change, so not super crazy. | 22:29 |
| adam_g | the latter sounds reasonable, perhaps only when a new 'check-merge: true' configuration option is set on the pipeline? | 22:31 |
| jeblair | adam_g: that's an option | 22:32 |
| jeblair | (note that even currently we sometimes send the change to the mergers before launching in the case of a zuul.yaml update, so that we can fetch the new config) | 22:33 |
| SpamapS | My wild ass guess is that a merge is at least 10x cheaper than a node allocation and recycle. | 22:33 |
| SpamapS | But if trigger sources are already doing it.. seems like the right thing to do is optimize for trigger sources doing it. | 22:34 |
| jeblair | SpamapS: yeah... actually oddly enough the worst and best cases line up: 30 seconds. but the general case you're probably right, or it might even be worse. :) | 22:34 |
| jeblair | (our fastest cloud gives us a node in 30 seconds and our slowest repo takes 30 seconds to merge a change) | 22:34 |
| adam_g | ha | 22:35 |
| jeblair | (our slowest cloud is 5 minutes and our fastest repo is, well, seconds at most) | 22:35 |
| SpamapS | In the "we don't pay for nodes" scenario, that's definitely a wash. | 22:35 |
| SpamapS | Dunno if any clouds bill in less than 1 hour increments. | 22:35 |
| adam_g | in either case, theres a bunch of reporting code that isn't used currently and shoudl either go away or be updated to support pre-launcher merge checking | 22:35 |
| SpamapS | I know early days, EC2 would not bill you if you shut down in the first 10 minutes or so. | 22:36 |
| SpamapS | Guessing RAX has a similar "oops" window. | 22:36 |
| SpamapS | (just thinking about zuul users who are paying for computers) | 22:36 |
| jeblair | yeah, i think i'm leaning toward just putting the pre-launch merge back in all cases for now. | 22:36 |
| SpamapS | it's not a complex path, so +1 | 22:36 |
| jeblair | (rack up another point for 'keep the separate mergers') | 22:37 |
| clarkb | we fixed grub to not have a timeout and just boot our images and I did rough math that showed we were saving more than a full cpu day per day with that small change | 22:37 |
| adam_g | i may take a stab at it if as part of re-enabling the existing merge conflict reporter tests | 22:37 |
| clarkb | (just pointing out that it does add up fast especially when using donated resources) | 22:37 |
| SpamapS | clarkb: yeah, the moral of the story is "Try hard to only allocate if you know you can use it" | 22:38 |
| jeblair | adam_g: cool... most of the code is still there for the getLayout() bits. that, plus some comparison with v2 should get you most of the way there | 22:39 |
| jeblair | adam_g: roughly speaking, "just do that for every change, but only use the custom layout if necessary" | 22:40 |
| adam_g | jeblair: yea, the harder part is going to be gutting all of the stuff built up around build_set.commit / build_set.unable_to_merge | 22:40 |
| SpamapS | I wonder if we should do a round of dead-code-hunting using coverage reports once we're done re-enabling all the old tests. | 22:40 |
| jeblair | adam_g: oh that shouldn't change i don't think | 22:40 |
| jeblair | adam_g: those bits are still active in the getLayout code path | 22:40 |
| jeblair | so if we run that path always, they'll be, erm, always active :) | 22:40 |
| jeblair | SpamapS: ++ | 22:41 |
| adam_g | jeblair: oh, ill need to take a closer look down there, then. the ways v2.5 used that stuff in other places seems to have changed drastically | 22:42 |
| jeblair | adam_g: yeah, the getLayout bit is still basically the v2.5 pre-launch merge check, just made conditional, so making it unconditional shouldn't be a huge thing and should basically plug right back in to the old stuff | 22:44 |
| mordred | SpamapS: re: billing - fwiw, here are three entries from fuga.io | 22:50 |
| mordred | Nov 01, 2016 — Nov 30, 2016 OpenStack Instance - mttest / openstack_flavour_c1.small €2.82 | 22:50 |
| mordred | Nov 01, 2016 — Nov 30, 2016 OpenStack Instance - testmt / openstack_flavour_c1.tiny €0.01 | 22:50 |
| mordred | Nov 01, 2016 — Nov 30, 2016 OpenStack Instance - testmt / openstack_flavour_c1.medium | 22:50 |
| mordred | €0.01 | 22:50 |
| mordred | the c1.medium has a rate of € 0,0600/hr | 22:51 |
| mordred | so it does seem they do fractional hour billing | 22:51 |
| mordred | with a floor of €0.01 | 22:51 |
| rcarrillocruz | fuga, wow, didn't know about this provider | 22:54 |
| rcarrillocruz | funny hearing so many 'openstack is dead' , yet openstack public clouds keep springing from everywhere | 22:55 |
| mordred | rcarrillocruz: openstack is only dead if you're a large already-not-very-good-at-tech US corporation | 22:58 |
| mordred | rcarrillocruz: especially one who was hoping openstack would somehow be a silver bullet to help you become the next amazon with no effort on your part | 22:58 |
| rcarrillocruz | hmm, that seems familiar :D | 22:59 |
| mordred | rcarrillocruz: kiss.cloud is another new public cloud | 22:59 |
| rcarrillocruz | yeah, heard that randomly from ng the other day | 22:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add missing pause fields to config-validate https://review.openstack.org/410992 | 23:13 |
| SpamapS | It's as dead as Linux was when VA Linux renamed themselves to VA Research and SCO started suing its users. ;) | 23:14 |
| SpamapS | Meaning "it's actually real and has users and has momentum" | 23:15 |
| clarkb | I dunno linux is only like 2% of steam installs. Clearly niche | 23:15 |
| clarkb | >_< | 23:15 |
| pabelanger | OpenStack is dead, long live OpenStack | 23:15 |
| pabelanger | clarkb: are you running steam on linux? I've tried it on my desktop and works pretty well. Minus the crappy graphics card I have | 23:16 |
| clarkb | pabelanger: not anymore, but I have a machine in a closet that has a hard drive with it on it | 23:17 |
| clarkb | ya it did work pretty well for don't starve | 23:17 |
| rcarrillocruz | my gaming time has gone to zero for some time | 23:21 |
| rcarrillocruz | i'm hoping i can make some room for a VR space to use my oculus from time to time tho | 23:21 |
| rcarrillocruz | room in my new house i mean | 23:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Validate configs when used by tests https://review.openstack.org/410995 | 23:26 |
| *** willthames has joined #zuul | 23:39 | |
| pabelanger | jeblair: Shrews: something to dig into tomorrow: http://logs.openstack.org/92/410992/1/check/gate-nodepool-python27-ubuntu-xenial/3ccd075/console.html#_2016-12-14_23_26_49_464170 | 23:40 |
| pabelanger | a failure in our test_image_removal job | 23:40 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/zuul: WIP Add per-repo public and private keys https://review.openstack.org/406382 | 23:41 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Validate configs when used by tests https://review.openstack.org/411002 | 23:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add missing pause fields to config-validate https://review.openstack.org/411004 | 23:52 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Validate configs when used by tests https://review.openstack.org/410995 | 23:53 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!