| *** saschpe- has quit IRC | 00:00 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Unbound: also override dhclient config on debian https://review.openstack.org/82406 | 00:01 |
|---|---|---|
| jeblair | lifeless: i think that will fix it ^ | 00:02 |
| jeblair | clarkb, mordred, fungi: ^ | 00:02 |
| *** rhsu has joined #openstack-infra | 00:02 | |
| jeblair | lifeless: i think that even without the unbound change, that server would be using the dhcp-supplied nameserver | 00:05 |
| lifeless | jeblair: it will have two nameservers | 00:06 |
| lifeless | 192.168.1.1 and 8.8.8.8 | 00:06 |
| lifeless | 1.1 will be a brain damaged dnsmasq | 00:06 |
| lifeless | e.g. nameserver 8.8.8.8 | 00:06 |
| lifeless | nameserver 192.168.1.1 | 00:06 |
| lifeless | would be the default without your scripts | 00:06 |
| *** saschpe has joined #openstack-infra | 00:07 | |
| jeblair | lifeless: i'm guessing the change is that we have disabled resolvconf, and perhaps dhclient is overwriting the nameserver when it gets the second lease whereas resolvconf may have added it. | 00:09 |
| jeblair | since 8.8.8.8 comes from the eth0 lease and 192.168.1.1 comes from the eth1 lease | 00:09 |
| *** yamahata__ has quit IRC | 00:09 | |
| *** dcramer_ has quit IRC | 00:09 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Unbound: also override dhclient config on debian https://review.openstack.org/82406 | 00:10 |
| lifeless | jeblair: disabling resolvconf - that worries me, because its exactly this sort of multi-interface thing where resolvconf solves lots and lots of problems | 00:11 |
| *** thedodd has joined #openstack-infra | 00:12 | |
| jeblair | lifeless: unfortunately, rackspace performs (or has performed) file injection on /etc/resolv.conf which breaks the symlink | 00:13 |
| *** CaptTofu has joined #openstack-infra | 00:13 | |
| jeblair | lifeless: if the symlink is broken, it's disabled | 00:13 |
| *** caleb_ has joined #openstack-infra | 00:13 | |
| jeblair | lifeless: so we ended up with a bunch of machines where it was not easy to make them consistent | 00:13 |
| nibalizer | mordred: whats the reason to have the slaves run puppet apply instead of puppet agent? | 00:13 |
| jeblair | lifeless: at any rate, we only want one thing in our resolv.conf files "nameserver 127.0.0.1", so it seemed the better way to go is to make that a static file, and disable resolvconf in all cases | 00:14 |
| *** krotscheck has quit IRC | 00:14 | |
| jeblair | lifeless: (though it does also apparently mean that we need to prevent dhclient from writing the file) | 00:14 |
| openstackgerrit | A change was merged to openstack-infra/config: Unbound: also override dhclient config on debian https://review.openstack.org/82406 | 00:15 |
| *** krotscheck has joined #openstack-infra | 00:15 | |
| jeblair | lifeless: any chance you can fix the quota? i can't build an image when we're over quota | 00:16 |
| lifeless | jeblair: I'm hoping to hear from mikal about what data he needs to examine the problem | 00:17 |
| jeblair | lifeless: okay, retrying managed to find a slot | 00:18 |
| lifeless | jeblair: btw you don't need to remove resolvconf to have a static /etc/resolv.conf | 00:18 |
| jeblair | lifeless: we didn't, we just broke the symlink | 00:19 |
| lifeless | jeblair: since resolvconf's database is separate; just removing the symlink is all you need. | 00:19 |
| lifeless | jeblair: ok | 00:19 |
| lifeless | jeblair: I haven't read your changes obviously ;) | 00:19 |
| lifeless | I'm surprised dhclient writes the file, since with resolvconf present it should never do that, rather push the data to resolvconf and let resolvconf DTRT | 00:20 |
| jeblair | lifeless: me too | 00:20 |
| jeblair | clarkb, mordred, fungi: ^ we should keep an eye on what happens after the tripleo image build is finished; if it does in fact fix the problem or make it worse | 00:20 |
| jeblair | clarkb, mordred, fungi: because it's also likely to have an effect on hpcloud precise and devstack-precise nodes | 00:21 |
| *** harlowja_at_home has quit IRC | 00:21 | |
| lifeless | jeblair: are you running your own caching nameserver on each node? Wouldn't it be better to run a caching nameserver in each cloud ? | 00:21 |
| lifeless | jeblair: we were going to put a recursive bind9 in ci-overcloud | 00:21 |
| lifeless | (and point DNS at that) | 00:22 |
| *** bhuvan has joined #openstack-infra | 00:22 | |
| jeblair | lifeless: each node is running a caching-forwarding nameserver; having a dedicated per-region caching-recursive nameserver is probably an option when all our providers support neutron | 00:22 |
| lifeless | jeblair: where does it forward to? That dhclient override squashes the DHCP supplies NS, right ? | 00:23 |
| jeblair | lifeless: but until then, running a public recursive nameserver is a bit too much of a commitment (esp with dns amplication attacks, etc, these days) | 00:23 |
| jeblair | lifeless: the supplied nameservers if they are well-behaved (eg rax), or google if they are not (hp) | 00:23 |
| lifeless | jeblair: oh, I wasn't going to run a public one :) | 00:23 |
| lifeless | jeblair: I was joing to put it on a node inside the cloud | 00:24 |
| jeblair | lifeless: i'm not either, but without neutron support, we can't run a network-private one (because we don't have a private network) | 00:24 |
| *** matsuhashi has joined #openstack-infra | 00:25 | |
| lifeless | jeblair: do you still get the dns dynamically, or do you manually store it in the puppet rules? | 00:25 |
| lifeless | jeblair: I'm asking so if/when we get these local cached resources in ci-overcloud we know whether to change our DHCP, or to just tell you. | 00:26 |
| *** bhuvan has quit IRC | 00:26 | |
| mordred | nibalizer: the slaves are single use slaves | 00:27 |
| mordred | nibalizer: we create and destroy as many as 20k of them in a day | 00:27 |
| mordred | nibalizer: which means that the cert stuff on the puppemaster would be a complete nightmare | 00:28 |
| mordred | nibalizer: also, we consider them hostile | 00:28 |
| mordred | nibalizer: make sense? | 00:29 |
| mordred | jeblair: kk | 00:29 |
| mordred | nibalizer: needless to say - puppet apply is SUPER important and breaking it is a non-starter :) | 00:30 |
| *** rhsu has quit IRC | 00:32 | |
| jeblair | lifeless: dynamically | 00:40 |
| lifeless | jeblair: ok, cool | 00:41 |
| *** jeremyb has quit IRC | 00:43 | |
| *** jeremyb has joined #openstack-infra | 00:43 | |
| *** Ryan_Lane has quit IRC | 00:50 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Shorten update spinner fadeout on status page https://review.openstack.org/82019 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to status page to collapse changesets https://review.openstack.org/82018 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add expand by default option to status page https://review.openstack.org/82290 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add a total progress bar to each change https://review.openstack.org/82017 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Format change headers like infra's status page https://review.openstack.org/82016 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add a filter box to status.html https://review.openstack.org/82279 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Build status page more jquery like https://review.openstack.org/81717 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Update zuul's status page to bootstrap 3.1.1 https://review.openstack.org/82013 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add demo support for openstack's live zuul feed https://review.openstack.org/82014 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add in progress bars for jobs https://review.openstack.org/82015 | 00:51 |
| *** jhesketh has quit IRC | 00:54 | |
| *** Ryan_Lane has joined #openstack-infra | 00:55 | |
| *** jhesketh has joined #openstack-infra | 00:55 | |
| *** yongli has joined #openstack-infra | 01:02 | |
| *** BadCub01 has quit IRC | 01:12 | |
| *** yamahata has joined #openstack-infra | 01:14 | |
| jeblair | clarkb, mordred, fungi: dns on new tripleo nodes look good, so i think https://review.openstack.org/#/c/82406/ worked as intended. i expect hpcloud to be fine, but just in case, be aware of that in case we need to kill new images tomorrow morning. | 01:20 |
| jeblair | lifeless: ^ | 01:20 |
| *** thedodd has quit IRC | 01:20 | |
| lifeless | jeblair: cool, thank you! | 01:22 |
| *** CaptTofu has quit IRC | 01:23 | |
| nibalizer | mordred: okay, that makes sense | 01:26 |
| *** mburned has quit IRC | 01:32 | |
| *** nosnos has joined #openstack-infra | 01:36 | |
| *** thuc has joined #openstack-infra | 01:41 | |
| *** thuc_ has joined #openstack-infra | 01:43 | |
| *** thuc has quit IRC | 01:47 | |
| *** mburned has joined #openstack-infra | 01:49 | |
| *** mburned has quit IRC | 01:56 | |
| *** mriedem has quit IRC | 01:56 | |
| *** mburned has joined #openstack-infra | 02:00 | |
| *** mriedem has joined #openstack-infra | 02:07 | |
| *** yjiang has joined #openstack-infra | 02:07 | |
| *** thedodd has joined #openstack-infra | 02:08 | |
| *** adalbas has joined #openstack-infra | 02:12 | |
| *** david-lyle has joined #openstack-infra | 02:14 | |
| *** CaptTofu has joined #openstack-infra | 02:15 | |
| *** mburned has quit IRC | 02:17 | |
| *** mburned has joined #openstack-infra | 02:18 | |
| *** zehicle_at_dell has quit IRC | 02:21 | |
| *** zehicle_at_dell has joined #openstack-infra | 02:22 | |
| *** mburned has quit IRC | 02:23 | |
| *** mriedem has quit IRC | 02:30 | |
| *** caleb_ has quit IRC | 02:32 | |
| *** mburned_out has joined #openstack-infra | 02:34 | |
| *** mburned_out has quit IRC | 02:42 | |
| *** thuc_ has quit IRC | 02:43 | |
| *** thuc has joined #openstack-infra | 02:43 | |
| *** thuc_ has joined #openstack-infra | 02:45 | |
| *** bknudson has quit IRC | 02:45 | |
| *** thuc has quit IRC | 02:47 | |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Fix run-mirror writing wrong project directory https://review.openstack.org/81469 | 02:48 |
| *** thedodd has quit IRC | 02:48 | |
| *** rcleere has joined #openstack-infra | 02:52 | |
| *** matsuhashi has quit IRC | 02:55 | |
| *** adalbas has quit IRC | 03:03 | |
| *** michchap_ has joined #openstack-infra | 03:08 | |
| *** michchap has quit IRC | 03:09 | |
| *** CaptTofu has quit IRC | 03:11 | |
| *** CaptTofu has joined #openstack-infra | 03:12 | |
| *** CaptTofu has quit IRC | 03:17 | |
| *** harlowja_at_home has joined #openstack-infra | 03:20 | |
| *** matsuhashi has joined #openstack-infra | 03:21 | |
| *** nati_ueno has joined #openstack-infra | 03:23 | |
| *** Ryan_Lane has quit IRC | 03:31 | |
| *** harlowja_at_home has quit IRC | 03:41 | |
| *** ekarlso has quit IRC | 03:48 | |
| *** ekarlso has joined #openstack-infra | 03:48 | |
| *** thuc_ has quit IRC | 03:53 | |
| *** thuc has joined #openstack-infra | 03:54 | |
| *** thuc has quit IRC | 03:59 | |
| *** thuc has joined #openstack-infra | 04:02 | |
| *** todd_dsm has joined #openstack-infra | 04:05 | |
| *** bhuvan has joined #openstack-infra | 04:06 | |
| *** bhuvan has quit IRC | 04:16 | |
| *** harlowja_at_home has joined #openstack-infra | 04:20 | |
| *** _Alexandra_ has joined #openstack-infra | 04:27 | |
| *** Alexandra has quit IRC | 04:30 | |
| *** _Alexandra_ has quit IRC | 04:30 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: disable the jeepyb trivial rebase hook on review.o.o https://review.openstack.org/82416 | 04:34 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update gerrit project configs to use new Workflow label https://review.openstack.org/79107 | 04:34 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update trigger to match new workflow label https://review.openstack.org/79103 | 04:34 |
| *** thuc_ has joined #openstack-infra | 04:39 | |
| *** Alexandra has joined #openstack-infra | 04:40 | |
| *** todd_dsm has quit IRC | 04:41 | |
| *** thuc has quit IRC | 04:42 | |
| *** alff has joined #openstack-infra | 04:42 | |
| *** thuc_ has quit IRC | 04:43 | |
| *** jepoy_ has joined #openstack-infra | 04:44 | |
| *** jepoy has quit IRC | 04:45 | |
| *** mestery has joined #openstack-infra | 04:46 | |
| *** nati_ueno has quit IRC | 04:46 | |
| *** nati_ueno has joined #openstack-infra | 04:47 | |
| *** nati_ueno has quit IRC | 04:51 | |
| *** nati_ueno has joined #openstack-infra | 04:52 | |
| *** harlowja_at_home has quit IRC | 05:06 | |
| *** bhuvan has joined #openstack-infra | 05:12 | |
| *** CaptTofu has joined #openstack-infra | 05:12 | |
| *** chandankumar has quit IRC | 05:14 | |
| *** mestery has quit IRC | 05:14 | |
| *** nosnos_ has joined #openstack-infra | 05:16 | |
| *** CaptTofu has quit IRC | 05:17 | |
| *** bhuvan has quit IRC | 05:19 | |
| *** nosnos has quit IRC | 05:19 | |
| *** nicedice has quit IRC | 05:23 | |
| *** zhiyan_ is now known as zhiyan | 05:24 | |
| *** chandan_kumar has joined #openstack-infra | 05:25 | |
| openstackgerrit | ZhiQiang Fan proposed a change to openstack/requirements: Rename PrettyTable to prettytable https://review.openstack.org/82422 | 05:31 |
| *** sdake_ has quit IRC | 05:47 | |
| yjiang | hi guys,I want to build a openstack 3rd part testing system.Should I use zuul or gerrit trigger as event stream trigger?Is there any related document about these details of configuration? | 05:48 |
| *** alff has quit IRC | 05:50 | |
| *** todd_dsm has joined #openstack-infra | 05:52 | |
| *** mihgen has joined #openstack-infra | 06:00 | |
| *** matsuhas_ has joined #openstack-infra | 06:00 | |
| *** matsuhashi has quit IRC | 06:01 | |
| *** Alexandra has quit IRC | 06:01 | |
| *** mestery has joined #openstack-infra | 06:13 | |
| *** mestery has quit IRC | 06:13 | |
| *** mestery has joined #openstack-infra | 06:14 | |
| *** nati_ueno has quit IRC | 06:14 | |
| *** mestery_ has joined #openstack-infra | 06:15 | |
| *** mestery has quit IRC | 06:15 | |
| *** nosnos has joined #openstack-infra | 06:19 | |
| *** mestery_ has quit IRC | 06:20 | |
| *** nosnos_ has quit IRC | 06:22 | |
| *** obondarev has quit IRC | 06:25 | |
| *** ildikov_ has quit IRC | 06:26 | |
| *** matsuhas_ has quit IRC | 06:31 | |
| *** matsuhashi has joined #openstack-infra | 06:31 | |
| *** saju_m has joined #openstack-infra | 06:35 | |
| openstackgerrit | Thomas Goirand proposed a change to openstack/requirements: Using happybase 0.8 https://review.openstack.org/82438 | 06:35 |
| *** moted has quit IRC | 06:38 | |
| *** amotoki has joined #openstack-infra | 06:41 | |
| *** mrda is now known as mrda_away | 06:45 | |
| *** jamielennox is now known as jamielennox|away | 06:51 | |
| *** saju_m has quit IRC | 06:59 | |
| *** saju_m has joined #openstack-infra | 07:12 | |
| *** CaptTofu has joined #openstack-infra | 07:13 | |
| openstackgerrit | Ivan Melnikov proposed a change to openstack-dev/hacking: Disable logging correctly https://review.openstack.org/82441 | 07:13 |
| *** ildikov_ has joined #openstack-infra | 07:13 | |
| *** saju_m has quit IRC | 07:17 | |
| *** saju_m has joined #openstack-infra | 07:17 | |
| *** michchap has joined #openstack-infra | 07:18 | |
| *** branen_ has joined #openstack-infra | 07:18 | |
| *** mihgen_ has joined #openstack-infra | 07:18 | |
| *** CaptTofu has quit IRC | 07:18 | |
| *** kmartin has joined #openstack-infra | 07:18 | |
| *** krotscheck has quit IRC | 07:19 | |
| *** nprivalova has quit IRC | 07:19 | |
| *** todd_dsm has quit IRC | 07:19 | |
| *** jeremyb has quit IRC | 07:19 | |
| *** branen__ has quit IRC | 07:19 | |
| *** KurtMartin has quit IRC | 07:19 | |
| *** jhesketh_ has quit IRC | 07:19 | |
| *** loquacities has quit IRC | 07:19 | |
| *** Steap has quit IRC | 07:19 | |
| *** michchap_ has quit IRC | 07:19 | |
| *** Steap_ has joined #openstack-infra | 07:19 | |
| *** jeremyb_ has joined #openstack-infra | 07:19 | |
| *** jhesketh_ has joined #openstack-infra | 07:19 | |
| *** mihgen has quit IRC | 07:20 | |
| *** mihgen_ is now known as mihgen | 07:20 | |
| *** loquacities has joined #openstack-infra | 07:20 | |
| *** saju_m has quit IRC | 07:22 | |
| *** jeremyb_ has quit IRC | 07:23 | |
| *** jeremyb_ has joined #openstack-infra | 07:23 | |
| *** saju_m has joined #openstack-infra | 07:23 | |
| *** jeremyb_ is now known as jeremyb | 07:24 | |
| *** krotscheck has joined #openstack-infra | 07:24 | |
| *** saju_m has quit IRC | 07:30 | |
| *** saju_m has joined #openstack-infra | 07:31 | |
| *** nprivalova has joined #openstack-infra | 07:33 | |
| *** flaper87|afk is now known as flaper87 | 07:40 | |
| *** jlibosva has joined #openstack-infra | 07:44 | |
| *** e0ne has joined #openstack-infra | 07:44 | |
| *** e0ne has quit IRC | 07:47 | |
| *** ttx has quit IRC | 07:51 | |
| *** todd_dsm has joined #openstack-infra | 07:57 | |
| *** bookwar has joined #openstack-infra | 08:02 | |
| *** ttx has joined #openstack-infra | 08:02 | |
| *** ttx has quit IRC | 08:02 | |
| *** ttx has joined #openstack-infra | 08:02 | |
| *** ttx has quit IRC | 08:06 | |
| *** ttx has joined #openstack-infra | 08:07 | |
| *** ttx has quit IRC | 08:07 | |
| *** ttx has joined #openstack-infra | 08:07 | |
| *** dizquierdo has joined #openstack-infra | 08:08 | |
| *** yolanda has joined #openstack-infra | 08:08 | |
| *** rlandy has joined #openstack-infra | 08:11 | |
| *** jpich has joined #openstack-infra | 08:21 | |
| *** bogdando has joined #openstack-infra | 08:26 | |
| *** hashar has joined #openstack-infra | 08:28 | |
| *** e0ne has joined #openstack-infra | 08:30 | |
| *** yolanda has quit IRC | 08:31 | |
| *** afazekas has quit IRC | 08:32 | |
| *** afazekas has joined #openstack-infra | 08:33 | |
| *** jgallard has joined #openstack-infra | 08:34 | |
| *** soren_ has quit IRC | 08:36 | |
| *** soren_ has joined #openstack-infra | 08:36 | |
| *** soren_ is now known as soren | 08:36 | |
| *** yolanda has joined #openstack-infra | 08:36 | |
| *** thuc has joined #openstack-infra | 08:39 | |
| *** roeyc has joined #openstack-infra | 08:41 | |
| *** pblaho has joined #openstack-infra | 08:43 | |
| *** thuc has quit IRC | 08:44 | |
| *** hashar has quit IRC | 08:46 | |
| *** mihgen has quit IRC | 08:50 | |
| *** ilyashakhat has quit IRC | 08:55 | |
| *** derekh has joined #openstack-infra | 08:57 | |
| *** jlibosva has quit IRC | 08:58 | |
| *** jepoy_ has quit IRC | 08:58 | |
| *** hashar has joined #openstack-infra | 08:59 | |
| *** rossella_s has joined #openstack-infra | 08:59 | |
| *** jlibosva has joined #openstack-infra | 09:00 | |
| *** yassine has joined #openstack-infra | 09:05 | |
| *** ihrachys|wfh is now known as ihrachys | 09:06 | |
| *** jooools has joined #openstack-infra | 09:09 | |
| *** jepoy has joined #openstack-infra | 09:13 | |
| *** jlibosva has quit IRC | 09:13 | |
| *** jlibosva has joined #openstack-infra | 09:14 | |
| *** CaptTofu has joined #openstack-infra | 09:14 | |
| openstackgerrit | Victor Stinner proposed a change to openstack/requirements: Allow trollius 0.2 https://review.openstack.org/79901 | 09:16 |
| *** CaptTofu has quit IRC | 09:19 | |
| *** ilyashakhat has joined #openstack-infra | 09:20 | |
| *** dpyzhov has joined #openstack-infra | 09:24 | |
| *** pbelanyi has joined #openstack-infra | 09:24 | |
| *** katyafervent has joined #openstack-infra | 09:25 | |
| *** jepoy has quit IRC | 09:26 | |
| *** e0ne_ has joined #openstack-infra | 09:27 | |
| *** johnthetubaguy has joined #openstack-infra | 09:27 | |
| *** johnthetubaguy has quit IRC | 09:28 | |
| *** johnthetubaguy has joined #openstack-infra | 09:28 | |
| *** mihgen has joined #openstack-infra | 09:29 | |
| *** e0ne has quit IRC | 09:29 | |
| *** dizquierdo has quit IRC | 09:31 | |
| *** ominakov has joined #openstack-infra | 09:33 | |
| *** vkozhukalov has joined #openstack-infra | 09:50 | |
| *** ociuhandu has quit IRC | 09:53 | |
| *** johnthetubaguy1 has joined #openstack-infra | 09:55 | |
| *** johnthetubaguy has quit IRC | 09:56 | |
| *** yjiang has quit IRC | 09:56 | |
| *** ociuhandu has joined #openstack-infra | 10:08 | |
| *** vkozhukalov has left #openstack-infra | 10:11 | |
| *** mkoderer has joined #openstack-infra | 10:23 | |
| *** jp_at_hp has joined #openstack-infra | 10:28 | |
| *** matsuhashi has quit IRC | 10:33 | |
| *** matsuhas_ has joined #openstack-infra | 10:35 | |
| *** david-lyle has quit IRC | 10:41 | |
| *** yamahata has quit IRC | 10:43 | |
| *** CaptTofu has joined #openstack-infra | 10:48 | |
| *** yassine has quit IRC | 10:52 | |
| *** vkozhukalov has joined #openstack-infra | 10:54 | |
| *** dizquierdo has joined #openstack-infra | 11:00 | |
| *** mburned_out has joined #openstack-infra | 11:05 | |
| *** matsuhas_ has quit IRC | 11:07 | |
| ihrachys | does anyone know whether any service is running on gate machines listening on :9000? We have a test failing due to this in stable/havana oslo-incubator. | 11:12 |
| *** jepoy has joined #openstack-infra | 11:12 | |
| *** mbacchi has joined #openstack-infra | 11:14 | |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Format json text Updated lp_projects definitions https://review.openstack.org/82477 | 11:17 |
| *** jasondotstar has quit IRC | 11:20 | |
| *** nosnos has quit IRC | 11:23 | |
| *** jhesketh has quit IRC | 11:27 | |
| *** chandankumar_ has joined #openstack-infra | 11:29 | |
| *** dkranz has quit IRC | 11:30 | |
| *** sergmelikyan has joined #openstack-infra | 11:32 | |
| *** gnuoy has joined #openstack-infra | 11:32 | |
| sergmelikyan | Guys, gate job is hang on gate-tempest-dsvm-neutron | 11:32 |
| sergmelikyan | https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron/10101/ | 11:32 |
| sergmelikyan | *hang = stuck | 11:33 |
| sergmelikyan | https://review.openstack.org/80259 | 11:33 |
| sergmelikyan | Any help? | 11:33 |
| *** salv-orlando has quit IRC | 11:37 | |
| *** _ruhe_ has joined #openstack-infra | 11:37 | |
| *** _ruhe_ has quit IRC | 11:38 | |
| *** _ruhe_ has joined #openstack-infra | 11:38 | |
| *** gmoro has quit IRC | 11:42 | |
| *** mburned_out is now known as mburned | 11:43 | |
| *** sandywalsh has joined #openstack-infra | 11:46 | |
| openstackgerrit | A change was merged to openstack/requirements: Updated oslo.messaging requirement https://review.openstack.org/80259 | 11:46 |
| *** weshay has joined #openstack-infra | 11:46 | |
| *** sandywalsh has quit IRC | 11:50 | |
| *** jgallard has quit IRC | 11:53 | |
| *** matsuhashi has joined #openstack-infra | 11:53 | |
| *** nosnos has joined #openstack-infra | 11:54 | |
| *** gmoro has joined #openstack-infra | 11:54 | |
| ihrachys | lifeless: ping re port 9000 on gate machines | 11:55 |
| *** ianw- has joined #openstack-infra | 12:01 | |
| *** ianw has quit IRC | 12:01 | |
| *** jamielenz has joined #openstack-infra | 12:01 | |
| *** ianw- is now known as ianw | 12:01 | |
| *** sandywalsh has joined #openstack-infra | 12:03 | |
| *** jamielennox|away has quit IRC | 12:03 | |
| *** rfolco has joined #openstack-infra | 12:05 | |
| *** yassine has joined #openstack-infra | 12:05 | |
| *** pdmars has joined #openstack-infra | 12:06 | |
| *** yolanda has quit IRC | 12:07 | |
| sdague | sergmelikyan: why did you think it was hung? | 12:09 |
| sergmelikyan | No new messages in console output for about 30m | 12:09 |
| sdague | it merged at 7:46 | 12:10 |
| sdague | EST | 12:10 |
| *** jepoy has quit IRC | 12:10 | |
| sdague | 11:46 UTC | 12:10 |
| *** pdmars has quit IRC | 12:10 | |
| *** salv-orlando has joined #openstack-infra | 12:11 | |
| *** chandankumar_ has quit IRC | 12:13 | |
| *** pdmars has joined #openstack-infra | 12:13 | |
| *** matsuhashi has quit IRC | 12:14 | |
| *** yassine has quit IRC | 12:15 | |
| *** matsuhas_ has joined #openstack-infra | 12:17 | |
| *** yassine has joined #openstack-infra | 12:17 | |
| ihrachys | sdague: do you have insight into software running on gate machines? | 12:17 |
| *** aysyd has joined #openstack-infra | 12:19 | |
| *** ArxCruz has joined #openstack-infra | 12:21 | |
| sdague | ihrachys: for what kinds of tests? | 12:21 |
| ihrachys | sdague: wsgi test that starts a service listening on :9000 | 12:22 |
| *** dkranz has joined #openstack-infra | 12:22 | |
| ihrachys | sdague: sometimes it suceeded, sometimes fail | 12:22 |
| ihrachys | sdague: http://logs.openstack.org/67/80967/1/check/gate-oslo-incubator-python27/b6dcfc1/testr_results.html.gz | 12:23 |
| cyeoh | sdague: hi - re: https://review.openstack.org/#/c/8247 - is that -2 a "don't want this sort of functionality" or just want it to interact it ok with xen overriding it? | 12:23 |
| sdague | using a common port like that seems really error prone. If the test really needs to bind to a port, it needs to look for a free port as part of the test | 12:24 |
| *** e0ne_ has quit IRC | 12:24 | |
| sdague | cyeoh: wrong url? | 12:24 |
| ihrachys | sdague: as per my understanding of _get_socket(), we get errno.EADDRINUSE from .listen() | 12:24 |
| cyeoh | sdague: oh sorry I meant https://review.openstack.org/#/c/82470 | 12:24 |
| ihrachys | sdague: indeed, though since the test is deprecated and removed in master, we search for an easy fix :) | 12:25 |
| *** salv-orlando_ has joined #openstack-infra | 12:25 | |
| sdague | ihrachys: sure, this is supposed to be unit tests though, runnable on anyone's machine, so you need to find a free port | 12:25 |
| sdague | not just die when a port is already used | 12:25 |
| ihrachys | sdague: for now, flaper87 changed 9000 to 9001, but it's not elegant enough :) | 12:25 |
| sdague | ihrachys: right, that's just going to fail somewhere else | 12:25 |
| sdague | cyeoh: more, I don't understand why you'd change that | 12:26 |
| ihrachys | sdague: it seems we see other failures with this port in other projects, see https://bugs.launchpad.net/openstack-ci/+bug/1285593 | 12:26 |
| uvirtbot | Launchpad bug 1285593 in openstack-ci "storyboard-webclient tests fail with "Port 9000 is already in use by another process"" [Medium,Triaged] | 12:26 |
| sdague | it's not meant to be a variable, it's meant to be a default | 12:26 |
| flaper87 | (note that the change to 9001 is just on stable/havana and that test doesn't exist in master anymore | 12:26 |
| flaper87 | ) | 12:26 |
| flaper87 | but yeah, I agree it's not the nicest fix | 12:27 |
| * flaper87 wasn't aware this was failing elsewhere too | 12:27 | |
| sdague | flaper87: sure, honestly, binding to ports shouldn't be happening in unit tests anyway. That's a bad pattern that's grown up | 12:27 |
| cyeoh | sdague: well its a default that gets used in a couple of places | 12:27 |
| sdague | cyeoh: right, and in all those places you can override already | 12:27 |
| ihrachys | flaper87: it doesn't mean your fix shouldn't go now though :) | 12:27 |
| flaper87 | sdague: agreed! | 12:27 |
| cyeoh | sdague: yep, it just means that it requires overriding in a couple of places, rather than one ... | 12:27 |
| ihrachys | flaper87: but there's smth to consider. like the way we assign ports. | 12:28 |
| cyeoh | sdague, but I can live without it | 12:28 |
| sdague | cyeoh: it's pretty much just FLAT_INTERFACE | 12:28 |
| ihrachys | sdague: I'm for mocking also, but I leave it up to flaper87 :_ | 12:28 |
| *** salv-orlando has quit IRC | 12:28 | |
| *** salv-orlando_ is now known as salv-orlando | 12:28 | |
| cyeoh | sdague: and VLAN_INTERFACE? | 12:28 |
| sdague | FLAT_INTERFACE and VLAN_INTERFACE shouldn't ever get used at the same time (you can set both if you care) | 12:28 |
| cyeoh | right, devstack does | 12:29 |
| ihrachys | sdague: we already mock .listen() in other test cases in this file, and the test checks thread pool, not socket interface, so... | 12:29 |
| cyeoh | sdague: anyway I'll just drop it - for my case I only care about FLAT_INTERFACE really | 12:29 |
| sdague | yeh | 12:30 |
| sdague | mostly because there is already a way to do it, I don't think we should make another way to do it, which then confuses people | 12:30 |
| sdague | especially as the existing variables match nova.conf variables | 12:31 |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Format json text Updated lp_projects definitions Fix climate.json reported error https://review.openstack.org/82477 | 12:31 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Sort ant builders settings for consistency https://review.openstack.org/82492 | 12:32 |
| *** jasondotstar has joined #openstack-infra | 12:32 | |
| *** adalbas has joined #openstack-infra | 12:37 | |
| *** _nadya__ has joined #openstack-infra | 12:39 | |
| *** jgallard has joined #openstack-infra | 12:41 | |
| anteaya | yjiang have you read http://ci.openstack.org/third_party.html | 12:44 |
| rustlebee | lifeless: it's up and running again (reviewstats) | 12:46 |
| *** yolanda has joined #openstack-infra | 12:47 | |
| *** yolanda has quit IRC | 12:48 | |
| *** yolanda has joined #openstack-infra | 12:49 | |
| *** salv-orlando has quit IRC | 12:49 | |
| *** vkozhukalov has quit IRC | 12:49 | |
| *** vkozhukalov1 has joined #openstack-infra | 12:50 | |
| *** jeckersb_gone is now known as jeckersb | 12:51 | |
| *** miqui has joined #openstack-infra | 12:51 | |
| *** yolanda has quit IRC | 12:52 | |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Format json text Updated lp_projects definitions https://review.openstack.org/82477 | 12:52 |
| *** mriedem has joined #openstack-infra | 12:54 | |
| *** e0ne has joined #openstack-infra | 12:55 | |
| *** thomasem has joined #openstack-infra | 12:55 | |
| openstackgerrit | Ivan Melnikov proposed a change to openstack-dev/hacking: Consider sys.real_prefix for _get_import_type https://review.openstack.org/82495 | 12:57 |
| *** thuc_ has joined #openstack-infra | 12:58 | |
| *** e0ne has quit IRC | 12:59 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Use OVERRIDE_ZUUL_BRANCH as ZUUL_BRANCH when defined https://review.openstack.org/82496 | 13:00 |
| *** alexpilotti has quit IRC | 13:00 | |
| *** dprince has joined #openstack-infra | 13:01 | |
| *** bknudson has joined #openstack-infra | 13:06 | |
| *** yassine has quit IRC | 13:08 | |
| *** mwagner_lap has quit IRC | 13:09 | |
| *** e0ne has joined #openstack-infra | 13:11 | |
| *** matsuhas_ has quit IRC | 13:12 | |
| *** matsuhashi has joined #openstack-infra | 13:13 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Use OVERRIDE_ZUUL_BRANCH as ZUUL_BRANCH when defined https://review.openstack.org/82496 | 13:17 |
| *** matsuhashi has quit IRC | 13:17 | |
| *** julim has joined #openstack-infra | 13:18 | |
| *** yassine has joined #openstack-infra | 13:21 | |
| sdague | afazekas: if you are trying to fix trove on stable/havana, that's not the right way to do it | 13:23 |
| *** jhesketh_ has quit IRC | 13:23 | |
| *** rustlebee is now known as russellb | 13:23 | |
| *** CaptTofu has quit IRC | 13:24 | |
| *** jamespage has quit IRC | 13:24 | |
| *** nosnos has quit IRC | 13:24 | |
| *** nosnos has joined #openstack-infra | 13:25 | |
| *** dcramer_ has joined #openstack-infra | 13:27 | |
| afazekas | sdague: What is purpose of the OVERRIDE_ZUUL_BRANCH variable ? | 13:29 |
| SergeyLukjanov | hey folks | 13:29 |
| sdague | afazekas: a bunch of internals in zuul, don't touch those | 13:29 |
| afazekas | sdague: where is the ZUUL_BRANCH defined normally ? | 13:29 |
| SergeyLukjanov | why we're not removing stable/havana from trove if it's not working? | 13:29 |
| sdague | afazekas: zuul creates those | 13:29 |
| sdague | SergeyLukjanov: exactly | 13:29 |
| *** nosnos has quit IRC | 13:29 | |
| fungi | afazekas: OVERRIDE_ZUUL_BRANCH is for when you want to force a specific branch to be used, rather than letting zuul specify it for you based on the branch the proposed commit appears in | 13:30 |
| afazekas | looks like the trove should not be enabled service when the ZUUL_BRANCH is stable/havana | 13:31 |
| sdague | afazekas: that's probably true | 13:31 |
| sdague | so go into devstack-vm-gate.sh and change it there where we do service selection | 13:31 |
| sdague | which is what I said in my first -1 in that review | 13:32 |
| afazekas | but http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-postgres-full-havana/545dc19/logs/devstacklog.txt.gz | 13:32 |
| *** aysyd has quit IRC | 13:32 | |
| afazekas | sdague: the service selection seams good in the devstack-vm-gate.sh | 13:33 |
| *** vhoward has joined #openstack-infra | 13:34 | |
| sdague | afazekas: the value is suspicious | 13:34 |
| sdague | devstack have a bad default on stable/havana ? | 13:34 |
| *** eharney has joined #openstack-infra | 13:35 | |
| afazekas | sdague: AFAIK the localrc overrides all default | 13:35 |
| *** andreaf has joined #openstack-infra | 13:36 | |
| sdague | sure, so figure out where that's getting set | 13:37 |
| afazekas | FIXME: True or False: If I made a change on devstack-gate it will use the 'master' as ZUUL_BRANCH even on the stable jobs | 13:38 |
| *** jgrimm has quit IRC | 13:39 | |
| mordred | morning all | 13:39 |
| mordred | afazekas: true. and hrm. | 13:41 |
| mordred | sdague: we don't really have a job that runs master d-g against stable/havana do we? | 13:42 |
| *** salv-orlando has joined #openstack-infra | 13:43 | |
| fungi | mordred: we do | 13:43 |
| *** yolanda has joined #openstack-infra | 13:43 | |
| *** njd002 has quit IRC | 13:43 | |
| *** jasondotstar has quit IRC | 13:43 | |
| fungi | mordred: just not a periodic job | 13:43 |
| fungi | we test proposed d-g changes against stable/havana | 13:44 |
| *** dkliban has joined #openstack-infra | 13:44 | |
| *** markmcclain has joined #openstack-infra | 13:44 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/storyboard: Remove legacy priority column from stories table https://review.openstack.org/82504 | 13:46 |
| *** wenlock has joined #openstack-infra | 13:47 | |
| mordred | fungi: when you awaken, could you take a peek at https://review.openstack.org/#/c/78249/ ? it's ready to go | 13:48 |
| mordred | oh - fungi is here | 13:48 |
| * mordred was in scrollback | 13:48 | |
| mordred | fungi: where? | 13:48 |
| mordred | fungi: I was just looking through layout.yaml and didn't see anything | 13:48 |
| mordred | oh - duh. in the check queue | 13:49 |
| *** thuc_ has quit IRC | 13:49 | |
| * mordred shuts up | 13:49 | |
| * mordred may not be awaken yet | 13:49 | |
| fungi | yup. non-voting for fear that people wouldn't be able to merge d-g changes since stable is broken so often | 13:49 |
| *** thuc has joined #openstack-infra | 13:49 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Comments controller https://review.openstack.org/81505 | 13:50 |
| *** jeckersb is now known as jeckersb_gone | 13:50 | |
| *** yolanda has quit IRC | 13:50 | |
| *** jepoy has joined #openstack-infra | 13:50 | |
| *** yolanda has joined #openstack-infra | 13:51 | |
| *** thuc_ has joined #openstack-infra | 13:51 | |
| fungi | mordred: does storyboard-db-manage load_projects rely on those projects being clonable via git? just making sure we're not introducing a new manage-projects race condition like we have with repo replication | 13:51 |
| fungi | mordred: and by ready to go you mean that storyboard has actual db migrations now (per clarkb's earlier -1)? | 13:53 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Adds second hiera data directory https://review.openstack.org/71739 | 13:53 |
| mordred | fungi: it does not | 13:53 |
| mordred | fungi: yes - | 13:54 |
| fungi | great | 13:54 |
| *** rcleere has quit IRC | 13:54 | |
| mordred | fungi: (it does not depend on git things, and yes, it has migrations - I have run that script by hand on the storyboard server and it works) | 13:54 |
| *** wchrisj has joined #openstack-infra | 13:54 | |
| *** thuc has quit IRC | 13:54 | |
| *** yamahata has joined #openstack-infra | 13:54 | |
| *** jeckersb_gone is now known as jeckersb | 13:54 | |
| mordred | fungi: also, have you logged in to storyboard yet? I'm making a patch that adds superusers, and it's easier if you've logged in already | 13:54 |
| mordred | fungi: ah - I see that you have | 13:55 |
| fungi | i think i havem but not sure if there have been any db wipes since. i'll log in again | 13:55 |
| fungi | done, though logging in gets me a completely empty page (i assume that's a known issue) | 13:55 |
| *** jamespage has joined #openstack-infra | 13:56 | |
| *** rossella_s has quit IRC | 13:56 | |
| *** vkozhukalov1 has left #openstack-infra | 13:57 | |
| anteaya | since we are in a soft freeze on puppet changes until https://review.openstack.org/#/c/51425/ merges, how far away are we from it merging? | 13:58 |
| *** nosnos has joined #openstack-infra | 13:58 | |
| *** _nadya__ has quit IRC | 13:59 | |
| *** matsuhashi has joined #openstack-infra | 13:59 | |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: provide more specific feedback to query authors https://review.openstack.org/81891 | 13:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Load storyboard projects from projects.yaml https://review.openstack.org/78249 | 14:00 |
| *** nosnos has quit IRC | 14:00 | |
| *** nosnos has joined #openstack-infra | 14:01 | |
| *** aysyd has joined #openstack-infra | 14:01 | |
| *** wenlock has quit IRC | 14:05 | |
| *** nosnos has quit IRC | 14:05 | |
| *** jepoy has quit IRC | 14:07 | |
| *** rlandy_ has joined #openstack-infra | 14:08 | |
| *** jepoy has joined #openstack-infra | 14:08 | |
| *** salv-orlando_ has joined #openstack-infra | 14:08 | |
| *** rossella_s has joined #openstack-infra | 14:08 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Load storyboard superusers from yaml file https://review.openstack.org/82507 | 14:08 |
| *** aysyd has quit IRC | 14:09 | |
| *** kashyap has quit IRC | 14:10 | |
| *** jasondotstar has joined #openstack-infra | 14:10 | |
| *** salv-orlando has quit IRC | 14:10 | |
| *** salv-orlando_ is now known as salv-orlando | 14:10 | |
| *** jp_at_hp has quit IRC | 14:10 | |
| *** CaptTofu has joined #openstack-infra | 14:10 | |
| *** nosnos has joined #openstack-infra | 14:11 | |
| *** rlandy has quit IRC | 14:11 | |
| *** thuc_ has quit IRC | 14:12 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Just make popularity-contest absent https://review.openstack.org/82509 | 14:12 |
| *** skraynev is now known as skraynev_afk | 14:12 | |
| *** jepoy has quit IRC | 14:12 | |
| *** thuc has joined #openstack-infra | 14:12 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/storyboard: Remove legacy priority column from stories table https://review.openstack.org/82504 | 14:14 |
| *** nosnos has quit IRC | 14:14 | |
| *** krtaylor has quit IRC | 14:15 | |
| *** adalbas has quit IRC | 14:16 | |
| *** matsuhashi has quit IRC | 14:16 | |
| *** jp_at_hp has joined #openstack-infra | 14:16 | |
| *** rwsu has joined #openstack-infra | 14:17 | |
| *** thuc has quit IRC | 14:17 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Make pip install from upstream better https://review.openstack.org/51425 | 14:17 |
| mordred | fungi: ^^ that fixes your comment - and thanks, you're right, we do not need to do that by hand anymore | 14:17 |
| fungi | mordred: lgtm | 14:19 |
| *** mfer has joined #openstack-infra | 14:19 | |
| openstackgerrit | mark mcclain proposed a change to openstack-infra/config: Create neutron-specs git repo https://review.openstack.org/82512 | 14:20 |
| mordred | nibalizer: at some point, not today, we should investigate whether we can merge https://review.openstack.org/51425 with the stankevitch-python module you used for puppetboard | 14:20 |
| *** jgrimm has joined #openstack-infra | 14:21 | |
| *** aysyd has joined #openstack-infra | 14:22 | |
| mordred | since that module provides pip, it seems like perhaps adding a few more flags to encompass installing latest pip and/or describing a thing for pip3 would be nice-ish | 14:22 |
| *** thuc has joined #openstack-infra | 14:23 | |
| *** yassine has quit IRC | 14:25 | |
| *** rcleere has joined #openstack-infra | 14:27 | |
| *** aysyd has quit IRC | 14:27 | |
| mordred | anteaya: I think as soon as either SergeyLukjanov, clarkb or jeblair get a chance to do a second pass on it, we can land it and then watch any carnage | 14:27 |
| * anteaya gets ready for carnage | 14:28 | |
| SergeyLukjanov | mordred, re pip? | 14:28 |
| anteaya | fortunately the new puppetdb.openstack.org gui should give us all the gory details | 14:28 |
| *** vkozhukalov1 has joined #openstack-infra | 14:29 | |
| jeblair | mordred: +2; are you going to try some manual applications of that? | 14:31 |
| *** adalbas has joined #openstack-infra | 14:31 | |
| fungi | i'd like to at least make sure it's been tried out selectively on *a* server whjich gets things from pip before we punch it through | 14:31 |
| fungi | er, what jeblair said | 14:31 |
| SergeyLukjanov | mordred, +2 from me too for 51425 | 14:31 |
| *** matsuhashi has joined #openstack-infra | 14:32 | |
| fungi | if we know it completes on one of our servers, then we can have some confidence that whatever carnage there is will hopefully not be widespread | 14:32 |
| openstackgerrit | A change was merged to openstack-infra/config: Just make popularity-contest absent https://review.openstack.org/82509 | 14:32 |
| fungi | we'll probably also want to try manual image updates of each basic puppeted slave class to make sure there's nothing obviously wrong (or to be able to catch it early if there is, since we're not going to see that via puppetboard) | 14:33 |
| *** DuncanT- has quit IRC | 14:36 | |
| *** yassine has joined #openstack-infra | 14:37 | |
| *** marktraceur has quit IRC | 14:38 | |
| *** thuc_ has joined #openstack-infra | 14:39 | |
| SergeyLukjanov | folks, let's make noop real! https://review.openstack.org/#/c/82162/ any objections? | 14:39 |
| *** saju_m has quit IRC | 14:39 | |
| *** homeless has joined #openstack-infra | 14:40 | |
| *** marktraceur has joined #openstack-infra | 14:41 | |
| jeblair | SergeyLukjanov: no objections! | 14:41 |
| * SergeyLukjanov breaking the world! | 14:41 | |
| SergeyLukjanov | ooops, I mean making noop real :) | 14:41 |
| *** maxbit has joined #openstack-infra | 14:41 | |
| fungi | i take it the functionality for that is already merged and zuul's been restarted? | 14:42 |
| jeblair | fungi: yes i did that yesterday | 14:42 |
| *** thuc has quit IRC | 14:42 | |
| fungi | awesome. i didn't pay close enough attention to scrollback | 14:43 |
| jeblair | fungi: i also restarted the mergers, so they should have their init scripts and auto-re-registration bugs fixed | 14:43 |
| *** hub_cap_ has quit IRC | 14:43 | |
| SergeyLukjanov | jeblair, cool | 14:43 |
| *** thuc_ has quit IRC | 14:43 | |
| *** david-lyle has joined #openstack-infra | 14:43 | |
| *** mwagner_lap has joined #openstack-infra | 14:43 | |
| *** krtaylor has joined #openstack-infra | 14:43 | |
| fungi | jeblair: excellent, so next zuul restart we just restart the daemon on zuul.o.o and everything should simply work, right? | 14:44 |
| jeblair | fungi: yep | 14:44 |
| jeblair | while we're breaking things, i also approved the revoke more sudo patch | 14:44 |
| fungi | wonderful | 14:45 |
| *** zns has joined #openstack-infra | 14:45 | |
| pabelanger | morning! Is there a list of jenkins plugins that nodepool needs / uses? I couldn't see anything obvious | 14:45 |
| fungi | pabelanger: the zeromq event publisher | 14:45 |
| *** marktraceur has quit IRC | 14:46 | |
| pabelanger | fungi, Yup, have that one. Looks to be the only one right? | 14:46 |
| fungi | pabelanger: as far as i know, yes | 14:46 |
| pabelanger | okay cool | 14:46 |
| *** marktraceur has joined #openstack-infra | 14:47 | |
| *** dims has quit IRC | 14:47 | |
| anteaya | SergeyLukjanov: could you state your reasons for your -1 on https://review.openstack.org/#/c/82512/ on the patch? | 14:47 |
| anteaya | so that markmcclain knows what he needs to chage | 14:48 |
| anteaya | change | 14:48 |
| *** thedodd has joined #openstack-infra | 14:48 | |
| fungi | pabelanger: yep, that's all we mentioned in https://git.openstack.org/cgit/openstack-infra/nodepool/tree/doc/source/installation.rst so i think you should be set | 14:48 |
| fungi | anteaya: he did in his comment immediately prior to the -1 vote | 14:48 |
| fungi | anteaya: looks like he simply forgot to set the vote on the previous comment and was adding it with a follow-up empty comment | 14:49 |
| SergeyLukjanov | anteaya, we've decided to set topic for new-projects-creation CRs 'new-project' | 14:49 |
| anteaya | SergeyLukjanov: yeah, didn't know that was the cause of the -1 | 14:49 |
| anteaya | okay thanks | 14:49 |
| afazekas | jeblair: https://review.openstack.org/#/c/82496/ looks like it changed all stable jobs to SUCCESS on the devstack-gate, what is the normal way to fix it ? | 14:50 |
| SergeyLukjanov | anteaya, it could be 0, but it's harder to identify later which CRs has been reviewed... | 14:50 |
| *** thuc has joined #openstack-infra | 14:50 | |
| fungi | also, i guess the nova experiment which is only just underway to try doing blueprints in a git repo is creating an avalanche of other programs wanting to try the same experiment without first waiting to see if it's a success? | 14:50 |
| jeblair | afazekas: did you look into sdague's suggestion? | 14:50 |
| *** atiwari has joined #openstack-infra | 14:51 | |
| pabelanger | fungi, great thanks. Wanted to check since I'm hitting an exception when nodepool tries to configure jenkins for a new build executor, just making sure no extra plugin was needed | 14:51 |
| afazekas | jeblair: the service adds are correct | 14:51 |
| openstackgerrit | mark mcclain proposed a change to openstack-infra/config: Create neutron-specs git repo https://review.openstack.org/82512 | 14:52 |
| afazekas | jeblair: the problem is it behaves like it were a master job even when it is stable/havana | 14:52 |
| anteaya | SergeyLukjanov: sure, that makes sense | 14:52 |
| mordred | jeblair: yes. I will now run it manually on a server or two before doing the +A | 14:52 |
| jeblair | afazekas: is this only on periodic jobs? (eg, jobs for changes on the branch are correct, but periodic jobs are incorrect?) | 14:53 |
| mordred | jeblair: how about graphite? that's heavy-python but also if it borks for a second because we missed something won't kill the world | 14:53 |
| afazekas | jeblair: it is a not voting job on the devstack-gate | 14:53 |
| jeblair | mordred: wfm | 14:53 |
| afazekas | jeblair: You can check the jenkins response on my change and on another one for example: https://review.openstack.org/#/c/81216/ | 14:54 |
| jeblair | afazekas: what is not a voting job? i'm trying to understand in what conditions it does not work as expected | 14:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove Oneiric conditional logic https://review.openstack.org/82335 | 14:54 |
| *** wchrisj has quit IRC | 14:54 | |
| afazekas | check-tempest-dsvm-postgres-full-havana FAILURE in 8m 27s (non-voting) | 14:55 |
| afazekas | vs check-tempest-dsvm-postgres-full-havana SUCCESS in 38m 04s (non-voting) | 14:55 |
| pabelanger | fungi, is it possible I need: https://wiki.jenkins-ci.org/display/JENKINS/SSH+Slaves+plugin ? | 14:56 |
| pabelanger | I _think_ I do | 14:56 |
| jeblair | afazekas: right, so it's the branch override jobs (which are mostly periodic, but also some of the cross-branch testing like the one you're pointing out) | 14:57 |
| *** salv-orlando_ has joined #openstack-infra | 14:57 | |
| jeblair | afazekas: so your change may be correct, but i didn't understand what you were fixing from the commit message | 14:57 |
| jeblair | afazekas: would you mind updating that a bit, and also adding a test for it in test-functions.sh? | 14:57 |
| *** salv-orlando has quit IRC | 14:58 | |
| *** salv-orlando_ is now known as salv-orlando | 14:58 | |
| nibalizer | mordred: definitely!! | 14:58 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Authorization models added https://review.openstack.org/82101 | 14:58 |
| jeblair | afazekas: oh, hrm, we don't have any setup_localrc tests... in that case, it may not be easy to test | 14:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Revoke sudo from almost all jobs https://review.openstack.org/82170 | 14:59 |
| jeblair | afazekas: at any rate, maybe you could at least update the commit message with more of an explanation? | 14:59 |
| jeblair | pabelanger: seems likely | 14:59 |
| *** dims has joined #openstack-infra | 15:00 | |
| fungi | pabelanger: i think so, yes | 15:00 |
| *** pcrews has joined #openstack-infra | 15:01 | |
| sdague | jeblair: do you have the log size / day current growth rate? I'd like to understand how bad we are at the moment, so how aggressively we need to dump things | 15:01 |
| fungi | sdague: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 15:02 |
| *** vkozhukalov1 has left #openstack-infra | 15:03 | |
| pabelanger | fungi, jeblair: okay, testing the theory now. | 15:03 |
| sdague | fungi: are we deleting old logs regularly as well? | 15:03 |
| fungi | sdague: yes, any which are >180 days | 15:03 |
| fungi | sdague: and we also currently run a weekly pass to compress anything compressable which was not already | 15:04 |
| sdague | ok, so the raw total is a little less useful than delta / week (once everything is accounted for, compression + deletes) | 15:04 |
| jeblair | clarkb: when you have a second, the thread "[Openstack-i18n] Horizon translations - and update of strings" has a question for you | 15:05 |
| sdague | is there a way to get that? | 15:05 |
| *** thuc_ has joined #openstack-infra | 15:05 | |
| jeblair | sdague: i would just zoom in and subtract | 15:05 |
| *** dhellmann_ is now known as dhellmann | 15:05 | |
| jeblair | sdague: (if you click-drag on cacti graphs they zoom) | 15:05 |
| *** mgagne has joined #openstack-infra | 15:05 | |
| jeblair | sdague: (zoom in to get one day and then read "current" value) | 15:06 |
| afazekas | jeblair: ok | 15:06 |
| yolanda | hi jeblair, do you have a second? i wanted some feedback from that bug: https://bugs.launchpad.net/openstack-ci/+bug/1274731 | 15:07 |
| uvirtbot | Launchpad bug 1274731 in openstack-ci "non overalapping change to requirements repo broke gate" [Medium,Confirmed] | 15:07 |
| *** salv-orlando has quit IRC | 15:07 | |
| *** thuc has quit IRC | 15:08 | |
| *** vkozhukalov has joined #openstack-infra | 15:08 | |
| *** dangers_` is now known as dangers | 15:08 | |
| *** salv-orlando has joined #openstack-infra | 15:09 | |
| *** todd_dsm has quit IRC | 15:11 | |
| *** SumitNaiksatam has quit IRC | 15:11 | |
| *** vkozhukalov has quit IRC | 15:12 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Use OVERRIDE_ZUUL_BRANCH as ZUUL_BRANCH when defined https://review.openstack.org/82496 | 15:13 |
| *** zhiyan is now known as zhiyan_ | 15:13 | |
| jeblair | yolanda: i think the idea was that if someone changed a requirement (instead of adding or removing one), then the script should try to make sure the old and new values overlapped | 15:14 |
| jeblair | yolanda: i don't know for certain if that's possible or how to do that, but it's worth looking into the api | 15:15 |
| yolanda | jeblair, i started looking at that scripts: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_bilder/config/requirements.yaml | 15:15 |
| yolanda | so is ok to look at what this job is doing and dig more on it? | 15:16 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Make pip install from upstream better https://review.openstack.org/51425 | 15:16 |
| *** marktraceur has quit IRC | 15:16 | |
| *** wenlock has joined #openstack-infra | 15:16 | |
| jeblair | yolanda: we were thinking about this job: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml#n230 | 15:17 |
| jeblair | gate-{name}-requirements | 15:17 |
| *** mwagner_lap has quit IRC | 15:17 | |
| *** jswarren has joined #openstack-infra | 15:18 | |
| *** marktraceur has joined #openstack-infra | 15:18 | |
| yolanda | ok, that script /usr/local/jenkins/slave_scripts/project-requirements-change.py | 15:18 |
| *** salv-orlando has quit IRC | 15:19 | |
| *** beekneemech is now known as bnemec | 15:20 | |
| *** prad has joined #openstack-infra | 15:20 | |
| *** prad has left #openstack-infra | 15:20 | |
| *** prad has joined #openstack-infra | 15:21 | |
| *** salv-orlando has joined #openstack-infra | 15:21 | |
| jeblair | yolanda: right | 15:21 |
| yolanda | ok, let me take a look at it | 15:21 |
| *** AJaeger has joined #openstack-infra | 15:22 | |
| AJaeger | Hi infra team, I would appreciate a review of the following patch to better handle the translation jobs: https://review.openstack.org/#/c/82384/ | 15:23 |
| *** matsuhashi has quit IRC | 15:25 | |
| jeblair | fungi, clarkb, mordred: glance -d --os-image-url=https://iad.images.api.rackspacecloud.com --os-image-api-version=2 image-list | 15:25 |
| *** thuc_ has quit IRC | 15:25 | |
| jeblair | fungi, clarkb, mordred: that works in rax ^ | 15:25 |
| mordred | jeblair: w00t | 15:25 |
| fungi | YES! | 15:26 |
| *** nati_ueno has joined #openstack-infra | 15:26 | |
| jeblair | fungi, clarkb, mordred: the first arg is to get around a bug in rackspace's non-keystone where it spits out the wrong endpoint url... or something like that | 15:26 |
| *** thuc has joined #openstack-infra | 15:26 | |
| * mordred cries | 15:26 | |
| jeblair | fungi, clarkb, mordred: the second is that for some reason, we have to force the v2 endpoint... i don't know if that's because of the first bug or what. | 15:26 |
| mordred | jeblair: I'm assuming we can probably set OS_IMAGE_URL ish | 15:26 |
| jeblair | mordred: yeah | 15:26 |
| mordred | jeblair: so now I suppose we've got some motiviation to move off of hpcloud 1.0 completely | 15:27 |
| *** mwagner_lap has joined #openstack-infra | 15:27 | |
| *** todd_dsm has joined #openstack-infra | 15:27 | |
| jeblair | SergeyLukjanov: oh no, noop failed (i think it needs to be updated because a new gate-noop job slipped in) | 15:28 |
| jeblair | mordred: i have a patch series up for that | 15:28 |
| *** matsuhas_ has joined #openstack-infra | 15:28 | |
| jeblair | fungi: yeah, so about -specs repos... | 15:28 |
| jeblair | i'm pretty sure the mailing list posts were about "doing an experiment" | 15:28 |
| fungi | i fear we're facing an explosion of -specs repos which will get abandoned soon | 15:29 |
| jeblair | not "changing the way all the projects handle enhancement proposals in an ad-hoc and inconsistent manner" | 15:29 |
| fungi | and then we get to cart them around indefinitely | 15:29 |
| *** packet has joined #openstack-infra | 15:29 | |
| pabelanger | fungi, do you have Global credentials setup for SSH statically, or is nodepool adding them each time a new node comes online. If I understand correctly, nodepool expect the Global credentials to be already setup? | 15:30 |
| jeblair | i'd personally rather avoid that... | 15:30 |
| *** thuc has quit IRC | 15:31 | |
| jeblair | we've already seen that nova and qa specs repos don't plan on following the same standards | 15:31 |
| fungi | pabelanger: nodepool's config had the uuid of the jenkins ssh credential from the master's config.xml | 15:31 |
| jeblair | so i'm really concerned about the experience for contributors | 15:32 |
| mordred | jeblair: the patch does not work - and on further inspection, this is not the same as the one we're running successfully inside of HP | 15:32 |
| jeblair | mordred, ttx: thoughts on specs repos? | 15:32 |
| jeblair | mordred: the pip patch? | 15:32 |
| mordred | jeblair: so, consider soft-freeze lifted until futher notice - this will require work | 15:32 |
| mordred | jeblair: yes | 15:32 |
| * mordred grumbles things about ruby | 15:32 | |
| jeblair | mordred: can you wip that? | 15:32 |
| mordred | jeblair: yes | 15:32 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Make pip install from upstream better https://review.openstack.org/51425 | 15:33 |
| mordred | pushing up latest edits first | 15:33 |
| *** matsuhas_ has quit IRC | 15:33 | |
| *** jpeeler has joined #openstack-infra | 15:33 | |
| mordred | k. | 15:33 |
| mordred | jeblair: so- specs repos - I'd like to stop creating any more | 15:33 |
| mordred | jeblair: I'm a bit annoyed that we have 2 divergent options already | 15:34 |
| mordred | jeblair: why are they not folloing the same standards? | 15:34 |
| jeblair | mordred: because they need "freedom to experiment" | 15:34 |
| mordred | I mean, of all the projects, shouldn't nova and qa know the value of collaboration? | 15:34 |
| mordred | jeblair: bah. they need freedom to be lazy and not talk | 15:35 |
| mordred | that's bullshit | 15:35 |
| markmcclain | mordred: so we're aligning with Nova here… what do you suggest? | 15:36 |
| mordred | sdague: ^^ I believe I'm calling you out ina rude manner | 15:36 |
| pabelanger | fungi, Great thanks. I think that was the last piece I was needing: http://jenkins.kickstand-project.org/computer/bare-precise-hpcloud-region-b-1/ | 15:36 |
| mordred | markmcclain: I suggest waiting until we have more agreement on what the heck is going on, tbh | 15:36 |
| ttx | jeblair: yes, i see it as an experiment | 15:36 |
| fungi | since storyboard is aimed at solving the blueprint situation, it would be nice to at least have the current experiment limited to figuring out what new blueprint workflow options storyboard should be considering | 15:36 |
| mordred | this is quickly escalating from "nova is experimenting with something new" | 15:36 |
| ttx | frankly, it looks all shiny but I want to see how it works in the wild | 15:37 |
| mordred | to "ZOMG PONIES!!! quick, everyone adopt the not-yet-finished-experiment" | 15:37 |
| mordred | mind you - I _LIKE_ the idea | 15:37 |
| ttx | because in the past, all attempts to add more process weas met with skepticism | 15:37 |
| fungi | ttx: well, we've now got several projects wanting to ditch lp blueprints in favor of whatever nova's experiment is | 15:37 |
| ttx | fungi: they don't ditch Lp | 15:37 |
| fungi | which feels premature to me | 15:37 |
| sdague | mordred: because I actually specifically want to try somethings differently to figure out what works and what doesn't | 15:38 |
| ttx | fungi: they just use an out-of-LP approval process | 15:38 |
| mordred | sdague: ok. you see what's happening though | 15:38 |
| markmcclain | fungi: the problem with lp is the process for commenting on, revising and versioning a blueprint spec is non-existent | 15:38 |
| sdague | because I think coming up with standardized process right now, with no real experience, just means we'll pick process before knowing what works | 15:38 |
| *** wchrisj has joined #openstack-infra | 15:38 | |
| mordred | sdague: how about we keep the experiment to one project | 15:39 |
| mordred | and try different things | 15:39 |
| *** gyee has joined #openstack-infra | 15:39 | |
| jeblair | mordred: indeed, that's usually how we experiment -- we let someone trailblaze, try a couple different paths, and then see what worked | 15:39 |
| *** marktraceur has quit IRC | 15:39 | |
| fungi | markmcclain: i understand it's broken, just worried that if nova's "approving blueprints through code review" experiment is aborted, we end up with neutron, qa and possibly a dozen other programs who have decided to follow suite which now have dead specs repos hanging around | 15:39 |
| *** CaptTofu has quit IRC | 15:39 | |
| sdague | mordred: because the current process is so broken on qa front, we basically told people to stop creating blueprints entirely | 15:39 |
| ttx | In the past, the idea would have been rejected fast, but it sounds like we've reached a critical mass of processs wonks / maturity | 15:39 |
| sdague | that's our current approach | 15:39 |
| jeblair | it's much easier to get people on board with the final outcome than having 20 experiments, all of which come up with something different, and all of which people think are successful | 15:40 |
| mordred | sdague: right. we did the same thing here | 15:40 |
| sdague | and it sucks | 15:40 |
| *** CaptTofu has joined #openstack-infra | 15:40 | |
| mordred | sdague: yes. I'm talking about two different groups expermenting independently | 15:40 |
| ttx | fungi: so the issue is... we don't know how to delete repositories | 15:40 |
| mordred | I think the idea of blueprint dev in code review is awesome | 15:40 |
| fungi | and what is the end-game we envision for these specs repos when storyboard blueprints are up and running? | 15:40 |
| fungi | ttx: the issue is that deleting repositories means erasing history | 15:40 |
| *** marktraceur has joined #openstack-infra | 15:41 | |
| ttx | fungi: as I commented on the ML, storyboard will be happy having -specs tasks | 15:41 |
| sdague | but a blueprint for nova looks radically different than a blueprint in qa program, by definition | 15:41 |
| fungi | even if we get the technical hurdles down | 15:41 |
| mordred | sdague: I disagre with teh "by definition" | 15:41 |
| mordred | that is an assertion you are making, I do not think that it is necessarily true | 15:41 |
| sdague | so if you really want we can punt on this entirely with qa, and just go completely out of process and use google docs are something | 15:41 |
| fungi | ttx: ahh, i'm back to missing context obviously--been too busy to read e-mail again. i'll just lurk this conversation | 15:41 |
| sdague | which is basically waht people are doing now | 15:41 |
| mordred | sdague: no - I'm asking you to help us try to keep the project from balkanizing | 15:42 |
| ttx | I'd prefer the experimentation to be limited before it is successful for at least one project | 15:42 |
| mordred | which is a hard enough task without introducing a billion divergent mechanisms for blueprint development | 15:42 |
| sdague | mordred: we already have a billion divergent mechanisms for blueprint development | 15:43 |
| mordred | sdague: and you don't consider tah a bug? | 15:43 |
| markmcclain | the only problem of letting one project experiment is the others who aren't chosen keep suffering from the chaos we have now | 15:43 |
| mordred | why would we CODIFY that inside of actual openstack systems? | 15:43 |
| ttx | markmcclain: they admittedly had the idea first :) | 15:43 |
| sdague | I do, and using a common tool, that lets me purge random etherpad, wiki, google docs, regardless in not being a perfect clone of nova, is a real step forward | 15:43 |
| markmcclain | ttx: agreed but it is a good one | 15:44 |
| *** andreaf2 has joined #openstack-infra | 15:44 | |
| mordred | so - what is it that is divergent in the nova and qa approaches? | 15:44 |
| sdague | templates | 15:44 |
| jeblair | mordred: directory structure, document types... | 15:44 |
| sdague | the document types aren't divergent | 15:45 |
| *** andreaf has quit IRC | 15:45 | |
| sdague | directory structure is | 15:45 |
| *** CaptTofu has quit IRC | 15:45 | |
| jeblair | i suggested that they should be sphinx docs, and we can generate actual sphinx docs from them | 15:45 |
| jeblair | but i could not get them to agree on that | 15:45 |
| mordred | jeblair: wow, why not? that seems obvious to me | 15:45 |
| sdague | they are both rst docs | 15:45 |
| mordred | ok - looking at nova, I'd liek to make some suggestions | 15:46 |
| mordred | "approved" is duplicative | 15:46 |
| mordred | the spec is approved when it lands in the repo | 15:46 |
| mordred | "implemeneted" is duplicative - status of a blueprint is tracked currently in blueprints | 15:46 |
| jeblair | mordred: i was hoping we could plug them into the project testing interface even, so we wouldn't have to build new frameworks | 15:46 |
| mordred | jeblair: that would be neat | 15:46 |
| mordred | it's almost like we alraedy have tools for that | 15:47 |
| sdague | we don't have tools for the style checker I hacked out | 15:47 |
| mordred | and why would there be a juno dir and not a branch? | 15:47 |
| *** dkliban_ has joined #openstack-infra | 15:48 | |
| mordred | sdague: I'm sure you can run a style checker on things that fit into the current publication toolchain | 15:48 |
| mordred | I'm sorry - the idea of specs in git repos is great - but this is a mess | 15:48 |
| *** dkliban has quit IRC | 15:49 | |
| jeblair | i don't see how we can say it's an experiment, and yet have all the projects doing it | 15:49 |
| jeblair | we need to either keep it small so that we actually know what we're experimenting with | 15:49 |
| jeblair | or it needs to grow up quick so we can scale it out to all the projects | 15:49 |
| mordred | ++ | 15:50 |
| ttx | My main issue with the idea is that the whole thing fails when what you want to discuss is cross-project | 15:51 |
| sdague | so I get that all the ducks are in a row at this point | 15:51 |
| ttx | would you then have parallel -specs change in various repos ? | 15:51 |
| sdague | but I don't think people realize how truly terrible the current situation isu | 15:51 |
| jeblair | sdague: i do | 15:51 |
| ttx | but I'm happy to trat cross-project as an exception procedure | 15:51 |
| sdague | and that even divergent specs repos is a giant step forward | 15:52 |
| sdague | in project commonality | 15:52 |
| sdague | jeblair: so that's what I'm trying to reconcile, because I don't understand how this can be seen as a bad thing given the current state | 15:53 |
| *** BadCub01 has joined #openstack-infra | 15:53 | |
| sdague | ttx: there is 0 ability to do cross project blueprints today, so this doesn't make that any worse | 15:54 |
| *** thuc has joined #openstack-infra | 15:55 | |
| ttx | sdague: agreed. I was just discussing using that system as the long-term solution | 15:55 |
| *** marktraceur has quit IRC | 15:55 | |
| *** marktraceur has joined #openstack-infra | 15:55 | |
| ttx | sdague: basically I agree it's a temporary improvement. the problem the -infra team has is in the "temporary" aspect | 15:55 |
| sdague | it's an incremental improvement | 15:56 |
| ttx | since creating repositories is not really "temporary" | 15:56 |
| sdague | the only way we get anywhere is incrementally | 15:56 |
| ttx | sdague: wouldn't cause a problem if we had more agility around repositories creation/deletion/renaming | 15:57 |
| mordred | sdague: right - the problem isn't that you're working on it, nor that the attempts aren't perfect yet | 15:57 |
| jeblair | sdague: the openstack project, as a development organization, scales by sharing process; every -spec repo that's different is more confusion and more work | 15:57 |
| sdague | ttx: so can we then solve that problem instead? | 15:57 |
| jeblair | ttx: i disagree about deleting -- even if we could, i don't think we would ever want to delete anything | 15:57 |
| mordred | sdague: the problem we're worried about is what jeblair just said | 15:57 |
| sdague | jeblair: and we'll get there, but we aren't sharing process today | 15:57 |
| sdague | we really aren't | 15:57 |
| sdague | maybe it's not visible so clearly until we get here | 15:58 |
| mordred | sdague: right. but encoding divergent processes here is what troubles us | 15:58 |
| sdague | but go try and propose and get approved a blueprint across heat, nova, neutron, qa | 15:58 |
| jeblair | you can even see it in our natural tendency -- two projects adopted this thing, so other projects start doing it, because so many people know the value of that consistency | 15:58 |
| sdague | mordred: it's not encoding it, it's let us evolve it. And trusting that people will actually try to diverge over time | 15:59 |
| jeblair | (we'll have stackforge repos wanting them soon to, no doubt) | 15:59 |
| mordred | yeah - I think what we're probably saying is - perhaps experimentation is not as important as just rolling is out | 15:59 |
| *** thuc_ has joined #openstack-infra | 15:59 | |
| mordred | or, what i'm probably saying - I don't want to speak for others | 15:59 |
| sdague | sorryr - And trusting that people will actually try to *converge( over time | 15:59 |
| *** blamar has joined #openstack-infra | 15:59 | |
| *** blamar has quit IRC | 15:59 | |
| jeblair | sdague: and there i was thinking i had won you over. :) | 15:59 |
| mordred | :) | 16:00 |
| *** blamar has joined #openstack-infra | 16:00 | |
| mordred | sdague: so, why don't we all just put our heads together, cook up one thing that's probably less bad than current, and then let the tidal wave of people who want to start using it start using it | 16:00 |
| mordred | sdague: and if we discover places it sucks, we can fix taht everywhere at once? | 16:00 |
| *** michchap has quit IRC | 16:01 | |
| sdague | because then we end up in oslo gridlock | 16:01 |
| jeblair | i really do want storyboard to handle this; i don't think this should be necessary in the long term, so if we make a lot of them, we really are going to need openstack-attic/ | 16:01 |
| mordred | oslo gridlock is caused by allowing divergence | 16:01 |
| sdague | mordred: nope | 16:01 |
| mordred | yes | 16:01 |
| sdague | no, it really isn't | 16:01 |
| mordred | :) | 16:01 |
| *** thuc has quit IRC | 16:01 | |
| * jeblair registers openstack-attic/ on github | 16:01 | |
| *** dkliban_ is now known as dkliban | 16:01 | |
| *** jnoller has joined #openstack-infra | 16:01 | |
| sdague | go look at the current state of oslo sync trees in projects | 16:02 |
| mordred | right | 16:02 |
| mordred | that's what I mean | 16:02 |
| sdague | there is massive divergence | 16:02 |
| mordred | because we allowed it because "the apis might not be solild yet" | 16:02 |
| sdague | because process requires getting everyone to agree to something | 16:02 |
| mordred | so we started copying stuff | 16:02 |
| mordred | except at different cadences | 16:02 |
| mordred | as opposed to eating it and using libraries from the start | 16:02 |
| mordred | whcih has a known mechanism for sharing | 16:02 |
| mordred | oslo-incubator hell is EXACTLY what I'm worried about with divergent spec repo approaches in the name of "freedom to experiment" | 16:03 |
| *** reed has joined #openstack-infra | 16:04 | |
| *** afazekas_ has joined #openstack-infra | 16:04 | |
| sdague | so we can either have a bike shedding up front about if you encode releases / states in directory names. Or we can try it a couple different ways. Then say "oh, that sucked when we had to do all those git moves" and give up on it | 16:04 |
| mordred | if we make one spec repo format and a set of tools that work with it, then iterating on that involves iterating on the one set of tools, and we can keep people in somewhat of a similar place | 16:04 |
| mordred | I would prefer the bikeshedding up front | 16:04 |
| mordred | it's called "design" | 16:04 |
| mordred | and it's actually not bikeshedding - it's important | 16:05 |
| sdague | but design without experience ends up going around in circles. I feel like we'd have a really useful design session on common format after a release and experiences | 16:05 |
| fungi | also, we won't be just be trying it a couple different ways if every program decides they want to jump on board while the one or two which are piloting the workflow haven't arrived at any recommendations yet | 16:05 |
| mordred | right | 16:06 |
| mordred | that's the problem | 16:06 |
| sdague | we'll be doing what we did with hacking.py | 16:06 |
| mordred | it's not going to be a couple | 16:06 |
| sdague | where we copy it around between like minded projects for a while | 16:06 |
| *** afazekas has quit IRC | 16:06 | |
| sdague | then figure out what's working common and extract | 16:06 |
| mordred | that's not what we do | 16:06 |
| sdague | anyway, that was my thinking | 16:06 |
| mordred | did | 16:06 |
| mordred | with hacking | 16:06 |
| sdague | mordred: that's exactly what we did with hacking | 16:06 |
| mordred | the instant I saw a copy hacking.py thing go into oslo | 16:06 |
| mordred | I made a library | 16:06 |
| sdague | nope | 16:06 |
| sdague | we were copying it around in tempest for a whole release prior to that | 16:07 |
| sdague | and back and forth fixes to nova | 16:07 |
| mordred | I just didn't see that :) | 16:07 |
| mordred | but this isn't code | 16:07 |
| mordred | this is process that involves tooling and repos | 16:07 |
| mordred | "just copy the file" doesn't have a good analogue here | 16:07 |
| *** sahumada has quit IRC | 16:08 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Replace gate-noop with noop https://review.openstack.org/82162 | 16:08 |
| jeblair | 16:07 < openstackgerrit> Matthew Treinish proposed a change to openstack/qa-specs: Add sphinx support to qa-specs repo https://review.openstack.org/82531 | 16:09 |
| jeblair | relevant ^ | 16:09 |
| mordred | yup. ESPECIALLY since it is a patch that's not in line with how our sphinx tooling works | 16:09 |
| jeblair | oh! i hadn't actually looked at it | 16:10 |
| jeblair | mtreinish: ^ | 16:10 |
| mordred | mtreinish: we know how to build sphinx things - and publish them | 16:10 |
| jeblair | yeah, that would need a tox.ini | 16:10 |
| *** asselin has joined #openstack-infra | 16:14 | |
| jeblair | so i'm going to reply to the -dev list and ask other projects to hold off at least a little bit so we can try to get some really basic stuff sorted out first | 16:14 |
| sdague | jeblair: it's probably worth getting deep into the threads on the nova specs repo than as well | 16:15 |
| *** ominakov has quit IRC | 16:15 | |
| jeblair | i don't think we have consensus on whether or not we should keep to the current repos for a whole cycle, or open the floodgates, but i do feel pretty strongly that if we're going to open the floodgates, we need to have a few basic things worked out | 16:16 |
| sdague | because the problem is this current need by projects is completely unaddressed in anything we have currently, so it doesn't surprise me that people are jumping at anything that might make their life better | 16:16 |
| sdague | jeblair: ok, so can you nail down what you think those things should be in a thread | 16:16 |
| reed | hello folks | 16:16 |
| sdague | so we can at least try to agree on the things that we think have to be identical between projects, and the places where they can adapt to local needs | 16:17 |
| markmcclain | sorry resuming the conversation, but the other important item is I'd prefer not the spend a long time iterating on this as folks are already writing specs for Juno | 16:17 |
| mordred | sdague: that would be great. I think that we can probably get a common scaffolding in place withouth years of bikeshedding | 16:17 |
| nibalizer | mordred: left you a short essay on the 'make pip better' change | 16:17 |
| mordred | that still allows for exploration per-project | 16:18 |
| *** michchap has joined #openstack-infra | 16:18 | |
| *** mihgen has quit IRC | 16:18 | |
| *** sdake_ has joined #openstack-infra | 16:18 | |
| *** sdake_ has quit IRC | 16:18 | |
| *** sdake_ has joined #openstack-infra | 16:18 | |
| jeblair | sdague: yeah. i think they are a) sphinx yes/no? b) testing/publishing interface c) what information is encoded (don't duplicate blueprint status) d) workflow process (how it interacts with blueprints) | 16:18 |
| mordred | I agree with that list | 16:19 |
| sdague | jeblair: sure, so get that out in a thread, especially if it contradicts what's in nova-specs today | 16:19 |
| mordred | tbh - spec template format divergence sounds like a fine place to play | 16:19 |
| sdague | I honestly think directory structure is probably going to vary as well | 16:20 |
| sdague | as keeping track of things with 150 active blueprints is different than with 20 | 16:20 |
| mordred | sure. | 16:20 |
| anteaya | so should we ask markmcclain to WIP https://review.openstack.org/#/c/82512/ for the moment? | 16:20 |
| fungi | maybe approach directories vs branches specifically as an item to be ironed out | 16:20 |
| sdague | which is why the qa-specs repo didn't put release in the directory structure, and had an implemented, largely as an attic | 16:21 |
| mordred | but I think that we can have a structure where you can have arbitrary directory structure that still has a common entry point | 16:21 |
| zaro | fungi: help with publication | 16:21 |
| sdague | sure | 16:21 |
| fungi | zaro: what's up? | 16:21 |
| zaro | fungi: so i modified the 'overview' slides for gerrit uc. would like to put that up on publications but not sure how i should do that. | 16:22 |
| zaro | fungi: i don't think i want to make a branch for it. | 16:22 |
| zaro | fungi: because it's just a modified version of overview. | 16:22 |
| fungi | zaro: is it an improvement to the current presentation, or a separate presentation? | 16:23 |
| mordred | sdague: sure - but adding a release dir probably wouldn't kill you if it that was just what folks felt was necessary | 16:23 |
| zaro | fungi: it's a shortened version of it, not really improvement. | 16:23 |
| sdague | mordred: tempest doesn't have releases | 16:23 |
| zaro | fungi: it was only 30 min talk. | 16:23 |
| sdague | not in the same way as server projects | 16:23 |
| fungi | zaro: mmm, that seems like fodder for a separate branch (an abbreviated version of the talk?) | 16:24 |
| sdague | and the qa-specs repo was going to be bigger than tempest anyway as it was going to be program level | 16:24 |
| mordred | sdague: sure. but you live in the release cycle, and you do tend to have divergence per release | 16:24 |
| mordred | like, temptest for icehouse and tempest for juno will be different things, no? | 16:24 |
| sdague | mordred: in all fairness the only reason for the tag is because the APIs are too unstable | 16:25 |
| sdague | otherwise it's just tempest, that works against openstack apis | 16:25 |
| mordred | sure. but, well, let me put it a different way - do you not have a set of things you plan to work on in the juno cycle? | 16:25 |
| mordred | that you plan to discuss some of at the juno design summit? | 16:25 |
| sdague | yes | 16:26 |
| zaro | fungi: it's actually abreviated but some added stuff as well. maybe what i'll do is just add the stuff i added (multimaster jenkins) to the slides in the overview branch. | 16:26 |
| sdague | but I thought we were not trying to encode the things that blueprints provide | 16:26 |
| sdague | like targets | 16:26 |
| sdague | being my logic | 16:26 |
| fungi | zaro: we can branch the latest state of the overview branch and then you can propose the edits for the abbreviated talk you gave, then we approve and tag that for the date/venue | 16:26 |
| mordred | right. I don't think there should be a juno dir | 16:26 |
| mordred | but - if, for some reason, it's needed everywhere else | 16:26 |
| mordred | then it would not destroy qa to do it too | 16:27 |
| mordred | is all I'm saying | 16:27 |
| sdague | sure | 16:27 |
| jeblair | zaro, fungi: or if it's a one-off, you can just push it up for review, and then one of us can tag that review as a tagged version of the overview talk as given | 16:27 |
| fungi | zaro: because someone else might also want to give an abbreviated version of the talk later, which allows that to be a good starting point | 16:27 |
| jeblair | (either way) | 16:27 |
| fungi | jeblair: sure, just a tag without really landing the commit on a particular branch. that could also work | 16:28 |
| zaro | sounds fine. should i mention that it should only be a tag in the commit message? | 16:29 |
| fungi | zaro: sure, that would help | 16:29 |
| zaro | thx. | 16:29 |
| *** bhuvan has joined #openstack-infra | 16:31 | |
| jeblair | sdague, mordred, ttx: msg sent to openstack-dev | 16:33 |
| jeblair | russellb: ^ | 16:34 |
| *** nati_ueno has quit IRC | 16:35 | |
| jeblair | hashar: are you or someone else at wmf interested in reviewing zuul status page javascript changes? | 16:35 |
| jeblair | hashar: jhesketh is working on merging our status page changes back into zuul and getting the zuul version in shape so that we can use it (with our styling, etc) | 16:36 |
| mtreinish | mordred: oops yeah I forgot the tox.ini | 16:36 |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Format JSON text https://review.openstack.org/82541 | 16:37 |
| jeblair | hashar: (i don't mean put our style in the zuul repo, i mean apply our style on top of what's there) | 16:37 |
| hashar | jeblair: I can ask our jQuery evangelist :-] | 16:39 |
| jeblair | hashar: https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul+branch:master+topic:javascript_updates,n,z | 16:39 |
| hashar | he rewrote the status page to be more javascript like | 16:39 |
| *** jasondotstar has quit IRC | 16:39 | |
| hashar | jeblair: passing the message to him :-] | 16:40 |
| jeblair | hashar: great, thanks! | 16:40 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: WIP Refactor kibana module to add kibana3 https://review.openstack.org/82314 | 16:41 |
| hashar | jeblair: mail pushed (he is not online right now). Thanks for the notification! | 16:43 |
| *** nicedice has joined #openstack-infra | 16:43 | |
| jeblair | hashar: thank you! | 16:43 |
| hashar | just in time since I need to rush back home now :] | 16:44 |
| mtreinish | mordred: oh and the build dir path doesn't match what's expected in the run-docs slave script | 16:44 |
| openstackgerrit | Akihiro Motoki proposed a change to openstack-infra/elastic-recheck: Add query for bug 1296825 https://review.openstack.org/82546 | 16:45 |
| uvirtbot | Launchpad bug 1296825 in openstack-ci "ERROR: InvocationError: '/home/jenkins/workspace/gate-neutron-python26/.tox/py26/bin/python" [Undecided,New] https://launchpad.net/bugs/1296825 | 16:45 |
| *** packet has quit IRC | 16:46 | |
| *** hashar has quit IRC | 16:47 | |
| *** dizquierdo has quit IRC | 16:49 | |
| *** harlowja has joined #openstack-infra | 16:49 | |
| mordred | mtreinish: you want me to lend a hand? or you got it? | 16:50 |
| mtreinish | mordred: I think I've got it | 16:50 |
| *** jlibosva has quit IRC | 16:52 | |
| *** pblaho has quit IRC | 16:52 | |
| mordred | mtreinish: cool. let me know - happy to help! | 16:52 |
| *** prad has quit IRC | 16:52 | |
| *** jpich has quit IRC | 16:53 | |
| *** marktraceur has quit IRC | 16:53 | |
| mtreinish | mordred: https://review.openstack.org/#/c/82531/2/ I think got everything. Did I miss anything on this revision? | 16:55 |
| *** mgagne has quit IRC | 16:56 | |
| *** hogepodge has joined #openstack-infra | 16:56 | |
| *** amcrn has joined #openstack-infra | 16:57 | |
| *** marktraceur has joined #openstack-infra | 16:58 | |
| *** sc68cal_ is now known as sc68cal | 16:59 | |
| *** mgagne has joined #openstack-infra | 16:59 | |
| *** mgagne has quit IRC | 16:59 | |
| *** mgagne has joined #openstack-infra | 16:59 | |
| *** maxbit has quit IRC | 16:59 | |
| *** vkozhukalov has joined #openstack-infra | 17:01 | |
| *** zns has quit IRC | 17:01 | |
| *** zns has joined #openstack-infra | 17:02 | |
| anteaya | does the wikipage: https://wiki.openstack.org/wiki/ProjectTestingInterface need a face lift under end results needed? | 17:02 |
| *** kiall_ is now known as Kiall | 17:02 | |
| anteaya | like mentioning py3? | 17:02 |
| *** todd_dsm has quit IRC | 17:03 | |
| mordred | mtreinish: looks good - have you tested that it makes the doc/build dir if there is no doc dir? also, you could probably leave teh version field out of setup.cfg | 17:03 |
| mordred | but it's also fine | 17:03 |
| mtreinish | that's a good idea, I probably should test it :) | 17:05 |
| fungi | mordred: did https://launchpad.net/bugs/1296200 come to your attention yet? | 17:05 |
| uvirtbot | Launchpad bug 1296200 in pbr "Unit tests are failing in Sid" [Undecided,New] | 17:05 |
| *** roeyc has quit IRC | 17:05 | |
| mtreinish | seems to work fine | 17:05 |
| *** markmcclain has quit IRC | 17:09 | |
| *** markmcclain has joined #openstack-infra | 17:09 | |
| *** pbelanyi has quit IRC | 17:09 | |
| *** jpich has joined #openstack-infra | 17:10 | |
| *** dprince has quit IRC | 17:11 | |
| clarkb | MORNING | 17:12 |
| fungi | Alex_Gaynor: any news on getting https://launchpad.net/~pypy/+archive/ppa updated with a workaround/solution to the empty __pycache__ dirs breaking pip problem? | 17:12 |
| clarkb | wow capslock | 17:12 |
| SergeyLukjanov | clarkb, morning :) | 17:12 |
| jeblair | clarkb: HAHA | 17:12 |
| derekh | Does nodepool have logs that I can look at? looks like its having trouble building the template for tripleo-f20 want to try and figure out why? | 17:12 |
| * clarkb fixes capslock on this machine | 17:12 | |
| fungi | derekh: http://nodepool.openstack.org/ | 17:12 |
| jeblair | derekh: yes! http://nodepool.openstack.org/ | 17:12 |
| derekh | fungi: jeblair thanks | 17:12 |
| *** dstufft has quit IRC | 17:13 | |
| clarkb | jeblair: looking at the horizon thread now | 17:14 |
| *** rlandy_ has quit IRC | 17:14 | |
| *** e0ne has quit IRC | 17:14 | |
| *** prad has joined #openstack-infra | 17:14 | |
| *** dstufft has joined #openstack-infra | 17:15 | |
| *** vhoward has left #openstack-infra | 17:16 | |
| *** thuc_ has quit IRC | 17:16 | |
| openstackgerrit | Jason Kincl proposed a change to openstack-infra/jenkins-job-builder: Add support for Copy Artifact project copy permissions https://review.openstack.org/82551 | 17:17 |
| *** nati_ueno has joined #openstack-infra | 17:17 | |
| *** thuc has joined #openstack-infra | 17:17 | |
| *** zehicle_at_dell has quit IRC | 17:19 | |
| *** melwitt has joined #openstack-infra | 17:19 | |
| *** wenlock_ has joined #openstack-infra | 17:21 | |
| *** wenlock has quit IRC | 17:21 | |
| *** yassine has quit IRC | 17:21 | |
| *** wenlock_ has quit IRC | 17:21 | |
| *** prad has quit IRC | 17:21 | |
| *** wenlock has joined #openstack-infra | 17:21 | |
| *** thuc has quit IRC | 17:22 | |
| *** rhsu has joined #openstack-infra | 17:23 | |
| *** dstufft has quit IRC | 17:23 | |
| *** marktraceur has quit IRC | 17:23 | |
| *** mihgen has joined #openstack-infra | 17:24 | |
| clarkb | jeblair: you didn't happen to test HPCloud US East yet did you? I haven't heard if they changed perms for us yet | 17:24 |
| *** dstufft has joined #openstack-infra | 17:24 | |
| *** marktraceur has joined #openstack-infra | 17:24 | |
| jeblair | clarkb: glance? no. | 17:24 |
| *** prad has joined #openstack-infra | 17:24 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/publications: Modifications for 2014 Gerrit User conference https://review.openstack.org/82559 | 17:26 |
| *** melwitt has quit IRC | 17:26 | |
| pleia2 | fungi: if you have time today, I still need the info (cron job, config.js and anything else that looks customized)) from old-wiki for bugdaystats, WIP patch that lacks the info here: https://review.openstack.org/#/c/82106/ | 17:27 |
| *** melwitt has joined #openstack-infra | 17:27 | |
| fungi | pleia2: i'll dig that up | 17:27 |
| *** Ryan_Lane has joined #openstack-infra | 17:28 | |
| *** ildikov_ has quit IRC | 17:29 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Allow merge failures to have unique reporters. https://review.openstack.org/70636 | 17:30 |
| *** mkoderer has quit IRC | 17:31 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Restart unbound with systemd if upstart failed https://review.openstack.org/82562 | 17:31 |
| *** khyati has joined #openstack-infra | 17:31 | |
| derekh | that ^^ should fix the error tripleo-f20 template build is currently hitting, I don't know if there are other problems | 17:32 |
| derekh | but the rest of that script looks ok | 17:32 |
| *** leakypipes is now known as jaypipes | 17:33 | |
| *** flashgordon is now known as jogo | 17:34 | |
| *** prad has quit IRC | 17:34 | |
| fungi | derekh: any reason not to just look for systemd-managed unbound there rather than assuming any initscript failure should be followed by a call to systemctl? i suspect what you have there could lead to some confusing logs if we ever have to dig into an image build failure related to an initscript issue. lefe a comment | 17:35 |
| fungi | er, left | 17:35 |
| derekh | fungi: no reason, will fix | 17:36 |
| fungi | thanks! | 17:37 |
| openstackgerrit | Jason Kincl proposed a change to openstack-infra/jenkins-job-builder: Add support for Copy Artifact project copy permissions https://review.openstack.org/82551 | 17:37 |
| *** vhoward has joined #openstack-infra | 17:40 | |
| *** _nadya_ has joined #openstack-infra | 17:40 | |
| *** thuc has joined #openstack-infra | 17:43 | |
| openstackgerrit | A change was merged to openstack-infra/config: Replace gate-noop with noop https://review.openstack.org/82162 | 17:44 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/publications: Modifications for 2014 Gerrit User conference https://review.openstack.org/82559 | 17:44 |
| *** thuc_ has joined #openstack-infra | 17:44 | |
| *** prad has joined #openstack-infra | 17:45 | |
| *** aysyd has joined #openstack-infra | 17:45 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Add support to restart unbound with systemd https://review.openstack.org/82562 | 17:45 |
| *** reed has quit IRC | 17:47 | |
| *** thuc has quit IRC | 17:48 | |
| *** sabari2 has joined #openstack-infra | 17:52 | |
| *** vhoward has left #openstack-infra | 17:53 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix centos image name in region-b https://review.openstack.org/82369 | 17:53 |
| *** derekh has quit IRC | 17:54 | |
| *** alff has joined #openstack-infra | 17:55 | |
| *** CaptTofu has joined #openstack-infra | 17:57 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/publications: update slides with multi-master jenkins and nodepool https://review.openstack.org/82567 | 17:59 |
| *** _nadya_ has quit IRC | 17:59 | |
| mordred | zaro: nice slides! how was our stuff receieved? | 17:59 |
| fungi | zaro: what tag do you want on http://git.openstack.org/cgit/openstack-infra/publications/commit/?id=f1d0dfc ? | 18:01 |
| zaro | mordred: most of the people there were dedicated to gerrit, not to the entire CI work. so lots of gerrit admins and developers, not much people working on CI in general. | 18:01 |
| mordred | nod | 18:02 |
| clarkb | jeblair: this is a fun mystery, I can't remember why but am digging through logs to see if I can figure it out | 18:02 |
| zaro | mordred: so they really liked how we automated gerrit and integrated our tools with it. but not super excited about zuul and jenkins | 18:03 |
| mordred | zaro: I guess I was just wondering if any of how we're using gerrit made sense to them? did they think we were crazy? both? neither? didn't care? :) | 18:03 |
| mordred | zaro: ah - interesting | 18:03 |
| mordred | so manage-projects is a thing they found interesting | 18:03 |
| zaro | yes, very interested. scaling git and gerrit were also very interesting to the audience | 18:03 |
| zaro | one of the guys created a UI to automate the creation of projects. | 18:04 |
| zaro | i think most of people liked our solution better because they liked that it was reviewable in gerrit. | 18:05 |
| *** signed8bit has joined #openstack-infra | 18:05 | |
| pleia2 | cool | 18:05 |
| *** salv-orlando has quit IRC | 18:06 | |
| zaro | fungi: how about 2014-gerrit-UC ? | 18:07 |
| *** alff has quit IRC | 18:07 | |
| fungi | zaro: 2014-gerrit_uc-overview maybe? | 18:08 |
| zaro | fungi: wfm | 18:08 |
| fungi | that seems to be a little more in keeping with our year-conference-shortname standard | 18:08 |
| anteaya | should I be able to view the .pngs in that patch somehow? | 18:09 |
| clarkb | anteaya: git review -d and open the file in your web browser | 18:09 |
| *** SumitNaiksatam has joined #openstack-infra | 18:09 | |
| *** markmcclain has quit IRC | 18:09 | |
| zaro | anteaya: i think you just need to view index.html to make sure no images are missing in the slides. | 18:10 |
| fungi | yep | 18:10 |
| anteaya | clarkb: ah | 18:10 |
| *** zul has quit IRC | 18:13 | |
| fungi | heh... 82559 just clicked... that's the board series number for a popular intel ethernet interface i used for years | 18:14 |
| *** dprince has joined #openstack-infra | 18:14 | |
| fungi | those were some of the most stable 100basetx pci cards ever produced | 18:15 |
| anteaya | I see there are tabs in index.html | 18:15 |
| *** zul has joined #openstack-infra | 18:16 | |
| fungi | anteaya: yeah, someone's text editor thinks embedded javascript deserves mixed tabs+spaces indents or something | 18:16 |
| zaro | anteaya: not from me :) | 18:17 |
| fungi | no, it might be inherited from slidy2 | 18:17 |
| anteaya | zaro: yes they aren't in your edits, they seem to be there from before | 18:18 |
| anteaya | zaro: commented | 18:18 |
| openstackgerrit | Solly Ross proposed a change to openstack/requirements: Limit psutil to <2.0.0 https://review.openstack.org/81373 | 18:18 |
| fungi | tab substitution is popular in html/javascript because it potentially reduces the number of bytes transmitted over the wire | 18:18 |
| anteaya | really | 18:19 |
| anteaya | huh | 18:20 |
| *** prad has quit IRC | 18:20 | |
| anteaya | I can never remember where the publications are published | 18:20 |
| anteaya | I would like to offer a patch to root/readme.rst so I remember | 18:20 |
| anteaya | can anyone tell me where they are published? | 18:20 |
| clarkb | anteaya: http://docs.openstack.org/infra/publications iirc | 18:21 |
| fungi | anteaya: http://docs.openstack.org/infra/publications/ | 18:21 |
| zaro | jeblair: could you please take a re-look at https://review.openstack.org/#/c/70014 ? i would like to land that so i can test it. | 18:21 |
| anteaya | awesome thanks | 18:23 |
| *** andreaf2 has quit IRC | 18:23 | |
| rcarrillocruz | clarkb: hi, when you have some time, may you please revisit https://review.openstack.org/#/c/79205/ ? i exposed the reasons for coding the ref-updated as a special case, as gerrit won't give a branch info on a tag event... | 18:23 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Set paramiko version > 1.9.0 https://review.openstack.org/82404 | 18:23 |
| *** morganfainberg_Z is now known as morganfainberg | 18:24 | |
| clarkb | rcarrillocruz: looking | 18:25 |
| *** vhoward has joined #openstack-infra | 18:25 | |
| *** jpeeler has quit IRC | 18:26 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/publications: Add the publication url to root/README.rst https://review.openstack.org/82570 | 18:26 |
| mtreinish | fungi, jeblair, clarkb: if I wanted to make a new branch on a project do I have to do anything special or just do a git review with the new branch name? | 18:27 |
| *** mestery has joined #openstack-infra | 18:27 | |
| clarkb | mtreinish: you have to do special things, it requires bit fiddling in the UI or a force push which no one should have perms for | 18:28 |
| *** jgallard has quit IRC | 18:28 | |
| jeblair | mtreinish: if it's an openstack project, ask the release management team | 18:28 |
| clarkb | mtreinish: they will want to know the project, branch name, and starting ref | 18:28 |
| *** prad has joined #openstack-infra | 18:28 | |
| mtreinish | jeblair: ok, I think cookiecutter is in openstack-dev | 18:28 |
| *** mihgen has quit IRC | 18:29 | |
| mtreinish | does that go through the same process? | 18:29 |
| *** mestery has quit IRC | 18:29 | |
| fungi | mtreinish: no, but what branch does it need (and why)? | 18:29 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added prod convenience invocation https://review.openstack.org/82571 | 18:29 |
| mtreinish | I was going to add a specs-repo branch for cookiecutter | 18:29 |
| mtreinish | so everything can have a common starting point | 18:29 |
| mtreinish | if we move forward with it | 18:30 |
| *** e0ne has joined #openstack-infra | 18:30 | |
| *** Hefeweizen has quit IRC | 18:30 | |
| fungi | mtreinish: for making new specs repo projects? | 18:30 |
| mtreinish | yeah | 18:30 |
| fungi | mtreinish: i though cookiecutter didn't support differentiating by branch? | 18:30 |
| fungi | mtreinish: if memory serves, that's why oslo has their own cookiecutter project separate from the main one | 18:30 |
| mtreinish | fungi: well I didn't know that | 18:30 |
| clarkb | rcarrillocruz: I understand now. I think it will just have to work that way. Will review latest patchset shortly | 18:30 |
| jeblair | mtreinish: we should get dhellmann and mordred looped into this conversation | 18:31 |
| rcarrillocruz | k, thx sir | 18:31 |
| dhellmann | fungi: yes, that's right, cookiecutter takes a git repo url, but I don't think it handles branches | 18:31 |
| dhellmann | at least, I didn't find any evidence that it does | 18:31 |
| dhellmann | having separate repositories feels more sane than branches anyway | 18:32 |
| dhellmann | although subdirs in one repo would be cool | 18:32 |
| dhellmann | but it doesn't handle that, either, afaict | 18:32 |
| *** Ajaeger1 has joined #openstack-infra | 18:32 | |
| dhellmann | when I mentioned these things to audrey, she was enthusiastic about adding them, but more so about having pull requests :-) | 18:33 |
| *** thedodd has quit IRC | 18:33 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Reduce number of files send for propose_translation https://review.openstack.org/82573 | 18:34 |
| mtreinish | dhellmann: ok good to know, so for right now adding a separate specs-repo template repo would be the only option | 18:35 |
| dhellmann | mtreinish: certainly the path of least resistance | 18:35 |
| fungi | zaro: is https://sites.google.com/site/gerritsummit2014/ the same conference? if so it looks like maybe they decided that "Gerrit User Summit" is the official name? thinking of doing 'git tag -s -m "Gerr | 18:35 |
| fungi | it User Summit, 2014" 2014-gerrit_user_summit-overview' in that case | 18:35 |
| fungi | (without the stray newline of course) | 18:36 |
| clarkb | rcarrillocruz: how does that look? | 18:36 |
| zaro | fungi: yes, and wfm | 18:37 |
| *** aysyd has quit IRC | 18:37 | |
| *** _nadya_ has joined #openstack-infra | 18:39 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add jenkins job to build gerrit plugins https://review.openstack.org/70014 | 18:39 |
| rcarrillocruz | sure thing, more robust | 18:39 |
| rcarrillocruz | thanks, i will amend | 18:39 |
| jeblair | zaro: do you know if _david_'s wip plugin is ready? do you want to take a look at it and see if we can/should use it when we upgrade to 2.8? | 18:40 |
| *** alff has joined #openstack-infra | 18:40 | |
| clarkb | rcarrillocruz: fwiw we don't do versions with /s in them but other people may maybe? | 18:40 |
| zaro | jeblair: it's ready but not available on 2.8, only 2.9+ | 18:41 |
| clarkb | is the old change screen still available in 2.9? | 18:41 |
| zaro | jeblair: i think it would require a patch from master for it to work on 2.8 | 18:42 |
| lifeless | russellb: ah ok, I'll tear down this backup copy then :) | 18:42 |
| zaro | clarkb: i'm not sure, don't think so, but will double check. | 18:42 |
| lifeless | ihrachys: hi ? | 18:42 |
| fungi | jeblair: weird... i get "fatal: reference is not a tree: <sha of tag>" from the publication jobs when pushing a tag for a not-approved change's gitsha. ideas where i went wrong there? the job parameters were https://jenkins01.openstack.org/job/infra-publications-publish/51/parameters/ | 18:43 |
| *** thuc_ has quit IRC | 18:45 | |
| *** thuc has joined #openstack-infra | 18:45 | |
| zaro | clarkb: yep, old change screen is still availabe: http://storage.googleapis.com/gerrit-talks/gerrit-new-features-2.9.html#%284%29 | 18:46 |
| *** thuc_ has joined #openstack-infra | 18:46 | |
| *** thuc_ has quit IRC | 18:47 | |
| *** thuc_ has joined #openstack-infra | 18:47 | |
| zaro | actually if you look thru the slides the new change screen has some very useful features that i didn't even notice when using upstream gerrit. | 18:47 |
| *** thuc_ has quit IRC | 18:48 | |
| clarkb | zaro: yes it has a lot of useful features but a lot of terrible ones too | 18:48 |
| *** thuc_ has joined #openstack-infra | 18:49 | |
| *** thuc has quit IRC | 18:49 | |
| fungi | jeblair: looking at the log, i think maybe ggp didn't see the new tag when it did remote update | 18:50 |
| jeblair | fungi: ah, yeah, we should probably see about having ggp update from the zuul merger | 18:51 |
| *** aysyd has joined #openstack-infra | 18:51 | |
| clarkb | jeblair: fungi: oh hey | 18:51 |
| fungi | replicated to the git servers with what looks correct to me... http://git.openstack.org/cgit/openstack-infra/publications/tag/?id=2014-gerrit_user_summit-overview | 18:51 |
| jeblair | fungi: since i believe it will have what it needs... | 18:51 |
| jeblair | fungi: it's probably a race | 18:51 |
| clarkb | about that, I wholeheartedly agree in making the post queue stuff in g-g-p less racy | 18:51 |
| *** prad has left #openstack-infra | 18:51 | |
| fungi | i'll stab at it mightily | 18:52 |
| clarkb | jeblair: fungi also what do you think about hosting /p/ as a valid root path on the replicas to replicate that path on review.o.o? | 18:52 |
| jeblair | fungi: git fetch --tags http://zm02.openstack.org/p/openstack-infra/publications refs/tags/2014-gerrit_user_summit-overview | 18:53 |
| jeblair | fungi: that got the tag for me | 18:53 |
| fungi | jeblair: perfect--i'll have it use that syntax | 18:53 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add gate jobs for oslo libraries https://review.openstack.org/76945 | 18:54 |
| jeblair | fungi: probably best to do that after a git remote update, because it might not have _all_ the tags | 18:54 |
| jeblair | clarkb: why /p/? | 18:54 |
| clarkb | jeblair: because review.o.o and apparently the zuul mergers both support it | 18:54 |
| clarkb | will make replacing the source of uri paths simpler if all sources support the same paths | 18:55 |
| *** mrodden has joined #openstack-infra | 18:55 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Make the oslo.test cross-testing jobs voting https://review.openstack.org/82580 | 18:56 |
| morganfainberg | pleia2, fyi i've been swamped. | 18:57 |
| *** afazekas_ has quit IRC | 18:57 | |
| pleia2 | morganfainberg: hah, no worries :) | 18:57 |
| morganfainberg | pleia2, might need to wait for the RC to ship before i can think about monitoring collaboration stuffs | 18:57 |
| morganfainberg | pleia2, just letting you know :) | 18:57 |
| *** mrodden1 has joined #openstack-infra | 18:57 | |
| pleia2 | morganfainberg: that's fine | 18:57 |
| anteaya | fungi: at this point my understanding of salt reactors (something that reacts to an event) is that once an event publishs a tag & data combo, this triggers a salt runner which is a module that executes a command on the salt master, the command can be anything you write and can poll minions for state information or what have you | 18:58 |
| pleia2 | thanks for letting me know | 18:58 |
| *** zns has quit IRC | 18:58 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/devstack-gate: Correcting the right path to mysql log in Fedora 19 https://review.openstack.org/79967 | 18:58 |
| anteaya | the part I am wondering is that the issue manage projects is facing is that the repo in question needs to be populated across the git farm before manage projects can be executed safely, is that correct? | 18:59 |
| morganfainberg | pleia2, didn't want you to think I wasn't interested still | 18:59 |
| *** rhsu has quit IRC | 18:59 | |
| anteaya | and if yes, how can we get this to send out a tag & data combo to trigger the salt runner | 18:59 |
| rcarrillocruz | clarkb: right, i took https://wiki.openstack.org/wiki/GerritJenkinsGit#Tagging_a_Release to code the stuff. But I rather look at http://git-scm.com/book/en/Git-Internals-Git-References a better understanding to cover more use cases... | 19:00 |
| fungi | anteaya: that was a big question for me, at any rate... how to make sure that 'create-cgitrepos' ran successfully on all five git farm servers before running manage-projects on the gerrit server | 19:00 |
| anteaya | I suppose we could poll, but having a completed event say it is completed is tidier | 19:00 |
| *** mrodden has quit IRC | 19:00 | |
| anteaya | ah so we both sit here with the same question | 19:00 |
| anteaya | okay will continue to read/dig | 19:00 |
| *** harlowja has quit IRC | 19:01 | |
| fungi | anteaya: i had tried to get an example first of using a job worker to trigger git updates on the puppet master and then fire puppet apply on all puppeted servers once that completes successfully, since i thought that might be an easier one to do as a first example | 19:01 |
| *** rhsu has joined #openstack-infra | 19:02 | |
| anteaya | did you have code for that? | 19:02 |
| *** mrda_away is now known as mrda | 19:02 | |
| fungi | UtahDave said he'd put together an example config for that, but doesn't seem to have had free time to work on it | 19:02 |
| anteaya | yeah, I keep trying to get 5 minutes of his time | 19:02 |
| anteaya | I'll keep trying | 19:03 |
| anteaya | my sense is that since we won't be moving away from puppet to salt that the powers that be have less free time for Dave to help us | 19:03 |
| anteaya | not unexpected | 19:03 |
| *** harlowja has joined #openstack-infra | 19:03 | |
| anteaya | and he his happy to help when he can | 19:03 |
| anteaya | just less | 19:03 |
| *** vhoward has left #openstack-infra | 19:04 | |
| *** alff has quit IRC | 19:04 | |
| anteaya | since I was asked that question point blank at saltconf and had to tell them no, we won't be moving off puppet | 19:04 |
| anteaya | since we can't | 19:04 |
| *** SumitNaiksatam has quit IRC | 19:05 | |
| *** _nadya_ has quit IRC | 19:06 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/reviewstats: Add requirements project config file https://review.openstack.org/82219 | 19:08 |
| *** melwitt has quit IRC | 19:09 | |
| *** melwitt has joined #openstack-infra | 19:09 | |
| clarkb | are the tenant ids and other ids in keystone md5 sums? | 19:11 |
| clarkb | I am taking a stab at making crm114 smarter about those things | 19:11 |
| *** ociuhandu has quit IRC | 19:13 | |
| *** tianst20 has quit IRC | 19:14 | |
| *** CaptTofu has quit IRC | 19:14 | |
| *** CaptTofu has joined #openstack-infra | 19:15 | |
| *** yonglihe_ has joined #openstack-infra | 19:15 | |
| *** tianst20 has joined #openstack-infra | 19:15 | |
| *** yongli has quit IRC | 19:16 | |
| *** johnthetubaguy1 has quit IRC | 19:16 | |
| fungi | clarkb: i believe they vary, diablo era were long int maybe? | 19:16 |
| fungi | oh, but at present | 19:17 |
| fungi | and for branches we still care about | 19:17 |
| *** alff has joined #openstack-infra | 19:17 | |
| anteaya | clarkb: I thought they are uuids now | 19:17 |
| fungi | i think they're a uuid, just commonly expressed without any punctuation/spacing | 19:17 |
| bknudson | clarkb: uuid.uuid4().hex | 19:17 |
| anteaya | and were pkis before | 19:17 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Don't treat IDs as uniquely special in CRM114 https://review.openstack.org/82581 | 19:18 |
| fungi | okay, glad to know i'm not imagining things ;) | 19:18 |
| mgagne | zaro: ping | 19:18 |
| clarkb | bknudson: oh they are rendered that way | 19:18 |
| clarkb | bknudson: I was curious because they don't have the typical -'s separating the chunks | 19:18 |
| *** UtahDave has joined #openstack-infra | 19:18 | |
| anteaya | UtahDave: we were just saying nice things about you | 19:18 |
| anteaya | like how much I missed you | 19:18 |
| clarkb | jeblair: https://review.openstack.org/82581 is not tested yet. How were you testing your crm114 changes? just redirecting a local file into crm114? | 19:19 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Comments UI https://review.openstack.org/82213 | 19:19 |
| clarkb | sdague: if you have an example of a log somewhere with PKI token things I can try matching those too | 19:19 |
| clarkb | sdague: so far the two things in 82581 seem to be pretty ubiquitous after scanning a few log files | 19:19 |
| *** CaptTofu has quit IRC | 19:19 | |
| *** che-arne has quit IRC | 19:20 | |
| sdague | clarkb: http://logs.openstack.org/26/81926/3/check/check-tempest-dsvm-full/7d507ab/logs/screen-n-api.txt.gz#_2014-03-24_17_42_59_568 | 19:21 |
| sdague | though I'm not sure we leak PKI into non debug any more | 19:21 |
| clarkb | sdague: it would make me so happy if we didn't :) | 19:22 |
| clarkb | sdague: but I suppose it could be leaked by any change so we should try to deal with it | 19:22 |
| clarkb | firefox is hating me right now | 19:22 |
| sdague | well, it hates these logs in general | 19:22 |
| *** jpich has quit IRC | 19:23 | |
| sdague | honestly, I can't interact with the logs at debug level live in anything other than chrome/chromium | 19:23 |
| *** thedodd has joined #openstack-infra | 19:25 | |
| *** e0ne has quit IRC | 19:25 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Fetch tag refs from zuul in gerrit-git-prep https://review.openstack.org/82583 | 19:26 |
| *** sandywalsh has quit IRC | 19:27 | |
| *** khyati has quit IRC | 19:29 | |
| *** khyati has joined #openstack-infra | 19:30 | |
| *** yolanda has quit IRC | 19:31 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:33 | |
| clarkb | sdague: ok PKI token is going to be tricky because you can only match up to 255 chars usin {N,M} in TRE | 19:34 |
| *** khyati has quit IRC | 19:34 | |
| clarkb | and doing things like ([asdfb]{255}){5} are very slow | 19:34 |
| *** jooools has quit IRC | 19:34 | |
| sdague | yeh, it's probably not worth it | 19:34 |
| *** hashar has joined #openstack-infra | 19:36 | |
| *** jnoller has quit IRC | 19:38 | |
| zaro | mgagne: yo! | 19:39 |
| mgagne | zaro: how hard would it be to implement it in JJB until it gets merged? https://code.launchpad.net/~zaro0508/python-jenkins/plugin-info-method/+merge/196653 | 19:39 |
| zaro | mgagne: maybe connect with a different library? | 19:41 |
| zaro | mgagne: i think jenkinsapi already has it that functionality | 19:42 |
| mgagne | zaro: would there be a way to patch it until it's supported? | 19:42 |
| mgagne | zaro: I fear backward compatibility breakages landing in master: https://review.openstack.org/#/c/78930/ | 19:43 |
| mgagne | zaro: I still use 1.x | 19:43 |
| zaro | jeblair: ^ | 19:44 |
| mgagne | zaro: which triggers a more general question: how far and for how long do we want to support older plugins? | 19:45 |
| zaro | mgagne: it's definately possible to do, but i think would require a 2nd connection to jenkins. jjb uses connection provided by python-jenkins. | 19:45 |
| * mgagne puts on cowboy hat | 19:46 | |
| mgagne | zaro: what about monkey patching? | 19:46 |
| zaro | mgagne: that's a good question, my comment on that change was the same question. would be good to get input from jeblair and clarbk | 19:47 |
| *** mestery has joined #openstack-infra | 19:47 | |
| mgagne | clarkb * | 19:47 |
| zaro | mgagne: what monkeys are you thinking? | 19:48 |
| mgagne | zaro: could you inject the method from JJB? | 19:49 |
| *** zns has joined #openstack-infra | 19:49 | |
| *** dcramer_ has quit IRC | 19:51 | |
| zaro | mgagne: yeah, i think that will work. but since jeblair can merge stuff to upstream project i think it would be good to get his feedback. | 19:53 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Don't enable savanna service for sahara https://review.openstack.org/81058 | 19:54 |
| mgagne | zaro: oh, I didn't know that | 19:55 |
| *** emagana has joined #openstack-infra | 20:00 | |
| zaro | mgagne: i have asked him to look and merge but i think he's been busy. it might help if you ask as well. | 20:00 |
| *** sabari2 has quit IRC | 20:00 | |
| mgagne | zaro: no rush =) | 20:00 |
| *** JoshNang has quit IRC | 20:00 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Stop logging obsolete channel #savanna https://review.openstack.org/82587 | 20:01 |
| *** sabari2 has joined #openstack-infra | 20:01 | |
| *** imcsk8_lap has joined #openstack-infra | 20:02 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Bump min hacking version to 0.8.1 https://review.openstack.org/82339 | 20:03 |
| jeblair | clarkb: yes | 20:08 |
| jeblair | clarkb: crm114 | 20:08 |
| *** ildikov_ has joined #openstack-infra | 20:09 | |
| *** imcsk8_lap has quit IRC | 20:10 | |
| *** prad has joined #openstack-infra | 20:11 | |
| *** hashar has quit IRC | 20:13 | |
| *** hashar has joined #openstack-infra | 20:14 | |
| *** jp_at_hp has quit IRC | 20:15 | |
| *** SumitNaiksatam has quit IRC | 20:17 | |
| *** prad has quit IRC | 20:18 | |
| * SergeyLukjanov fyi, it's possible now just to upload picture for organisations/user profiles at github - https://github.com/blog/1803-switch-your-picture-with-ease | 20:21 | |
| SergeyLukjanov | w/o using gravatar | 20:21 |
| SergeyLukjanov | fungi, ^^ | 20:21 |
| jeblair | wow. that was years in the making. | 20:21 |
| *** prad has joined #openstack-infra | 20:21 | |
| fungi | crazy | 20:21 |
| fungi | when i do an image search for stackforge logo, this is one of the top hits ;) http://i1.ytimg.com/vi/hiAelteHfzM/0.jpg | 20:23 |
| portante | SergeyLukjanov: thanks, updated | 20:23 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: Update Murano gate job config https://review.openstack.org/82316 | 20:26 |
| jogo | clarkb: https://review.openstack.org/#/c/82255/ | 20:27 |
| jogo | FYI | 20:27 |
| *** harlowja_ has joined #openstack-infra | 20:27 | |
| *** prad has quit IRC | 20:28 | |
| *** jeckersb is now known as jeckersb_gone | 20:28 | |
| *** dcramer_ has joined #openstack-infra | 20:31 | |
| *** mwagner_lap has quit IRC | 20:31 | |
| *** khyati has joined #openstack-infra | 20:31 | |
| *** khyati has quit IRC | 20:31 | |
| *** harlowja has quit IRC | 20:31 | |
| *** prad has joined #openstack-infra | 20:31 | |
| *** marun has quit IRC | 20:32 | |
| zaro | is there anyway to manually trigger a periodic job? | 20:32 |
| *** rlandy has joined #openstack-infra | 20:34 | |
| *** imcsk8 has joined #openstack-infra | 20:34 | |
| *** khyati has joined #openstack-infra | 20:35 | |
| *** dcramer_ has quit IRC | 20:36 | |
| fungi | zaro: zaro a zuul admin can run http://git.openstack.org/cgit/openstack-infra/zuul/tree/tools/trigger-job.py on the zuul master server, though no idea whether that has all the necessary parameters for periodic pipeline | 20:36 |
| *** marun has joined #openstack-infra | 20:40 | |
| sdague | jogo: yeh, that should shoot the conductor logs down a bunch | 20:40 |
| sdague | there are some other goofy oslo log defaults that should be dealt with as well | 20:40 |
| *** rediskin has joined #openstack-infra | 20:42 | |
| zaro | fungi: so would it be difficult to run this periodic job? http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n2732 | 20:43 |
| fungi | zaro: periodic-gerrit-plugin-javamelody-gerrit-plugin ? | 20:44 |
| imcsk8 | hello, we're having a problem in review.openstack.org. when packstack tests run on the jenkins jobs, one of the tests is no passing with NetworkError: Unknown hostname google-public-dns-a.google.com. which is odd because that address is active. do you know who can help me with this? this is one of the outputs: http://logs.openstack.org/25/82525/1/check/gate-packstack-python26/b7c5991/console.html | 20:44 |
| zaro | fungi: yes. | 20:44 |
| fungi | zaro: i guess if you know what parameters zuul is likely to pass for it when it runs, i can try to trigger it from the command line with the appropriate options | 20:46 |
| fungi | i could probably try to infer relevant parameters from another periodic job which has run recently | 20:46 |
| fungi | but i assume that job hasn't been run before | 20:46 |
| rediskin | hi all. tell me please how differs "check-*" and "gate-*" named jobs? | 20:46 |
| *** markmcclain has joined #openstack-infra | 20:47 | |
| fungi | rediskin: some jobs have "check-" and "gate-" variants so that we can run the check versions on lower-performing or experimental systems | 20:47 |
| *** e0ne has joined #openstack-infra | 20:47 | |
| fungi | without having them impact jobs running in the gate pipeline | 20:47 |
| zaro | fungi: ohh actually. i can just wait to see if it spits out anything tommorrow i guess. | 20:48 |
| rediskin | fungi: thanks | 20:48 |
| jeblair | imcsk8: 'google-public-dns-a.google.com' resolves correctly on centos6 node in hpcloud az2 | 20:48 |
| *** dcramer_ has joined #openstack-infra | 20:48 | |
| *** markmcclain1 has joined #openstack-infra | 20:49 | |
| jeblair | imcsk8: i can tell you that node is running a local caching resolver that forwards queries to 8.8.8.8, and that it's the only configured namesever. | 20:49 |
| jeblair | imcsk8: maybe that will help. | 20:49 |
| *** vkozhukalov has quit IRC | 20:49 | |
| *** markmcclain1 has quit IRC | 20:49 | |
| *** salv-orlando has joined #openstack-infra | 20:49 | |
| zaro | fungi: do you know how i would be able to get the log for that periodic job? would i just need to look thru all the gerrit masters for the log? | 20:49 |
| jeblair | imcsk8: (on a side note, a lot of folks, especially packagers, aren't too keen on unit tests that access the network) | 20:50 |
| fungi | zaro: it will appear under http://logs.openstack.org/periodic/ | 20:50 |
| *** markmcclain1 has joined #openstack-infra | 20:50 | |
| *** markmcclain1 has quit IRC | 20:50 | |
| zaro | fungi: ohh cool. thanks. | 20:50 |
| jogo | sdague: my take on logs is we should have a log level that is: openstack_debug and debug_all | 20:50 |
| jogo | or something | 20:50 |
| jogo | where openstack_debug doesn't set debug for all dependencies | 20:51 |
| jogo | is that feasable? | 20:51 |
| *** rossella_s has quit IRC | 20:51 | |
| imcsk8 | jeblair: yes, that test is not my favorite. i wanted to know if there was a problem on your end before i modify the test | 20:51 |
| *** markmcclain has quit IRC | 20:51 | |
| sdague | jogo: yeh, this was my debug2 idea | 20:51 |
| jogo | sdague: you think its too ale to do for icehuose? | 20:52 |
| imcsk8 | jeblair: the weird part is that it just started failing today | 20:52 |
| *** mrodden1 is now known as mrodden | 20:53 | |
| jeblair | imcsk8: not that i can tell, but the addition of the local nameserver is a recent change, so it may be causally related | 20:53 |
| jeblair | (not that i can tell == there is not a problem on the test node, that i can tell) | 20:53 |
| *** jpeeler has joined #openstack-infra | 20:53 | |
| *** jpeeler has quit IRC | 20:53 | |
| *** jpeeler has joined #openstack-infra | 20:53 | |
| *** jpeeler has quit IRC | 20:54 | |
| *** jpeeler has joined #openstack-infra | 20:54 | |
| *** jpeeler has quit IRC | 20:54 | |
| *** jpeeler has joined #openstack-infra | 20:54 | |
| clarkb | sorry back from lunch | 20:55 |
| *** markmcclain has joined #openstack-infra | 20:55 | |
| imcsk8 | jeblair: i guess this test wasn't designed with this kind of setup in mind, i'll change it so it works with this type of setup. thanks for the help! | 20:55 |
| sdague | jogo: no | 20:55 |
| sdague | I think all that's too disruptive right now | 20:56 |
| sdague | I think it's juno 1 stuff | 20:56 |
| clarkb | it is too nice outside to have a short lunch | 20:56 |
| *** thuc_ has quit IRC | 20:56 | |
| jogo | sdague: in that case just disabling oslo.messaging logs? | 20:57 |
| *** luis_ has joined #openstack-infra | 20:57 | |
| *** SumitNaiksatam has joined #openstack-infra | 20:58 | |
| sdague | jogo: and lockutils while you are at it | 20:58 |
| jogo | so there may be another way actually | 20:58 |
| jogo | logging_sample.conf | 20:59 |
| *** luisg has quit IRC | 20:59 | |
| *** sabari2 is now known as sabari | 20:59 | |
| jogo | looking into it now | 20:59 |
| sdague | jogo: there are a lot of ways to skin this cat | 21:00 |
| *** jeckersb_gone is now known as jeckersb | 21:01 | |
| jogo | if this works it solves the problem of needing the oslo sync every time | 21:01 |
| sdague | jogo: well the issue there is it requires logging.conf installation to get the right behavior | 21:01 |
| sdague | the defaults end up being wrong | 21:01 |
| sdague | the real issue is we have to get the logging default levels out of oslo code | 21:01 |
| sdague | and able to set that in nova code | 21:02 |
| jogo | sdague: we can fix logging.conf | 21:02 |
| sdague | that's bothered me a lot | 21:02 |
| jogo | and this wouldn't solve the oslo part but work around it for now | 21:02 |
| sdague | jogo: sure, but that requires people install that | 21:02 |
| sdague | which we've never required before | 21:02 |
| jogo | sdague: yeah ... true | 21:02 |
| jogo | it would fix gate but no one else | 21:02 |
| dhellmann | sdague: https://blueprints.launchpad.net/oslo/+spec/graduate-oslo-log is planned for juno, so if you could write up your thoughts on that we can take them into account | 21:02 |
| sdague | and logging.conf is powerful, but it's super complicated | 21:02 |
| *** Sukhdev has joined #openstack-infra | 21:03 | |
| sdague | especially as we don't use well understood names in openstack components | 21:03 |
| dhellmann | sdague: even a quick email to me or the ml would do it, I can stick them into the release requirements for that lib | 21:03 |
| sdague | just randomly whatever module you happen to be in | 21:03 |
| sdague | dhellmann: sure | 21:03 |
| jogo | if we update the logging.conf sample and better document it, I was thinking this may be a good interam solution for icehouse | 21:04 |
| *** rlandy has quit IRC | 21:06 | |
| *** bogdando has quit IRC | 21:06 | |
| jogo | sdague: we don't use even copy the file in devstack ... so not as quick a solution as I thought | 21:06 |
| *** afazekas has joined #openstack-infra | 21:07 | |
| *** bogdando has joined #openstack-infra | 21:08 | |
| *** pdmars has quit IRC | 21:09 | |
| *** thuc has joined #openstack-infra | 21:09 | |
| *** Ajaeger1 has quit IRC | 21:09 | |
| *** thuc has quit IRC | 21:10 | |
| jogo | dhellmann: thoughts on putting https://review.openstack.org/#/c/82255/ in oslo-incubator and cherry picking it out to nova etc | 21:10 |
| jogo | for icehouse | 21:10 |
| dhellmann | jogo: looking | 21:10 |
| *** thuc has joined #openstack-infra | 21:10 | |
| jogo | it drops file size dramatically | 21:10 |
| jogo | and an operator can turn this back on via logging.conf | 21:11 |
| *** luis_ has quit IRC | 21:11 | |
| sdague | jogo: can you do the lockutils one as well :) | 21:11 |
| jogo | sdague: yup | 21:11 |
| sdague | lockutils debug is pointless | 21:11 |
| *** thuc has quit IRC | 21:11 | |
| dhellmann | jogo: that looks like a reasonable change | 21:11 |
| *** thuc has joined #openstack-infra | 21:11 | |
| dhellmann | jogo: you could also just say "oslo = INFO" | 21:11 |
| jogo | dhellmann: ohh I like that | 21:11 |
| * dhellmann is writing up plans for 9 new oslo libraries next cycle | 21:12 | |
| *** thuc has quit IRC | 21:12 | |
| jogo | sdague: lockutils is incubator right? | 21:12 |
| jogo | if yes, how do we turn it off? | 21:13 |
| *** thuc has joined #openstack-infra | 21:13 | |
| sdague | jogo: yes | 21:13 |
| sdague | oh, right, grrr | 21:13 |
| sdague | that's the whole oslo sync problem | 21:14 |
| jogo | sdague: heh yeah | 21:14 |
| sdague | because it renames things | 21:14 |
| *** thuc_ has joined #openstack-infra | 21:14 | |
| sdague | dhellmann: is there a sane way to override this from nova code? | 21:14 |
| *** ociuhandu has joined #openstack-infra | 21:14 | |
| dhellmann | jogo: if you put oslo.openstack.common.lockutils in that config option default, I think the sync code will replace "oslo" with "$project" | 21:14 |
| dhellmann | if that doesn't work, you'd have to call log.set_defaults() | 21:15 |
| jogo | dhellmann: would that break a sync, if I add oslo=INFO into oslo-sync? | 21:15 |
| dhellmann | jogo: heh, yeah :-/ | 21:15 |
| sdague | jogo: also n-cpu dropped by half there | 21:15 |
| *** andreaf has joined #openstack-infra | 21:15 | |
| jogo | dhellmann: lolz | 21:16 |
| dhellmann | set_defaults() doesn't override this value, but I think we're doing it elsewhere | 21:16 |
| dhellmann | jogo: the problems we make for ourselves, right? | 21:17 |
| jogo | dhellmann: oh yeah | 21:17 |
| fungi | anteaya: do you happen to know whether http://lists.openstack.org/pipermail/openstack-infra/2014-March/001077.html came up for a vote in the neutron weekly meeting or something? | 21:17 |
| *** thuc has quit IRC | 21:18 | |
| jogo | dhellmann: heh yup just tried an update.sh and it broke things | 21:18 |
| jogo | I now have nova=INFO | 21:18 |
| dhellmann | jogo: yeah, I should have seen that one coming | 21:19 |
| jogo | so whats the right step forward for icehouse? | 21:19 |
| jogo | sdague dhellmann: | 21:19 |
| sdague | jogo: oslo.openstack=INFO ? | 21:20 |
| sdague | jogo: oslo.openstack=INFO, won't that translate well? | 21:21 |
| jogo | that would work for lockutils but not oslo.messaging | 21:21 |
| sdague | right, do them both | 21:21 |
| sdague | or, oslo -> nova ? | 21:21 |
| sdague | so we get nova.messaging ? | 21:21 |
| jogo | we can't add oslo=INFO because, sync turns that into nova | 21:21 |
| jogo | bingo | 21:21 |
| sdague | ffs | 21:21 |
| clarkb | wow crm114 really chews through CPU :) | 21:21 |
| * jogo hides while sdague rages | 21:21 | |
| clarkb | jeblair: I am just about done testing my crm114 change | 21:21 |
| sdague | honestly, fork log.py | 21:21 |
| dhellmann | jogo: there's a special dispensation for oslo.config in update.py (line 158) so we could add oslo.messaging handling there, then you could add a default for oslo.messaging and it should work | 21:22 |
| sdague | or that ^^^ | 21:22 |
| sdague | I am going to try not to rage quit over oslo sync | 21:22 |
| * dhellmann adds another step to https://wiki.openstack.org/wiki/Oslo/CreatingANewLibrary | 21:22 | |
| sdague | I'll save that for juno | 21:22 |
| *** e0ne has quit IRC | 21:23 | |
| dhellmann | sdague: I am, seriously, doing everything I can to move us to as many libraries as possible during juno. promise. | 21:23 |
| sdague | dhellmann: I know, it's not your fault | 21:23 |
| sdague | we have a bunch of debt to dig out of | 21:23 |
| *** dkranz has quit IRC | 21:23 | |
| sdague | just this config issue has driven me nuts for a while | 21:24 |
| dhellmann | sdague: yep, I'm just trying to cheer you up :-) | 21:24 |
| *** dizquierdo has joined #openstack-infra | 21:24 | |
| clarkb | jeblair: yup seems to work | 21:24 |
| dhellmann | fwiw, I'm going to need some input from the qa and infra teams about how to deal with releases of those libs during the cycle -- alphas, published where, etc. | 21:24 |
| *** alff has quit IRC | 21:25 | |
| jogo | dhellmann: that worked | 21:25 |
| dhellmann | jogo: \o/ | 21:25 |
| sdague | clarkb: so on crm114, this is a reason that I was suggesting just cut the line until the ] and be no more clever than that | 21:25 |
| clarkb | sdague: jogo jeblair https://review.openstack.org/#/c/82581/ is ready to review | 21:25 |
| clarkb | sdague: right but how do you make the rule specific enough | 21:25 |
| *** sandywalsh has joined #openstack-infra | 21:25 | |
| clarkb | sdague: cut until the ] is too generic | 21:26 |
| clarkb | sdague: and there are UUIDs everywhere | 21:26 |
| clarkb | sdague: routes.middleware [-] Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x2e4ac10>, 'project_id': u'e6372917a0b3465eb8a85b9733e656c5' for example | 21:26 |
| sdague | that's at INFO? | 21:26 |
| *** dprince has quit IRC | 21:26 | |
| clarkb | sdague: that particular example was a debug line /me filters down and looks for more examples | 21:27 |
| sdague | yeh, this was mostly about INFO | 21:27 |
| sdague | you are right, there are some UUIDs for sure there | 21:27 |
| clarkb | sdague: nova.osapi_compute.wsgi.server [req-d88d7c64-d8b0-4bad-9663-cc9258946b58 FlavorsExtraSpecsNegativeTestXML-185396789 FlavorsExtraSpecsNegativeTestXML-1496176899] 127.0.0.1 "PUT /v2/7d77450c960a46329c0c63d1ddc55244/flavors/1803403985/os-extra_specs/key1 HTTP/1.1 in the url | 21:27 |
| sdague | request urls for sure | 21:27 |
| sdague | yep | 21:27 |
| clarkb | so this way we catch all of them and replace them | 21:27 |
| sdague | cool | 21:27 |
| jogo | dhellmann sdague: https://review.openstack.org/82613 | 21:28 |
| sdague | jogo: beside the fact that it will fail pep8? | 21:28 |
| mattoliverau | Morning | 21:28 |
| jogo | sdague: fixed' | 21:28 |
| sdague | oslo.test a released library now as well? | 21:28 |
| *** rossella_ has joined #openstack-infra | 21:29 | |
| sdague | I'm trying to think of what else might need it | 21:29 |
| dhellmann | sdague: yes | 21:29 |
| dhellmann | sdague: although that's oslotest | 21:29 |
| dhellmann | and nothing in the incubator is using it, yet | 21:29 |
| sdague | are we going to make it osloconfig and oslomessaging in juno as well? | 21:29 |
| sdague | and solve our namespace problem | 21:29 |
| dhellmann | https://wiki.openstack.org/wiki/Oslo/CreatingANewLibrary#Avoid_Rewriting_References_in_Incubator.27s_update.py | 21:29 |
| dhellmann | sdague: no, because that would mask another issue | 21:30 |
| dhellmann | the real problem there is we have 2 versions of the library installed | 21:30 |
| dhellmann | we get a hard error because the namespace package breaks | 21:30 |
| dhellmann | without that, we would have weird "why is it using the wrong version of my code" errors | 21:30 |
| dhellmann | more subtle, and hard to track down | 21:30 |
| *** fifieldt has quit IRC | 21:30 | |
| *** e0ne has joined #openstack-infra | 21:30 | |
| *** jamielenz is now known as jamielennox | 21:33 | |
| *** HenryG has quit IRC | 21:33 | |
| *** miqui has quit IRC | 21:33 | |
| jeblair | clarkb: how do you tell testr to run a single scenario? | 21:34 |
| clarkb | jeblair: test run --parallel testnameregex | 21:34 |
| clarkb | in the case of testscenarios they create new tests with unique names, you should be able to pass a regex to match only the scenarios you want | 21:35 |
| *** julim has quit IRC | 21:35 | |
| jeblair | clarkb: okay, i'm guessing there's something funny going on with regex and the () in the name from scenarios | 21:36 |
| jeblair | i'll throw backslashes at the problem and see if it goes away | 21:36 |
| clarkb | jeblair: ok, let me know how it goes and if you give me an example I can try debugging too (assuming it persists) | 21:36 |
| *** amcrn has quit IRC | 21:36 | |
| *** dangers is now known as dangers_away | 21:38 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add check-rally-install-* template and jobs https://review.openstack.org/82617 | 21:38 |
| anteaya | fungi: I will get some info about http://lists.openstack.org/pipermail/openstack-infra/2014-March/001077.html | 21:39 |
| anteaya | fungi: I don't know anything off the top of my head | 21:39 |
| fungi | anteaya: okay, thanks. as usual i don't want to be the one making that decision because the outcome doesn't really affect me | 21:41 |
| jogo | clarkb: one more for you https://review.openstack.org/82618 | 21:42 |
| * anteaya nods | 21:42 | |
| *** mfer has quit IRC | 21:42 | |
| sdague | dhellmann: I thought it was venv vs. namespace vs. editable install | 21:43 |
| sdague | and the venv did the right thing if it wasn't namespace module | 21:43 |
| dhellmann | sdague: we've had cases where they were installed outside of a venv, too | 21:44 |
| sdague | because otherwise we'd hit issues with unit tests all the time | 21:44 |
| dhellmann | most of the unit tests configs don't allow site-packages, afaik | 21:44 |
| dhellmann | the *global* site-packages, that is | 21:44 |
| sdague | dhellmann: nova does | 21:44 |
| *** jpeeler has left #openstack-infra | 21:44 | |
| sdague | otherwise libvirt tests don't run | 21:44 |
| dhellmann | sdague: right, I said most don't | 21:44 |
| sdague | and this is definitely a rage quit issue for nova folks atm | 21:45 |
| sdague | because it means they actually often can't run unit tests on devstack nodes | 21:45 |
| sdague | in any way | 21:45 |
| dhellmann | sigh | 21:45 |
| *** mriedem has quit IRC | 21:46 | |
| *** fifieldt has joined #openstack-infra | 21:46 | |
| dhellmann | sdague: if a non-namespace version of a library ends up installed editable globally and from pypi locally, which copy is importable from the venv? | 21:47 |
| sdague | dude, I have no idea :) | 21:47 |
| *** nati_ueno has quit IRC | 21:47 | |
| sdague | this is all so far out of my depth | 21:47 |
| sdague | but at least it won't be | 21:47 |
| sdague | ERROR: oslo.config not found | 21:47 |
| dhellmann | because that's the bigger concern for me: tests that fail in subtle ways, rather than with this hard issue we know how to fix | 21:47 |
| sdague | and it's like ffs python, it's right there -> | 21:48 |
| dhellmann | right, it will be AttributeError: oslo.config.cfg.CONF has no attribute some_new_thing | 21:48 |
| *** nati_ueno has joined #openstack-infra | 21:48 | |
| sdague | dhellmann: so if you have a reproducable way to get us out of this instead | 21:48 |
| sdague | that's goodness | 21:48 |
| dhellmann | sdague: which global libraries does nova need? just libvirt? | 21:49 |
| sdague | right now, I think that's the killer one | 21:49 |
| dhellmann | ok, well, there are other ways to manipulate the import path to add that one package to it | 21:49 |
| dhellmann | perhaps we can update nova's tox.ini to do that, instead of allowing all globally-installed packages | 21:49 |
| *** thomasem has quit IRC | 21:50 | |
| dhellmann | sdague: https://bitbucket.org/dhellmann/virtualenvwrapper/src/4b2ed5cfa333e2f373868f33b7f287ab35d6a4d8/virtualenvwrapper.sh?at=master#cl-770 | 21:50 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add check-rally-install-* template and jobs https://review.openstack.org/82617 | 21:53 |
| *** alexpilotti has joined #openstack-infra | 21:54 | |
| *** mestery has quit IRC | 21:55 | |
| *** dims has quit IRC | 21:56 | |
| sdague | dhellmann: sure, that would make tons of people happy if it worked | 21:57 |
| sdague | like I said, any fix would be a win | 21:57 |
| sdague | I'm just super out of my depth in helping on this one | 21:57 |
| dhellmann | sdague: yeah; and like I said, I'm worried the "don't use namespace packages" fix is just going to lead to other issues that are harder to track down -- I'll think about this more | 21:58 |
| sdague | yep, no worries | 21:58 |
| sdague | it was an unrelated rant | 21:59 |
| dhellmann | sdague: it looks like the approach taken in add2virtualenv doesn't work, because the argument has to be the parent dir of the package :-/ | 21:59 |
| dhellmann | so, dist-packages, instead of dist-packages/libvirt | 21:59 |
| *** _ruhe_ has quit IRC | 21:59 | |
| hogepodge | Hm. It looks like the gates for our proejct broke because rake now needs ruby >= 1.9 | 22:02 |
| *** khyati has quit IRC | 22:03 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1295876 https://review.openstack.org/82245 | 22:03 |
| uvirtbot | Launchpad bug 1295876 in libvirt "libvirtError: internal error unable to add domain xxx to cgroup: No space left on device" [Undecided,New] https://launchpad.net/bugs/1295876 | 22:03 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gearman-plugin: change logging level from info to debug https://review.openstack.org/82627 | 22:04 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add check-rally-install-* template and jobs https://review.openstack.org/82617 | 22:04 |
| *** mestery has joined #openstack-infra | 22:04 | |
| *** mestery has quit IRC | 22:06 | |
| *** che-arne has joined #openstack-infra | 22:07 | |
| *** salv-orlando has quit IRC | 22:09 | |
| anteaya | fungi: markmcclain will respond to the radware email one way or another | 22:12 |
| fungi | anteaya: great--thanks | 22:12 |
| anteaya | np | 22:12 |
| *** sandywalsh has quit IRC | 22:13 | |
| *** dims_ has joined #openstack-infra | 22:13 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add requirements project config file https://review.openstack.org/82219 | 22:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add tests for the allocator https://review.openstack.org/82628 | 22:16 |
| jeblair | nodepool's first tests! <sniff> :) | 22:16 |
| clarkb | it is a grown up project now | 22:17 |
| anteaya | congratulations | 22:17 |
| *** Sukhdev has quit IRC | 22:18 | |
| fungi | hogepodge: we use rake to run puppet lint for changes to openstack-infra/config... should we expect it to be failing as well now? | 22:20 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Don't treat IDs as uniquely special in CRM114 https://review.openstack.org/82581 | 22:20 |
| clarkb | sdague: jeblair jogo ^ I think I am actually happy with that | 22:20 |
| mgagne | fungi: if you use Gemfile and install rake from there, yes | 22:20 |
| clarkb | latest patchset will get all TOKENS on a line, first one would only replace the first | 22:20 |
| hogepodge | fungi that depends. If you’re using < Ruby 1.9 you’ll see failures | 22:20 |
| fungi | mgagne: oh, i think we might use ubuntu's packaged rake instead... checking | 22:20 |
| *** jhesketh_ has joined #openstack-infra | 22:20 | |
| mgagne | fungi: I think that you are using system packages only, not Gemfile | 22:21 |
| jogo | clarkb: re we using CRM114 for anything right now | 22:22 |
| clarkb | jogo: yes, all log files are piped through it | 22:22 |
| jhesketh_ | Morning | 22:22 |
| clarkb | jogo: and we index the results | 22:22 |
| fungi | mgagne: yep, looks that way... okay | 22:22 |
| clarkb | jogo: it has a bug where it falls over in soem cases on lines with those UUIDs, theory is the statistics file is keeping track of too much unique data and flattening it like that will make it better | 22:23 |
| *** markmcclain has quit IRC | 22:23 | |
| clarkb | jogo: more accurate and less falling over | 22:23 |
| jogo | ahh error_pr | 22:23 |
| clarkb | jhesketh_: ohai | 22:23 |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Allow the neutron functional job to vote https://review.openstack.org/82631 | 22:23 |
| *** jhesketh has joined #openstack-infra | 22:24 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Run all python jobs for nodepool https://review.openstack.org/82632 | 22:24 |
| jogo | clarkb: what do I do with something like this: | 22:24 |
| jogo | -1.1314,-0.2453,-0.0786,0.0239,-0.0789,-0.0106,-0.0764,-0.0825,-0.0795,0.0218,-0.0531,-0.1149,-0.0419,-0.0175,-0.0534,0.1464,-0.0138,0.103,0.0092,-0.0189,0.0098,-0.0909,-0.005,-0.1173,0.071,-0.065,0.0368,-0.118,-0.0378,-0.0072,-0.0487,-0.0407,0.0023,0.137,-0.0555,0.151,0.0588,-0.0772,0.0522,-0.0975,0.0599,-0.0755,1.2372,-0.2477,-0.163,0.0231,-0.0034,0.0288,-0.0224,0.0248,-0.1164,0.0307,-0.0199,0.3081,-0.117,0.4014,-0.091,-0.1631,1.2273 | 22:24 |
| jogo | error_pr for a stacktrace | 22:25 |
| jeblair | clarkb: (also, uuids defeate the algorithm and make it less effective) | 22:25 |
| clarkb | jogo: you can do searches like error_pr:[-1000.0 TO -10.0] | 22:27 |
| *** khyati has joined #openstack-infra | 22:27 | |
| clarkb | jogo: those are the lines that are identified as being "bad" | 22:27 |
| jogo | ohh negative are more likly to be errors? | 22:28 |
| jeblair | clarkb: that seems strange to duplicate the match/alter lines, why can't you just have the inner loop | 22:28 |
| jeblair | ? | 22:28 |
| jeblair | clarkb, jogo: also, before we get too far, should we reverse that? | 22:28 |
| jogo | jeblair: I think so, | 22:29 |
| jeblair | clarkb, jogo: really the pR that it's an error should be a positive value, so we should either invert the sign, or change the column name to "not_an_error_pr" or equiv | 22:29 |
| clarkb | jeblair: you can if you don't want the <fromnext> I am optimizing, potentially prematurely | 22:29 |
| jeblair | clarkb: oh fromnext, didn't see that | 22:30 |
| clarkb | jeblair: the first match starts from beginning of line, subsequent ones start after the last match | 22:30 |
| jogo | heh: Build step 'Execute shell' marked build as failure | 22:30 |
| jogo | has a pr of -3 | 22:30 |
| *** jeckersb is now known as jeckersb_gone | 22:30 | |
| *** maxbit has joined #openstack-infra | 22:31 | |
| anteaya | fungi: UtahDave is educating me on this etherpad: https://etherpad.openstack.org/p/OVMqR5SOHq | 22:31 |
| anteaya | do join us | 22:31 |
| anteaya | and anyone else is welcome too | 22:31 |
| clarkb | jeblair: the changes in https://review.openstack.org/#/c/82628/1/nodepool/allocation.py are making my brain melt. won't that cause all allocations for particular images to go to one provider when we are not busy as grant will be the min value? | 22:32 |
| *** signed8bit has quit IRC | 22:33 | |
| *** andreaf has quit IRC | 22:33 | |
| *** thedodd has quit IRC | 22:34 | |
| jeblair | clarkb: so each time you add a provider to a request, the allocator redistributes the request across all providers | 22:35 |
| *** prad has quit IRC | 22:35 | |
| jeblair | clarkb: so if we're requesting 4 nodes from 2 providers, the actuall requests it's working with are 2 nodes from each | 22:35 |
| clarkb | jeblair: the ratio calculation in that diff is used for resource constrained situations? | 22:36 |
| jeblair | clarkb: the bug was that when granting those requests, it was shrinking them even more based on sibling requests which may not have even been an issue | 22:36 |
| jeblair | clarkb: yeah, it _should_ be used when constrained, but was actually used all the time | 22:36 |
| jeblair | clarkb: so we ended up always allocating half of the first request when there were two requests | 22:37 |
| jeblair | clarkb: instead, what we want to do is give the first request _up to_ half of what's available | 22:37 |
| jeblair | clarkb: (so the second request can have up to the second half) | 22:37 |
| clarkb | jeblair: gotcha | 22:38 |
| jeblair | clarkb: and yeah, it's kinda brain melty, thus the unit tests :) they do test the scenario you were worried about | 22:39 |
| clarkb | jeblair: great, I will keep reading then | 22:39 |
| *** gyee has quit IRC | 22:40 | |
| *** e0ne has quit IRC | 22:41 | |
| hogepodge | Pinning in my Gemfile fixed the gates, so definitly ‘our’ problem. | 22:41 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Run all python jobs for nodepool https://review.openstack.org/82632 | 22:42 |
| *** thuc_ has quit IRC | 22:42 | |
| *** afazekas has quit IRC | 22:42 | |
| *** thuc has joined #openstack-infra | 22:43 | |
| nibalizer | jeblair: clarkb want to see why the puppet agent isnt running on puppetdb.o.o http://puppetdb.openstack.org/reports/puppetdb.openstack.org | 22:43 |
| jeblair | nibalizer: ack will look | 22:44 |
| * clarkb puts money on mordred stopping it to debug over weekend | 22:44 | |
| *** zns has quit IRC | 22:46 | |
| openstackgerrit | Gregory Haynes proposed a change to openstack-infra/pypi-mirror: Add blurb on configuration and run_mirror in README https://review.openstack.org/80861 | 22:46 |
| jeblair | nibalizer: my guess is that it was never started | 22:47 |
| *** thuc has quit IRC | 22:47 | |
| jeblair | nibalizer: it _is_ configured to start on boot, but i'm guessing the host has never been rebooted, and no one started it manually | 22:47 |
| jeblair | at least, that's what it looks like from the logs | 22:48 |
| JayF | The jenkins servers have both self-signed and expired certs (see: https://jenkins06.openstack.org/job/check-tempest-dsvm-ironic/186/). Is it valuable to file a bug for this? I mean, self-signed is OK, but self-signed and expired just seems a little strange is all. | 22:48 |
| jeblair | anyone know of a reason it shouldn't be started? | 22:48 |
| *** CaptTofu has joined #openstack-infra | 22:48 | |
| clarkb | jeblair: run puppet agent --test --noop and see what the delta looks like? | 22:48 |
| openstackgerrit | Gregory Haynes proposed a change to openstack-infra/pypi-mirror: Add blurb on configuration and run_mirror in README https://review.openstack.org/80861 | 22:49 |
| clarkb | JayF: I think we are just using the ubuntu provided snake oil certs | 22:49 |
| clarkb | JayF: for simplicity | 22:49 |
| JayF | clarkb: ah, so it just means that machine is older than a year :) | 22:49 |
| clarkb | no looks like the snake oil certs may only be good for a motnh | 22:49 |
| JayF | oooh. | 22:49 |
| JayF | Like a failed cert is a failed cert, it was just strange enough to see a self-signed+expired warning that I thought it bore mentioning | 22:50 |
| jeblair | nibalizer, clarkb: http://paste.openstack.org/show/74201/ | 22:50 |
| JayF | Is there a reason there's not a *.openstack.org cert tossed on those boxes? Just money? Or management? | 22:50 |
| blkperl | jeblair: nice :) | 22:51 |
| clarkb | we don't have a wildcard cert and those machiens were deemed to not need a valid cert | 22:51 |
| clarkb | very few people actually look at jenkins anymore | 22:51 |
| nibalizer | jeblair: thats on puppetdb.o.o and not ci-puppetmaster.o.o ? | 22:51 |
| jeblair | nibalizer: correct | 22:51 |
| nibalizer | thats fine to be on there | 22:51 |
| clarkb | nibalizer: jeblair: I think the puppet.conf file is the same everywhere and puppetdb is just lagging | 22:52 |
| clarkb | jeblair: should be ok to restart puppet on that node if that was the only diff | 22:52 |
| openstackgerrit | Gregory Haynes proposed a change to openstack-infra/pypi-mirror: Use openstack git urls instead of github https://review.openstack.org/82649 | 22:52 |
| nibalizer | yep | 22:52 |
| jeblair | clarkb: only relevant one. started. | 22:52 |
| *** david-lyle has quit IRC | 22:52 | |
| jogo | wow in a 24 hour period we run 600 to 900 copies of cehck-temepst-dsvm-full | 22:52 |
| clarkb | jogo: is that all? | 22:54 |
| *** afazekas has joined #openstack-infra | 22:54 | |
| jogo | clarkb: that is only that one job | 22:55 |
| clarkb | jogo: during a busy week (feature freeze time, we run ~25k tests a day half of which are tempest give or take) | 22:55 |
| clarkb | jeblair: reviewed the nodepool tests change. | 22:55 |
| jogo | clarkb: http://tinyurl.com/mvv7lwc | 22:55 |
| jogo | clarkb: that is only for the one job | 22:55 |
| jogo | so figure that corrilates closely to number of patches that go through check queue that have an inegrated check queue test | 22:56 |
| clarkb | oh and its check only | 22:56 |
| jogo | clarkb: yeah was trying to get numbers for tripleo to grok | 22:56 |
| *** rhsu1 has joined #openstack-infra | 22:58 | |
| *** CaptTofu has quit IRC | 22:58 | |
| *** CaptTofu has joined #openstack-infra | 22:59 | |
| *** salv-orlando has joined #openstack-infra | 23:00 | |
| *** rhsu has quit IRC | 23:00 | |
| *** salv-orlando_ has joined #openstack-infra | 23:01 | |
| fungi | note to self... stop trying to browse a requirements integration job log on a netbook with only 0.5gib ram | 23:01 |
| clarkb | fungi: gg | 23:02 |
| *** gokrokve has joined #openstack-infra | 23:02 | |
| anteaya | ha ha ha | 23:03 |
| sdague | jogo: 6 - 9 00 sounds right, a little higher than I remember, but there are a bunch of new jobs | 23:03 |
| sdague | I noticed this morning, devstack is running 16 d-g jobs on every check change | 23:03 |
| *** CaptTofu has quit IRC | 23:03 | |
| clarkb | sdague: yeah d-g and devstack both have quite a few check jobs | 23:03 |
| sdague | yeh, the 3 ironic and savana ones that are nv are kind of brutal | 23:04 |
| *** maxbit has quit IRC | 23:04 | |
| *** maxbit has joined #openstack-infra | 23:04 | |
| sdague | the savana ones should be able to largely go away | 23:04 |
| jogo | I wonder how much infra tripleo is gonna need to keep up with check | 23:04 |
| anteaya | fungi: UtahDave gave me a place to begin, so I will start with what he outlined and then follow up with questions | 23:04 |
| sdague | once we get past release | 23:04 |
| *** salv-orlando has quit IRC | 23:04 | |
| *** salv-orlando_ is now known as salv-orlando | 23:04 | |
| fungi | d-g has the issue of being branchless, so needs to be tested against stable branches of everything too | 23:04 |
| sdague | right, d-g I understand it more | 23:05 |
| fungi | anteaya: sounds good--looked like it might be a good start anyway | 23:05 |
| anteaya | and you are back on the etherpad again | 23:05 |
| sdague | I just hadn't counted the jobs on devstack until this morning | 23:05 |
| anteaya | great | 23:05 |
| anteaya | yay for a good direction | 23:05 |
| fungi | anteaya: yeah, i stopped trying to look at job logs from here | 23:05 |
| anteaya | ha ha ha | 23:05 |
| *** maxbit has quit IRC | 23:05 | |
| jogo | sdague: 600-900 was on'y check-dsvm-tempest-full' | 23:06 |
| jogo | and thats it | 23:06 |
| *** maxbit has joined #openstack-infra | 23:06 | |
| *** CaptTofu has joined #openstack-infra | 23:07 | |
| *** ArxCruz_ has joined #openstack-infra | 23:07 | |
| *** ArxCruz has quit IRC | 23:07 | |
| *** rcleere has quit IRC | 23:08 | |
| clarkb | jeblair: https://review.openstack.org/#/c/82632/ reviewed but not approved (comment explains why) | 23:08 |
| jeblair | clarkb: yeah, i didn't want to change the allocator from under the multinode change (which also changes the allocator) | 23:09 |
| jeblair | i actually spotted the bug while developing multinode, but decided to push it off until after | 23:10 |
| clarkb | jeblair: agreed | 23:10 |
| clarkb | jeblair: fungi https://review.openstack.org/#/c/82631/ now that we sort of gate on check tests, would it be inappropriate to -1 that change because it isn't gating on that test? | 23:10 |
| clarkb | imo if we are running check tests and voting on them then we should gate on them too | 23:10 |
| jeblair | clarkb: yes, and in fact because of the hole in "require check for gate" logic, we _need_ to do that | 23:11 |
| openstackgerrit | Jamie Lennox proposed a change to openstack-infra/config: Add kite notifications to the barbican channel https://review.openstack.org/82660 | 23:11 |
| sdague | jogo: yeh, check-dsvm-tempest-full is a pretty good proxy for "all jobs" | 23:11 |
| jeblair | clarkb: otherwise we can wedge the gate | 23:11 |
| *** aysyd has quit IRC | 23:11 | |
| clarkb | oh right | 23:12 |
| jogo | sdague: yeah that was my thinking although I wonder how many stackforge projects use it | 23:12 |
| sdague | jogo: it's under a different name | 23:12 |
| jogo | sdague: oh even better | 23:13 |
| sdague | jeblair: the neutron job won't wedge the gate if the neutron functional tests aren't gated on | 23:13 |
| sdague | it will just wedge neutron | 23:13 |
| sdague | right? | 23:13 |
| sdague | or am I missing a piece of logic | 23:13 |
| *** salv-orlando_ has joined #openstack-infra | 23:14 | |
| nibalizer | zaro: you're running puppet actively on zuul-dev.o.o ? | 23:14 |
| nibalizer | it should be appearing with its stuff in puppetboard | 23:14 |
| jeblair | sdague: what you say is correct, but for neutron, that's "the gate" | 23:14 |
| clarkb | nibalizer: I don't think it is | 23:15 |
| clarkb | I am probably to blame for that, will look into it now | 23:15 |
| *** dangers_away is now known as dangers | 23:15 | |
| *** SumitNaiksatam has quit IRC | 23:15 | |
| sdague | jeblair: right, but that's fixable completely by neutron cores | 23:15 |
| sdague | it's not like they are broken by something else | 23:15 |
| sdague | I'm actually less comfortable with them putting this in the gate, given that it's depending on unstable tempest interfaces. | 23:16 |
| sdague | which actually makes me think it probably shouldn't vote at all | 23:16 |
| *** dcramer_ has quit IRC | 23:16 | |
| jeblair | sdague: correct, however, i think it is undesirable for them to be able to accidentally break their own gate due to a missing piece of logic in zuul, so i believe that if they are voting on it in check, it should be voting in gate for them | 23:16 |
| *** salv-orlando has quit IRC | 23:17 | |
| *** salv-orlando_ is now known as salv-orlando | 23:17 | |
| clarkb | ok I am restarting puppet on zuul-dev, the diff shows the unbound changes and the gate-noop to noop zuul layout change | 23:18 |
| sdague | so I definitely don't think that something which depends on tempest unstable interfaces should be able to cause resets for everyone else in the gate | 23:18 |
| clarkb | it looked like puppet agent had been stuck since february 25th | 23:18 |
| sdague | so maybe we just say it has to stay non voting | 23:18 |
| *** hashar has quit IRC | 23:19 | |
| fungi | clarkb: github remotes for vcsrepos in /opt maybe? | 23:19 |
| fungi | clarkb: i've been cleaning those up as i stumble across them | 23:19 |
| fungi | (just rm -fr the repo and let vcsrepo reclone it from our git servers on the next puppet run) | 23:20 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add debugging metrics to RPC https://review.openstack.org/78813 | 23:20 |
| clarkb | fungi: oh thanks, there is indeed a hung git process | 23:20 |
| *** dizquierdo has quit IRC | 23:21 | |
| *** maxbit has quit IRC | 23:21 | |
| * clarkb reruns puppet | 23:21 | |
| clarkb | it recloned using git.o.o as origin | 23:22 |
| jogo | sdague clarkb: this turn off debug logs for oslo.messaging is shrinking a lot of logs | 23:22 |
| clarkb | jeblair: zaro I think zuul on that host may need to be hit with a hammer to make the zuul merger stuff work right | 23:22 |
| sdague | jogo: yeh, I honestly think oslo.messaging was the thing that caused the explosion | 23:23 |
| sdague | the dates line up right | 23:23 |
| jogo | heh | 23:24 |
| *** CaptTofu has quit IRC | 23:25 | |
| *** CaptTofu has joined #openstack-infra | 23:25 | |
| jogo | ceilo logs are really bad still | 23:26 |
| jogo | full of /opt/stack/new/ceilometer/ceilometer/openstack/common/db/sqlalchemy/session.py:526: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 | 23:27 |
| jeblair | clarkb: i did actually run the tests on https://review.openstack.org/#/c/82628/ and they pass. :) | 23:27 |
| clarkb | jeblair: huh, I really didn't expect that | 23:27 |
| *** amcrn has joined #openstack-infra | 23:27 | |
| * clarkb runs them | 23:27 | |
| *** bknudson has quit IRC | 23:28 | |
| jeblair | clarkb: what you describe was the case if the class is in test_allocator.py, so i moved it to __init__ and it did what i wanted | 23:28 |
| clarkb | jeblair: interesting and good to know thanks | 23:29 |
| clarkb | I will update my review shortly | 23:29 |
| jeblair | clarkb: if that's a bug, don't tell anyone, i think it's good behavior. :) | 23:29 |
| *** CaptTofu has quit IRC | 23:30 | |
| clarkb | sure enough it doesn't run them out of __init__.py | 23:30 |
| *** homeless has quit IRC | 23:35 | |
| clarkb | zaro: jeblair https://review.openstack.org/#/c/82627/1 is that change being made after operational observations? | 23:35 |
| *** wenlock has quit IRC | 23:36 | |
| clarkb | I think zaro is commuting right now so may not be able to answer, but a bit more background would be handy | 23:36 |
| sdague | jogo: the tempest logs should be about 60% smaller now | 23:36 |
| jogo | nice | 23:37 |
| jogo | I saw you had a flurry of patches doing things there | 23:38 |
| *** salv-orlando has quit IRC | 23:38 | |
| *** yamahata has quit IRC | 23:39 | |
| *** afazekas has quit IRC | 23:40 | |
| clarkb | jeblair: http://docs.python.org/2/library/unittest.html#unittest.TestLoader.discover looks like that behavior is intentional. you need to explicitly load tests in __init__.py | 23:41 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Shorten update spinner fadeout on status page https://review.openstack.org/82019 | 23:42 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to status page to collapse changesets https://review.openstack.org/82018 | 23:42 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add expand by default option to status page https://review.openstack.org/82290 | 23:42 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add a total progress bar to each change https://review.openstack.org/82017 | 23:42 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Format change headers like infra's status page https://review.openstack.org/82016 | 23:42 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add a filter box to status.html https://review.openstack.org/82279 | 23:42 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Build status page more jquery like https://review.openstack.org/81717 | 23:42 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Update zuul's status page to bootstrap 3.1.1 https://review.openstack.org/82013 | 23:42 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add demo support for openstack's live zuul feed https://review.openstack.org/82014 | 23:42 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add in progress bars for jobs https://review.openstack.org/82015 | 23:42 |
| *** thuc has joined #openstack-infra | 23:43 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 23:46 |
| *** atiwari has quit IRC | 23:46 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client https://review.openstack.org/68828 | 23:47 |
| *** gokrokve has quit IRC | 23:48 | |
| morganfainberg | clarkb, removing a bunch of keystone log lines from testing https://review.openstack.org/#/c/82674/ | 23:49 |
| clarkb | morganfainberg: great thanks (keep in mind I think we are happy to keep verbose logs if they are helpful, but would like to see unneeded verbosity trimmed) | 23:51 |
| *** mriedem has joined #openstack-infra | 23:52 | |
| clarkb | jeblair: fungi mordred heads up I just responeded to the hpcloud dnssec thread | 23:55 |
| clarkb | hopefully we can get a proper answer | 23:55 |
| fungi | clarkb: thanks! | 23:56 |
| clarkb | I also just approved a change to stop recording #savanna, I checked the wiki meeting schedule page and no tuesday 0000 meetings are scheduled | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
