| dstufft | hm strange | 00:00 |
|---|---|---|
| dstufft | I'm not entirely sure that we preserve ordering in that | 00:01 |
| mordred | I believe you are correct | 00:01 |
| *** e0ne has joined #openstack-infra | 00:01 | |
| *** dcramer_ has quit IRC | 00:01 | |
| *** e0ne_ has joined #openstack-infra | 00:03 | |
| *** e0ne has quit IRC | 00:03 | |
| *** dolphm has quit IRC | 00:06 | |
| *** e0ne_ has quit IRC | 00:08 | |
| *** rlandy has quit IRC | 00:08 | |
| *** dolphm has joined #openstack-infra | 00:09 | |
| *** dstanek has joined #openstack-infra | 00:13 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Support Keystone V3 based gate https://review.openstack.org/79212 | 00:15 |
| *** jamielennox|away is now known as jamielennox | 00:17 | |
| *** yamahata has quit IRC | 00:23 | |
| *** matsuhashi has joined #openstack-infra | 00:23 | |
| *** dizquierdo has joined #openstack-infra | 00:40 | |
| *** alexpilotti has quit IRC | 00:46 | |
| *** mriedem has quit IRC | 00:48 | |
| *** Alexandra is now known as alex-TOTHECOLESD | 01:01 | |
| *** e0ne has joined #openstack-infra | 01:01 | |
| *** alex-TOTHECOLESD is now known as alex-TOCOLES | 01:01 | |
| *** enikanorov_ has joined #openstack-infra | 01:04 | |
| *** HenryG_ has joined #openstack-infra | 01:05 | |
| *** KurtMartin has joined #openstack-infra | 01:05 | |
| *** hogepodge_ has joined #openstack-infra | 01:07 | |
| *** jeblair_ has joined #openstack-infra | 01:08 | |
| *** Hefeweiz1n has joined #openstack-infra | 01:09 | |
| *** kiall_ has joined #openstack-infra | 01:10 | |
| *** esmute_ has joined #openstack-infra | 01:12 | |
| *** gaelL_ has joined #openstack-infra | 01:13 | |
| *** ruhe2 has joined #openstack-infra | 01:13 | |
| *** cody-somerville has quit IRC | 01:13 | |
| *** hogepodge has quit IRC | 01:13 | |
| *** adalbas has quit IRC | 01:13 | |
| *** zul has quit IRC | 01:13 | |
| *** dims has quit IRC | 01:13 | |
| *** sandywalsh has quit IRC | 01:13 | |
| *** krtaylor has quit IRC | 01:13 | |
| *** HenryG has quit IRC | 01:13 | |
| *** esmute has quit IRC | 01:13 | |
| *** jeblair has quit IRC | 01:13 | |
| *** pfallenop has quit IRC | 01:13 | |
| *** gaelL has quit IRC | 01:13 | |
| *** nprivalova has quit IRC | 01:13 | |
| *** ruhe has quit IRC | 01:13 | |
| *** DinaBelova has quit IRC | 01:13 | |
| *** Daviey has quit IRC | 01:13 | |
| *** e0ne has quit IRC | 01:13 | |
| *** che-arne has quit IRC | 01:13 | |
| *** kiall has quit IRC | 01:13 | |
| *** aglarendil has quit IRC | 01:13 | |
| *** jgrimm has quit IRC | 01:13 | |
| *** ogelbukh has quit IRC | 01:13 | |
| *** katyafervent has quit IRC | 01:13 | |
| *** enikanorov has quit IRC | 01:13 | |
| *** kmartin has quit IRC | 01:13 | |
| *** Hefeweizen has quit IRC | 01:13 | |
| *** gmoro has quit IRC | 01:13 | |
| *** NikitaKonovalov has quit IRC | 01:13 | |
| *** hogepodge_ is now known as hogepodge | 01:13 | |
| *** esmute_ is now known as esmute | 01:13 | |
| *** ruhe2 is now known as ruhe | 01:13 | |
| *** Daviey_ has joined #openstack-infra | 01:13 | |
| *** Daviey_ is now known as Daviey | 01:13 | |
| *** katyafervent has joined #openstack-infra | 01:14 | |
| *** DinaBelova has joined #openstack-infra | 01:14 | |
| *** pfallenop has joined #openstack-infra | 01:14 | |
| *** pfallenop has quit IRC | 01:14 | |
| *** pfallenop has joined #openstack-infra | 01:14 | |
| *** fifieldt has joined #openstack-infra | 01:17 | |
| *** gmoro has joined #openstack-infra | 01:19 | |
| *** ogelbukh has joined #openstack-infra | 01:19 | |
| *** adalbas has joined #openstack-infra | 01:19 | |
| *** cody-somerville has joined #openstack-infra | 01:20 | |
| *** aglarendil has joined #openstack-infra | 01:20 | |
| *** dims has joined #openstack-infra | 01:20 | |
| *** yamahata has joined #openstack-infra | 01:21 | |
| *** jgrimm has joined #openstack-infra | 01:21 | |
| *** che-arne has joined #openstack-infra | 01:21 | |
| *** zul has joined #openstack-infra | 01:21 | |
| *** nprivalova has joined #openstack-infra | 01:21 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Document zuul generated log paths https://review.openstack.org/84040 | 01:24 |
| *** matsuhashi has quit IRC | 01:24 | |
| fifieldt | morning | 01:26 |
| *** matsuhashi has joined #openstack-infra | 01:26 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client https://review.openstack.org/68828 | 01:26 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add debugging metrics to RPC https://review.openstack.org/78813 | 01:31 |
| *** nosnos has joined #openstack-infra | 01:31 | |
| *** krtaylor has joined #openstack-infra | 01:31 | |
| *** sandywalsh has joined #openstack-infra | 01:31 | |
| *** NikitaKonovalov has joined #openstack-infra | 01:31 | |
| *** zehicle has joined #openstack-infra | 01:38 | |
| *** alex-TOCOLES is now known as Alexandra | 01:40 | |
| *** yaguang has joined #openstack-infra | 01:40 | |
| *** zehicle_at_dell has quit IRC | 01:40 | |
| *** dizquierdo has quit IRC | 01:41 | |
| *** saschpe has quit IRC | 01:44 | |
| *** gothicmindfood has left #openstack-infra | 01:45 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Remove duplicate git setup in tests. https://review.openstack.org/80448 | 01:45 |
| *** gothicmindfood has joined #openstack-infra | 01:45 | |
| *** saschpe has joined #openstack-infra | 01:46 | |
| *** thuc has joined #openstack-infra | 01:49 | |
| *** sdake has quit IRC | 01:54 | |
| *** thuc has quit IRC | 01:54 | |
| *** wenlock has joined #openstack-infra | 01:55 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Add support for nested requirements files https://review.openstack.org/83149 | 01:56 |
| *** timrc has joined #openstack-infra | 01:56 | |
| *** alexandra_ has joined #openstack-infra | 01:57 | |
| *** Daviey has quit IRC | 02:00 | |
| *** Alexandra has quit IRC | 02:00 | |
| *** alexandra_ is now known as Alexandra | 02:01 | |
| *** e0ne has joined #openstack-infra | 02:01 | |
| *** yjiang has joined #openstack-infra | 02:04 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add debugging metrics to RPC https://review.openstack.org/78813 | 02:06 |
| *** e0ne has quit IRC | 02:06 | |
| *** Ryan_Lane has quit IRC | 02:15 | |
| *** rlandy has joined #openstack-infra | 02:37 | |
| *** rlandy has quit IRC | 02:37 | |
| *** sdake has joined #openstack-infra | 02:45 | |
| *** Ryan_Lane has joined #openstack-infra | 02:46 | |
| StevenK | Can someone please kick godb from #openstack-swift? I'm happy to provide reasoning. | 02:50 |
| *** yjiang has quit IRC | 02:50 | |
| fifieldt | right | 02:52 |
| fifieldt | moment | 02:52 |
| *** yjiang|2 has joined #openstack-infra | 02:52 | |
| fifieldt | argh, looks like my patch to get op didn't go through yet | 02:52 |
| *** jhesketh_ has quit IRC | 02:53 | |
| fifieldt | jeblair_, jeremyb - IRCop needed in #openstack-swift | 02:54 |
| *** Ryan_Lane has quit IRC | 02:55 | |
| *** jhesketh_ has joined #openstack-infra | 02:55 | |
| jhesketh | mordred: ping | 02:57 |
| *** khyati has joined #openstack-infra | 02:59 | |
| *** jhesketh_ has quit IRC | 02:59 | |
| notmyname | fifieldt: StevenK: just got online. on it | 03:00 |
| *** Ryan_Lane has joined #openstack-infra | 03:00 | |
| *** e0ne has joined #openstack-infra | 03:01 | |
| *** alff has joined #openstack-infra | 03:01 | |
| *** alff_ has joined #openstack-infra | 03:01 | |
| *** khyati has quit IRC | 03:02 | |
| *** e0ne has quit IRC | 03:03 | |
| *** Ryan_Lane has quit IRC | 03:05 | |
| *** lcheng has joined #openstack-infra | 03:10 | |
| *** gtt116__ has quit IRC | 03:16 | |
| *** matsuhashi has quit IRC | 03:19 | |
| *** hogepodge has quit IRC | 03:27 | |
| *** nosnos has quit IRC | 03:34 | |
| *** Alexandra_ has joined #openstack-infra | 03:39 | |
| *** mihgen has joined #openstack-infra | 03:41 | |
| *** vogxn has joined #openstack-infra | 03:46 | |
| *** Ryan_Lane has joined #openstack-infra | 03:47 | |
| cody-somerville | What sort of machine does http://static.openstack.org/ run on? | 03:50 |
| *** thuc has joined #openstack-infra | 03:50 | |
| *** vogxn has quit IRC | 03:51 | |
| mordred | cody-somerville: it's an 8G instance on rackspace cloud | 03:54 |
| *** jhesketh_ has joined #openstack-infra | 03:55 | |
| *** hogepodge has joined #openstack-infra | 03:55 | |
| *** markwash has joined #openstack-infra | 03:56 | |
| *** lcheng has quit IRC | 03:57 | |
| *** matsuhashi has joined #openstack-infra | 03:57 | |
| *** mgagne has joined #openstack-infra | 03:59 | |
| *** adalbas has quit IRC | 03:59 | |
| *** hogepodge has quit IRC | 04:00 | |
| *** e0ne has joined #openstack-infra | 04:01 | |
| *** nosnos has joined #openstack-infra | 04:02 | |
| *** thuc has quit IRC | 04:02 | |
| *** thuc has joined #openstack-infra | 04:02 | |
| *** e0ne has quit IRC | 04:06 | |
| *** thuc has quit IRC | 04:07 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:08 | |
| *** adalbas has joined #openstack-infra | 04:12 | |
| *** alff_ has quit IRC | 04:13 | |
| *** alff has quit IRC | 04:13 | |
| *** dstanek has quit IRC | 04:16 | |
| *** jepoy has quit IRC | 04:17 | |
| *** jepoy has joined #openstack-infra | 04:18 | |
| *** dstanek has joined #openstack-infra | 04:18 | |
| *** lcheng has joined #openstack-infra | 04:19 | |
| *** jepoy_ has joined #openstack-infra | 04:21 | |
| *** jepoy has quit IRC | 04:22 | |
| *** thuc has joined #openstack-infra | 04:23 | |
| *** nati_ueno has joined #openstack-infra | 04:23 | |
| *** nati_ueno has quit IRC | 04:23 | |
| *** marun has joined #openstack-infra | 04:27 | |
| *** marun is now known as marun_afk | 04:27 | |
| *** thuc has quit IRC | 04:28 | |
| *** thuc has joined #openstack-infra | 04:29 | |
| *** thuc has quit IRC | 04:33 | |
| *** mgagne has quit IRC | 04:36 | |
| *** NikitaKonovalov2 has joined #openstack-infra | 04:38 | |
| *** matsuhashi has quit IRC | 04:39 | |
| *** marun_afk has quit IRC | 04:39 | |
| *** adalbas has quit IRC | 04:39 | |
| *** jhesketh_ has quit IRC | 04:39 | |
| *** krtaylor has quit IRC | 04:39 | |
| *** sandywalsh has quit IRC | 04:39 | |
| *** NikitaKonovalov has quit IRC | 04:39 | |
| *** sdake has quit IRC | 04:39 | |
| *** yamahata has quit IRC | 04:39 | |
| *** NikitaKonovalov2 is now known as NikitaKonovalov | 04:39 | |
| *** gokrokve has joined #openstack-infra | 04:41 | |
| *** matsuhashi has joined #openstack-infra | 04:43 | |
| *** sdake has joined #openstack-infra | 04:45 | |
| *** adalbas has joined #openstack-infra | 04:46 | |
| *** marun has joined #openstack-infra | 04:46 | |
| *** yamahata has joined #openstack-infra | 04:46 | |
| *** jhesketh_ has joined #openstack-infra | 04:46 | |
| *** krtaylor has joined #openstack-infra | 04:47 | |
| *** sandywalsh has joined #openstack-infra | 04:51 | |
| *** gokrokve has quit IRC | 04:52 | |
| *** gokrokve has joined #openstack-infra | 04:52 | |
| *** sdake has quit IRC | 04:54 | |
| *** e0ne has joined #openstack-infra | 05:01 | |
| *** e0ne has quit IRC | 05:04 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/jeepyb: WIP Add a "reviews please" msg to welcome_message https://review.openstack.org/84056 | 05:06 |
| *** amotoki has joined #openstack-infra | 05:07 | |
| *** gokrokve_ has joined #openstack-infra | 05:10 | |
| *** gokrokve has quit IRC | 05:13 | |
| *** gokrokve_ has quit IRC | 05:15 | |
| *** gokrokve has joined #openstack-infra | 05:16 | |
| *** Alexandra_ has quit IRC | 05:16 | |
| *** gokrokve has quit IRC | 05:20 | |
| *** kevinbenton has left #openstack-infra | 05:25 | |
| *** ildikov_ has quit IRC | 05:27 | |
| *** _nadya_ has joined #openstack-infra | 05:28 | |
| *** _nadya_ has quit IRC | 05:31 | |
| *** e0ne has joined #openstack-infra | 05:32 | |
| *** dstanek has quit IRC | 05:33 | |
| *** dstanek has joined #openstack-infra | 05:34 | |
| *** e0ne has quit IRC | 05:36 | |
| *** dcramer_ has joined #openstack-infra | 05:40 | |
| *** _nadya_ has joined #openstack-infra | 05:46 | |
| *** gokrokve has joined #openstack-infra | 05:46 | |
| *** gokrokve_ has joined #openstack-infra | 05:49 | |
| *** chandan_kumar has joined #openstack-infra | 05:50 | |
| *** gokrokve has quit IRC | 05:51 | |
| *** nosnos_ has joined #openstack-infra | 05:53 | |
| *** nosnos has quit IRC | 05:53 | |
| *** matsuhashi has quit IRC | 05:54 | |
| *** matsuhashi has joined #openstack-infra | 05:55 | |
| *** _nadya_ has quit IRC | 05:57 | |
| *** gokrokve_ has quit IRC | 05:59 | |
| *** e0ne has joined #openstack-infra | 06:01 | |
| *** e0ne has quit IRC | 06:02 | |
| *** dstanek has quit IRC | 06:08 | |
| *** yolanda has joined #openstack-infra | 06:08 | |
| *** markwash has quit IRC | 06:10 | |
| *** dstanek has joined #openstack-infra | 06:10 | |
| *** ildikov_ has joined #openstack-infra | 06:11 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add a slave-script to Jenkins for pushing to swift https://review.openstack.org/71189 | 06:17 |
| *** jcoufal has joined #openstack-infra | 06:20 | |
| *** afazekas has joined #openstack-infra | 06:25 | |
| *** lcheng has quit IRC | 06:26 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: New geard server address for tripleo jobs. https://review.openstack.org/84061 | 06:26 |
| *** afazekas has quit IRC | 06:31 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/jeepyb: Remove dupe bug detection from notify_impact https://review.openstack.org/84067 | 06:34 |
| *** jcoufal has quit IRC | 06:40 | |
| *** Alexandra is now known as alex-gone | 06:47 | |
| mattoliverau | I'm off, have a good evening (monday morning) infra peeps. | 06:49 |
| fifieldt | cheers mattoliverau | 06:49 |
| *** talluri has joined #openstack-infra | 06:55 | |
| *** gokrokve has joined #openstack-infra | 06:56 | |
| *** jhesketh has quit IRC | 06:57 | |
| *** jhesketh_ has quit IRC | 06:57 | |
| *** roeyc has joined #openstack-infra | 06:58 | |
| *** mihgen has quit IRC | 07:00 | |
| lifeless | clarkb: ^ is fairly crucial to getting us back online | 07:00 |
| *** gokrokve has quit IRC | 07:01 | |
| *** jgallard has joined #openstack-infra | 07:01 | |
| *** flaper87|afk is now known as flaper87 | 07:02 | |
| *** zhiyan_ is now known as zhiyan | 07:05 | |
| *** chandan_kumar has quit IRC | 07:06 | |
| *** matsuhashi has quit IRC | 07:08 | |
| *** matsuhashi has joined #openstack-infra | 07:08 | |
| *** nosnos_ has quit IRC | 07:09 | |
| *** nosnos has joined #openstack-infra | 07:10 | |
| *** jhesketh has joined #openstack-infra | 07:10 | |
| *** jhesketh_ has joined #openstack-infra | 07:10 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Move status's js build tools into status folder https://review.openstack.org/84069 | 07:12 |
| *** pcrews_ has joined #openstack-infra | 07:12 | |
| *** Ryan_Lane has quit IRC | 07:13 | |
| *** afazekas has joined #openstack-infra | 07:14 | |
| *** jamielennox is now known as jamielennox|away | 07:17 | |
| *** matsuhashi has quit IRC | 07:22 | |
| *** jhesketh has quit IRC | 07:23 | |
| *** hashar has joined #openstack-infra | 07:24 | |
| *** irenab has joined #openstack-infra | 07:24 | |
| *** alff_ has joined #openstack-infra | 07:24 | |
| *** alff has joined #openstack-infra | 07:24 | |
| *** mkoderer has joined #openstack-infra | 07:25 | |
| *** matsuhashi has joined #openstack-infra | 07:28 | |
| *** mihgen has joined #openstack-infra | 07:32 | |
| *** dstanek has quit IRC | 07:34 | |
| *** julienvey has joined #openstack-infra | 07:42 | |
| *** pcrews_ has quit IRC | 07:44 | |
| *** talluri has quit IRC | 07:44 | |
| *** matsuhashi has quit IRC | 07:46 | |
| *** matsuhashi has joined #openstack-infra | 07:47 | |
| *** vponomaryov has joined #openstack-infra | 07:47 | |
| *** Ryan_Lane has joined #openstack-infra | 07:57 | |
| *** jcoufal has joined #openstack-infra | 07:58 | |
| *** mkerrin has joined #openstack-infra | 07:59 | |
| *** shardy has joined #openstack-infra | 08:02 | |
| *** alff_ has quit IRC | 08:04 | |
| *** alff has quit IRC | 08:04 | |
| *** jpich has joined #openstack-infra | 08:05 | |
| *** derekh has joined #openstack-infra | 08:09 | |
| *** Ryan_Lane has quit IRC | 08:09 | |
| *** rossella_ has joined #openstack-infra | 08:10 | |
| *** rossella_ has quit IRC | 08:11 | |
| *** jcoufal has quit IRC | 08:12 | |
| *** jcoufal has joined #openstack-infra | 08:13 | |
| openstackgerrit | Manuel Desbonnet proposed a change to openstack-infra/pypi-mirror: Add --force-reinstall to pip install cmd https://review.openstack.org/84081 | 08:13 |
| *** yassine has joined #openstack-infra | 08:13 | |
| *** e0ne has joined #openstack-infra | 08:17 | |
| *** alff_ has joined #openstack-infra | 08:18 | |
| *** alff has joined #openstack-infra | 08:18 | |
| *** jooools has joined #openstack-infra | 08:19 | |
| *** alff_ has quit IRC | 08:23 | |
| *** alff has quit IRC | 08:23 | |
| *** thuc has joined #openstack-infra | 08:29 | |
| *** thuc has quit IRC | 08:33 | |
| *** pblaho has joined #openstack-infra | 08:36 | |
| *** Daviey has joined #openstack-infra | 08:40 | |
| *** mattymo has quit IRC | 08:49 | |
| *** dachary has quit IRC | 08:49 | |
| *** mattymo has joined #openstack-infra | 08:49 | |
| *** dachary has joined #openstack-infra | 08:50 | |
| *** julienvey has quit IRC | 08:54 | |
| *** markmc has joined #openstack-infra | 08:56 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Add the check-tripleo-overcloud-f20 jenkins job https://review.openstack.org/81793 | 08:56 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Add tripleo overcloud job to check queue https://review.openstack.org/81794 | 08:56 |
| *** gokrokve has joined #openstack-infra | 08:57 | |
| *** jp_at_hp has joined #openstack-infra | 08:59 | |
| *** jp_at_hp has quit IRC | 08:59 | |
| *** jp_at_hp has joined #openstack-infra | 08:59 | |
| *** basha has joined #openstack-infra | 08:59 | |
| *** jcoufal has quit IRC | 09:01 | |
| *** gokrokve has quit IRC | 09:02 | |
| *** irenab has left #openstack-infra | 09:07 | |
| *** e0ne_ has joined #openstack-infra | 09:08 | |
| *** gnuoy has quit IRC | 09:09 | |
| *** gnuoy has joined #openstack-infra | 09:10 | |
| *** e0ne has quit IRC | 09:12 | |
| *** sahumada__ has joined #openstack-infra | 09:22 | |
| *** sahumada__ is now known as sahumada | 09:23 | |
| *** jlibosva has joined #openstack-infra | 09:24 | |
| *** basha has quit IRC | 09:29 | |
| *** dstanek has joined #openstack-infra | 09:32 | |
| *** ominakov has joined #openstack-infra | 09:34 | |
| *** dizquierdo has joined #openstack-infra | 09:35 | |
| *** dstanek has quit IRC | 09:36 | |
| *** julienvey has joined #openstack-infra | 09:47 | |
| *** jcoufal has joined #openstack-infra | 09:51 | |
| *** gokrokve has joined #openstack-infra | 09:58 | |
| *** gokrokve has quit IRC | 10:03 | |
| *** dstanek has joined #openstack-infra | 10:03 | |
| zhiyan | tristanC: ping | 10:06 |
| *** dstanek has quit IRC | 10:08 | |
| *** michchap has quit IRC | 10:11 | |
| *** dstanek has joined #openstack-infra | 10:13 | |
| *** wenlock has quit IRC | 10:14 | |
| *** dstanek has quit IRC | 10:18 | |
| *** ociuhandu has quit IRC | 10:20 | |
| *** yamahata has quit IRC | 10:21 | |
| *** alexpilotti has joined #openstack-infra | 10:22 | |
| *** alexpilotti has quit IRC | 10:26 | |
| *** jgallard has quit IRC | 10:28 | |
| *** basha has joined #openstack-infra | 10:28 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/config: Checks if requirements overlap between head and branch reqs https://review.openstack.org/84100 | 10:28 |
| sdague | ttx: so are we sorted on the requirements yet? | 10:32 |
| ttx | sdague: the happybase one? | 10:32 |
| sdague | yeh | 10:32 |
| ttx | sdague: last time I looked (one hour ago) what was proposed was different from the ML conclusion and commit message | 10:33 |
| yjiang|2 | Hi,guys,I want to ask a question:Did any one know if I use git-review a patch on review.openstack.org,how did Jenkins make sure the deployed openstack code in VM including that patch??I'm really confused about this detail.... | 10:33 |
| sdague | ok | 10:33 |
| *** jcoufal has quit IRC | 10:33 | |
| sdague | yjiang|2: that's what zuul does | 10:33 |
| ttx | sdague: I could go with anything. Looks like anything but >=0.8 is broken anyway | 10:33 |
| ttx | sdague: I just want what is proposed to match what we discussed | 10:34 |
| ttx | sdague: it's probably just an oversight | 10:34 |
| ttx | The whole thing doesn't matter that much, since apparently none of it is tested | 10:36 |
| sdague | yeh, would have been nice if any of this code path was tested, then we'd have know the requirement was broken | 10:36 |
| sdague | yeh | 10:36 |
| yjiang|2 | sdague: Oh,thx.I want to build a 3rd part testing platform..Could I use gerrit-triger to replace zuul?Because my OS is redhat 6.4 and I think may be zuul don't support redhat.. | 10:38 |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/config: Checks if requirements overlap between head and branch reqs https://review.openstack.org/84100 | 10:38 |
| sdague | yjiang|2: I don't know the folks that do are typically not awake yet | 10:38 |
| yjiang|2 | sdague: OK.still thx:) | 10:41 |
| *** jcoufal has joined #openstack-infra | 10:43 | |
| *** jhesketh has joined #openstack-infra | 10:48 | |
| *** e0ne_ has quit IRC | 10:54 | |
| *** alexpilotti has joined #openstack-infra | 10:57 | |
| *** jepoy_ has quit IRC | 10:57 | |
| *** jepoy has joined #openstack-infra | 10:58 | |
| *** andreaf has joined #openstack-infra | 10:58 | |
| *** gokrokve has joined #openstack-infra | 10:59 | |
| *** lcostantino has joined #openstack-infra | 11:00 | |
| *** jepoy has quit IRC | 11:02 | |
| *** basha has quit IRC | 11:02 | |
| *** gokrokve has quit IRC | 11:03 | |
| *** mwagner_zzz has quit IRC | 11:04 | |
| *** BobBall has joined #openstack-infra | 11:06 | |
| *** michchap has joined #openstack-infra | 11:11 | |
| rcarrillocruz | hey guys, for stackforge projects, is there a specific namespace in the wiki? I have a project and I was thinking to put some documentation for it... | 11:11 |
| *** rlandy has joined #openstack-infra | 11:12 | |
| *** mbacchi has joined #openstack-infra | 11:17 | |
| *** ihrachys|afk is now known as ihrachys | 11:23 | |
| *** jepoy has joined #openstack-infra | 11:24 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: periodic cleanup of bugs https://review.openstack.org/83874 | 11:25 |
| *** yaguang has quit IRC | 11:26 | |
| sdague | ttx: so what do you want that requirements value to be? I can just update the review and you can +A it through? | 11:26 |
| *** matsuhashi has quit IRC | 11:26 | |
| sdague | because I think another day of going around in circles on it is probably only of medium value :) | 11:27 |
| tristanC | zhiyan: pong! | 11:27 |
| *** dkranz has quit IRC | 11:27 | |
| *** nosnos has quit IRC | 11:27 | |
| SergeyLukjanov | sdague, ttx, are you going to approve >=0.8? | 11:28 |
| ttx | sdague: no kidding :) I'd update it to match the commit message | 11:29 |
| ttx | i.e. >=0.5,!=0.7 | 11:29 |
| sdague | ttx: cool | 11:30 |
| SergeyLukjanov | ttx, ++, it sounds correctly | 11:30 |
| SergeyLukjanov | nprivalova, are you around? | 11:30 |
| nprivalova | >=0.5,!=0.7 is ok | 11:30 |
| *** yjiang|2 has quit IRC | 11:30 | |
| ttx | I'm done with it. It's not properly tested so I consider it broken, whatever the requirements file says | 11:32 |
| ttx | so yeah, >=0.5,!=0.7 is as good as anything | 11:33 |
| *** e0ne has joined #openstack-infra | 11:34 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Making a token expiration time configurable https://review.openstack.org/84107 | 11:34 |
| sdague | ttx: ok, so are yuo pushing the new rev? or should I? | 11:35 |
| ttx | sdague: I'll do it and you approve it | 11:35 |
| sdague | sounds good | 11:35 |
| nprivalova | ttx, are we gonna bring HBase to gating? | 11:36 |
| sdague | nprivalova: I think the whole issue in how we got into this is there were no tests for it anywhere | 11:36 |
| sdague | nprivalova: is there something proposed to do that? | 11:37 |
| nprivalova | ttx, sdague: maybe wrong question... would you mind if somebody start to work on make HBase tested on gating :) | 11:37 |
| nprivalova | sdague: we may start testing on real HBase in Ceilo | 11:37 |
| openstackgerrit | Thierry Carrez proposed a change to openstack/requirements: Using happybase >=0.5,!=0.7 https://review.openstack.org/82438 | 11:38 |
| nprivalova | sdague: afaik, sahara and trove may be interested in HBase too in future | 11:40 |
| sdague | nprivalova: I think that would be fine. Though it probably would have to be in ceilometer unit | 11:40 |
| *** yolanda has quit IRC | 11:43 | |
| *** ociuhandu has joined #openstack-infra | 11:43 | |
| nprivalova | sdague: I'll start investigations shortly. Will keep you posted. Maybe Thomas Goirand may help us with packets in Debian | 11:44 |
| sdague | nprivalova: cool, thankis | 11:44 |
| sdague | thanks | 11:44 |
| *** e0ne has quit IRC | 11:44 | |
| *** e0ne has joined #openstack-infra | 11:44 | |
| ruhe | the main problem with hbase is that it is not available from the official ubuntu packages | 11:45 |
| ruhe | that was the reason for infra core team to decline our previous attempt to install hbase in gating vms | 11:46 |
| ruhe | and that's a valid reason | 11:46 |
| isviridov | ruhe, hi | 11:46 |
| ruhe | isviridov: hi | 11:46 |
| isviridov | Seems to be we have similar issue with Cassandra | 11:47 |
| ruhe | isviridov: probably | 11:47 |
| isviridov | It is available in datastax repos | 11:47 |
| openstackgerrit | Alessandro Pilotti proposed a change to openstack/requirements: Upgrades Pbr to exclude 0.7 due to Windows issue https://review.openstack.org/84030 | 11:47 |
| ruhe | those vendor-hosted repos aren't considered to be reliable. they can go away any time and block the whole gating system | 11:48 |
| ruhe | i see two options: | 11:48 |
| ruhe | 1. create mirrors in openstack infra | 11:48 |
| ruhe | 2. someone to maintain hbase,cassandra,etc in official ubuntu repos | 11:48 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/config: Storyboard token expiration increased https://review.openstack.org/84111 | 11:51 |
| sdague | hmmm... | 11:51 |
| sdague | is debian packaging them? | 11:51 |
| *** saju_m has joined #openstack-infra | 11:51 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: periodic cleanup of bugs https://review.openstack.org/83874 | 11:51 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Fix up bootstrap in status page https://review.openstack.org/84112 | 11:53 |
| ttx | sdague: let's just say that properly packaging a decently-sized java stack in debian is... difficult at best | 11:53 |
| sdague | ttx: hmmm | 11:53 |
| sdague | yeh, that's unfortunate | 11:53 |
| ttx | sdague: http://fnords.wordpress.com/2010/09/24/the-real-problem-with-java-in-linux-distros/ | 11:54 |
| ruhe | vendors do these packages | 11:54 |
| ruhe | we could probably just keep local mirrors of those packages | 11:54 |
| *** ArxCruz has joined #openstack-infra | 11:57 | |
| *** jgallard has joined #openstack-infra | 12:00 | |
| sdague | ruhe: so to head down this path, we probably want to make a lib/nosql in devstack like lib/database | 12:00 |
| *** gokrokve has joined #openstack-infra | 12:00 | |
| *** saju_m has quit IRC | 12:00 | |
| sdague | and make mongo, hbase, casandra available as the nosql backend selectably | 12:00 |
| sdague | maybe we can issolate some of this with using devstack for node setup | 12:01 |
| *** zhiyan is now known as zhiyan_ | 12:01 | |
| isviridov | sdague, we have done some efforts for Cassandra integration into devstack in MagnetoDB project | 12:02 |
| isviridov | https://github.com/stackforge/magnetodb/tree/master/contrib/devstack | 12:02 |
| isviridov | Cassandra is installed from Datastax repo now | 12:03 |
| sdague | ok | 12:03 |
| *** rfolco has joined #openstack-infra | 12:03 | |
| *** markmcclain has joined #openstack-infra | 12:04 | |
| sdague | so if we could put these all into a standard mechanism of lib/nosql and lib/nosql/XXXXXXX | 12:04 |
| sdague | then we could probably standardize some mechanism in installing these engines, which all basically seem to need to punt on using distro packaging | 12:04 |
| *** markmcclain has quit IRC | 12:04 | |
| sdague | though - https://github.com/stackforge/magnetodb/blob/master/contrib/devstack/lib/magnetodb seems to indicate that you are downloading java as well, that will need to come from the base platform | 12:05 |
| *** gokrokve has quit IRC | 12:05 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Fix up bootstrap in status page https://review.openstack.org/84112 | 12:05 |
| *** saju_m has joined #openstack-infra | 12:06 | |
| *** ArxCruz has quit IRC | 12:07 | |
| isviridov | sdague, Cassandra required or at least recommends Oracle JDK, that is why we are downloading it | 12:07 |
| *** ArxCruz has joined #openstack-infra | 12:08 | |
| *** saju_m has quit IRC | 12:08 | |
| isviridov | probably we should cache it somehow | 12:08 |
| sdague | well that starts to become it's own challenge, if then we are also managing jvms as well | 12:09 |
| sdague | lets start without that | 12:09 |
| ruhe | isviridov: according to hortonworks guys (main contributors to hadoop), hadoop and hbase works fine on openjdk | 12:09 |
| *** sandywalsh has quit IRC | 12:10 | |
| ruhe | i guess cassandra should work fine too. at least for testing purposes | 12:10 |
| isviridov | We will test it. AFAIK there are some performance optimizations tightly connected with Oracle JDK. | 12:12 |
| *** mugsie has joined #openstack-infra | 12:13 | |
| *** jpeeler has joined #openstack-infra | 12:19 | |
| sdague | so I think that's where this gets sticky, because the oracle jdk is very much not open source | 12:22 |
| sdague | and there has been a pretty strong feeling that we shouldn't be putting non open code into infra for testing | 12:22 |
| *** pdmars has joined #openstack-infra | 12:23 | |
| sdague | but if it's functional, even if a little slower, on openjdk, that would be fine | 12:23 |
| *** sandywalsh has joined #openstack-infra | 12:23 | |
| openstackgerrit | Jason Kincl proposed a change to openstack-infra/jenkins-job-builder: Add support for Copy Artifact project copy permissions https://review.openstack.org/82551 | 12:24 |
| *** dcramer_ has quit IRC | 12:25 | |
| *** pdmars has quit IRC | 12:26 | |
| isviridov | sdague, you are right. After first talk with Cassandra guys it looks like we really can use OpenJDK for gating | 12:26 |
| sdague | cool | 12:27 |
| ttx | sdague: https://review.openstack.org/#/c/82438/ up for your review pleasure | 12:27 |
| *** aysyd has joined #openstack-infra | 12:27 | |
| *** jgallard has quit IRC | 12:28 | |
| sdague | done | 12:28 |
| *** jgallard has joined #openstack-infra | 12:28 | |
| nprivalova | ttx, sdague, thank you very much for happybase! | 12:29 |
| *** pdmars has joined #openstack-infra | 12:30 | |
| *** maxbit has joined #openstack-infra | 12:30 | |
| fungi | things seem not horribly broken this morning! | 12:32 |
| *** changbl has quit IRC | 12:32 | |
| therve | fungi, Are some changes landing though? I'm seeing some pretty bad cinder errors | 12:34 |
| therve | http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkNhdWdodCBlcnJvcjogJ21vZHVsZScgb2JqZWN0IGhhcyBubyBhdHRyaWJ1dGUgJ0ZhaWx1cmUnXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjQzMjAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NjI2OTIxMTg0NX0= | 12:35 |
| *** ominakov has quit IRC | 12:36 | |
| *** ominakov has joined #openstack-infra | 12:37 | |
| *** mwagner_zzz has joined #openstack-infra | 12:38 | |
| *** dims has quit IRC | 12:38 | |
| *** ominakov has quit IRC | 12:38 | |
| *** ominakov has joined #openstack-infra | 12:39 | |
| fungi | therve: http://git.openstack.org/cgit/openstack/openstack/log/ suggests that the last thing to pass the integrated gate on master was a tempest change 3 hours ago | 12:40 |
| therve | fungi, That matches the taskflow commit I think | 12:41 |
| therve | https://review.openstack.org/83965 | 12:41 |
| fungi | and yeah, that error message you linked in logstash starts around the same time we stopped merging things | 12:42 |
| therve | "Breaking change: any client should be updated" | 12:42 |
| fungi | timing on the taskflow patch is consistent with that | 12:42 |
| therve | Who thinks it's a good idea :/ | 12:43 |
| fungi | um, yeah. revert that noise | 12:43 |
| fungi | sudden change with no backward compatibility and no deprecation in a lib used by other projects? | 12:43 |
| fungi | i take it taskflow is asymmetrically gated | 12:44 |
| *** yolanda has joined #openstack-infra | 12:44 | |
| therve | Yeah apparently | 12:45 |
| fungi | yep... no devstack/tempest/grenade jobs on openstack/taskflow! http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n1015 | 12:45 |
| fungi | that too should get fixed, apparently | 12:45 |
| *** bknudson has quit IRC | 12:47 | |
| fungi | dhellmann_: if you're around, we need an emergency revert of https://review.openstack.org/83965 and integration tests on openstack/taskflow | 12:47 |
| *** mugsie_ has joined #openstack-infra | 12:47 | |
| *** dprince has joined #openstack-infra | 12:49 | |
| *** maxbit has quit IRC | 12:50 | |
| *** dims has joined #openstack-infra | 12:52 | |
| *** yamahata has joined #openstack-infra | 12:52 | |
| *** mugsie has quit IRC | 12:55 | |
| *** jeckersb_gone is now known as jeckersb | 12:56 | |
| *** hashar has quit IRC | 13:00 | |
| *** gokrokve has joined #openstack-infra | 13:01 | |
| *** ameade has joined #openstack-infra | 13:01 | |
| *** jlibosva has quit IRC | 13:01 | |
| *** markmcclain has joined #openstack-infra | 13:01 | |
| *** jlibosva has joined #openstack-infra | 13:03 | |
| *** e0ne has quit IRC | 13:03 | |
| *** dhellmann_ is now known as dhellmann | 13:04 | |
| *** matrohon has joined #openstack-infra | 13:04 | |
| *** e0ne has joined #openstack-infra | 13:04 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Perform integration testing on openstack/taskflow https://review.openstack.org/84123 | 13:05 |
| therve | Hum I tried "Revert change" in gerrit :) | 13:05 |
| *** hashar has joined #openstack-infra | 13:05 | |
| fungi | therve: i was waiting until i could link that ^ in the commit message | 13:06 |
| *** gokrokve has quit IRC | 13:06 | |
| fungi | therve: https://review.openstack.org/84124 | 13:06 |
| therve | fungi, Great thanks | 13:06 |
| *** HenryG_ has quit IRC | 13:07 | |
| *** bknudson has joined #openstack-infra | 13:08 | |
| *** jhesketh has quit IRC | 13:10 | |
| *** adarazs has joined #openstack-infra | 13:11 | |
| *** mbacchi has quit IRC | 13:11 | |
| fungi | dhellmann: https://review.openstack.org/84124 is the proposed revert | 13:13 |
| *** zhiyan_ is now known as zhiyan | 13:14 | |
| *** weshay has joined #openstack-infra | 13:16 | |
| sdague | fungi: can you force the revert? | 13:21 |
| *** zehicle has quit IRC | 13:21 | |
| sdague | otherwise, we could always drop taskflow from devstack | 13:21 |
| sdague | the sequence here was definitely incorrect | 13:21 |
| fungi | sdague: i can force it in, just don't want to step on any toes | 13:22 |
| *** dcramer_ has joined #openstack-infra | 13:22 | |
| sdague | actually, we need to back out more of these repos | 13:22 |
| sdague | I think I'll do it via a devstack change instead | 13:22 |
| fungi | you mean test them on their released versions from pypi instead of on the current state of their master branches? | 13:23 |
| *** mriedem has joined #openstack-infra | 13:24 | |
| fungi | sdague: ^ | 13:24 |
| sdague | yes | 13:25 |
| fungi | ahh, got it | 13:25 |
| sdague | there are 4 oslo trees in devstack which aren't in the gate | 13:25 |
| fungi | argh | 13:25 |
| sdague | apparently a piece of this was missed | 13:25 |
| sdague | https://review.openstack.org/84129 | 13:26 |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/devstack-gate: Support XenServer in the gate https://review.openstack.org/80241 | 13:27 |
| sdague | missed, one, apparently pycadf in the same situation as well | 13:27 |
| *** julim has joined #openstack-infra | 13:27 | |
| *** HenryG has joined #openstack-infra | 13:29 | |
| *** signed8bit has joined #openstack-infra | 13:31 | |
| *** jlibosva1 has joined #openstack-infra | 13:31 | |
| *** julim has quit IRC | 13:33 | |
| *** jlibosva has quit IRC | 13:34 | |
| *** julim has joined #openstack-infra | 13:36 | |
| dhellmann | fungi: catching up -- that's a long patch, did you generate the revert with a tool? | 13:40 |
| fungi | dhellmann: git revert 42ca240 | 13:40 |
| fungi | dhellmann: if that counts as a tool | 13:40 |
| dhellmann | yeah | 13:40 |
| *** thuc has joined #openstack-infra | 13:40 | |
| dhellmann | +2a | 13:40 |
| dhellmann | I didn't want to compare them by eye | 13:41 |
| fungi | i don't blame you | 13:41 |
| *** thomasem has joined #openstack-infra | 13:42 | |
| fungi | dhellmann: apparently part of the problem is that devstack started checking out the state of the git repos for several oslo libs which aren't being tested in the integrated gate yet, so sdague's cleaning that up and then i guess those will get jobs proposed when we're ready to try again with testing newer than the released versions of those from pypi | 13:42 |
| *** thuc has quit IRC | 13:43 | |
| *** mbacchi has joined #openstack-infra | 13:43 | |
| *** thuc has joined #openstack-infra | 13:43 | |
| fungi | but i expect that taskflow change to present problems when that time comes if it's not reverted first, so i think that's still warranted | 13:43 |
| *** primeministerp has quit IRC | 13:43 | |
| dhellmann | fungi: why are we reverting those changes? we want devstack testing those libraries | 13:44 |
| dhellmann | taskflow is an oslo library now, we need to set up the symmetric gating | 13:44 |
| dhellmann | so if we're missing a job, let's add that instead of taking away one | 13:44 |
| dhellmann | sdague: ^^ | 13:44 |
| fungi | dhellmann: well, we need to add integration test jobs for those libraries before telling devstack to use the latest branch state on them | 13:46 |
| fungi | but it seems they're all working save taskflow at the moment, so it might make sense to just cram in the additional jobs on them asap, i agree | 13:46 |
| *** primeministerp has joined #openstack-infra | 13:47 | |
| dhellmann | fungi: Do you mean tests specific for taskflow? Testing the apps that use taskflow isn't enough (at least to start)? | 13:47 |
| fungi | dhellmann: just the devstack-tempest jobs (so we make sure the jobs other projects are running won't break when a change to one of those libs is introduced) | 13:48 |
| *** jgrimm has quit IRC | 13:48 | |
| fungi | specifically for the integration jobs shared by multiple projects. less important for things like unit tests where you're aiming with oslo.test, though that's important too certainly | 13:49 |
| *** nkinder has quit IRC | 13:49 | |
| *** jcoufal has quit IRC | 13:49 | |
| dhellmann | fungi: I'm not sure what the names of the individual jobs are, so I may use the wrong words. We have the devstack gate job(s) installing taskflow, since the app tests run there broke. We should run that same set of jobs as a gate on taskflow itself, right? | 13:50 |
| *** thuc_ has joined #openstack-infra | 13:50 | |
| *** zigo has quit IRC | 13:50 | |
| fungi | dhellmann: yep, that's what i'm suggesting. perhaps something like https://review.openstack.org/84123 i linked in the commit message of the revert and in bug 1300161 | 13:51 |
| uvirtbot | Launchpad bug 1300161 in devstack "Error import taskflow Failure" [Undecided,In progress] https://launchpad.net/bugs/1300161 | 13:51 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Making a token expiration time configurable https://review.openstack.org/84107 | 13:51 |
| *** vhoward- has joined #openstack-infra | 13:51 | |
| dhellmann | fungi: yes, that's what I mean | 13:52 |
| *** amotoki_ has joined #openstack-infra | 13:52 | |
| *** thuc has quit IRC | 13:53 | |
| *** eharney has joined #openstack-infra | 13:53 | |
| fungi | because removing them from the git checkouts in devstack only moves the issue forward from commit proposal time to library release time, at which point the "solution" is more complicated when something like that breaking change does find its way in | 13:54 |
| dhellmann | fungi: I have several changes pending related to making it easier to set up testing for oslo libs. I assume we're holding off on merging those until the FF lifts? | 13:54 |
| dhellmann | fungi: right, we definitely want those libraries included in the integrated gate | 13:54 |
| fungi | dhellmann: i don't think we're holding off necessarily, unless there are features needed in frozen projects to support that. infra is in "tread carefully" mode at the moment, but hasn't gone to slush yet | 13:55 |
| dhellmann | ok, let me find my list | 13:55 |
| fungi | it's probably more that we're just under water for reviews again, so apologies | 13:56 |
| *** dkliban has joined #openstack-infra | 13:56 | |
| *** jlibosva1 is now known as jlibosva | 13:56 | |
| dhellmann | this one adds a bunch of jobs for oslo libs: https://review.openstack.org/#/c/76945/10 | 13:56 |
| dhellmann | no worries, I thought we were frozen or I would have bugged you | 13:56 |
| dhellmann | heh, that change includes adding gate jobs for taskflow | 13:56 |
| fungi | i see that! abandoning mine | 13:57 |
| *** prad_ has joined #openstack-infra | 13:57 | |
| *** AJaeger has joined #openstack-infra | 13:58 | |
| *** zigo has joined #openstack-infra | 13:58 | |
| ttx | damn, must be the 5th reset since we enqueued ceilometer open-juno commit 3 hours ago | 13:59 |
| fungi | ttx: yep, bug discussed above ^ | 13:59 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Parameterize library cross-test job https://review.openstack.org/83759 | 13:59 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add tox env for testing jenkins-job-builder https://review.openstack.org/83776 | 13:59 |
| *** dkranz has joined #openstack-infra | 13:59 | |
| *** zns has joined #openstack-infra | 14:00 | |
| ttx | fungi: is it blocking or just introducing a high fail rate ? | 14:00 |
| fungi | dhellmann: looks like 76945 should be perfectly safe since it's just adding jobs to libs which aren't part of the integrated release, so worst case you just block changes to those libs | 14:00 |
| fungi | ttx: full on gate block | 14:00 |
| dhellmann | fungi: and we're frozen now anyway :-) | 14:00 |
| ttx | fungi: ok :) | 14:00 |
| dhellmann | at least we were supposed to be, I sent email to josh about taskflow | 14:00 |
| fungi | yeah, to ivan's credit he called out clearly in the commit message on that one "this is a breaking change" and it got a slew of +2 votes anyway | 14:01 |
| *** gokrokve has joined #openstack-infra | 14:01 | |
| dhellmann | fungi: https://review.openstack.org/#/c/83759/5 changes the unit test cross-test job so we can use it for libs other than oslotest | 14:01 |
| sdague | gah - http://logs.openstack.org/29/84129/2/check/check-tempest-dsvm-full/d6364bb/logs/devstacklog.txt.gz#_2014-03-31_13_37_03_941 | 14:01 |
| dhellmann | fungi: yeah | 14:01 |
| sdague | how I hate namespace packages | 14:02 |
| fungi | let you count the ways? | 14:02 |
| dhellmann | sdague: where's the log of things being installed in that job? | 14:02 |
| sdague | dhellmann: in devstack log there | 14:03 |
| sdague | devstack logs a pretty verbose | 14:03 |
| *** dstanek has joined #openstack-infra | 14:04 | |
| sdague | http://logs.openstack.org/29/84129/2/check/check-tempest-dsvm-full/d6364bb/logs/devstacklog.txt.gz#_2014-03-31_13_34_06_271 we install it right there | 14:04 |
| dhellmann | sdague: it looks like something is downloading oslo.config at http://logs.openstack.org/29/84129/2/check/check-tempest-dsvm-full/d6364bb/logs/devstacklog.txt.gz#_2014-03-31_13_34_03_765 | 14:04 |
| dhellmann | the oslo libs should be installed from source, right? | 14:05 |
| sdague | dhellmann: that's the uninstall / install / uninstall that we do | 14:05 |
| sdague | it's uninstalled about 3s later | 14:06 |
| dhellmann | why don't you just delete the library with rm? | 14:06 |
| *** gokrokve has quit IRC | 14:06 | |
| *** rwsu has joined #openstack-infra | 14:06 | |
| *** jnoller has joined #openstack-infra | 14:07 | |
| sdague | because pip often believed it was still installed | 14:08 |
| sdague | I don't remember how we got there fully, it's been like that in the code for a year | 14:08 |
| dhellmann | ok | 14:08 |
| dhellmann | there's probably a safer way to do that with rm, but we can look at that laster | 14:08 |
| dhellmann | later | 14:08 |
| *** mugsie_ is now known as mugsie | 14:08 | |
| *** jnoller has quit IRC | 14:09 | |
| *** bnemec has joined #openstack-infra | 14:09 | |
| AJaeger | Hi, could somebody review https://review.openstack.org/#/c/82573/ , please? It should fix cases like https://review.openstack.org/#/c/82719/ | 14:10 |
| *** mrodden has quit IRC | 14:12 | |
| dhellmann | sdague: here's the problem: http://logs.openstack.org/29/84129/2/check/check-tempest-dsvm-full/d6364bb/logs/devstacklog.txt.gz#_2014-03-31_13_35_22_605 | 14:12 |
| sdague | dhellmann: why's that a problem | 14:13 |
| sdague | that was one of the points of the change | 14:13 |
| sdague | because that library isn't co-gating either | 14:13 |
| dhellmann | the fix for that is to add the gate job: https://review.openstack.org/#/c/76945/10 | 14:13 |
| dhellmann | all oslo libraries must be installed from source, otherwise it triggers this ns package issue | 14:13 |
| dhellmann | sdague: the gate job isn't there because I thought the CI system was frozen so I didn't push for reviews. | 14:14 |
| sdague | dhellmann: ok, so I agree, however, I still don't understand why this is breaking | 14:14 |
| sdague | is this just because we have a 2nd namespace package? | 14:15 |
| dhellmann | no | 14:15 |
| dhellmann | the problem is triggered when one lib is installed pip -e and another is installed pip | 14:15 |
| dhellmann | into the same namespace | 14:15 |
| dhellmann | so we have oslo.config and oslo.message going in via "pip -e" and oslo.vmware going in via "pip" | 14:16 |
| *** homeless has joined #openstack-infra | 14:16 | |
| sdague | hmph | 14:17 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add gate jobs for oslo libraries https://review.openstack.org/76945 | 14:18 |
| sdague | so, that would also mean that if we ever install oslo libs in the wrong order based on their dependencies they'll break, right? | 14:18 |
| dhellmann | sdague: yes, I suppose that's true | 14:19 |
| dhellmann | we will have to be careful about how we add them to devstack | 14:19 |
| *** yamahata has quit IRC | 14:19 | |
| dhellmann | that file probably needs a nice big warning comment | 14:19 |
| *** afazekas has quit IRC | 14:19 | |
| sdague | and changing requirements ever in an olso library can cause similar issues | 14:19 |
| *** yamahata has joined #openstack-infra | 14:19 | |
| sdague | because it could trigger a need to reorder | 14:20 |
| sdague | which might put you in a death lock | 14:20 |
| dhellmann | yeah, but I don't expect that to be a problem -- that would indicate some sort of circular dependency anyway | 14:20 |
| dhellmann | and if that's the case, we wouldn't be able to install the libs in the first place | 14:20 |
| sdague | dhellmann: well, it means you can't never invert the install order either | 14:20 |
| dhellmann | the reason we have so many oslo libs is so they can depend on each other without cycles, and so they don't have common dependencies on third-party libs | 14:21 |
| dhellmann | why would we want to invert the installation order? | 14:21 |
| sdague | I don't know, I'm just looking at the constraints being put on the system | 14:21 |
| dhellmann | ok | 14:21 |
| dhellmann | pip figures out dependencies for things in requirements.txt; we're installing from source; we will have to do that work; that work is done in the oslo wiki | 14:22 |
| sdague | and wishing that we'd never gone down the namespace package route | 14:22 |
| *** zhiyan is now known as zhiyan_ | 14:22 | |
| dhellmann | we would have other problems if we didn't | 14:22 |
| sdague | like ? | 14:22 |
| dhellmann | we have some cases where we're installing the same package twice and triggering this problem | 14:22 |
| dhellmann | rather than a hard "you've broken your libraries" error, we would have something that looked like it didn't have the latest source code | 14:23 |
| dhellmann | so we would be trying to figure out why installing a library wasn't getting the fixes that were clearly in the version being installed | 14:23 |
| sdague | if this wasn't a namespace though, this would have worked | 14:23 |
| dhellmann | this one case, yes | 14:23 |
| dhellmann | this is not the only way this problem has been triggered, though | 14:23 |
| sdague | that's fair, it's just the one that tends to ruin my mondays :) | 14:23 |
| dhellmann | we had systems where oslo.config was installed from source and oslo.messaging was installed both from source and from a package -- that was happning *a lot* earlier this cycle | 14:24 |
| sdague | well, we could aslo decide that oslo libraries aren't the same as the rest of openstack | 14:25 |
| sdague | and not install them -e | 14:25 |
| dhellmann | sdague: that might also solve this problem, yes | 14:25 |
| *** david-lyle has joined #openstack-infra | 14:25 | |
| sdague | because you aren't going to just ctrl-c a thing to restart it | 14:25 |
| dhellmann | I hadn't thought about that. I wonder what other impact that would have. | 14:25 |
| *** mrodden has joined #openstack-infra | 14:25 | |
| sdague | like you would with nova-compute | 14:26 |
| dhellmann | well, if I'm working on oslo.messaging I might want to do it with one of the apps | 14:26 |
| dhellmann | but yeah, in general we are more likely to develop with the unit tests | 14:26 |
| sdague | well, then yuo'd be using wrong oslo.messagin on all the other services | 14:26 |
| dhellmann | ./unstack.sh then | 14:26 |
| dhellmann | which I guess is going to trigger a re-install | 14:26 |
| sdague | sure, but if you unstack / stack, then it's installed | 14:26 |
| *** alff_ has joined #openstack-infra | 14:27 | |
| *** alff has joined #openstack-infra | 14:27 | |
| dhellmann | right | 14:27 |
| dhellmann | yeah, this may be the right way to go, but I'd like to think about what it means first | 14:27 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Parameterize library cross-test job https://review.openstack.org/83759 | 14:29 |
| *** dstanek has quit IRC | 14:30 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add gate jobs for oslo libraries https://review.openstack.org/76945 | 14:30 |
| dhellmann | rebasing these old changes is a pita | 14:30 |
| fungi | yep, large files with lots of churn | 14:31 |
| boris-42 | fungi hi | 14:31 |
| dhellmann | it's too bad we can't split up projects.yaml and layout.yaml | 14:31 |
| dhellmann | one file each per program | 14:31 |
| *** zehicle_at_dell has joined #openstack-infra | 14:33 | |
| jeblair_ | dhellmann: that would not likely affect the churn... that tends to happen when we make changes across the board | 14:34 |
| dhellmann | jeblair_: fair enough | 14:34 |
| boris-42 | fungi could you pls take a look at patch that enables "sudo" on bare-precise nodes https://review.openstack.org/#/c/72780/ | 14:34 |
| fungi | boris-42: yep, it's in my list of things to review (all the infra project changes are) | 14:35 |
| *** nkinder has joined #openstack-infra | 14:35 | |
| boris-42 | fungi hehe=) | 14:35 |
| *** jgrimm has joined #openstack-infra | 14:36 | |
| boris-42 | fungi it's seems like a quite old patch=) | 14:36 |
| fungi | boris-42: yes, i'm very behind on reviews | 14:37 |
| *** damnsmith is now known as dansmith | 14:40 | |
| jeblair_ | dhellmann: while you're here, is there a reason you went from 'oslo.test' to 'oslotest' in 83759? | 14:40 |
| *** belliott has quit IRC | 14:41 | |
| *** changbl has joined #openstack-infra | 14:41 | |
| *** thuc_ has quit IRC | 14:42 | |
| jeblair_ | (for the job name) | 14:42 |
| *** wenlock has joined #openstack-infra | 14:44 | |
| *** hashar_ has joined #openstack-infra | 14:45 | |
| jeblair_ | sdague: would you please review the integrated gate changes in https://review.openstack.org/#/c/76945/12 ? | 14:45 |
| *** belliott has joined #openstack-infra | 14:48 | |
| *** hashar has quit IRC | 14:48 | |
| *** rcleere has joined #openstack-infra | 14:48 | |
| *** pcrews_ has joined #openstack-infra | 14:49 | |
| *** belliott has quit IRC | 14:50 | |
| dhellmann | jeblair_: the library name is "oslotest" -- we need to rename the repo to match after the rc is cut | 14:50 |
| jeblair_ | dhellmann: that explains all, thanks! | 14:51 |
| *** mkerrin has quit IRC | 14:53 | |
| *** freyes has joined #openstack-infra | 14:54 | |
| dhellmann | jeblair_ or fungi : do the templates in layout.yaml support recursion? could I create an "oslo-library" template that had all of the jobs an oslo lib is expected to pass, and use the other templates? | 14:56 |
| jeblair_ | dhellmann: i don't _think_ so (though i don't think it would be too difficult to add that to zuul) | 14:58 |
| *** jeblair_ is now known as jeblair | 14:59 | |
| jeblair | ttx: is it the new storyboard meeting time? | 14:59 |
| dhellmann | it feels less error-prone | 14:59 |
| openstackgerrit | Akihiro Motoki proposed a change to openstack-infra/config: Add language spepcific font package to Jenkins slave https://review.openstack.org/84149 | 14:59 |
| *** wenlock_ has joined #openstack-infra | 15:00 | |
| *** alff has quit IRC | 15:00 | |
| *** alff_ has quit IRC | 15:00 | |
| fungi | jeblair: yes | 15:01 |
| *** mihgen has quit IRC | 15:01 | |
| *** gokrokve has joined #openstack-infra | 15:02 | |
| sdague | jeblair: lgtm | 15:03 |
| ttx | jeblair: oh. Yes. | 15:04 |
| *** wenlock_ has quit IRC | 15:04 | |
| *** gokrokve has quit IRC | 15:06 | |
| *** alexpilotti has quit IRC | 15:07 | |
| *** blamar has quit IRC | 15:07 | |
| *** changbl has quit IRC | 15:08 | |
| *** CaptTofu has joined #openstack-infra | 15:09 | |
| *** belliott has joined #openstack-infra | 15:09 | |
| *** mgagne has joined #openstack-infra | 15:09 | |
| markmcclain | is there a way to get something promoted to the gate that is failing check for a test that does not even load Neutron? | 15:09 |
| markmcclain | or do I need to keep reverifying until I get lucky? | 15:10 |
| markmcclain | https://review.openstack.org/#/c/83762/ | 15:10 |
| *** mkerrin has joined #openstack-infra | 15:11 | |
| *** BadCub01 has joined #openstack-infra | 15:11 | |
| Daviey | Hi, is this a new CI bug http://logs.openstack.org/81/83981/4/check/check-tempest-dsvm-full/b78800c/console.html ? | 15:11 |
| Daviey | can't see anything similar in the recheck page | 15:11 |
| *** belliott has left #openstack-infra | 15:12 | |
| *** BadCub01 has left #openstack-infra | 15:12 | |
| *** mgagne1 has joined #openstack-infra | 15:12 | |
| *** BadCub01 has joined #openstack-infra | 15:13 | |
| therve | Daviey, 1300161, should be fixed now | 15:13 |
| fungi | markmcclain: looking | 15:13 |
| *** mgagne has quit IRC | 15:14 | |
| fungi | markmcclain: server fault on rest client calls? | 15:15 |
| markmcclain | yeah… Neutron isn't even loaded for check-tempest-dsvm-full | 15:17 |
| *** gokrokve has joined #openstack-infra | 15:17 | |
| markmcclain | and this is last review to merge before we can cut our RC | 15:17 |
| fungi | what's the bug for that serverfault exception? | 15:18 |
| Daviey | therve: ah, thanks | 15:19 |
| *** lcostantino has quit IRC | 15:19 | |
| mtreinish | markmcclain: why is a non-neutron job run on a neutron patch? That just seems like a waste of resources | 15:19 |
| *** gokrokve_ has joined #openstack-infra | 15:20 | |
| mtreinish | fungi: it's the taskflow thing: http://logs.openstack.org/62/83762/2/check/check-tempest-dsvm-full/a11f4ca/logs/screen-c-api.txt.gz?level=TRACE | 15:21 |
| *** gokrokve has quit IRC | 15:21 | |
| fungi | mtreinish: still? i thought the revert merged? | 15:22 |
| anteaya | I had thought the taskflow patch was reverted | 15:22 |
| anteaya | it was abandoned: https://review.openstack.org/#/c/84122/ | 15:22 |
| fungi | anteaya: no, i meant https://review.openstack.org/#/c/84124/ | 15:23 |
| mtreinish | fungi: I thought that was the error right? http://logs.openstack.org/62/83762/2/check/check-tempest-dsvm-full/a11f4ca/logs/screen-c-api.txt.gz?level=TRACE#_2014-03-31_14_26_36_573 | 15:23 |
| anteaya | your revert, yes | 15:23 |
| mtreinish | that's what markmcclain's run shows in the cinder logs | 15:23 |
| *** CaptTofu has quit IRC | 15:23 | |
| fungi | mtreinish: yep | 15:24 |
| *** andreaf has quit IRC | 15:24 | |
| mtreinish | fungi: oh it was probably racing with the revert and the fail is in the check queue | 15:25 |
| anteaya | it was a devstack-precise-rax-dfw node | 15:25 |
| markmcclain | mtreinish: not sure why that check configured that way was added | 15:26 |
| fungi | aha. 13:42 is when testing started on markmcclain's linked change, 13:46 is when the taskflow revert merged | 15:29 |
| fungi | markmcclain: reverify bug 1300161 | 15:29 |
| fungi | should work now | 15:29 |
| uvirtbot | Launchpad bug 1300161 in devstack "Error import taskflow Failure" [Undecided,In progress] https://launchpad.net/bugs/1300161 | 15:29 |
| *** KurtMartin is now known as kmartin | 15:29 | |
| fungi | mtreinish: not racing per se. taskflow isn't integration-tested yet | 15:29 |
| fungi | which is how things got in that state to start with | 15:29 |
| markmcclain | fungi: thanks | 15:29 |
| annegentle | hey jeblair: if I'm looking for a replacement cms for the docs site, what licenses are allowable? | 15:30 |
| annegentle | jeblair: for example, jekyll is MIT | 15:30 |
| *** gyee has joined #openstack-infra | 15:30 | |
| annegentle | jeblair: so not just cms but build chains | 15:30 |
| mtreinish | fungi: fair enough, there are probably better ways to describe it | 15:30 |
| *** alexpilotti has joined #openstack-infra | 15:31 | |
| *** CaptTofu has joined #openstack-infra | 15:31 | |
| annegentle | jeblair: that's probably not super clear, but as I look to replace file-based scp, what are the first class licenses? | 15:31 |
| *** mkerrin has quit IRC | 15:31 | |
| *** thuc has joined #openstack-infra | 15:31 | |
| fungi | annegentle: running mit-licensed stuff is fine (i assume you're talking about the mit expat license, not other mit licenses like the x11 license or something) | 15:32 |
| *** mihgen has joined #openstack-infra | 15:34 | |
| *** alexpilotti has quit IRC | 15:36 | |
| *** UtahDave has joined #openstack-infra | 15:38 | |
| *** mkerrin has joined #openstack-infra | 15:38 | |
| *** dizquierdo has quit IRC | 15:41 | |
| *** roeyc has quit IRC | 15:42 | |
| *** dstanek has joined #openstack-infra | 15:43 | |
| *** thuc_ has joined #openstack-infra | 15:44 | |
| *** freyes has quit IRC | 15:45 | |
| *** khyati has joined #openstack-infra | 15:47 | |
| *** mfer has joined #openstack-infra | 15:48 | |
| *** thuc has quit IRC | 15:48 | |
| *** markmcclain has quit IRC | 15:48 | |
| annegentle | fungi: seems like Sphinx is BSD | 15:49 |
| annegentle | fungi: so that might also qualify | 15:49 |
| annegentle | fungi: what about something like https://github.com/mozilla/kitsune/blob/master/LICENSE where it's open but not MIT, BSD, GPL, Apache 2, who would I ask or talk to? | 15:49 |
| *** ok_delta has joined #openstack-infra | 15:53 | |
| *** CaptTofu has quit IRC | 15:56 | |
| *** hashar_ has quit IRC | 15:56 | |
| *** dstanek has quit IRC | 15:56 | |
| *** jcoufal has joined #openstack-infra | 15:58 | |
| pleia2 | good morning | 16:00 |
| *** mkoderer has quit IRC | 16:01 | |
| *** e0ne_ has joined #openstack-infra | 16:01 | |
| jeblair | annegentle: you're talking about something that will run on a git tree, and produce a bunch of static files that are uploaded to the server, right? | 16:01 |
| jeblair | annegentle: (as opposed to like running a full server-side cms like wordpress or zope or drupal...?) | 16:01 |
| *** jpich has quit IRC | 16:02 | |
| anteaya | morning pleia2 | 16:02 |
| *** hogepodge has joined #openstack-infra | 16:03 | |
| *** e0ne has quit IRC | 16:04 | |
| jeblair | annegentle: kitsune looks like a 3-clause bsd...? | 16:05 |
| *** jepoy has quit IRC | 16:06 | |
| *** thuc_ has quit IRC | 16:07 | |
| *** thuc has joined #openstack-infra | 16:07 | |
| *** changbl has joined #openstack-infra | 16:09 | |
| *** e0ne has joined #openstack-infra | 16:12 | |
| *** thuc has quit IRC | 16:12 | |
| jogo | is taskflow gated on tempest? | 16:12 |
| jeblair | jogo: in about 30 minutes it will be https://review.openstack.org/#/c/76945/ | 16:14 |
| *** sdake has joined #openstack-infra | 16:14 | |
| *** sdake has quit IRC | 16:14 | |
| *** sdake has joined #openstack-infra | 16:14 | |
| jogo | jeblair: sweet | 16:14 |
| *** e0ne_ has quit IRC | 16:15 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add gate jobs for oslo libraries https://review.openstack.org/76945 | 16:16 |
| * dhellmann does a kermit flail | 16:16 | |
| *** hogepodge has quit IRC | 16:16 | |
| *** markmc has quit IRC | 16:18 | |
| *** hogepodge has joined #openstack-infra | 16:20 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Fix missing attribute error in subnodes https://review.openstack.org/83843 | 16:20 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add a test for subnodes https://review.openstack.org/83602 | 16:20 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add a very basic functional test https://review.openstack.org/82952 | 16:20 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add per-test database fixture https://review.openstack.org/82865 | 16:20 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add tests for the allocator https://review.openstack.org/82628 | 16:20 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Finish initial docs https://review.openstack.org/82854 | 16:20 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add ready-script and multi-node support https://review.openstack.org/82252 | 16:20 |
| *** jerryz_ has quit IRC | 16:21 | |
| openstackgerrit | James Michael DuPont proposed a change to openstack-infra/git-review: list feedback command https://review.openstack.org/84165 | 16:22 |
| *** CaptTofu has joined #openstack-infra | 16:24 | |
| *** mfer has quit IRC | 16:24 | |
| jeblair | pleia2: can you provide the author with advice to find the equivalent fedora/rhel package in this change? https://review.openstack.org/#/c/84149/1 | 16:25 |
| *** jgallard has quit IRC | 16:25 | |
| pleia2 | jeblair: will do | 16:25 |
| pleia2 | jeblair: oh, this again, there isn't one in the main repos | 16:26 |
| pleia2 | jeblair: https://review.openstack.org/#/c/72287/ was related | 16:26 |
| *** che-arne has quit IRC | 16:27 | |
| *** lcostantino has joined #openstack-infra | 16:28 | |
| *** ildikov_ has quit IRC | 16:28 | |
| *** jlibosva has quit IRC | 16:29 | |
| openstackgerrit | A change was merged to openstack-infra/config: Enable Gearman as default on Jenkins slaves https://review.openstack.org/73214 | 16:29 |
| jeblair | pleia2: ok. i think the current change is probably good then; it has the idea from the old one that clarkb liked | 16:30 |
| openstackgerrit | A change was merged to openstack-infra/config: New geard server address for tripleo jobs. https://review.openstack.org/84061 | 16:33 |
| *** esker has joined #openstack-infra | 16:34 | |
| *** markmcclain has joined #openstack-infra | 16:36 | |
| *** SumitNaiksatam has quit IRC | 16:38 | |
| *** e0ne has quit IRC | 16:39 | |
| *** bearhands is now known as comstud | 16:40 | |
| clarkb | morning | 16:40 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1300161 https://review.openstack.org/84181 | 16:40 |
| uvirtbot | Launchpad bug 1300161 in devstack "Error import taskflow Failure" [Undecided,In progress] https://launchpad.net/bugs/1300161 | 16:40 |
| jogo | clarkb: morning | 16:41 |
| jogo | clarkb: ^ did I use the tag correctly? | 16:41 |
| openstackgerrit | James Michael DuPont proposed a change to openstack-infra/git-review: list feedback command https://review.openstack.org/84165 | 16:41 |
| *** alff has joined #openstack-infra | 16:41 | |
| *** alff_ has joined #openstack-infra | 16:41 | |
| clarkb | looking | 16:41 |
| clarkb | jogo: yes lgtm | 16:44 |
| *** lcostantino has quit IRC | 16:45 | |
| jogo | clarkb: thanks | 16:45 |
| *** dcramer_ has quit IRC | 16:45 | |
| jeblair | clarkb, fungi: regarding this change: https://review.openstack.org/#/c/71189/4 | 16:46 |
| jeblair | clarkb, fungi: i'm a little skittish about adding new system installed pip packages on tests slaves, particularly for something that runs _after_ the tests... | 16:46 |
| jeblair | clarkb, fungi: do you have thoughts on that? | 16:46 |
| openstackgerrit | A change was merged to openstack-infra/config: upgrade review-dev.o.o to gerrit 2.8.3 https://review.openstack.org/81618 | 16:47 |
| openstackgerrit | James Michael DuPont proposed a change to openstack-infra/git-review: list feedback command https://review.openstack.org/84165 | 16:47 |
| *** mrmartin has joined #openstack-infra | 16:48 | |
| jeblair | clarkb, fungi: both of those are available as os packages, and i'm pretty sure requests at least could probably be done with stdlib functions if needed... | 16:48 |
| fungi | jeblair: any chance we could cache those eggs on node images and then run zuul_swift_upload.py inside a virtualenv | 16:48 |
| fungi | ? | 16:48 |
| fungi | oh! if there are debs/rpms available, then even better | 16:48 |
| clarkb | jeblair: ya if we can get away with apt-get installing those packages that would be much preferred | 16:48 |
| *** harlowja_away is now known as harlowja | 16:48 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add non voting gate-rally-python33 check https://review.openstack.org/84013 | 16:49 |
| fungi | magic and requests aren't things i would try to reimplement on stdlib though | 16:49 |
| jeblair | fungi: def not magic, but the stdlib does have an httplib that should work... | 16:49 |
| jeblair | fungi, clarkb: but anyway, it sounds like we all think debs would be the least problematic | 16:49 |
| jeblair | fungi, clarkb: so i'll start with that. if for some reason that's untenable, i think we could do fungi's idea of the venv... | 16:50 |
| fungi | jeblair: i though httplib didn't validate x.509 certs for https connections, but maybe it just takes extra work | 16:50 |
| openstackgerrit | A change was merged to openstack-infra/config: Adds second hiera data directory https://review.openstack.org/71739 | 16:50 |
| jeblair | fungi: i think the nodepool prep script could just create that venv when it runs, so it would already be there and set up for the script at the end | 16:51 |
| fungi | jeblair: great point, and easier than setting up an egg cache to install from later | 16:51 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Add new job to build misc Japanese Guides https://review.openstack.org/83979 | 16:52 |
| fungi | but yeah, make it work with python-requests and python-magic versions carried in precise and centos6 is definitely preferred | 16:52 |
| *** mihgen has quit IRC | 16:52 | |
| mtreinish | jeblair: do you have any thoughts on this?: https://review.openstack.org/#/c/83192/ | 16:53 |
| *** SumitNaiksatam has joined #openstack-infra | 16:54 | |
| jeblair | mtreinish: can you push a new version with the topic "new-project" ? | 16:54 |
| mtreinish | jeblair: sure | 16:54 |
| jeblair | mtreinish: (we didn't see it on friday, otherwise we would have created it then) | 16:54 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add specs-cookiecutter to openstack-dev https://review.openstack.org/83192 | 16:55 |
| mtreinish | jeblair: ^^^ | 16:55 |
| *** Ryan_Lane has joined #openstack-infra | 16:56 | |
| openstackgerrit | Thomas Goirand proposed a change to openstack/requirements: Using happybase >=0.5,!=0.7 https://review.openstack.org/82438 | 16:56 |
| devananda | good morning, all | 16:56 |
| jeblair | mtreinish: thanks, lgtm | 16:56 |
| devananda | clarkb: it looks like that fix for translations on friday did not work | 16:56 |
| jeblair | mtreinish: should get made on friday | 16:56 |
| mtreinish | jeblair: ok cool | 16:57 |
| clarkb | devananda: have a link to your change that should've fixed it? | 16:58 |
| mtreinish | I'll send something to ML about it then | 16:58 |
| clarkb | devananda: and a log to a failing test if you have it would be handy | 16:58 |
| nibalizer | wooo second hiera dir landed! | 16:58 |
| devananda | clarkb: https://review.openstack.org/83935 | 16:58 |
| *** pblaho has quit IRC | 16:58 | |
| * nibalizer does his happy dance | 16:58 | |
| *** CaptTofu has quit IRC | 16:58 | |
| devananda | clarkb: i'm not sure how to find the jenkins translation job logs | 16:59 |
| clarkb | devananda: I can find it, but it should be http://logs.openstack.org/post/firsttwocharsofgitsha/completesha | 16:59 |
| *** sarob has joined #openstack-infra | 17:00 | |
| devananda | clarkb: https://review.openstack.org/#/c/83956/ is the currenly-proposed update from jenkins | 17:00 |
| *** krotscheck has quit IRC | 17:00 | |
| devananda | clarkb: http://logs.openstack.org/periodic/ironic-propose-translation-update/9061d0d/console.html is the log from the latest run AFAICT | 17:01 |
| *** krotscheck has joined #openstack-infra | 17:01 | |
| clarkb | devananda: http://logs.openstack.org/fb/fbc47c51fb20a52b40a21f4ce7237a31630650c1/post/ is the job run after 83935 merged | 17:01 |
| devananda | clarkb: new error: Could not import file: We're not able to extract any string from the file uploaded for language Portuguese (Brazil) (pt_BR) in resource | 17:02 |
| *** derekh has quit IRC | 17:04 | |
| *** mgagne1 has quit IRC | 17:04 | |
| clarkb | devananda: progress! | 17:05 |
| clarkb | devananda: http://logs.openstack.org/fb/fbc47c51fb20a52b40a21f4ce7237a31630650c1/post/ironic-upstream-translation-update/8367d2c/console.html.gz shows el was pushed | 17:06 |
| clarkb | and sr | 17:06 |
| clarkb | possibly a format issue in the other fiels? | 17:06 |
| clarkb | looks like those files may have not had any duplicates in the first place | 17:07 |
| *** alexpilotti has joined #openstack-infra | 17:07 | |
| openstackgerrit | A change was merged to openstack-infra/config: Upload oslo.messaging final releases to pypi https://review.openstack.org/83740 | 17:07 |
| devananda | hmm | 17:08 |
| openstackgerrit | Thomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates https://review.openstack.org/83761 | 17:09 |
| jeblair | fungi: i think havana is broken; do you know if that is known? | 17:10 |
| jeblair | 2014-03-31 15:55:27 CRITI [ceilometer] __init__() got an unexpected keyword argument 'match' | 17:10 |
| jeblair | http://logs.openstack.org/43/80643/5/check/check-tempest-dsvm-full-havana/d7f3b4d/logs/devstacklog.txt.gz | 17:10 |
| fungi | jeblair: i hadn't spotted it yet--presumably recent? | 17:11 |
| *** rwsu has quit IRC | 17:11 | |
| jeblair | sdague, mtreinish, adam_g: ^ | 17:12 |
| clarkb | devananda: I wonder if the mention about remote time vs local time indicates the problem? Looks like transifex was in the past compared to the slave | 17:13 |
| clarkb | devananda: maybe they reject things from the future? | 17:13 |
| clarkb | devananda: I am still doing more digging | 17:13 |
| devananda | clarkb: thanks | 17:13 |
| devananda | clarkb: i'm looking at the diff of sr and pt_BR | 17:13 |
| devananda | not a lot | 17:13 |
| devananda | clarkb: http://paste.openstack.org/show/74673/ | 17:14 |
| *** gokrokve has joined #openstack-infra | 17:15 | |
| *** markwash has joined #openstack-infra | 17:16 | |
| *** jooools has quit IRC | 17:16 | |
| *** mgagne has joined #openstack-infra | 17:17 | |
| clarkb | yeah, small diff | 17:17 |
| *** dripton has quit IRC | 17:17 | |
| jeblair | clarkb: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQ1JJVEkgW2NlaWxvbWV0ZXJdIF9faW5pdF9fKCkgZ290IGFuIHVuZXhwZWN0ZWQga2V5d29yZCBhcmd1bWVudCAnbWF0Y2gnXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjM2MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzk2Mjg2MjE2ODU1fQ== | 17:17 |
| *** dripton has joined #openstack-infra | 17:18 | |
| jeblair | clarkb: it looks like a huge chunk of the log was stored as a single entry | 17:18 |
| *** gokrokve_ has quit IRC | 17:18 | |
| mtreinish | jeblair: yeah it looks like the last periodic tempest stable failed with that error on both jobs with mysql | 17:19 |
| jeblair | (and that explains why if i searched for more than 60m my browser hung) | 17:19 |
| mtreinish | but the postgres one worked | 17:19 |
| clarkb | jeblair: probably broke multiline parsing when we went to devstack log | 17:19 |
| jeblair | mtreinish: looks like it's alembic related | 17:19 |
| clarkb | jeblair: yup, no milliseconds | 17:20 |
| clarkb | sdague: ^ | 17:20 |
| devananda | clarkb: based on Revision-Date, most of the translations havent' been updated since 2013-11-12 16:15+0000 | 17:21 |
| clarkb | devananda: ya, the two that did push have more recent dates so my theory about the time thing seems bad | 17:21 |
| clarkb | jeblair: mtreinish sdague also no | | 17:21 |
| clarkb | devananda: https://github.com/transifex/transifex-client/issues/16 | 17:24 |
| clarkb | devananda: are those files simply empty of anything to push? | 17:24 |
| *** mgagne has quit IRC | 17:24 | |
| clarkb | it may be working properly now | 17:24 |
| clarkb | at least that bit | 17:24 |
| devananda | pt_BR, for example, has a 46% completion in the web UI | 17:25 |
| devananda | but nothing in the .po file | 17:25 |
| devananda | there are several other languages with non-zero translations in the UI | 17:25 |
| devananda | but zero in the .po files | 17:25 |
| clarkb | right so thats a different problem | 17:25 |
| devananda | ? | 17:25 |
| clarkb | devananda: I/we were looking at why transifex was rejecting our pushes and it appears because there are no strings to push | 17:25 |
| clarkb | now we need to find out why it isn't pushing any strings to gerrit | 17:26 |
| *** sarob has quit IRC | 17:26 | |
| devananda | clarkb: ah. "it isn't pushing any strings to gerrit" is the problem I've been after since the start. sorry if that wasnt' clear | 17:26 |
| *** sarob has joined #openstack-infra | 17:26 | |
| devananda | clarkb: https://bugs.launchpad.net/ironic/+bug/1298645 is the bug, if that helps | 17:26 |
| uvirtbot | Launchpad bug 1298645 in openstack-i18n "translated .po files do not contain any translation text" [Undecided,New] | 17:26 |
| devananda | clarkb: also affects other projects | 17:27 |
| clarkb | devananda: http://logs.openstack.org/fb/fbc47c51fb20a52b40a21f4ce7237a31630650c1/post/ironic-upstream-translation-update/8367d2c/console.html.gz#_2014-03-29_01_01_17_605 | 17:27 |
| asselin | Hi, I'm working on setting up local certification tests for cinder. I'm trying to get our jenkins setup with nodepool. I see the image 'state' in 'building'. How long does that take? Anyone know how to debug what's it's doing? I want to make sure it's not stuck. | 17:27 |
| clarkb | devananda: that is the symptom I think | 17:27 |
| clarkb | asselin: you should have an image.log in /var/log/nodepool | 17:27 |
| clarkb | asselin: it takes a while, the images are built then snapshotted so that test slaves can be booted off of that snapshot | 17:28 |
| clarkb | depending on network speeds and disk sizes this can take quite a bit of time | 17:28 |
| jeblair | asselin: ours take 30-60 minutes usually | 17:28 |
| *** jcoufal has quit IRC | 17:29 | |
| clarkb | devananda: http://support.transifex.com/customer/portal/questions/1045471-tx-pull-f-skipping-everything that could be biting us, /me digs | 17:29 |
| *** khyati has quit IRC | 17:29 | |
| devananda | i've got an env locally now where I can reproduce the skipping everythign issue | 17:29 |
| clarkb | devananda: awesome | 17:29 |
| asselin | clarkb, I don't see this file. I'm running everything on a single blade. Icehouse provider, jenkins, nodepool, etc. I'm running nodepoold -d | 17:30 |
| fungi | asselin: you can also 'nodepool image-update providername imagename' to see it working in your terminal session instead of going to the log | 17:30 |
| *** mgagne has joined #openstack-infra | 17:30 | |
| clarkb | devananda: what if you do tx pull -a --force? | 17:30 |
| devananda | clarkb: same thing | 17:30 |
| devananda | clarkb: there doesn't seem to be any --verbose option :( | 17:31 |
| *** gokrokve has quit IRC | 17:31 | |
| *** sarob has quit IRC | 17:31 | |
| devananda | it just skips the files | 17:31 |
| devananda | with no explanation | 17:31 |
| *** gokrokve has joined #openstack-infra | 17:32 | |
| clarkb | :/ | 17:32 |
| *** hashar has joined #openstack-infra | 17:34 | |
| clarkb | the slave's .transifexrc looks good at least as compared to their documentation | 17:34 |
| *** gokrokve_ has joined #openstack-infra | 17:34 | |
| *** ildikov_ has joined #openstack-infra | 17:34 | |
| devananda | clarkb: do I need any special credentials to get "tx pull" to work? | 17:34 |
| clarkb | devananda: you may, I think it depends on the project ACLs /me looks at those | 17:35 |
| clarkb | devananda: you definitely need them for push but I would expect pull to work without creds | 17:35 |
| openstackgerrit | A change was merged to openstack-infra/config: update comment filter for gerrit 2.8 https://review.openstack.org/70864 | 17:35 |
| devananda | clarkb: i'm sure I do for push... right | 17:35 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Remove maven from operations-guide and openstack-manuals https://review.openstack.org/77446 | 17:35 |
| *** gokrokve has quit IRC | 17:36 | |
| *** gokrokve has joined #openstack-infra | 17:37 | |
| asselin | fungi, thank you. I now get the error message: "ERROR:nodepool.utils:Exception while testing ssh access" I'll investigate from here. | 17:37 |
| *** morganfainberg_Z is now known as morganfainberg | 17:38 | |
| *** gokrokve_ has quit IRC | 17:38 | |
| *** vhoward- has left #openstack-infra | 17:38 | |
| devananda | clarkb: ahh. --debug | 17:40 |
| devananda | clarkb: looks like pull needs creds. http://paste.openstack.org/show/ruvLee3Tt6VxMFRVi0kc/ | 17:40 |
| *** e0ne has joined #openstack-infra | 17:41 | |
| clarkb | devananda: well, jenkins should have them, let me run the --debug in the workspace | 17:41 |
| devananda | checked teh URL directly, looks ilke I have enough at least for pull. lemme see how to set this up | 17:42 |
| clarkb | devananda: you need a .transifexrc | 17:42 |
| devananda | clarkb: right. so, when I add that, pull works fine | 17:43 |
| devananda | well pull with -s -f | 17:43 |
| devananda | ahhh | 17:43 |
| devananda | Local is newer than remote for lang pt_BR | 17:43 |
| devananda | Skipping 'pt_BR' translation (file: ironic/locale/pt_BR/LC_MESSAGES/ironic.po). | 17:43 |
| devananda | so yes, time is the problem | 17:43 |
| clarkb | devananda: we should update the jobs to do pull -a -f then? | 17:44 |
| clarkb | also I was able to run the command you pasted in the workspace | 17:44 |
| devananda | yep, that seems to fix pull for me | 17:44 |
| devananda | and sure enough, there's a TON of diffs | 17:45 |
| devananda | clarkb: should I wait for jenkins, or do this pull -f -a now and propose to gerrit? | 17:45 |
| devananda | clarkb: this is the only change blocking our RC at this point that doesn't have code up for review | 17:45 |
| clarkb | devananda: its up to you, would be nice to see jenkins do it for us though | 17:46 |
| *** amcrn has joined #openstack-infra | 17:46 | |
| fungi | jeblair: is cmd.nodepool unaware of the labels config change? http://paste.openstack.org/show/74676/ (just starting to dig into it now) | 17:46 |
| clarkb | devananda: I can push the change really quickly then we can see how reviews go | 17:46 |
| devananda | clarkb: sounds good. just didn't want to wait for the periodic tonight :) | 17:46 |
| *** rustlebee is now known as russellb | 17:47 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1300161 https://review.openstack.org/84181 | 17:47 |
| uvirtbot | Launchpad bug 1300161 in taskflow "Error import taskflow Failure" [Critical,Fix committed] https://launchpad.net/bugs/1300161 | 17:47 |
| jeblair | fungi: ah, probably. i'll fix it. | 17:47 |
| fungi | jeblair: thanks! also the rebased stack lgtm if we want to press forward merging the remainder | 17:48 |
| *** gokrokve has quit IRC | 17:49 | |
| *** sandywalsh_ has joined #openstack-infra | 17:49 | |
| fungi | also, clarkb: jeblair: any reason not to approve https://review.openstack.org/72780 ? (lgtm) | 17:49 |
| *** gokrokve has joined #openstack-infra | 17:49 | |
| *** packet has joined #openstack-infra | 17:50 | |
| fungi | seems we already got the other machinery in place to default to disabling sudo in jobs using those nodes | 17:50 |
| *** dstanek has joined #openstack-infra | 17:50 | |
| *** sandywalsh has quit IRC | 17:50 | |
| *** Ajaeger1 has joined #openstack-infra | 17:50 | |
| *** jaypipes has joined #openstack-infra | 17:50 | |
| Ajaeger1 | clarkb: are you around? | 17:51 |
| fungi | yeah, i guess that was 72779 | 17:51 |
| *** hashar has quit IRC | 17:51 | |
| *** DuncanT- has joined #openstack-infra | 17:51 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Fix update-image command https://review.openstack.org/84190 | 17:52 |
| jeblair | fungi: ^; checking on your other Q | 17:52 |
| *** branen has quit IRC | 17:52 | |
| clarkb | Ajaeger1: yes | 17:53 |
| clarkb | fungi: I think 72780 should be good | 17:53 |
| Ajaeger1 | clarkb, great! I have a question about the duplicate strings where you commented on the ironic patch (bug and references at https://bugs.launchpad.net/heat/+bug/1299349 ). | 17:53 |
| uvirtbot | Launchpad bug 1299349 in nova "upstream-translation-update Jenkins job failing" [Undecided,In progress] | 17:53 |
| Ajaeger1 | clarkb: duplicate strings in po files. | 17:53 |
| *** gokrokve has quit IRC | 17:53 | |
| Ajaeger1 | clarkb: DuncanT- and myself discussed how to avoid something like this in the future. | 17:53 |
| fungi | jeblair: clarkb: cool, thanks! | 17:54 |
| Ajaeger1 | Do you have any idea why those where generated - and what kind of gating can we add to get notified about it? | 17:54 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make transifex the only source of translations https://review.openstack.org/84191 | 17:54 |
| Ajaeger1 | clarkb: Just run "msgfmt -c " on all po files would be an option | 17:54 |
| *** alff has quit IRC | 17:54 | |
| *** alff_ has quit IRC | 17:54 | |
| clarkb | Ajaeger1: devananda ^ something like that should fix ironics pull problem | 17:55 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add python27 job to nova-specs https://review.openstack.org/84192 | 17:55 |
| clarkb | Ajaeger1: is it a bug in Babel? | 17:55 |
| fungi | clarkb: though that will probably make transitioning from transifex to not_transifex harder | 17:55 |
| Ajaeger1 | clarkb: that one implements probably what we discussed via email ;) | 17:55 |
| clarkb | Ajaeger1: but yes, we can probably add a gating test that runs msgfmt -c or similar | 17:55 |
| *** jaypipes has quit IRC | 17:55 | |
| jogo | not sure if I did that patch correctly ^^ | 17:55 |
| clarkb | fungi: no it shouldn't | 17:55 |
| jogo | russellb: ^^^ | 17:56 |
| clarkb | fungi: we would just treat not transifex as the only source of truth when we switch to that | 17:56 |
| devananda | Ajaeger1: I just replied to the i18n thread and posted on the ironic / i18n bug. | 17:56 |
| fungi | clarkb: right, but we can't do a phased move of the community on a given project from one interface to the other. maybe we never could though | 17:56 |
| Ajaeger1 | clarkb: I don't know when it got introduced since nobody watches the post/periodic jobs, haven't isolated the cause. | 17:56 |
| Ajaeger1 | devananda: thanks! | 17:56 |
| devananda | Ajaeger1: I'm not conviced that duplicate msgs were the problem. all the "pull" were failing | 17:56 |
| clarkb | Ajaeger1: we could protect against it in the changes that update the files | 17:57 |
| clarkb | Ajaeger1: with a gating job | 17:57 |
| devananda | Ajaeger1: its posible there may have been two different problems | 17:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Allow bare-precise nodes to sudo https://review.openstack.org/72780 | 17:57 |
| clarkb | Ajaeger1: so that we will at least know broken is being introduced which could help in debugging the problem | 17:57 |
| Ajaeger1 | devananda: yes, these are two different problems. | 17:57 |
| clarkb | devananda: yes, I think there were two problems. The duplicate messages and needing to force pulls beacuse git can update the files directly outside of transifex's knowledge | 17:57 |
| devananda | ah, ok | 17:58 |
| *** denis_makogon_ has joined #openstack-infra | 17:58 | |
| Ajaeger1 | clarkb: agreed. | 17:58 |
| * Ajaeger1 now really reviews clarkb's patch | 17:58 | |
| *** denis_makogon_ is now known as denis_makogon | 17:59 | |
| clarkb | fungi: ya I am not sure how easy that would've been | 17:59 |
| clarkb | fungi: IMO we would really need to completely kill the openstack org on transifex all at once otherwise there will be confusion | 17:59 |
| jogo | clarkb fungi: we are hoping to get https://review.openstack.org/#/c/84192/ up so we can start accepting nova-specs patches | 18:00 |
| *** imcsk8_lap has quit IRC | 18:02 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revoke sudo on more jobs https://review.openstack.org/84193 | 18:02 |
| fungi | clarkb: probably so--fair enough | 18:02 |
| *** imcsk8_lap has joined #openstack-infra | 18:02 | |
| jeblair | fungi: ^ | 18:02 |
| *** thuc has joined #openstack-infra | 18:02 | |
| Ajaeger1 | clarkb: propose_translation_update_manual.sh could remove the "Push .pot changes to transifex" section as well - shouldn't it? | 18:03 |
| *** thuc has quit IRC | 18:03 | |
| *** markmcclain has quit IRC | 18:03 | |
| clarkb | Ajaeger1: looking | 18:03 |
| *** thuc has joined #openstack-infra | 18:03 | |
| *** jp_at_hp has quit IRC | 18:03 | |
| Ajaeger1 | clarkb: the if starting at line 85 | 18:04 |
| clarkb | Ajaeger1: ah yup, I didn't see it there, I looked at the end of the file. New patchset incoming | 18:04 |
| *** khyati_ has joined #openstack-infra | 18:04 | |
| clarkb | Ajaeger1: should I add the -f to the tx pull -a in that file too? | 18:04 |
| jogo | jeblair: https://review.openstack.org/#/c/84192/ hoping to get that in soonish so we can gate nova-specs more | 18:05 |
| Ajaeger1 | clarkb: wouldn't hurt IMO | 18:05 |
| jogo | (assuming I didn't make mistake in that patch) | 18:05 |
| jeblair | fungi: did you mean to aprv the revoke sudo on more jobs change? because you left a comment about it on the nodepool image-update change approval. | 18:05 |
| fungi | gah! too many tabs open at once | 18:06 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make transifex the only source of translations https://review.openstack.org/84191 | 18:06 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add ready-script and multi-node support https://review.openstack.org/82252 | 18:06 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add tests for the allocator https://review.openstack.org/82628 | 18:06 |
| mtreinish | jogo: are there any of those improvements you've made to the nova-specs repo that you think would be good to get in the specs-cookiecutter? | 18:06 |
| Ajaeger1 | clarkb: regarding the "msgfmt -c " checking job - should this be a complete separate job? | 18:06 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Finish initial docs https://review.openstack.org/82854 | 18:06 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add per-test database fixture https://review.openstack.org/82865 | 18:06 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add a very basic functional test https://review.openstack.org/82952 | 18:06 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add a test for subnodes https://review.openstack.org/83602 | 18:06 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Fix missing attribute error in subnodes https://review.openstack.org/83843 | 18:06 |
| clarkb | Ajaeger1: yes I think so. We would then add it to the check/gate jobs for projects. As an alternative we could add it to the pep8 job | 18:06 |
| jogo | mtreinish: probably | 18:06 |
| jogo | mtreinish: link for specs cookie cutter | 18:06 |
| clarkb | since the pep8 job has become a collection point for static checks | 18:06 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Fix update-image command https://review.openstack.org/84190 | 18:06 |
| Ajaeger1 | DuncanT-: proposed the pep8 job as well ;) | 18:07 |
| mtreinish | jogo: https://github.com/mtreinish/specs-cookiecutter | 18:07 |
| clarkb | Ajaeger1: in that case I am happy with that solution :) | 18:07 |
| mtreinish | jogo: it should be added to openstack-dev on Fri | 18:07 |
| fungi | jeblair: yep, thanks | 18:07 |
| clarkb | Ajaeger1: we can rely on Babel possibly if Babel can do a msgfmt -c for us if we use the pep8 job | 18:07 |
| jogo | mtreinish: https://review.openstack.org/#/c/83613/ | 18:08 |
| Ajaeger1 | clarkb, the problem is that right now it would fail until these patches are in: https://review.openstack.org/#/c/83951/ , https://review.openstack.org/#/c/83954/ , https://review.openstack.org/#/c/83960/ , https://review.openstack.org/#/c/83961/ , | 18:08 |
| clarkb | Ajaeger1: yup | 18:08 |
| openstackgerrit | A change was merged to openstack-infra/config: Add python27 job to nova-specs https://review.openstack.org/84192 | 18:08 |
| jogo | mtreinish: https://review.openstack.org/#/c/84157/ | 18:08 |
| clarkb | Ajaeger1: we will probably need to coordinate the change with the projects | 18:08 |
| *** dcramer_ has joined #openstack-infra | 18:08 | |
| jogo | mtreinish: https://review.openstack.org/#/c/83934/ | 18:08 |
| Ajaeger1 | clarkb: yep. | 18:08 |
| clarkb | Ajaeger1: if we use the pep8 job that is easy, we can make the pep8 changes depend on the changes you linked | 18:08 |
| clarkb | Ajaeger1: if we go with a new job it is a bit more complicated because the new job config is in the infra/config repo | 18:09 |
| *** markmcclain has joined #openstack-infra | 18:09 | |
| jogo | mtreinish: and https://review.openstack.org/#/c/83617/ | 18:09 |
| jogo | what is the new job? | 18:09 |
| jogo | one downside of adding directly to pep8 is we can't phase it in as non-voting first | 18:09 |
| clarkb | jogo: thats fine, we wouldn't need to | 18:09 |
| Ajaeger1 | clarkb: ok, let me work prototype this in cinder with DuncanT- ... | 18:10 |
| clarkb | jogo: because the change to ad dit to pep8 won't merge until it passes | 18:10 |
| mtreinish | jogo: ok, yeah I'll add those to the cookiecutter. The unit test one though is kinda dependent on the template format | 18:10 |
| jogo | clarkb: link? | 18:10 |
| clarkb | jogo: link to what? | 18:10 |
| jogo | mtreinish: yeah agreed | 18:10 |
| jogo | clarkb: what do you want to add to pep8? | 18:10 |
| clarkb | jogo: msgfmt -c or equivalent | 18:11 |
| *** alff has joined #openstack-infra | 18:11 | |
| *** alff_ has joined #openstack-infra | 18:11 | |
| jogo | clarkb: ahh | 18:11 |
| *** _nadya_ has joined #openstack-infra | 18:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revoke sudo on more jobs https://review.openstack.org/84193 | 18:13 |
| *** gyee has quit IRC | 18:13 | |
| jogo | heh that failed in nova | 18:14 |
| clarkb | Babel doesn't seem ot have msgfmt -c functionality | 18:14 |
| openstackgerrit | Akihiro Motoki proposed a change to openstack-infra/config: Add language spepcific font package to Jenkins slave https://review.openstack.org/84149 | 18:14 |
| clarkb | but msgfmt should work fine | 18:14 |
| jogo | adding to pep8 seems reasonable -- I just don't want another conf.sample snafu | 18:14 |
| jogo | fungi jeblair: thank you | 18:15 |
| clarkb | jogo: conf.sample is specail because external changes make you break | 18:15 |
| *** gyee has joined #openstack-infra | 18:15 | |
| clarkb | jogo: this would all be internal to repo stuff | 18:15 |
| jogo | clarkb: agreed | 18:15 |
| Ajaeger1 | clarkb: indeed, it does not have it, so need to run msgfmt -c. | 18:16 |
| jogo | new job configs get pushed out on the hour? | 18:17 |
| *** kevinbenton has joined #openstack-infra | 18:17 | |
| jogo | (https://review.openstack.org/#/c/84192/) | 18:17 |
| Ajaeger1 | clarkb: what's the best way to add it? A script in tools of cinder that runs a for loop over all files and files if one is wrong? | 18:17 |
| clarkb | Ajaeger1: I would use find in the tox.ini directly | 18:17 |
| clarkb | Ajaeger1: I don't think we need an actual script | 18:17 |
| *** mrodden has quit IRC | 18:18 | |
| jeblair | jogo: it's complicated and slightly random. should happen within 10 minutes or so after the first quarter hour mark after the change merged. | 18:18 |
| fungi | is openstack-ci the best place to leave a toci bug? and who can do a good job of triaging those? i added a toci tag to bug 1300208 but would like to give someone a heads up on it since that isn't (currently) an official tag to which anyone can subscribe | 18:19 |
| uvirtbot | Launchpad bug 1300208 in openstack-ci "ERROR: the main setup script run by this job " [Undecided,New] https://launchpad.net/bugs/1300208 | 18:19 |
| *** mrodden has joined #openstack-infra | 18:19 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Replace more searches with filters. https://review.openstack.org/83828 | 18:19 |
| clarkb | mtreinish: sdague jogo ^ that fixes a merge conflict of a change that was approved once | 18:19 |
| sdague | clarkb: lgtm | 18:20 |
| clarkb | fungi: agreed, I am told the reason it went there is toci is an openstack-infra project | 18:20 |
| clarkb | fungi: maybe we can open up the bug drivers list? | 18:20 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/requirements: Bump minimum requirement for dogpile.cache https://review.openstack.org/84200 | 18:20 |
| Ajaeger1 | clarkb: you need to call msgfmt on each file individually and cannot pass all files to it. | 18:21 |
| clarkb | Ajaeger1: right, find can do that | 18:21 |
| * Ajaeger1 was thinking to complicated I guess... | 18:21 | |
| mtreinish | clarkb: oh, it's because someone went and self approved a patch this morning that deleted the queries out from under you... | 18:22 |
| clarkb | find . -type f -name'.pot?' -execdir msgfmt -c {} \; | 18:24 |
| Ajaeger1 | clarkb: find cinder/locale/ -name "cinder.po" -exec msgfmt -c {} \; | 18:24 |
| clarkb | Ajaeger1: ^ Ithink that should do it assuming I got the msgfmt command correct | 18:24 |
| Ajaeger1 | clarkb: let's merge ours - problem is that it returns with 0 while called individually msgfmt returns with 1 | 18:25 |
| clarkb | Ajaeger1: -execdir is apparently prefered but that should work | 18:25 |
| clarkb | find should return >0 if any command returns >0 I thought | 18:25 |
| * Ajaeger1 tested and was surprised as well ;( | 18:26 | |
| clarkb | oh right it returns not 0 if find itself fails | 18:27 |
| clarkb | xargs can help here | 18:27 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Replace more searches with filters. https://review.openstack.org/83828 | 18:28 |
| Ajaeger1 | clarkb: ok, next try... | 18:28 |
| *** gokrokve has joined #openstack-infra | 18:29 | |
| clarkb | find . -type f -name'*.pot?' -print0 | xargs --null -n 1 msgfmt -c | 18:29 |
| Ajaeger1 | clarkb: yeah, that should work - tested something similar... | 18:29 |
| Ajaeger1 | Ok, will update cinder now. | 18:29 |
| *** gokrokve has quit IRC | 18:34 | |
| *** ominakov has quit IRC | 18:34 | |
| *** gokrokve has joined #openstack-infra | 18:37 | |
| *** gokrokve_ has joined #openstack-infra | 18:38 | |
| Ajaeger1 | clarkb, here's the patch: https://review.openstack.org/#/c/84207/ | 18:39 |
| Ajaeger1 | I used --check-format since "-c" also warns about: | 18:39 |
| Ajaeger1 | "./cinder/locale/tl/LC_MESSAGES/cinder.po:7: warning: header field 'Language' missing in header" | 18:39 |
| clarkb | Ajaeger1: lgtm, we will know when the tests report back | 18:39 |
| Ajaeger1 | clarkb: it passes now on my local checkout - and failed before ;) | 18:40 |
| devananda | Ajaeger1: i haven't followed the whole conversation (juggling other issues in -ironic). | 18:41 |
| *** gokrokve has quit IRC | 18:41 | |
| devananda | Ajaeger1: are there similar steps that I need to take for ironic? | 18:41 |
| Ajaeger1 | devananda: I'll send you patch like the one I did for cinder, ok? | 18:42 |
| devananda | Ajaeger1: thanks! | 18:42 |
| *** gokrokve_ has quit IRC | 18:43 | |
| fungi | okay, this is weird... puppet agent on nodepool seems to have not updated for nearly 4 hours now... http://puppetdb.openstack.org/node/nodepool.openstack.org | 18:43 |
| phschwartz | jeblair, mordred: sorry for any delays in the Blueprint I promised you guys. I got stuck with a bit of internal work that was needed as a rush and a vacation for my son's 4th Bday. I will get it out for you guys as soon as posible. | 18:43 |
| fungi | i don't see it with any child processes | 18:43 |
| Ajaeger1 | devananda: https://review.openstack.org/84210 | 18:44 |
| fungi | aha, it broke *and* stopped reporting for some reason | 18:45 |
| mordred | fungi: excellent | 18:45 |
| fungi | Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item nodepool_mysql_password in any Hiera data file and no default supplied at /opt/config/production/manifests/site.pp:504 on node nodepool.openstack.org | 18:46 |
| mordred | fungi: that seems somewhat like a possible bug | 18:46 |
| fungi | changed between 16:58:21 and 17:08:15 | 18:46 |
| fungi | looking to see which change that was | 18:47 |
| devananda | Ajaeger1: ty | 18:47 |
| fungi | oh, 2 hours ago not 4 | 18:48 |
| *** thuc has quit IRC | 18:48 | |
| *** thuc has joined #openstack-infra | 18:48 | |
| *** thuc has quit IRC | 18:48 | |
| fungi | i'm thinking it's due to nibalizer's https://review.openstack.org/71739 | 18:48 |
| *** thuc has joined #openstack-infra | 18:49 | |
| clarkb | fungi: good catch, probably | 18:49 |
| clarkb | sdague: did you see the comments about devstacklog.txt being in the wrong format? | 18:50 |
| *** gokrokve has joined #openstack-infra | 18:51 | |
| *** thuc has quit IRC | 18:51 | |
| sdague | ? | 18:51 |
| Ajaeger1 | clarkb: while we're talking translations, I would like to see this one merged (after your patch 84191 is in): https://review.openstack.org/#/c/82573/ and would appreciate your review on it. Also, the translators like to see .tx/config in - so could you reconsider your -1 at https://review.openstack.org/#/c/82384/, please? | 18:51 |
| sdague | no, what's up? | 18:51 |
| *** thuc has joined #openstack-infra | 18:51 | |
| fungi | and then i logged into zuul to see how widespread this is... puppet stopped running about 19 hours ago, and / is 100% full | 18:51 |
| clarkb | sdague: the devstacklog file timestamps don't have milliseconds and the pipe is missing | 18:52 |
| sdague | reference? | 18:52 |
| clarkb | sdague: so it doesn't match the console filters for multiline | 18:52 |
| clarkb | sdague: http://logs.openstack.org/06/81306/1/check/check-tempest-dsvm-full/c449192/logs/devstacklog.txt.gz | 18:52 |
| *** mrmartin has quit IRC | 18:52 | |
| fungi | jeblair: clarkb: mordred: any objection to me deleting some older rotated logs from zuul.o.o? | 18:52 |
| clarkb | fungi: not from me, I don't think we are debugging any outstanding issues | 18:52 |
| mordred | fungi: go for it | 18:53 |
| sdague | clarkb: that's havana | 18:53 |
| sdague | but we should backport that | 18:53 |
| asselin | clarkb, fungi I can't seem to find why ssh is timing out. I'm using the default ubuntu image. SSH works when I create the instance manually. Any ideas what nodepool does diffiferently? I stepped through the code and didn't find anything other than the use of it's own keypair. | 18:53 |
| clarkb | pabelanger: you around? re https://review.openstack.org/#/c/83980/1 isn't it possible that we could delete nodes above min ready that were created due to demand? | 18:54 |
| fungi | looks like recent zuul debug logs are up to 2.2g apiece and we don't compress those when they're rotated | 18:54 |
| mordred | fungi: the problem exists on review.o.o as well - so I'd say it's widespread | 18:54 |
| clarkb | pabelanger: I think that change may want to compare n_ready to demand and no min_ready | 18:54 |
| *** dstanek has quit IRC | 18:54 | |
| clarkb | sdague: roger thanks | 18:54 |
| mordred | fungi: looking at ci-puppetmaster hiera config | 18:55 |
| fungi | mordred: okay, do you want to debug the hiera thing in production or should we revert that change for now? | 18:55 |
| mordred | fungi: let's look at it for a few minutes before reverting | 18:55 |
| mordred | otherwise we'll just be scared of it in the future | 18:55 |
| fungi | mordred: that's why i asked! | 18:55 |
| fungi | feel free | 18:55 |
| sdague | clarkb: https://review.openstack.org/84213 | 18:56 |
| sdague | should be the backport | 18:56 |
| fungi | okay, i deleted zuul debug logs older than a week, which has free'd up 75% of / now | 18:56 |
| *** sarob has joined #openstack-infra | 18:57 | |
| mordred | fungi, nibalizer: removing the two additional datadir stanzas made it work | 18:57 |
| clarkb | sdague: thank you | 18:58 |
| lifeless | clarkb: do we logstash infra systems yet? | 19:00 |
| mordred | lifeless: nope | 19:00 |
| *** david-lyle has quit IRC | 19:00 | |
| *** hashar has joined #openstack-infra | 19:01 | |
| clarkb | lifeless: I don't think there is much intention to either | 19:02 |
| clarkb | pabelanger: nevermind, I see how this works. min_ready there isn't the same as label.min_ready it is a calculated value | 19:02 |
| boris-42 | mordred btw I have some cagy idea about gates .. | 19:02 |
| boris-42 | mordred hi* | 19:02 |
| boris-42 | mordred so current goal of gates is to check that every change in every project doesn't brake anything else | 19:03 |
| mordred | clarkb: I'd love to - it's just low on the priority list I Think | 19:03 |
| mordred | boris-42: that's right | 19:03 |
| boris-42 | mordred but seems like such approach doesn't scale, e.g. if we add 10 new projects | 19:03 |
| mordred | boris-42: no | 19:03 |
| boris-42 | mordred and improve coverage 10 times | 19:03 |
| mordred | boris-42: let me stop you right there | 19:04 |
| clarkb | mordred: I really don't want to run a second ES cluster | 19:04 |
| boris-42 | mordred I know what you thought=) | 19:04 |
| mordred | boris-42: the solution scales fine | 19:04 |
| clarkb | mordred: we can, but its more than just where on the priority list it falls | 19:04 |
| sdague | clarkb: what, running the first one isn't a ton of fun? :) | 19:04 |
| mordred | and we'd like to continue to make it scale, rather than give up on it | 19:04 |
| boris-42 | mordred so how we are going to scale when tempest will work 10 times more? | 19:04 |
| mordred | boris-42: the same way we've handled the last bazillion scaling issues | 19:04 |
| *** jcoufal has joined #openstack-infra | 19:04 | |
| clarkb | sdague: yes "fun" :P | 19:04 |
| mordred | by solving them with technology | 19:04 |
| boris-42 | mordred sure sure | 19:05 |
| mordred | boris-42: :) | 19:05 |
| boris-42 | mordred but i have some ideas | 19:05 |
| boris-42 | mordred how to resolve tempest stuff | 19:05 |
| clarkb | sdague: it is completely worth it for the test logs as e-r and other things have shown. But I am not sure we are at that point where it would be worth it for system logs | 19:05 |
| dstufft | mordred: To the Cloud! (tm) | 19:05 |
| sdague | clarkb: yeh, I'm not disagreeing | 19:05 |
| mordred | boris-42: great - as long as they don't involved "run less things" or "dont' run things" or "dont' test as many things" | 19:05 |
| boris-42 | mordred currently we are running all bunch of tests in any case | 19:05 |
| *** e0ne has quit IRC | 19:05 | |
| mordred | boris-42: that's right - because the number of unexpected thigns that can break is too large to be able to assert that a change only needs a certain subset of tests | 19:06 |
| boris-42 | mordred probably we should split all bunch of tempest tests to "cross" projects | 19:06 |
| *** e0ne has joined #openstack-infra | 19:06 | |
| sdague | clarkb: honestly, it would be interesting to wrap up the lessons learned into an openstack project on it's own right and provide logging as a service, which I know jogo was shopping around last summit over drinks | 19:06 |
| mordred | they're all cross projectrs | 19:06 |
| mordred | boris-42: I promise - the system is REALLY interconnected | 19:06 |
| boris-42 | mordred sure | 19:06 |
| mordred | we catch _Crazy_ unexpected thigns all the time | 19:06 |
| boris-42 | =) | 19:06 |
| clarkb | sdague: meniscus does that, and laying the foundation for something along those lines was certainly one of the reasons we used tools like logstash and elasticsearch (because people are already using them in their deployments) | 19:07 |
| boris-42 | mordred but for example creation flavors shouldn't be run in ironic gate | 19:07 |
| jogo | clarkb: ++ I was trying to convice mtreinish to do it | 19:07 |
| boris-42 | mordred cause it is not related absolutely to ironic | 19:07 |
| boris-42 | mordred and key-pairs | 19:07 |
| boris-42 | mordred and a billions of other operation | 19:07 |
| clarkb | boris-42: ironic doesn't do flavors and ssh keys? | 19:08 |
| mordred | it does | 19:08 |
| boris-42 | clarkb mordred I didn't mean that.. | 19:08 |
| mordred | well, it does flavors for sure | 19:08 |
| mordred | I don't see why it wouldn't do key-pairs | 19:08 |
| *** e0ne has quit IRC | 19:08 | |
| boris-42 | mordred it is impossible to break flavors in nova from paths in ironic | 19:08 |
| boris-42 | patch* | 19:08 |
| boris-42 | mordred cause only patch in nova could affect nova | 19:08 |
| sdague | clarkb: Note: 1/10/2014 - Project Meniscus is being put on hold and the fulltime team resources are being moved to Barbican | 19:09 |
| *** _nadya_ has quit IRC | 19:09 | |
| sdague | https://github.com/ProjectMeniscus/meniscus | 19:09 |
| boris-42 | mordred so checking flavor-create in ironic gate is something useless | 19:09 |
| *** yolanda has quit IRC | 19:09 | |
| boris-42 | mordred cause ironic patch won't break nova stuff | 19:09 |
| openstackgerrit | Samuel Stavinoha proposed a change to openstack-dev/pbr: Don't overwrite cmd classes spec'd in setup.cfg https://review.openstack.org/84218 | 19:09 |
| boris-42 | mordred I know that it is simpler to have a bunch of test and to run them always | 19:09 |
| boris-42 | mordred but such approach won't scale.. | 19:10 |
| sdague | boris-42: it's a little more complicated than that | 19:10 |
| mtreinish | jogo: heh I've got a WIP tree on one of my laptops somewhere | 19:10 |
| boris-42 | sdague ? | 19:10 |
| sdague | there are definitely white spaces in the tests, where it is impossible for certain changes to be provably not able to break certain tests | 19:10 |
| mtreinish | jogo: but it doesn't do anything besides build | 19:10 |
| sdague | but most of those tests are pretty cheap | 19:11 |
| * clarkb pulls out https://etherpad.openstack.org/p/juno-test-maxtrices again | 19:11 | |
| sdague | and the cost of getting those whitespace assessments wrong is a gate wedge | 19:11 |
| clarkb | boris-42: ^ is the biggest problem imo | 19:11 |
| clarkb | boris-42: it isn't a matter of figuring out where we might able to not run tets | 19:11 |
| sdague | yeh, it's a meta testing problem then as well | 19:12 |
| clarkb | boris-42: instead we need to be able to have a more standard set of tests | 19:12 |
| sdague | anyway, I definitely encourage ideas | 19:12 |
| clarkb | everyone wants to optimize what tests they run, but fixing what features they support fixes most of these issues | 19:13 |
| sdague | however, I'd say that you should also pick up some experience in tracking down the gate wedges and races that we have first | 19:13 |
| sdague | clarkb: true :) | 19:13 |
| clarkb | because then we can rely on a few tests rather than specialized tests everywhere | 19:13 |
| boris-42 | sdague actually I am heavy working on part=) | 19:13 |
| boris-42 | sdague of fixing crap=) | 19:13 |
| boris-42 | sdague whole project started lol | 19:14 |
| boris-42 | =) | 19:14 |
| sdague | ok, well, I haven't seen you in the middle of most of the race bugs, so sorry if I missed that part | 19:14 |
| *** mrodden1 has joined #openstack-infra | 19:14 | |
| boris-42 | sdague ^ if you are think only about nowadays you won't be able to think about feature | 19:14 |
| boris-42 | sdague agree that we should fix "today" bugs | 19:15 |
| boris-42 | sdague but we should invest in future as well | 19:15 |
| sdague | boris-42: fair, but if you don't have experience with the way things work and break today, you are liable to design a solution that make it worse | 19:15 |
| *** mrodden has quit IRC | 19:15 | |
| boris-42 | sdague I have a lot of expirience | 19:15 |
| sdague | there is a lot of elbow grease to keep the current system running | 19:15 |
| boris-42 | sdague cause I was fixing a lot of big clouds 200/300 servers | 19:16 |
| boris-42 | sdague sure and thank you for your work | 19:16 |
| boris-42 | sdague I don't want to say that I am hero and you don't know anything lol=) | 19:16 |
| sdague | boris-42: I'm not talking about that :) I'm talking about the specifics of how we can block the whole gate and make everyone lose a day of development :) | 19:16 |
| mtreinish | jogo: I just pushed the updates the specs-cookiecutter, let me know there is anything else you think it needs | 19:17 |
| boris-42 | sdague sure sure I remember that time=) | 19:17 |
| jogo | mtreinish: looks good | 19:17 |
| sdague | clarkb: are you going to submit the test matrix bit into the cross project day? | 19:17 |
| clarkb | sdague: I should /me does that | 19:18 |
| *** zehicle_at_dell has quit IRC | 19:18 | |
| mtreinish | clarkb: yeah I was going to ask you todo that too. It's on my list of things to make sure gets discussed at summit | 19:18 |
| dstufft | "and by embrace I mean actually care about them instead of ignoring them" sounds like a defintiion of embrace that i'm familar with | 19:18 |
| sdague | I actually kind of think that one might need a double slot | 19:18 |
| clarkb | dstufft: :) its openstack's dirty secret that we tend to sweep that stuff under the rug imo | 19:19 |
| sdague | but we have such big rugs | 19:19 |
| sdague | red shag carpet | 19:19 |
| *** alff has quit IRC | 19:19 | |
| *** alff_ has quit IRC | 19:19 | |
| *** zehicle_at_dell has joined #openstack-infra | 19:19 | |
| *** Sukhdev has joined #openstack-infra | 19:21 | |
| *** thomasem has quit IRC | 19:21 | |
| * anteaya fires up the lava lamp | 19:24 | |
| *** yassine has quit IRC | 19:24 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Multiple datadir entries are not supported https://review.openstack.org/84221 | 19:25 |
| fungi | mordred: are you suggesting we should remove the two lines added at the end of https://review.openstack.org/#/c/71739/4/modules/openstack_project/files/puppetmaster/hiera.yaml ? | 19:25 |
| fungi | oh, that | 19:25 |
| mordred | fungi: ^^ that should fix | 19:25 |
| mordred | fungi, nibalizer we're going to have to re-think the puppet apply approach | 19:25 |
| mordred | I think specifically we're going to need a second specific hiera.yaml that is for puppet apply | 19:25 |
| *** yassine has joined #openstack-infra | 19:25 | |
| mordred | that is not a symlink to the one we install in the master | 19:26 |
| mordred | this means SOME copying, but since we dont' change that file, it hould still work | 19:26 |
| mordred | whereas the othe rthing is jut actually not functional | 19:26 |
| Ajaeger1 | clarkb: do I need to add find to whitelist_externals for the change? see http://logs.openstack.org/10/84210/1/check/gate-ironic-pep8/aacc4c2/console.html | 19:26 |
| clarkb | Ajaeger1: It shouldn't be necessary but will clean up the warning messages if you add find and xargs | 19:27 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/nodepool: Create snapshots when min-ready is >= 0 https://review.openstack.org/83975 | 19:28 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/nodepool: Delete surplus ready nodes if greater then min_ready https://review.openstack.org/83980 | 19:28 |
| Ajaeger1 | clarkb: I should have looked closer, error is: | 19:28 |
| Ajaeger1 | /usr/bin/find: unknown predicate `-print0|xargs' | 19:28 |
| clarkb | Ajaeger1: ah, needs whitespace after print0 | 19:29 |
| clarkb | Ajaeger1: oh actually | 19:29 |
| clarkb | Ajaeger1: these commands are execed directly, you may need to wrap the whole thing in bash -c | 19:29 |
| Ajaeger1 | clarkb: thanks, will check | 19:29 |
| clarkb | sdague: mtreinish: cross project workshops is the correct topic? | 19:30 |
| mtreinish | clarkb: yeah I think so | 19:30 |
| mtreinish | we need everyone to be there | 19:30 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Allow configurable large-ops tests https://review.openstack.org/80643 | 19:30 |
| sdague | clarkb: yeh | 19:33 |
| *** denis_makogon has quit IRC | 19:33 | |
| mordred | fungi, nibalizer: I'm going to test a followup patch to see what might work with applky | 19:34 |
| mordred | boris-42: I think the point from above is not that you may not have good ideas - I know you do - but simply that there is a team already working on the subject, and we've foudn in the past that if you don't start working directly with the current team on the current set of problems, you will, no matter how smart you are, lack context that's needed to make future-looking solutions | 19:36 |
| mordred | boris-42: so what I'd really liek to see before suggestions on how to deal with future scaling problems is to jump in to the current work to deal with the current set of scaling challenges we're working on now | 19:36 |
| mtreinish | clarkb: on https://review.openstack.org/#/c/83172/ how exactly do I use the branch designator variable to make separate jobs? | 19:37 |
| boris-42 | mordred btw how can I join?) | 19:37 |
| anteaya | silly question alert, is this how the logs are supposed to appear? http://logs.openstack.org/62/83762/2/gate/gate-grenade-dsvm/9afe402/logs/etc/swift/backups/1396288286.account.ring.gz.txt.gz | 19:37 |
| mtreinish | clarkb: I agree that copy and pasted job definitions aren't the best, but I wasn't sure how else to do it | 19:37 |
| mordred | in doing that, the parts of the system that seem at first to be bugs or oversights but are quite on purpose should become clear - and then when you make suggestions on new radical things to do (Which we all do from time to time) - they'll be ina good context :) | 19:37 |
| Ajaeger1 | clarkb: yes, using bash -c works - and we need to avoid using the .tox subdirectory within find - patch https://review.openstack.org/#/c/84210/ for ironic send | 19:37 |
| anteaya | because there are more than one characters in this log file that I don't know how to interpret | 19:37 |
| mordred | boris-42: we've got a list of current priorities - I don't have the etherpad link on me right now though | 19:38 |
| sdague | anteaya: I'm pretty sure that's binary packed data | 19:38 |
| mtreinish | anteaya: yeah it's probably not gunzipping it | 19:38 |
| anteaya | okay how do I get it so that I can read it? | 19:38 |
| boris-42 | mordred it will be nice to see=) it's a bit hard to join infra to much to learn=) | 19:38 |
| mtreinish | anteaya: well I'm not sure about on the infra side to fix it. But to view it you can download the file and just run zcat on it. | 19:39 |
| clarkb | sdague: mtreinish boris-42 http://summit.openstack.org/cfp/details/198 | 19:39 |
| *** _nadya_ has joined #openstack-infra | 19:39 | |
| anteaya | mtreinish: thanks I will try that to start | 19:39 |
| mtreinish | clarkb: cool thanks | 19:40 |
| fungi | mordred: boris-42: https://etherpad.openstack.org/p/infrastructure-priorities | 19:40 |
| *** dangers_away is now known as dangers | 19:40 | |
| *** gokrokve has quit IRC | 19:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Configure large-ops test and add a larger one for testing https://review.openstack.org/80718 | 19:40 |
| *** gokrokve has joined #openstack-infra | 19:40 | |
| boris-42 | clarkb fungi mordred btw is there some small tasks | 19:41 |
| boris-42 | clarkb fungi mordred that I can take a think about them | 19:41 |
| anteaya | mtreinish: now it is an html file, so somewhat better | 19:41 |
| boris-42 | cause I should start playing with this stuff | 19:41 |
| boris-42 | otherwise I won't learn it=) | 19:42 |
| fungi | boris-42: https://bugs.launchpad.net/openstack-ci/+bugs?field.tag=low-hanging-fruit | 19:42 |
| boris-42 | fungi thx | 19:42 |
| *** gokrokve_ has joined #openstack-infra | 19:42 | |
| *** khyati_ has quit IRC | 19:43 | |
| fungi | boris-42: lurking in the weekly infra team meeting and/or reading teh meeting logs might also help keep on top of where we're focusing efforts https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting | 19:43 |
| *** e0ne has joined #openstack-infra | 19:44 | |
| boris-42 | fungi I'll try to fix some bug tomorrow thanks for introduction | 19:44 |
| fungi | boris-42: also, the openstack-infra@lists.openstack.org ml http://lists.openstack.org/pipermail/openstack-infra/ | 19:44 |
| mordred | jeblair, fungi: you probably already know this - but there are no puppet 2.7 packages for raring - which makes me think there will be none for trusty | 19:44 |
| boris-42 | fungi why do you have separated mailing list? | 19:44 |
| *** gokrokve has quit IRC | 19:45 | |
| nibalizer | mordred: i thought of that the other night, then forgot about it | 19:45 |
| clarkb | mordred: correct, because puppet 2.7 doesn't work well with ruby 1.9 | 19:45 |
| fungi | boris-42: mainly because we discuss a lot of infrastructure-specific stuff which isn't of interest to the majority of the dev community (we have a lot of additional projects which aren't directly related to openstack) | 19:45 |
| mordred | clarkb: awesome | 19:45 |
| clarkb | mordred: so puppetlabs stopped making pacakges for puppet 2.7 on platforms with newer ruby | 19:46 |
| * mordred deletes the raring server and spins up a precise server | 19:46 | |
| nibalizer | what we could do is add /root/config/data as a data directory for both ihera yaml | 19:46 |
| boris-42 | fungi ah ok | 19:46 |
| nibalizer | hiera.yamls i mean | 19:46 |
| clarkb | mordred: however, pleia2 has made it possible to use puppet3 on slaves | 19:46 |
| nibalizer | clarkb: :(:( | 19:46 |
| mordred | clarkb: yah | 19:46 |
| clarkb | mordred: so we are compatible at least in that way (nothing prevents us from testing on trusty or raring best I can tell) | 19:46 |
| mordred | nibalizer: but /root/config/data is not where the data goes on our puppetmaster | 19:46 |
| boris-42 | fungi subscribed | 19:47 |
| fungi | boris-42: and conversely, there are a lot of organizations outside of the openstack community making use of some of the infra tools, so that gives them a much lower-volume list to which they can subscribe to follow and discuss things | 19:47 |
| mordred | nibalizer: one sec - I'm trying out a possible solution | 19:47 |
| nibalizer | mordred: that won't cause a failure, it will just skip it and look in the next data dir | 19:47 |
| nibalizer | okay | 19:47 |
| mordred | nibalizer: that is not the experience I just had | 19:47 |
| *** andreaf has joined #openstack-infra | 19:47 | |
| *** david-lyle has joined #openstack-infra | 19:47 | |
| mordred | nibalizer: it seems taht we cannot have multiple datadirs specified | 19:47 |
| nibalizer | hrm wut | 19:48 |
| mordred | nibalizer: yah. puppet broke teh everywhere | 19:48 |
| nibalizer | paste the hiera.yaml you're using? | 19:48 |
| nibalizer | oshi oops | 19:48 |
| mordred | nibalizer: the one from your patch :) | 19:48 |
| mordred | nibalizer: I shelled in to the pupeptmaster, removed teh extra two datadir lines, and then it worked again | 19:49 |
| *** markmcclain has quit IRC | 19:49 | |
| fungi | nibalizer: we get errors from puppet agent on all the servers along the lines of "Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item nodepool_mysql_password in any Hiera data file and no default supplied at /opt/config/production/manifests/site.pp:504 on node nodepool.openstack.org" | 19:50 |
| jogo | the new nova-specs py27 hasn't kicked in yet ... | 19:50 |
| fungi | nibalizer: or "Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item gerrit_ssh_rsa_pubkey_contents in any Hiera data file and no default supplied at /opt/config/production/manifests/site.pp:528 on node zuul.openstack.org" | 19:50 |
| fungi | et cetera | 19:50 |
| *** _nadya_ has quit IRC | 19:50 | |
| *** pdmars has quit IRC | 19:51 | |
| *** pdmars has joined #openstack-infra | 19:51 | |
| jogo | fungi: and its been about an hour | 19:51 |
| jogo | so not sure if my patch was bad or something else | 19:51 |
| fungi | jogo: puppet is broken right now on everything, so changes aren't taking effect yet | 19:52 |
| jogo | fungi: ohh | 19:52 |
| jogo | ETA until fixed (no rush) | 19:52 |
| jogo | russellb: ^ | 19:52 |
| russellb | jogo: ah, k | 19:52 |
| fungi | we're batting around ideas for how to solve it, but will hopefully settle on something shortly | 19:53 |
| *** Sukhdev has quit IRC | 19:53 | |
| jogo | awesome thanks | 19:53 |
| annegentle | The following command failed with exit code 1 | 19:53 |
| annegentle | "scp -P 29418 klevenstein@review.openstack.org:hooks/commit-msg .git/hooks/commit-msg" | 19:53 |
| annegentle | huh I'm sorta stumped on this one ^^ | 19:53 |
| annegentle | had her check her public key, matches with review.openstack.org | 19:53 |
| annegentle | git remote -v shows gerrit remote already there | 19:53 |
| annegentle | so why wouldn't the hook get set? | 19:54 |
| nibalizer | wow, you really can't have multiple data dirs | 19:54 |
| nibalizer | thats so broken | 19:54 |
| nibalizer | so what i've done in the past, and forgot to do here, is set your datadir really high up like /etc or /etc/puppet then drill down into subdirectories with the hierarchy | 19:54 |
| nibalizer | to reach into the three directories specified currently we'd have to put the datadir at / | 19:55 |
| jeblair | fungi, clarkb, mordred, sdague: i have confirmation that our nodes in rackspace are erroneously being blocked from using rax dns servers | 19:55 |
| fungi | annegentle: have her try 'git review -s -v' | 19:55 |
| sdague | jeblair: cool. knowning is half the battle | 19:55 |
| fungi | jeblair: great news! (for some definition of "great" anyway) | 19:56 |
| jeblair | fungi, clarkb, mordred, sdague: basically, there is a delay in their auto-blacklist system, and we end up with ip's previously held by bad tenants | 19:56 |
| clarkb | jeblair: even with the unbound resolver? | 19:56 |
| clarkb | I guess it is forwarding though | 19:56 |
| *** ociuhandu has quit IRC | 19:56 | |
| clarkb | jeblair: ugh | 19:56 |
| annegentle | fungi: oh good tip | 19:56 |
| jeblair | clarkb, fungi: so i think we may want to switch to forwarding to 8.8.8.8 everywhere, or switch to fully-recursive everywhere... | 19:56 |
| fungi | annegentle: then stick the output up on paste.openstack.org and drop a link in here--i'll work it ou | 19:56 |
| fungi | t | 19:56 |
| jeblair | clarkb, fungi: at least, until we have neutron support and cloud-local private recursive resolvers | 19:57 |
| clarkb | jeblair: ++ | 19:57 |
| jeblair | clarkb, fungi: thoughts on what to do in the interim? | 19:57 |
| clarkb | jeblair: forwarding to google makes us consistent which I like | 19:57 |
| fungi | jeblair: i don't think we have much other choice, unfortunately. so rackspace confirmed that their blacklisting system has a delay and that they don't intend to work on solving that any time soon? | 19:57 |
| fungi | they should just stop providing resolvers at all if they can't do it reliably | 19:58 |
| jeblair | fungi: well, the tech on the ticket didn't have much to say about future plans; i emailed pvo and gabe. | 19:58 |
| *** ominakov has joined #openstack-infra | 19:59 | |
| *** sarob has quit IRC | 19:59 | |
| annegentle | fungi: thanks | 19:59 |
| clarkb | jeblair: why don't we unspecial case the google thing as a quick first step | 19:59 |
| fungi | jeblair: ahh, got it. so yeah short term fix is to hit google's nameservers, because... they're there and free | 19:59 |
| clarkb | since it is tested and all that | 19:59 |
| *** Ajaeger1 has quit IRC | 19:59 | |
| *** sarob_ has joined #openstack-infra | 20:00 | |
| *** mrodden1 is now known as mrodden | 20:00 | |
| fungi | and we already do it for some subset of our providers because that's how the providers (intentionally or not) configure us | 20:00 |
| jeblair | okiedokie. | 20:00 |
| *** markmcclain has joined #openstack-infra | 20:00 | |
| mordred | ++ | 20:00 |
| *** ominakov has quit IRC | 20:00 | |
| * clarkb lunches | 20:01 | |
| *** dstanek has joined #openstack-infra | 20:01 | |
| annegentle | fungi: http://paste.openstack.org/show/74690/ | 20:01 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Work around for single hiera data dir https://review.openstack.org/84229 | 20:01 |
| anteaya | annegentle: it is a key error | 20:02 |
| nibalizer | mordred: fungi i think this will work better, but i'm not sure if 'confdir' works the way we want it to | 20:03 |
| mordred | nibalizer: nice. | 20:03 |
| anteaya | what is the id for her account? | 20:03 |
| mordred | nibalizer: I'm testing confdir right now | 20:03 |
| anteaya | review.o.o > settings | 20:03 |
| mordred | nibalizer: from the puppetmaster, confdir is actually ::settings::confdir | 20:03 |
| mordred | but I don't know if it shows up that way from the apply commadn line | 20:03 |
| annegentle | anteaya: so beyond looking at her ~/.ssh/id_rsa.pub and matching to review.openstack.org, what else to check? | 20:04 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Always forward to 8.8.8.8 on test nodes https://review.openstack.org/84230 | 20:04 |
| anteaya | if you post her account id, fungi can check the db | 20:04 |
| jeblair | clarkb, mordred, fungi, sdague: https://review.openstack.org/84230 | 20:04 |
| anteaya | sometimes (rarely) what is in the gui and what is in the db don't match | 20:04 |
| davidlenwell | jeblair: why don't you have gears listed here? http://gearman.org/download/ just curious | 20:05 |
| annegentle | fungi: can you check on klevenstein (account) | 20:05 |
| annegentle | anteaya: is that the account ID ? | 20:05 |
| *** dstanek has quit IRC | 20:05 | |
| anteaya | it is in settings | 20:05 |
| jeblair | also, apparently today is one of the two days per year where we have thunderstorms in the bay area, so there's no telling what will happen to my connection. :) | 20:05 |
| nibalizer | mordred: maybe puppet config print confdir, or puppet config --confidr=thing print confdir ( a truly useless command) | 20:05 |
| anteaya | mine is 6316 | 20:05 |
| *** sarob has joined #openstack-infra | 20:05 | |
| anteaya | it will be a number | 20:05 |
| nibalizer | puppet apply --confdir=<thing> --configprint confdir | 20:06 |
| fungi | annegentle: right, "Permission denied (publickey)." generally is either that the username is wrong or the private key doesn't correspond to the public key entered into gerrit. i'll check logs and the db for anything obvious first | 20:06 |
| annegentle | fungi: ok thanks | 20:06 |
| *** zns_ has joined #openstack-infra | 20:06 | |
| nibalizer | mordred: okay now I grok what you're saying with ::settings::confdir | 20:06 |
| jeblair | davidlenwell: we should do that. mordred: do you know the edit process for http://gearman.org/download/ ? is it in a repo, or email brian? | 20:06 |
| annegentle | fungi: I had her check for the matching, I can look myself too but looking in the logs will probably see what we're not seeing | 20:06 |
| mordred | jeblair: I believe 'email brian' | 20:06 |
| davidlenwell | jeblair: I just hate to see the best gearman/python thingy not get some mention | 20:07 |
| jeblair | davidlenwell: aww shucks | 20:07 |
| mordred | nibalizer: ::settings::confdir seems to work in hiera.yaml from apply | 20:07 |
| fungi | annegentle: well, more than once i've found users who somehow changed their private key without changing the public key to match, and didn't realize they keypair they had in their ~/.ssh was inconsistent | 20:07 |
| nibalizer | okay ill update my patch | 20:07 |
| mordred | nibalizer: I think you can make it much simplier... | 20:08 |
| *** vhoward- has joined #openstack-infra | 20:08 | |
| fungi | annegentle: if it comes down to that, i have a command she can run to see what her public key should be (instead of what she has in ~/.ssh/id_rsa.pub or whatever) | 20:08 |
| annegentle | fungi: ok | 20:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Make apply and agent play nice https://review.openstack.org/84232 | 20:08 |
| mordred | nibalizer: ^^ like that | 20:08 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Work around for single hiera data dir https://review.openstack.org/84229 | 20:08 |
| mordred | nibalizer: (although I think puppet stuff into the hierarchy works too | 20:09 |
| *** zns has quit IRC | 20:09 | |
| fungi | annegentle: looks like username:karin-levenstein | 20:10 |
| annegentle | fungi: ah ha! | 20:10 |
| annegentle | fungi: ok | 20:10 |
| fungi | annegentle: not username:klevenstein | 20:10 |
| annegentle | fungi: best way forward? Change in gerrit? | 20:10 |
| anteaya | annegentle: does her gerrit say klevenstein? | 20:11 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Make apply and agent play nice https://review.openstack.org/84232 | 20:11 |
| *** markwash_ has joined #openstack-infra | 20:11 | |
| anteaya | as username in settings? | 20:11 |
| annegentle | anteaya: askin | 20:11 |
| *** markwash has quit IRC | 20:11 | |
| *** markwash_ is now known as markwash | 20:11 | |
| anteaya | yup | 20:11 |
| annegentle | asking that is | 20:11 |
| * anteaya nods | 20:12 | |
| mordred | nibalizer: of like that ^^ (taking your idea to put env in hierarchy) - the above has been tested to wrok with apply ona fresh machine | 20:12 |
| mordred | nibalizer: oh - wait - I don't need to break the symlink with that file | 20:12 |
| fungi | annegentle: git config --add gitreview.username karin-levenstein | 20:13 |
| fungi | annegentle: though she may need to update any existing git remotes which got created with the wrong username | 20:13 |
| annegentle | fungi: anteaya: so which is truth: Launchpad username or OpenStack.org/profile username? | 20:13 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Make apply and agent play nice https://review.openstack.org/84232 | 20:13 |
| annegentle | fungi: ok, yup, I get it. I think this happened because she had to sign the CLA after having already signed it | 20:13 |
| fungi | annegentle: gerrit initially gets the username from launchpad, but won't change if launchpad gets changed | 20:14 |
| annegentle | fungi: so her OpenStack.org/profile username "won" | 20:14 |
| devananda | clarkb: I see https://review.openstack.org/#/c/84191/ is still pending any other core reviews. I am holding back our RC tag until this is fixed in Ironic | 20:14 |
| nibalizer | mordred: im not sure thats true, for the master is confdir /etc/puppet or /opt/config ? it can't be both | 20:14 |
| fungi | annegentle: it's originally imported through the openid credentials lp provides on login | 20:14 |
| annegentle | fungi: (just guessing at root cause) | 20:14 |
| devananda | clarkb: so if it's going to be a while (which, I think, is reasonable given the nature of this change to -infra) I'm going to just propose the fix to Ironic directly | 20:14 |
| fungi | annegentle: but after that it's set and can't be changed in gerrit without mucking with the backend database | 20:14 |
| mordred | nibalizer: it's /etc/puppet | 20:14 |
| nibalizer | so then how do we teach the master to reach into /opt/config/data without setting the datadir to '/' ? | 20:15 |
| sdague | question: how much hate would there be for another nodepool type getting added? | 20:15 |
| mordred | nibalizer: oh! I see what you're saying - we need your versionso we can pick up the stuff in tree | 20:15 |
| mordred | yeah | 20:15 |
| fungi | annegentle: also i forgot --global in that command. should be 'git config --global --add gitreview.username karin-levenstein' | 20:15 |
| mordred | nibalizer: I'm with you | 20:15 |
| annegentle | fungi: what's the rename git remote command? | 20:15 |
| nibalizer | yea i kinda hate that datadir is '/' but its a consequence of the master's git checkout being in /opt | 20:15 |
| jeblair | davidlenwell, mordred: email sent | 20:15 |
| sdague | because recent experiences with ubuntu 13.10 vs. devstack makes me think there is going to be a bunch of unwind to make this work right on 14.04 | 20:16 |
| mordred | well, I kinda hate that we can't give a path | 20:16 |
| fungi | annegentle: she could just 'git remote --delete gerrit' and git review will later re-add it | 20:16 |
| davidlenwell | jeblair: awesome | 20:16 |
| sdague | and having 13.10 nodes running check non-voting on devstack might help us get there faster | 20:16 |
| anteaya | annegentle: and git review only knows about review.o.o credentials, it knows nothing about lp credentials | 20:16 |
| fungi | annegentle: also, git-review normally prompts for your gerrit username if it can't log in, and instructs you on how to set it globally, so not sure how that ended up getting missed | 20:17 |
| mordred | sdague: it's probably not a terrible idea | 20:17 |
| annegentle | fungi: anteaya: I think it's a case of having set it up over a year ago, when things worked a little differently | 20:17 |
| anteaya | ah yeah, that might explain things | 20:17 |
| fungi | anteaya: technically git-review knows about no credentials at all other than what it finds on your local hard drive and what it prompts you for if it fails | 20:17 |
| nibalizer | since there is 'data' in the hierachy I dont think anyone could convince hiera to cough up /etc/password or /etc/shadow | 20:17 |
| anteaya | fungi: right | 20:18 |
| fungi | annegentle: probably--a lot can change ina year | 20:18 |
| annegentle | fungi: yeah I was originally thinking it was a username mismatch due to her having a "single sign on" username on the Operating System | 20:18 |
| sdague | though, now that I think about it, without the zuul change dependencies that's going to be a grind | 20:18 |
| annegentle | fungi: anteaya: much gratitude, you all rock | 20:18 |
| anteaya | np, hope she is good to commit now | 20:18 |
| mordred | nibalizer: I kinda hate that we can't do multiple datadir paths ... | 20:18 |
| anteaya | fungi: though git review will only ever prompt you for information pertaining to review.o.o, in the cases when review.o.o credentials and lp credentials don't equal each other | 20:19 |
| fungi | annegentle: since https://launchpad.net/~karin-levenstein exists and https://launchpad.net/~klevenstein doesn't, seems that gerrit and lp still actually happen to match in this case | 20:20 |
| nibalizer | mordred: yea, serously | 20:20 |
| *** Sukhdev has joined #openstack-infra | 20:20 | |
| nibalizer | where is hunner | 20:20 |
| nibalizer | Hunner: you are summoned | 20:20 |
| annegentle | fungi: heh heh heh. Just a run-of-the-mill case of forgetfulness and entering the wrong username when prompted :) | 20:20 |
| mordred | nibalizer: I gotta run for a sec - I leave follow up in your capable hands | 20:21 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Re-enable cross-tests in oslo.test https://review.openstack.org/84235 | 20:21 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Enable cross-testing of oslo.test with dib and t-i-e https://review.openstack.org/83885 | 20:21 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Run unit tests in tripleo-image-elements https://review.openstack.org/83886 | 20:21 |
| *** rwsu has joined #openstack-infra | 20:22 | |
| *** Ajaeger1 has joined #openstack-infra | 20:22 | |
| *** sarob_ has quit IRC | 20:22 | |
| Ajaeger1 | clarkb, regarding the discussion we had with patch https://review.openstack.org/#/c/84191/ and the obsolete entries. What about adding --ignore-obsolete to "python setup.py update_catalog " ? That way, these obsoletes get removed automatically... | 20:24 |
| *** khyati_ has joined #openstack-infra | 20:28 | |
| *** julim has quit IRC | 20:29 | |
| *** ok_delta has left #openstack-infra | 20:30 | |
| *** sarob has quit IRC | 20:31 | |
| *** JoshNang has joined #openstack-infra | 20:32 | |
| *** sarob has joined #openstack-infra | 20:32 | |
| *** zzelle has joined #openstack-infra | 20:32 | |
| *** sarob_ has joined #openstack-infra | 20:34 | |
| *** sarob has quit IRC | 20:36 | |
| *** dizquierdo has joined #openstack-infra | 20:36 | |
| annegentle | thanks again fungi and anteaya she's all squared away now | 20:37 |
| anteaya | annegentle: glad to hear it | 20:37 |
| fungi | annegentle: great! | 20:37 |
| openstackgerrit | A change was merged to openstack-infra/config: Run all python jobs for nodepool https://review.openstack.org/82632 | 20:40 |
| zzelle | fungi, hi | 20:42 |
| zzelle | fungi, thanks for your approvals | 20:42 |
| *** salv-orlando has joined #openstack-infra | 20:43 | |
| zzelle | fungi, but https://review.openstack.org/79281 has been hit by the gate, could you reapprove it ? | 20:44 |
| fungi | zzelle: oh, i meant to double-check that one. thanks@ | 20:44 |
| fungi | zzelle: it looks like the ssh hash mismatch issue again. leave a comment of "reverify bug 1048724" on failures of that nature (at least until someone has more time to look into solving them) | 20:46 |
| uvirtbot | Launchpad bug 1048724 in git-review "git-review should be tested" [Medium,Fix committed] https://launchpad.net/bugs/1048724 | 20:46 |
| *** thomasem has joined #openstack-infra | 20:46 | |
| zzelle | fungi, reverify works for non-cores ? | 20:47 |
| fungi | zzelle: yes, but i went ahead and did it this time | 20:47 |
| zzelle | fungi, thanks | 20:48 |
| openstackgerrit | Doug Hellmann proposed a change to openstack/requirements: Work around atexit bug in python < 2.7.4 https://review.openstack.org/84245 | 20:50 |
| *** e0ne has quit IRC | 20:50 | |
| *** e0ne has joined #openstack-infra | 20:51 | |
| sdague | jeblair: because I've seen us hack around this a few times in the last couple of weeks - I started writing out some thoughts on - https://etherpad.openstack.org/p/zuul-depends-on | 20:52 |
| sdague | currious to see your feedback, and if you think it's a good infra meeting topic | 20:52 |
| jeblair | sdague: cool, that reminds me i owe you a link to previous gerrit discussions on the topic; i'll dig that up now | 20:53 |
| sdague | sure | 20:53 |
| sdague | I went and looked at the review set | 20:53 |
| jeblair | oh you found it cool | 20:53 |
| sdague | the gerrit feature was different | 20:53 |
| anteaya | I so wish one could save good etherpad colours | 20:53 |
| sdague | I tried to document that in the Alternatives | 20:54 |
| sdague | perhaps poorly | 20:54 |
| *** sarob_ has quit IRC | 20:54 | |
| anteaya | boo | 20:54 |
| jeblair | sdague: cool; i mostly wanted to make sure it informed our thinking on it. | 20:55 |
| sdague | yeh | 20:55 |
| anteaya | I know it wasn't very manly but it was very pretty | 20:55 |
| sdague | I think that has the problems that mordred brought up, in that it would completely wreck CD | 20:55 |
| *** e0ne has quit IRC | 20:55 | |
| jeblair | https://gerrit-review.googlesource.com/#/q/topic:cross-repo-dependencies,n,z | 20:56 |
| sdague | yeh, let me add it to the etherpad | 20:56 |
| jeblair | sdague: "Calculation of dependencies across zuul boundaries" what's that mean? | 20:57 |
| JayF | So for ironic-python-agent, we have an interesting issue. JoshNang filed a change (https://review.openstack.org/#/c/83604/) to fix bug (https://bugs.launchpad.net/ironic/+bug/1297019) and the bug didn't update. lifeless suggested there may be some needed mapping since ironic-python-agent is using the ironic launchpad for bug/bp tracking. Any ideas? | 20:57 |
| uvirtbot | Launchpad bug 1297019 in ironic "agent quits on a single error during lookup" [Undecided,New] | 20:57 |
| clarkb | back from lunch catching up on sb | 20:58 |
| jeblair | JayF: there's a 'group' field in ./modules/openstack_project/files/review.projects.yaml | 20:58 |
| clarkb | jeblair: comment on the dns forwarder change, did not -1 though | 20:58 |
| openstackgerrit | A change was merged to openstack-infra/git-review: Git review assumes the wrong ssh default port https://review.openstack.org/79281 | 20:58 |
| jeblair | JayF: in the openstack-infra/config repo; if you set that, you should be good | 20:59 |
| sdague | I think it meant that I was blathering | 20:59 |
| *** Ajaeger1 has quit IRC | 20:59 | |
| sdague | I just mean the dependency calculation | 20:59 |
| sdague | in general | 20:59 |
| JayF | jeblair: I'll check up on that, thanks! | 20:59 |
| jeblair | sdague: ok cool; i'll add some notes | 20:59 |
| mgagne | JayF: should be the "group" option in modules/openstack_project/files/review.projects.yaml | 21:00 |
| mgagne | JayF: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/review.projects.yaml | 21:00 |
| *** harlowja is now known as harlowja_away | 21:00 | |
| sdague | jeblair: thanks | 21:00 |
| jogo | is puppet alive again? | 21:00 |
| jeblair | sdague: lgtm, and that was the only thing i could think to add right now | 21:01 |
| fungi | mordred: nibalizer: clarkb: so we think that https://review.openstack.org/84229 will get puppet working again, do what we want for hiera file separation and is safe insofar as not serving up files it shouldn't? | 21:01 |
| openstackgerrit | Jay Faulkner proposed a change to openstack-infra/config: Set group 'ironic' for ironic-python-agent https://review.openstack.org/84250 | 21:02 |
| fungi | if so, i vote we cram it through | 21:02 |
| JayF | jeblair: mgagne: https://review.openstack.org/#/c/84250/ this appears to be what you guys were saying. I would appreciate a review. | 21:02 |
| jeblair | fungi: or we could revert | 21:03 |
| clarkb | fungi: question left on the review | 21:03 |
| fungi | jeblair: i'm in favor of that too. just want to get back to the things i was waiting on working puppet for which is why i noticed it had broken in the first place | 21:04 |
| jeblair | fungi: i'd rather revert to give people more time to explore/plan/test appropriately | 21:04 |
| jeblair | i'm not actually okay with puppet being broken for long periods of time | 21:05 |
| *** ociuhandu has joined #openstack-infra | 21:05 | |
| jeblair | where long periods means more than like 15 minutes. :) | 21:05 |
| lifeless | I believe tripleo-ci is back up | 21:05 |
| lifeless | we need to add images to glance for nodepool, doing that now | 21:05 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revert "Adds second hiera data directory" https://review.openstack.org/84251 | 21:06 |
| jeblair | fungi: it was that one that broke yeah? ^ | 21:06 |
| anteaya | I have a question about 84250 which is academic since it is already approved, why shouldn't there be a group for openstack/ironic and openstack/python-ironicclient | 21:07 |
| clarkb | jeblair: oui | 21:07 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Revert "Adds second hiera data directory" https://review.openstack.org/84252 | 21:07 |
| fungi | jeblair: oh, oops. i'll revert mine | 21:07 |
| clarkb | anteaya: typically the clienst have the same reviewer base, eg if you are a server core the client is part of your review responsiblity | 21:07 |
| clarkb | anteaya: so we don't bother with multiple groups | 21:07 |
| anteaya | I guess I don't understand about groups enough | 21:08 |
| anteaya | and thanks | 21:08 |
| jeblair | anteaya: it's a misnamed option | 21:08 |
| *** pdmars has quit IRC | 21:09 | |
| jeblair | anteaya: it used to be something like 'launchpad-project'; i don't recall | 21:09 |
| fungi | anteaya: "group" in this case is storyboard terminology | 21:09 |
| *** JoshNang has left #openstack-infra | 21:09 | |
| *** rfolco has quit IRC | 21:09 | |
| lifeless | oh ffs vim modules/openstack_project/templates/nodepool/nodepool.yaml.erb | 21:09 |
| clarkb | anteaya: oh whoops | 21:09 |
| lifeless | Vim: Caught deadly signal SEGV | 21:09 |
| anteaya | okay | 21:09 |
| jeblair | anyway, it's a mapping that says "bugs for this project are acutally in this other project" | 21:09 |
| anteaya | thanks | 21:09 |
| clarkb | anteaya: sorry I assumed I knoew what I was talking about but apparently didn't. jeblair nad fungi have it right thouh | 21:09 |
| fungi | anteaya: it's (future) storyboard's bug group, but for now it maps a launchpad project for use with bug updates | 21:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Set group 'ironic' for ironic-python-agent https://review.openstack.org/84250 | 21:09 |
| jeblair | i don't think storyboard actually has that feature yet though. | 21:10 |
| anteaya | so project and client don't need it since lp/storyboard already understands they are the group/lp-project | 21:10 |
| *** dprince has quit IRC | 21:10 | |
| jeblair | so in the mean time, the option doesn't actually make sense for either piece of software. :/ | 21:10 |
| fungi | agreed | 21:10 |
| anteaya | clarkb: I listen to you anyway, :D | 21:10 |
| anteaya | the twilight zone | 21:10 |
| anteaya | I know that feel | 21:10 |
| jeblair | lifeless: we're putting buffer overflows in that file now to add vim users to our botnet | 21:11 |
| * fungi has to step away for a bit to help evacuate more unwanted furniture from his third floor hallway | 21:11 | |
| jeblair | lifeless: your workstation will start running devstack gate tests soon. ;) | 21:11 |
| lifeless | :P | 21:14 |
| lifeless | oh, forgot quotas | 21:14 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Adds second hiera data directory" https://review.openstack.org/84251 | 21:15 |
| *** pblaho has joined #openstack-infra | 21:15 | |
| *** ameade has quit IRC | 21:16 | |
| mgagne | guys: https://github.com/puppetlabs/hiera/blob/master/lib/hiera/backend.rb#L25-L26 | 21:16 |
| mgagne | jeblair: ^ | 21:18 |
| *** harlowja_away is now known as harlowja | 21:19 | |
| *** sarob has joined #openstack-infra | 21:19 | |
| *** e0ne has joined #openstack-infra | 21:21 | |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/config: Add experimental keystonev3 jobs https://review.openstack.org/79314 | 21:21 |
| *** thuc has quit IRC | 21:22 | |
| andreaf | jeblair | 21:22 |
| devananda | clarkb: while waiting for https://review.openstack.org/#/c/84191/ is this going to break anything? https://review.openstack.org/84259 | 21:22 |
| andreaf | jeckersb: thanks for the review, I created the job-templates as you suggested ^^^^ | 21:22 |
| clarkb | devananda: I don't think so | 21:23 |
| clarkb | devananda: especially since you just pulled from transifex | 21:23 |
| andreaf | jeblair: https://review.openstack.org/79314 | 21:23 |
| clarkb | devananda: if you had done them by hand then when the transifex run happens it could overwrite then propose new stuff | 21:23 |
| clarkb | devananda: but you didn't so it should be fine | 21:23 |
| *** e0ne has quit IRC | 21:23 | |
| devananda | clarkb: awesome, thanks | 21:23 |
| jeblair | anteaya: can you replace 'check' in the name with '{pipeline}' like the other jobs there? | 21:24 |
| jeblair | andreaf: ^ | 21:24 |
| *** e0ne has joined #openstack-infra | 21:24 | |
| jeblair | anteaya: sorry | 21:24 |
| jeblair | andreaf: also, set node to '{node}' | 21:24 |
| lifeless | hi | 21:24 |
| *** sarob has quit IRC | 21:24 | |
| lifeless | can we get tripleo's hp region back in nodepool please | 21:25 |
| anteaya | np | 21:25 |
| *** zns_ has quit IRC | 21:25 | |
| anteaya | andreaf: welcome to teh we get each others mail group | 21:25 |
| anteaya | anne gentle is also a member | 21:25 |
| lifeless | ah bah, false alarm, something not -quite- there yet | 21:25 |
| *** jgrimm has quit IRC | 21:25 | |
| andreaf | jeblair: ok | 21:26 |
| *** e0ne has quit IRC | 21:26 | |
| andreaf | anteaya: hi :) | 21:27 |
| mattoliverau | Morning everyone! | 21:27 |
| *** miguelzuniga has joined #openstack-infra | 21:27 | |
| anteaya | andreaf: :D | 21:28 |
| anteaya | morning mattoliverau | 21:28 |
| mattoliverau | Morning anteaya :) | 21:28 |
| anteaya | tell me how lovely and warm it is there today | 21:28 |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/config: Add experimental keystonev3 jobs https://review.openstack.org/79314 | 21:29 |
| fungi | welp, managed not to get crushed under a 400lb dresser coming down two flights of stairs. doesn't look like i missed too much here | 21:30 |
| mattoliverau | It's going to be a warm 31 today :P I'm actually looking forward to colder weather... because it'll never be Canada cold ;P | 21:30 |
| *** SumitNaiksatam has quit IRC | 21:31 | |
| * clarkb top posts | 21:31 | |
| clarkb | feels bad | 21:31 |
| *** alff has joined #openstack-infra | 21:31 | |
| *** alff_ has joined #openstack-infra | 21:31 | |
| clarkb | fungi: o_O | 21:31 |
| anteaya | fungi: congrats for being not crushed | 21:31 |
| anteaya | how did the moving dresser fare? | 21:31 |
| anteaya | mattoliverau: true | 21:32 |
| anteaya | ah 31, I was happy with 8 today | 21:32 |
| mattoliverau | anteaya: well 8 is considered hot compared to your winder :P | 21:33 |
| anteaya | yes | 21:33 |
| fungi | anteaya: it (and the rest of that hideous-looking bedroom set) are now no longer taking up real estate. for once, craig's list came through | 21:33 |
| anteaya | the winder is colder | 21:33 |
| anteaya | fungi: ha ha ha | 21:33 |
| lifeless | ok, tested vms work | 21:33 |
| anteaya | fungi: who picks out the next bedroom set? | 21:33 |
| *** jcoufal has quit IRC | 21:34 | |
| lifeless | jeblair: clarkb: ^ can we reactivate nodepool for tripleo's HP region please. AFAICT we're all good to go | 21:34 |
| anteaya | and might I recommend a trip to bangkok for furniture shopping? | 21:34 |
| fungi | lifeless: did we actually remove the tripleo cloud from nodepool? if so, i missed it | 21:34 |
| marun | fungi, clarkb, jeblair: It would be great if you could approve the addition of the neutron functional job as non-voting check on devstack: https://review.openstack.org/#/c/82631/ | 21:34 |
| lifeless | fungi: it may have given p or something | 21:34 |
| jeblair | fungi: i also don't recall any deactivation... | 21:34 |
| marun | sdague already +1'd it (reminder that this isn't functional api, just system interaction) | 21:34 |
| jeblair | lifeless: ^ | 21:34 |
| lifeless | fungi: and certainly from prior experience since its a new cloud it means new tokens + new flavour ids etc. | 21:34 |
| clarkb | jeblair: neither do I | 21:34 |
| lifeless | so I suspect a nodepool restart is needed | 21:34 |
| clarkb | marun: looking | 21:35 |
| marun | I'd like to turn on voting before too long as we ramp up our testing of neutron system interaction | 21:35 |
| clarkb | jeblair: I double checked on the glance access thing via horizon for hpcloud account and I don't see the role yet | 21:35 |
| marun | and I was told that a devstack check job was the next step towards that | 21:35 |
| fungi | lifeless: yeah, it's still in the list. may just need ghost nodes blown away | 21:35 |
| lifeless | since I haven't had time to teach nodepool how to requery flavour ids when they are going | 21:35 |
| clarkb | jeblair: going to prod people about it now | 21:35 |
| fungi | lifeless: i'll take a look now | 21:35 |
| lifeless | fungi: nope - ^ see under 'lessons learnt from last failure' | 21:35 |
| fungi | lifeless: oh, that, we can kick that without a restart | 21:35 |
| clarkb | jeblair: actually, they say we have the role... | 21:36 |
| * clarkb double checks | 21:36 | |
| fungi | lifeless: just need to tweak a parameter on the provider and then switch it back (a lesson *i* learnt after the last failure) | 21:36 |
| * clarkb grabs a trusty cloud image | 21:36 | |
| fungi | lifeless: first, we have 35 nodes for tripleo which nodepool thinks it knows about--shall i blow them away? | 21:37 |
| lifeless | fungi: they certainly do not exist any more | 21:37 |
| anteaya | marun: so 82631 keeps the job as non-voting in the devstack check tests, correct? | 21:37 |
| marun | anteaya: correct | 21:37 |
| anteaya | marun: thanks | 21:37 |
| *** andreaf has quit IRC | 21:38 | |
| *** jhesketh_ has quit IRC | 21:38 | |
| *** jhesketh__ has joined #openstack-infra | 21:38 | |
| fungi | lifeless: same goes for the existing tripleo ready images? | 21:38 |
| jhesketh__ | Morning | 21:39 |
| anteaya | morning jhesketh__ | 21:39 |
| JayF | with https://review.openstack.org/#/c/84250/ merged, is there some other trigger I have to wait for before it takes effect? because it doesn't seem to have worked atm | 21:39 |
| anteaya | he of many underscores | 21:39 |
| anteaya | JayF: that might be related to our puppet not working right now | 21:40 |
| anteaya | I'm not 100% understanding all the details | 21:40 |
| JayF | anteaya: is there a place I can see that is the case, or maybe see the SHA1 of the openstack-infra/config used for the last successful puppet run? | 21:40 |
| anteaya | but puppet sad is what I do have | 21:40 |
| fungi | JayF: bugs only get updated when you upload the first patchset of a change mentions the bug in its commit message, or when that change is merged to the target repository | 21:40 |
| anteaya | JayF: might be a place to start: http://puppetdb.openstack.org/ | 21:41 |
| JayF | fungi: for the existing one, then, what needs to happen? just have him close and reopen, or just on merge it'll update? | 21:41 |
| *** aysyd has quit IRC | 21:41 | |
| fungi | JayF: it should update the bug when the change merges, but in the interim he can just assign the bug to himself, set it in progress and add a comment with a url to the change | 21:41 |
| JayF | okay, thanks. | 21:42 |
| fungi | lifeless: i'm assuming yes to my second question (about deleting old images) and am proceeding | 21:42 |
| *** dcramer_ has quit IRC | 21:43 | |
| *** dangers is now known as dangers_away | 21:44 | |
| *** andreaf has joined #openstack-infra | 21:44 | |
| lifeless | fungi: yes, its a brand new cloud | 21:45 |
| fungi | gotta love that "new cloud" smell | 21:45 |
| fungi | lifeless: BadRequest: Network f69ac547-db64-4e69-ae70-e5233634aff0, b54c5011-519a-4364-8256-a9214eec2387 could not be found. (HTTP 400) (Request-ID: req-e4df10c5-6b2b-427b-8673-90b19c88c9bd) | 21:45 |
| fungi | lifeless: new network ids? | 21:46 |
| lifeless | fungi: ohcrapIforgottosubmitachangeforthat | 21:46 |
| fungi | (is there a change up for that i've overlooked?) | 21:46 |
| fungi | ahh | 21:46 |
| lifeless | eth0 a395fac5-adec-4ab9-a605-963558349cfd | 21:47 |
| anteaya | that's not a unique id | 21:47 |
| lifeless | eth1 063ec9df-2fec-4723-8d4b-92c60193bd08 | 21:47 |
| lifeless | fungi: putting it into a git change now | 21:47 |
| fungi | lifeless: perfect | 21:47 |
| mattoliverau | anteaya: jhesketh__ like under scores. I like to think it's how many coffee's he needs to start filling himself in the morning :P | 21:47 |
| *** signed8bit has quit IRC | 21:47 | |
| anteaya | the coffee measurement | 21:47 |
| mattoliverau | Yup, so me thinks I need a few underscores myself :P | 21:48 |
| *** mriedem has quit IRC | 21:48 | |
| *** dizquierdo has quit IRC | 21:48 | |
| fungi | also, puppet is still broken... mordred did you stop puppet agent on ci-puppetmaster? | 21:49 |
| clarkb | jeblair: fungi mordred SergeyLukjanov woot! do an image-list against hpcloud region-b | 21:49 |
| anteaya | mattoliverau: :D | 21:49 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Fixup tripleo net ids. https://review.openstack.org/84263 | 21:49 |
| lifeless | fungi: ^ | 21:49 |
| *** rfolco has joined #openstack-infra | 21:49 | |
| anteaya | clarkb: what would I see were I root? | 21:49 |
| clarkb | we can boot Trusty Beta nodes assuming I uploaded the right thing | 21:49 |
| anteaya | yay | 21:49 |
| clarkb | anteaya: ^ I uploaded an image directly | 21:49 |
| anteaya | woohoo | 21:49 |
| anteaya | so glance is active? | 21:50 |
| clarkb | anteaya: yup | 21:50 |
| anteaya | yay | 21:50 |
| anteaya | \o/ | 21:50 |
| anteaya | woohoo | 21:50 |
| anteaya | well done region-b | 21:50 |
| fungi | now if only they could do something about all those error nodes | 21:50 |
| fungi | though actually no idea if we've continued to see them | 21:51 |
| *** alex-gone is now known as Alexandra_ | 21:51 | |
| *** jhesketh__ has quit IRC | 21:51 | |
| clarkb | jeblair: what was the last word on rax glance | 21:52 |
| *** jhesketh has joined #openstack-infra | 21:52 | |
| lifeless | fungi: hows it looking ? | 21:54 |
| *** jepoy has joined #openstack-infra | 21:55 | |
| *** tnurlygayanov_ has quit IRC | 21:55 | |
| *** jepoy has quit IRC | 21:55 | |
| *** sarob has joined #openstack-infra | 21:55 | |
| fungi | bwahahaha... chicken and egg on the puppet hiera revert: ci-puppetmaster can't apply the change because its hiera.yaml is broken *sigh* fixing by hand | 21:55 |
| *** jepoy has joined #openstack-infra | 21:56 | |
| *** tnurlygayanov has joined #openstack-infra | 21:56 | |
| *** signed8bit has joined #openstack-infra | 21:56 | |
| *** yassine has quit IRC | 21:56 | |
| *** flaper87 is now known as flaper87|afk | 21:57 | |
| *** dkranz has quit IRC | 21:57 | |
| *** dims has quit IRC | 21:57 | |
| *** thomasem has quit IRC | 21:57 | |
| lifeless | fungi: ah, so yak shaving? | 21:58 |
| *** pblaho has quit IRC | 21:58 | |
| *** e0ne has joined #openstack-infra | 21:59 | |
| fungi | lifeless: i've had an all-day yak it started out with me wanting to re-trigger a tag job which needed some nodepool image updates to include the changes to the slave script... then three or four other broken things eventually lead me to needing to get puppet going | 21:59 |
| *** andreaf has quit IRC | 22:00 | |
| fungi | nodepool image-update needed a patch for the config file changes which merged over the weekend, which then didn't get updated on the nodepool server because puppet... | 22:00 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Fixed contenteditable https://review.openstack.org/83847 | 22:00 |
| lifeless | fungi: mine was like that yesterday. | 22:00 |
| lifeless | fungi: I started out rebuilding testenvs to deliver Ironic undercloud tests | 22:00 |
| lifeless | fungi: found mysql down on ci-overcloud | 22:01 |
| lifeless | fungi: and from there.... | 22:01 |
| *** sarob has quit IRC | 22:01 | |
| lifeless | fungi: are you going to monkey the nodepool change in or is it queued behind puppet? [So that I can schedule what I do here, now] | 22:01 |
| *** thomasem has joined #openstack-infra | 22:02 | |
| fungi | lifeless: well, i tried manually changing it, but nodepool doesn't seem to have dynamically reloaded the net-id change (maybe that isn't handled correctly in its config reloading?) | 22:02 |
| *** packet has quit IRC | 22:02 | |
| fungi | lifeless: also, it looks like when we dropped the max-servers for tripleo to 35 we didn't lower the min-ready for the different images so that they don't add up to more than that | 22:03 |
| *** rlandy has quit IRC | 22:03 | |
| lifeless | fungi: you should be able to run 55 now, quota for ram is 960000 instance quota is 60 | 22:03 |
| fungi | lifeless: maybe amend your 84263 change with that then | 22:04 |
| lifeless | ok | 22:04 |
| fungi | give the min-ready parameters for your image lavels and max-server for the provider a once-over | 22:04 |
| jeblair | clarkb: word on rax is it needs some manually configured options (i think i mentioned that in here a few days ago) | 22:04 |
| lifeless | fungi: min-ready is 35 atm | 22:05 |
| lifeless | fungi: what are you suggesting it should be ? | 22:06 |
| fungi | lifeless: max-servers might ought to be raised | 22:06 |
| fungi | for the provider | 22:06 |
| lifeless | yes got that | 22:06 |
| lifeless | 55 | 22:06 |
| lifeless | fungi: ^ | 22:07 |
| clarkb | jeblair: fungi: thoughts on whether or not we should spin up some test slaves using the trusty beta image I uploaded to hpcloud region b? Do we think we should start poking at that yet? | 22:07 |
| fungi | lifeless: just suggesting that min-ready for all your tripleo-specific image labels should not add up to more than the max-servers for the provider where they're being created | 22:07 |
| lifeless | ok so tripleo-precise + tripleo-f20 <= max-servers? | 22:07 |
| fungi | lifeless: yep | 22:07 |
| fungi | sounds like raising your max-servers should solve it | 22:08 |
| *** andreaf has joined #openstack-infra | 22:08 | |
| clarkb | fungi: the max-servers in that env is intentionally low | 22:08 |
| clarkb | because we keep hitting quota limits | 22:08 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Fixup tripleo nodepool config. https://review.openstack.org/84263 | 22:08 |
| fungi | clarkb: it was, but supposedly it should not be now? | 22:08 |
| clarkb | fungi: supposedly but it returns to that behavior often | 22:08 |
| *** dansmith has quit IRC | 22:08 | |
| fungi | clarkb: i'll defer to lifeless on whether he thinks that's expected to continue | 22:09 |
| clarkb | see I51b9ef99880ba1bc0e36ba72bc6de97b276f1497 | 22:09 |
| *** dansmith has joined #openstack-infra | 22:09 | |
| *** hashar has quit IRC | 22:09 | |
| fungi | i really don't care what we set it to as long as we can avoid nodepool constantly hitting over quota errors on it when it tries to add nodes it thinks it should have room for | 22:09 |
| *** eharney has quit IRC | 22:10 | |
| clarkb | ++ | 22:10 |
| *** zns has joined #openstack-infra | 22:10 | |
| fungi | yay! i'm able to successfully nodepool image-update a rax-iad bare-precise image. this morning's yak is finally nearly shaven | 22:11 |
| *** nkinder has quit IRC | 22:11 | |
| lifeless | so my commit sort that | 22:11 |
| anteaya | the nearly shaven yak | 22:12 |
| anteaya | sounds like counting chickens before hatched | 22:12 |
| *** dims has joined #openstack-infra | 22:12 | |
| *** changbl has quit IRC | 22:13 | |
| *** e0ne has quit IRC | 22:13 | |
| *** wenlock_ has joined #openstack-infra | 22:13 | |
| fungi | anteaya: more like counting chickens before dinner, which i need to take a shaving break and go find | 22:13 |
| lifeless | fungi: nics should be dynamic | 22:13 |
| lifeless | fungi: from my reading of the nodepool code | 22:13 |
| *** mwagner_zzz has quit IRC | 22:14 | |
| jeblair | clarkb: i think it's worth seeing if we can use the base images in both providers, just as an experiment | 22:14 |
| fungi | lifeless: okay, in that case either there's a subtle bug there making them not, or nodepoold was ignoring changes to that file. or maybe i changed the values i tweaked back too quickly | 22:14 |
| anteaya | try to keep the yak hair out of the chopped onions | 22:14 |
| jeblair | clarkb: i don't think we should expect the outcome of that to be usable nodes at this point | 22:14 |
| lifeless | createServer uses self.provider.nic | 22:14 |
| clarkb | jeblair: that == glance uploads? | 22:14 |
| jeblair | clarkb: i think the next step after that is to have nodepool use dib to build its nodes | 22:15 |
| jeblair | clarkb: yes | 22:15 |
| lifeless | provider.nics is set from loadConfig | 22:15 |
| fungi | lifeless: oh, though i didn't change net-ids back, so nodepool should definitely have noticed changes to the file | 22:15 |
| clarkb | jeblair: I think we can have a step between now and dib, and that is using our own base images + current snapshot process | 22:15 |
| clarkb | jeblair: since typically we wait forever for the provider base images to show up | 22:15 |
| jeblair | clarkb: possibly. but there's a _really_ big question as to whether we can use them unmodified in both providers | 22:16 |
| jeblair | that needs answering first | 22:16 |
| jeblair | and if the answer is that we need to make any modification at all, then i think dib for nodepool is a hard requirement | 22:16 |
| jeblair | because that process needs to be reproducible | 22:17 |
| clarkb | good point, I think that is potentially more of an issue with Xen than KVM particularly if we want pvhvm | 22:17 |
| *** wenlock_ has quit IRC | 22:17 | |
| fungi | lifeless: still getting BadRequest: Network f69ac547-db64-4e69-ae70-e5233634aff0, b54c5011-519a-4364-8256-a9214eec2387 could not be found. (HTTP 400) (Request-ID: req-a6340ba3-33b3-413c-9778-a6a72a9b1822) | 22:17 |
| fungi | lifeless: that's after correcting nodepool.yaml with the net-ids from 84263 | 22:18 |
| lifeless | fungi: neither of those are env vaguely valid | 22:18 |
| jeblair | clarkb: luckily, mordred wrote a patch to use dib with nodepool :) | 22:18 |
| fungi | lifeless: right, i'm saying nodepool's still trying to use the old net-ids | 22:18 |
| lifeless | http://paste.openstack.org/show/74701/ | 22:18 |
| lifeless | yeah | 22:18 |
| lifeless | fungi: found the bug | 22:19 |
| * fungi is being dragged away to dinner... back in a little while | 22:19 | |
| fungi | lifeless: great! | 22:19 |
| *** jepoy has quit IRC | 22:19 | |
| lifeless | fungi: I can give you a workaround | 22:19 |
| nibalizer | fungi: i think its safe | 22:19 |
| lifeless | fungi: change the boot_timeout | 22:19 |
| fungi | lifeless: what's the workaround? | 22:19 |
| fungi | ahh | 22:19 |
| *** thuc has joined #openstack-infra | 22:19 | |
| nibalizer | the 'data' string appears in all hierachy lookups | 22:20 |
| *** Sukhdev has quit IRC | 22:20 | |
| *** rcleere has quit IRC | 22:20 | |
| nibalizer | so no one could possibly get at /etc/passwd /etc/shadow etc | 22:20 |
| *** thuc_ has joined #openstack-infra | 22:21 | |
| *** thuc_ has quit IRC | 22:21 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Detect neutron net-changes on reconfigure. https://review.openstack.org/84267 | 22:21 |
| lifeless | fungi: so if you could change the boot timeout and then run to dinner that would rock :) | 22:22 |
| nibalizer | clarkb: responded | 22:22 |
| *** thuc_ has joined #openstack-infra | 22:22 | |
| fungi | lifeless: no good. i upped the boot-timeout for that provider from 120 to 180 and it's still claiming "BadRequest: Network..." | 22:22 |
| lifeless | fungi: ok thats super odd | 22:22 |
| lifeless | shoo | 22:22 |
| lifeless | lets debug after your dinner | 22:22 |
| lifeless | I'll context switch for a bit | 22:22 |
| lifeless | fungi: please ping me ASAP after you return | 22:22 |
| fungi | anyway, really disappearing for a couple hours, then i'll pop back in and see where you got to | 22:22 |
| fungi | will do | 22:22 |
| fungi | g'luck! | 22:23 |
| *** mfer has joined #openstack-infra | 22:23 | |
| lifeless | fungi: ah true I can see if clarkb has a few minutes | 22:23 |
| *** jeckersb is now known as jeckersb_gone | 22:23 | |
| lifeless | clarkb: - can you see if there is a 'Creating new ProviderManager object for' log message for tripleo in the debug log ? | 22:23 |
| *** thuc has quit IRC | 22:24 | |
| *** CaptTofu has joined #openstack-infra | 22:24 | |
| *** thuc_ has quit IRC | 22:24 | |
| clarkb | looking | 22:24 |
| *** thuc has joined #openstack-infra | 22:25 | |
| clarkb | lifeless: 2014-03-31 22:22:35,290 DEBUG nodepool.NodePool: Creating new ProviderManager object for tripleo-test-cloud | 22:25 |
| clarkb | that was ~2.5 minutes ago | 22:25 |
| clarkb | nibalizer: and responded again :) | 22:26 |
| *** bknudson has quit IRC | 22:27 | |
| lifeless | so that *should* have the new nics then | 22:27 |
| *** prad_ has quit IRC | 22:27 | |
| lifeless | whats a SubNode | 22:28 |
| clarkb | lifeless: new feature to support multinode test environments. It is a node associated with the main node, but only the main node is attached to jenkins | 22:29 |
| lifeless | ah | 22:29 |
| clarkb | lifeless: basically nodepool drops bread crumbs in main nodes and in subnodes (/etc/nodepool) so they can all find each other | 22:29 |
| clarkb | lifeless: when you delete the main node the subnodes are deleted too and so on | 22:30 |
| lifeless | for devstack based multinode? | 22:30 |
| clarkb | yup (or any other multinode really, we can probably convert your non physical test envs to it) | 22:30 |
| lifeless | clarkb: its pointless for tripleo | 22:31 |
| lifeless | clarkb: since we spawn nodes to do the deploy | 22:31 |
| lifeless | clarkb: and there isn't a nova API around the test cluster stuff - and we shouldn't add one; better to make nova and neutron be able to host testenvs | 22:32 |
| lifeless | clarkb: so fullness of time, perhaps. | 22:32 |
| clarkb | lifeless: ya I don't think there is a pressing need to convert your stuff | 22:32 |
| *** mfer has quit IRC | 22:32 | |
| clarkb | the gearman env locking mechanism seems to work well | 22:32 |
| anteaya | so out of 21 programs I still have 9 that need at least one ptl candidate | 22:32 |
| *** zzelle has quit IRC | 22:32 | |
| anteaya | hint hint hint | 22:33 |
| lifeless | I don't see how this nic thing can't be updated | 22:33 |
| lifeless | can we restart nodepool ? | 22:33 |
| clarkb | lifeless: that was going to be my next suggestion. | 22:33 |
| clarkb | jeblair: ^ opinions before we take a sledge hammer to it? | 22:33 |
| nibalizer | clarkb: back atcha! | 22:33 |
| *** julim has joined #openstack-infra | 22:34 | |
| jogo | what does NOT_REGISTERED mean? | 22:34 |
| jogo | gate-nova-specs-python27: NOT_REGISTERED | 22:34 |
| lifeless | jogo: no slave matching the job spec is known to that jenkins AIUI | 22:34 |
| clarkb | nibalizer: we probably hsould've just used IRC :) responded | 22:35 |
| clarkb | jogo: lifeless correct | 22:35 |
| clarkb | jogo: lifeless: more specifically no jenkins slave has registered that job with gearman | 22:36 |
| clarkb | jogo: it means you need to be more patient | 22:36 |
| clarkb | jogo: and let puppet run on the jenkins masters | 22:36 |
| lifeless | clarkb: sleeeeedge haaaaaamer | 22:36 |
| jogo | clarkb: ahh thanks | 22:37 |
| nibalizer | clarkb: okay | 22:37 |
| jogo | clarkb: is there anyway to find out how patient I need to be? | 22:40 |
| clarkb | jogo: yes, http://puppetdb.openstack.org/ when the jenkins* nodes move from unreported long time period to changed you should be good | 22:40 |
| clarkb | now I need to figure out why puppet hasn't run on those nodes | 22:41 |
| nibalizer | clarkb: this isn't bad, this way i can say complete ideas. | 22:41 |
| jogo | clarkb: awesome thanks! | 22:42 |
| *** esker has quit IRC | 22:42 | |
| lifeless | clarkb: is there a timeout on the jeblair query ? | 22:43 |
| clarkb | lifeless: ya going to restart shortly | 22:45 |
| lifeless | cool | 22:45 |
| clarkb | jogo: the jenkins masters have broken puppet, SERVER: Invalid relationship: File[/var/lib/jenkins/hudson.plugins.gearman.GearmanPluginConfig.xml] { require => Class[Zuul] }, because Class[Zuul] doesn't seem to be in the catalog | 22:47 |
| clarkb | switching back to nodepool now | 22:47 |
| jeblair | clarkb: i will propose the revert to the patch that broke that | 22:48 |
| *** atiwari has joined #openstack-infra | 22:48 | |
| * jeblair is pretty sure no one tests puppet patches :( | 22:48 | |
| clarkb | jeblair: :/ | 22:49 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revert "Enable Gearman as default on Jenkins slaves" https://review.openstack.org/84271 | 22:49 |
| jeblair | clarkb: ^ | 22:49 |
| clarkb | lifeless: no dice | 22:49 |
| clarkb | lifeless: f69ac and b45c are in the config | 22:50 |
| clarkb | lifeless: are they wrong? | 22:50 |
| clarkb | I think this may be a case of fungi leaving things in a halfway state | 22:50 |
| *** signed8bit has quit IRC | 22:51 | |
| *** _nadya_ has joined #openstack-infra | 22:51 | |
| *** hogepodge has quit IRC | 22:52 | |
| lifeless | clarkb: https://review.openstack.org/#/c/84263/ has teh correct values | 22:52 |
| clarkb | lifeless: want to address the comment and we can merge that change? | 22:53 |
| lifeless | clarkb: I don't understand what you want | 22:53 |
| clarkb | lifeless: 35 tripleo-precise slaves as the min ready value is pretty silly | 22:53 |
| lifeless | clarkb: fungi said the sum should be < max-servers | 22:53 |
| lifeless | clarkb: why is it silly? | 22:53 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Enable Gearman as default on Jenkins slaves" https://review.openstack.org/84271 | 22:54 |
| clarkb | lifeless: we only do 42 for bare-precise | 22:54 |
| clarkb | lifeless: which run an order of magnitude more tests | 22:54 |
| clarkb | lifeless: it is overkill | 22:54 |
| lifeless | clarkb: its enough for 7 reviews | 22:54 |
| lifeless | clarkb: once fedora is added, 3 reviews. | 22:54 |
| lifeless | clarkb: what benefit will it have to lower it? | 22:54 |
| clarkb | the fedora node has 1 min ready | 22:55 |
| lifeless | fedora isn't active yet | 22:55 |
| clarkb | lifeless: fewer machiens connected to jenkins | 22:55 |
| *** _nadya_ has quit IRC | 22:55 | |
| *** dcramer_ has joined #openstack-infra | 22:56 | |
| lifeless | clarkb: I don't understand all the implications enough to guess at the right value. Tell me the number you want, and I'll change the review. My preference would be to have it higher not lower, to stay ahead of pushes, but if you're hitting resource limits elsewhere, maybe we can run a jenkins slave or something for you ? | 22:57 |
| clarkb | lifeless: I did suggest a number in the chang | 22:57 |
| lifeless | you suggested Maybe a min ready count for ~2 changes to be tested? | 22:57 |
| clarkb | its not that we are hitting resource limits necessarily as much as it is letting the autoscaling do its job | 22:57 |
| clarkb | lifeless: yes | 22:57 |
| clarkb | lifeless: which appears to be 10 | 22:57 |
| lifeless | sure, I don't entirely get the autoscaling aspect since this clouds entire reason for existence is hosting these nodes | 22:58 |
| *** Sukhdev has joined #openstack-infra | 22:59 | |
| jeblair | lifeless: for one, it gives more flexibility to spin up ubuntu or fedora nodes as needed | 22:59 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Fixup tripleo nodepool config. https://review.openstack.org/84263 | 22:59 |
| *** saschpe has quit IRC | 23:00 | |
| *** julim has quit IRC | 23:00 | |
| *** CaptTofu has quit IRC | 23:00 | |
| jeblair | is it possible to use names instead of uuids for the networks? | 23:00 |
| lifeless | jeblair: no | 23:01 |
| lifeless | jeblair: I wish it was | 23:01 |
| jeblair | :( | 23:01 |
| lifeless | jeblair: but I haven't had time to a) debate it to death to get agreement and then b) implement it | 23:01 |
| lifeless | multi-nic use with nova+neutron is at best described as unpleasant, today. | 23:02 |
| *** thuc has quit IRC | 23:02 | |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack-infra/config: Sync up designate repo ACLs https://review.openstack.org/84273 | 23:02 |
| *** thuc has joined #openstack-infra | 23:02 | |
| *** kiall has joined #openstack-infra | 23:03 | |
| *** thuc has quit IRC | 23:03 | |
| *** thuc has joined #openstack-infra | 23:03 | |
| *** atiwari has quit IRC | 23:03 | |
| kiall | Heya - I couldn't convince someone (well, 2 people) to +2 this Gerrit ACL change? It should allow me to create the milestone-proposed branch on the Designate repos.. Thanks :) | 23:03 |
| kiall | https://review.openstack.org/#/c/84273/ | 23:04 |
| *** saschpe has joined #openstack-infra | 23:06 | |
| *** jamielennox|away is now known as jamielennox | 23:06 | |
| clarkb | kiall: reviewed | 23:07 |
| kiall | thanks clarkb :) | 23:07 |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack-infra/config: Sync up designate repo ACLs https://review.openstack.org/84273 | 23:09 |
| *** wenlock has quit IRC | 23:10 | |
| lifeless | clarkb: so - any happier ? | 23:11 |
| *** alexpilotti has quit IRC | 23:11 | |
| clarkb | lifeless: not yet, I have a sudden worry that restarting nodepool was a bad idea | 23:11 |
| * clarkb pokes at it more | 23:11 | |
| clarkb | I did it the graceful way which I thought would be happy | 23:12 |
| clarkb | oh I know, we delete all nodes on startup now? | 23:12 |
| clarkb | jeblair: is that right? so we have a bunch of nodes building right now | 23:12 |
| mordred | clarkb, jeblair: I _MIGHT have to re-work the nodepool dib patch at this point slightly :) | 23:13 |
| lifeless | clarkb: for reference http://paste.openstack.org/show/74701/ are the networks | 23:13 |
| clarkb | note to self, don't restart nodepool | 23:15 |
| lifeless | clarkb: heh, so if you need to do it twice, do it quick ? | 23:16 |
| *** ameade has joined #openstack-infra | 23:17 | |
| clarkb | lifeless: nodepool deletes all nodes that it had when it restarts | 23:20 |
| clarkb | lifeless: so there is a delay after restarting wheere it deletes and rebuilds all the things | 23:20 |
| lifeless | clarkb: yah | 23:21 |
| lifeless | clarkb: so I guess my question is is the config in place now, or will we be dancing with it again in a few minutes | 23:21 |
| *** ameade has quit IRC | 23:21 | |
| *** yamahata has quit IRC | 23:21 | |
| clarkb | lifeless: it isn't in place yet because we need nodes to run the tests to merge it | 23:21 |
| *** nkinder has joined #openstack-infra | 23:21 | |
| lifeless | clarkb: so I'm worried that if a restart is needed for it, that you'll pay the price twice | 23:22 |
| lifeless | clarkb: thus I'd like to propose putting it in place now, and if it doesn't pick up automatically doing a new restart right now, rather than having this high cost again later. | 23:23 |
| clarkb | lifeless: I don't think it is needed | 23:23 |
| clarkb | jeblair IOError: [Errno 2] No such file is happening a lot when doing sftp to write out roles | 23:23 |
| clarkb | and I think that may actually be why we are not getting any nodes right now | 23:24 |
| *** khyati_ has quit IRC | 23:25 | |
| clarkb | jeblair: I think new nodepool code is broken, looking at fixing now | 23:26 |
| andreaf | jeblair: new patchset, {node} included - https://review.openstack.org/#/c/79314/4 | 23:27 |
| clarkb | jeblair: pretty sure we need to mkdir /etc/nodepool before writing files in that dir | 23:28 |
| *** mwagner_zzz has joined #openstack-infra | 23:28 | |
| lifeless | nodepool won't have privilege to do that will it ? | 23:28 |
| lifeless | clarkb: ^ | 23:28 |
| *** rfolco has quit IRC | 23:28 | |
| clarkb | lifeless: it will, it runs as root | 23:29 |
| Hunner | nibalizer: Hmm? | 23:30 |
| *** mgagne has quit IRC | 23:30 | |
| *** david-lyle has quit IRC | 23:30 | |
| lifeless | clarkb: wut :P | 23:30 |
| lifeless | clarkb: I wouldn't like to see that baked in as an assumption | 23:31 |
| *** thuc has quit IRC | 23:31 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Make /etc/nodepool before writing to it. https://review.openstack.org/84278 | 23:31 |
| clarkb | jeblair: lifeless ^ | 23:31 |
| clarkb | lifeless: uh but it is an assumption | 23:32 |
| *** zns has quit IRC | 23:32 | |
| *** thuc has joined #openstack-infra | 23:32 | |
| clarkb | lifeless: the subnode stuff won't work without it | 23:32 |
| clarkb | jeblair: an opinion on that would be great, though I think it is safe to go back to the old code, in fact I may do that right now | 23:33 |
| clarkb | lifeless: I am not sure how the thing tasked with spinning up and configuring your test nodes can get away with not having root | 23:34 |
| clarkb | lifeless: you may be able to move most of the work to image build time, but at some point the thing has roots | 23:34 |
| nibalizer | Hunner: i was going to grump about hiera, but I got over it :P | 23:35 |
| *** thuc has quit IRC | 23:36 | |
| sdague | clarkb / jeblair - I believe this to be largely sound - https://review.openstack.org/#/c/81216/ | 23:38 |
| mordred | nibalizer: did you solve things? | 23:38 |
| sdague | I'm running another check on it to clear out the hopefully crud on the havana trees | 23:38 |
| nibalizer | mordred: clarkb and I have some ideas on the review | 23:38 |
| mordred | nibalizer: neat! | 23:39 |
| nibalizer | i think we'll move /etc/puppet/hieradata into /etc/puppet/data | 23:39 |
| nibalizer | which gets us very close to 2 stanzas instead of 1 | 23:39 |
| nibalizer | er 2 instaed of 3 | 23:39 |
| clarkb | jeblair: lifeless et al I ahve applied lifeless' nodepool yaml change by hand and downgraded nodepool to where it was at earlier today | 23:39 |
| clarkb | running restart now | 23:39 |
| nibalizer | but we still need to decide what to do with /etc/puppet/data/%{environment} | 23:39 |
| nibalizer | are the rooters honestly using that feature? | 23:39 |
| nibalizer | in my experience that kind of data doesn't change much at all | 23:39 |
| clarkb | it is waiting for graceful shutdown | 23:40 |
| nibalizer | and when it does, it doesn't go by environment | 23:40 |
| lifeless | clarkb: because root is local machine admin, cloud-API access is cloud API access | 23:41 |
| lifeless | clarkb: totally separate concerns | 23:41 |
| clarkb | lifeless: correct, but nodepool is responsible for creating the images | 23:42 |
| clarkb | which requires root | 23:42 |
| lifeless | clarkb: on the spun up image yes, but not the nodepool daemon | 23:42 |
| clarkb | lifeless: I don't understand your concenr then | 23:42 |
| lifeless | clarkb: I am making the assumption that any reference to /etc/nodepool is on the daemon machine | 23:42 |
| clarkb | lifeless: wrong assumption | 23:43 |
| lifeless | clarkb: I assumed breadcrumbs or state markers would be in an FHS friendly place (e.g. /var/lib/nodepool or /var/run/nodepool) | 23:43 |
| clarkb | in multinode scenarios nodepool writes to /etc/nodepool on the spun up machines | 23:43 |
| clarkb | or attempts to at least | 23:43 |
| clarkb | currently it is failing to do thta and is broken. Once the graceful restart happens we should be back in business | 23:43 |
| lifeless | clarkb: ah - so thats surprising to me | 23:43 |
| lifeless | clarkb: thanks for looking at the nodepool config for us | 23:44 |
| *** ociuhandu has quit IRC | 23:44 | |
| clarkb | np, I think we are currently waiting for all deleting nodes to be deleted and all building nodes to finish building. | 23:44 |
| clarkb | then nodepool will restart | 23:45 |
| clarkb | down to 541 total nodes from 708 | 23:45 |
| clarkb | note to self, check /opt/nodepool for new code before restarting nodepool | 23:47 |
| *** mrodden has quit IRC | 23:49 | |
| *** jhesketh__ has joined #openstack-infra | 23:49 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: turn back on log enforcement https://review.openstack.org/81216 | 23:50 |
| *** jhesketh__ has quit IRC | 23:51 | |
| *** david-lyle has joined #openstack-infra | 23:53 | |
| *** david-lyle has quit IRC | 23:58 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: normalize the services for a post grizzly world https://review.openstack.org/84287 | 23:58 |
| *** gokrokve_ has quit IRC | 23:59 | |
| *** gokrokve has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!