| *** erlon has quit IRC | 00:01 | |
| *** mchestr has quit IRC | 00:01 | |
| *** ngoracke has joined #openstack-dev | 00:02 | |
| *** mchestr has joined #openstack-dev | 00:02 | |
| *** electrocucaracha has quit IRC | 00:05 | |
| *** emagana_ has quit IRC | 00:06 | |
| *** stevemar has joined #openstack-dev | 00:06 | |
| *** Raildo_ has joined #openstack-dev | 00:06 | |
| *** emagana has joined #openstack-dev | 00:06 | |
| *** mchestr has quit IRC | 00:07 | |
| *** ybathia has quit IRC | 00:07 | |
| *** xingchao has quit IRC | 00:08 | |
| *** ho has joined #openstack-dev | 00:09 | |
| *** timello has quit IRC | 00:10 | |
| *** zz_jgrimm has quit IRC | 00:10 | |
| *** emagana has quit IRC | 00:10 | |
| *** Raildo has quit IRC | 00:11 | |
| *** zz_jgrimm has joined #openstack-dev | 00:11 | |
| *** timello has joined #openstack-dev | 00:11 | |
| *** otter768 has joined #openstack-dev | 00:13 | |
| *** spzala has quit IRC | 00:13 | |
| *** _cjones_ has joined #openstack-dev | 00:13 | |
| *** Raildo_ has quit IRC | 00:13 | |
| *** sputnik13 has quit IRC | 00:15 | |
| *** timello has quit IRC | 00:16 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 00:16 | |
| *** zz_jgrimm has quit IRC | 00:17 | |
| *** otter768 has quit IRC | 00:17 | |
| *** chutwig has joined #openstack-dev | 00:18 | |
| *** timello has joined #openstack-dev | 00:18 | |
| *** zz_jgrimm has joined #openstack-dev | 00:18 | |
| *** galstrom_zzz is now known as galstrom | 00:19 | |
| *** xingchao has joined #openstack-dev | 00:20 | |
| *** thomasem has quit IRC | 00:23 | |
| *** asalkeld has joined #openstack-dev | 00:24 | |
| *** geguileo has quit IRC | 00:26 | |
| *** thomasem_ has joined #openstack-dev | 00:27 | |
| *** thomasem_ has quit IRC | 00:27 | |
| *** thomasem has joined #openstack-dev | 00:29 | |
| *** zhiwei has joined #openstack-dev | 00:31 | |
| *** stevemar has quit IRC | 00:32 | |
| *** armax has joined #openstack-dev | 00:33 | |
| *** geguileo has joined #openstack-dev | 00:34 | |
| *** promulo_ has joined #openstack-dev | 00:35 | |
| *** pradeep has joined #openstack-dev | 00:37 | |
| *** asettle has quit IRC | 00:37 | |
| *** thomasem has quit IRC | 00:38 | |
| *** thomasem has joined #openstack-dev | 00:38 | |
| *** promulo has quit IRC | 00:39 | |
| *** tiswanso has quit IRC | 00:40 | |
| *** hashar has quit IRC | 00:42 | |
| *** _cjones_ has quit IRC | 00:43 | |
| *** mpaolino has joined #openstack-dev | 00:44 | |
| *** mpaolino has quit IRC | 00:45 | |
| *** mwang2_ has quit IRC | 00:49 | |
| *** leeantho has quit IRC | 00:49 | |
| *** xarses has joined #openstack-dev | 00:51 | |
| *** kgriffs is now known as kgriffs|afk | 00:51 | |
| *** carl_baldwin has quit IRC | 00:52 | |
| *** btulchinsky has quit IRC | 00:52 | |
| *** dprince has joined #openstack-dev | 00:54 | |
| *** Qiming has joined #openstack-dev | 00:55 | |
| *** Qiming_ has joined #openstack-dev | 00:56 | |
| *** zzzeek has quit IRC | 00:56 | |
| *** btulchinsky has joined #openstack-dev | 00:57 | |
| *** fallenpegasus has joined #openstack-dev | 00:57 | |
| *** chutwig has quit IRC | 00:57 | |
| *** chutwig has joined #openstack-dev | 00:58 | |
| *** Qiming has quit IRC | 01:01 | |
| *** Qiming__ has joined #openstack-dev | 01:01 | |
| *** Qiming_ has quit IRC | 01:04 | |
| *** btulchinsky has quit IRC | 01:05 | |
| *** etoews has quit IRC | 01:09 | |
| *** ssurana1 has quit IRC | 01:11 | |
| *** tiswanso has joined #openstack-dev | 01:11 | |
| *** tnovacik has quit IRC | 01:15 | |
| *** asettle has joined #openstack-dev | 01:16 | |
| *** rawl has joined #openstack-dev | 01:17 | |
| *** pradeep has left #openstack-dev | 01:20 | |
| *** Qiming has joined #openstack-dev | 01:21 | |
| *** igordcard has quit IRC | 01:22 | |
| *** eghobo has quit IRC | 01:22 | |
| *** bpokorny has quit IRC | 01:22 | |
| *** eghobo has joined #openstack-dev | 01:22 | |
| *** Marga_ has quit IRC | 01:23 | |
| *** fallenpegasus has quit IRC | 01:23 | |
| *** fallenpegasus has joined #openstack-dev | 01:23 | |
| *** Qiming__ has quit IRC | 01:24 | |
| *** kacieee has joined #openstack-dev | 01:26 | |
| *** sarob has quit IRC | 01:27 | |
| *** MIG- has quit IRC | 01:27 | |
| *** pixelbeat has joined #openstack-dev | 01:27 | |
| *** fallenpegasus has quit IRC | 01:28 | |
| *** Lee1092 has joined #openstack-dev | 01:28 | |
| *** xingchao has quit IRC | 01:29 | |
| *** fallenpegasus has joined #openstack-dev | 01:31 | |
| *** rawl has quit IRC | 01:32 | |
| *** stanzgy has joined #openstack-dev | 01:37 | |
| *** rtheis has quit IRC | 01:37 | |
| *** _cjones_ has joined #openstack-dev | 01:39 | |
| *** rlrevell has quit IRC | 01:40 | |
| *** alanf-mc has quit IRC | 01:44 | |
| *** eghobo has quit IRC | 01:44 | |
| *** _cjones_ has quit IRC | 01:44 | |
| *** Longgeek has joined #openstack-dev | 01:46 | |
| *** Ephur has joined #openstack-dev | 01:47 | |
| *** mikeoutland has joined #openstack-dev | 01:52 | |
| *** ncoghlan has joined #openstack-dev | 01:52 | |
| *** tdruiva has joined #openstack-dev | 01:53 | |
| *** mikeoutland has quit IRC | 01:53 | |
| *** unicell has quit IRC | 01:54 | |
| *** dprince has quit IRC | 01:56 | |
| *** galstrom is now known as galstrom_zzz | 01:58 | |
| *** Longgeek has quit IRC | 01:58 | |
| *** patrickeast has quit IRC | 02:04 | |
| *** ssurana has joined #openstack-dev | 02:04 | |
| *** baoli has joined #openstack-dev | 02:05 | |
| *** bharath_ has quit IRC | 02:10 | |
| *** bharath has joined #openstack-dev | 02:10 | |
| *** achanda has quit IRC | 02:11 | |
| *** otter768 has joined #openstack-dev | 02:13 | |
| *** bharath has quit IRC | 02:15 | |
| *** zz_dimtruck is now known as dimtruck | 02:16 | |
| *** rlrevell has joined #openstack-dev | 02:16 | |
| *** ssurana has quit IRC | 02:18 | |
| *** tdruiva has quit IRC | 02:18 | |
| *** ssurana has joined #openstack-dev | 02:18 | |
| *** otter768 has quit IRC | 02:18 | |
| *** tdruiva has joined #openstack-dev | 02:19 | |
| *** kacieee has quit IRC | 02:20 | |
| *** rawl has joined #openstack-dev | 02:21 | |
| *** kbrierly has quit IRC | 02:21 | |
| *** baoli has quit IRC | 02:24 | |
| *** rawl has quit IRC | 02:25 | |
| *** rawl has joined #openstack-dev | 02:26 | |
| *** sarob has joined #openstack-dev | 02:28 | |
| *** asettle has quit IRC | 02:31 | |
| *** sarob has quit IRC | 02:33 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 02:33 | |
| *** baoli has joined #openstack-dev | 02:36 | |
| *** enarciso has quit IRC | 02:37 | |
| *** rlrevell has quit IRC | 02:37 | |
| *** yamahata has quit IRC | 02:37 | |
| *** sarob has joined #openstack-dev | 02:39 | |
| *** ijw has quit IRC | 02:39 | |
| *** novas0x2a|laptop has quit IRC | 02:39 | |
| *** xingchao has joined #openstack-dev | 02:39 | |
| *** ijw has joined #openstack-dev | 02:39 | |
| *** dimtruck is now known as zz_dimtruck | 02:41 | |
| *** btulchinsky has joined #openstack-dev | 02:42 | |
| *** ijw has quit IRC | 02:43 | |
| *** enarciso has joined #openstack-dev | 02:44 | |
| *** xingchao has quit IRC | 02:45 | |
| *** enarciso has quit IRC | 02:45 | |
| *** FunnyLookinHat has quit IRC | 02:46 | |
| *** djinni has quit IRC | 02:47 | |
| *** galstrom_zzz is now known as galstrom | 02:49 | |
| *** rawl has quit IRC | 02:52 | |
| *** mikeoutland has joined #openstack-dev | 02:52 | |
| *** jyuso1 has joined #openstack-dev | 02:52 | |
| *** sputnik13 has joined #openstack-dev | 02:53 | |
| *** Longgeek has joined #openstack-dev | 02:56 | |
| *** diogogmt has joined #openstack-dev | 02:56 | |
| *** diogogmt has quit IRC | 03:00 | |
| *** djinni has joined #openstack-dev | 03:01 | |
| *** mchestr has joined #openstack-dev | 03:03 | |
| *** Longgeek has quit IRC | 03:08 | |
| *** bpokorny has joined #openstack-dev | 03:10 | |
| *** mchestr has quit IRC | 03:10 | |
| *** vuntz has quit IRC | 03:10 | |
| *** spandhe has quit IRC | 03:10 | |
| *** vuntz has joined #openstack-dev | 03:10 | |
| *** kacieee has joined #openstack-dev | 03:10 | |
| *** pradeep has joined #openstack-dev | 03:12 | |
| *** pradeep has left #openstack-dev | 03:13 | |
| *** tiswanso has quit IRC | 03:14 | |
| *** alanf-mc has joined #openstack-dev | 03:16 | |
| *** mikeoutland has quit IRC | 03:22 | |
| *** coolsvap|afk is now known as coolsvap | 03:23 | |
| *** dims_ has quit IRC | 03:23 | |
| *** alanf-mc has quit IRC | 03:23 | |
| *** alanf-mc has joined #openstack-dev | 03:23 | |
| *** yamahata has joined #openstack-dev | 03:24 | |
| *** otter768 has joined #openstack-dev | 03:25 | |
| *** signed8bit has quit IRC | 03:25 | |
| *** _cjones_ has joined #openstack-dev | 03:28 | |
| *** asettle has joined #openstack-dev | 03:30 | |
| *** beisner- has joined #openstack-dev | 03:30 | |
| *** asettle has quit IRC | 03:31 | |
| *** beisner has quit IRC | 03:31 | |
| *** beisner- is now known as beisner | 03:31 | |
| *** asettle has joined #openstack-dev | 03:31 | |
| *** _cjones_ has quit IRC | 03:32 | |
| *** otter768 has quit IRC | 03:33 | |
| *** bpokorny has quit IRC | 03:34 | |
| *** otter768 has joined #openstack-dev | 03:35 | |
| *** sputnik13 has quit IRC | 03:38 | |
| *** egafford has joined #openstack-dev | 03:38 | |
| *** alanf-mc has quit IRC | 03:40 | |
| *** alanf-mc has joined #openstack-dev | 03:40 | |
| *** zz_dimtruck is now known as dimtruck | 03:41 | |
| *** stevemar has joined #openstack-dev | 03:43 | |
| *** egafford has quit IRC | 03:45 | |
| *** MarkAtwood has quit IRC | 03:47 | |
| *** armax has quit IRC | 03:47 | |
| *** fallenpegasus has quit IRC | 03:49 | |
| *** fallenpegasus has joined #openstack-dev | 03:49 | |
| *** eghobo has joined #openstack-dev | 03:50 | |
| *** achanda has joined #openstack-dev | 03:51 | |
| *** jckasper has joined #openstack-dev | 03:51 | |
| *** baoli has quit IRC | 03:52 | |
| *** zaitcev has quit IRC | 03:53 | |
| *** fallenpegasus has quit IRC | 03:54 | |
| *** tdruiva has quit IRC | 03:56 | |
| *** galstrom is now known as galstrom_zzz | 03:56 | |
| *** chutwig has quit IRC | 03:57 | |
| *** achampion has quit IRC | 03:58 | |
| *** achampion has joined #openstack-dev | 03:58 | |
| *** deepthi has joined #openstack-dev | 03:58 | |
| *** Longgeek has joined #openstack-dev | 04:05 | |
| *** ayoung-dadmode has quit IRC | 04:05 | |
| *** alanf-mc has quit IRC | 04:07 | |
| *** alanf-mc has joined #openstack-dev | 04:08 | |
| *** kacieee has quit IRC | 04:08 | |
| *** gamado has joined #openstack-dev | 04:09 | |
| *** otter768 has quit IRC | 04:09 | |
| *** achanda has quit IRC | 04:12 | |
| *** Qiming_ has joined #openstack-dev | 04:15 | |
| *** gokrokve has joined #openstack-dev | 04:16 | |
| *** galstrom_zzz is now known as galstrom | 04:17 | |
| *** eghobo has quit IRC | 04:18 | |
| *** Qiming has quit IRC | 04:18 | |
| *** pixelbeat has quit IRC | 04:18 | |
| *** dimtruck is now known as zz_dimtruck | 04:19 | |
| *** gokrokve has quit IRC | 04:20 | |
| *** tdruiva has joined #openstack-dev | 04:23 | |
| *** dims has joined #openstack-dev | 04:23 | |
| *** mikeoutland has joined #openstack-dev | 04:24 | |
| *** eghobo has joined #openstack-dev | 04:25 | |
| *** yatin has joined #openstack-dev | 04:25 | |
| *** RoozbehShafiee has joined #openstack-dev | 04:27 | |
| *** _cjones_ has joined #openstack-dev | 04:28 | |
| *** dims has quit IRC | 04:28 | |
| *** Longgeek has quit IRC | 04:29 | |
| *** sarob has quit IRC | 04:30 | |
| *** jckasper has quit IRC | 04:31 | |
| *** mikeoutland has quit IRC | 04:32 | |
| *** _cjones_ has quit IRC | 04:33 | |
| *** unicell has joined #openstack-dev | 04:35 | |
| *** Networkn3rd has quit IRC | 04:40 | |
| *** SumitNaiksatam has joined #openstack-dev | 04:43 | |
| *** maishsk has quit IRC | 04:43 | |
| *** gokrokve has joined #openstack-dev | 04:45 | |
| *** inhumanitas has quit IRC | 04:46 | |
| *** AnilV4 has joined #openstack-dev | 04:48 | |
| *** galstrom is now known as galstrom_zzz | 04:49 | |
| *** vCloudernBeer has joined #openstack-dev | 04:50 | |
| *** abregman has joined #openstack-dev | 05:03 | |
| *** bobmel has joined #openstack-dev | 05:04 | |
| *** bobmel_ has quit IRC | 05:06 | |
| *** asettle has quit IRC | 05:07 | |
| *** inhumanitas has joined #openstack-dev | 05:08 | |
| *** kacieee has joined #openstack-dev | 05:08 | |
| *** achanda has joined #openstack-dev | 05:10 | |
| *** bharath has joined #openstack-dev | 05:11 | |
| *** lpetrut has joined #openstack-dev | 05:12 | |
| *** eghobo_ has joined #openstack-dev | 05:13 | |
| *** bharath has quit IRC | 05:15 | |
| *** emagana has joined #openstack-dev | 05:16 | |
| *** BharatK has joined #openstack-dev | 05:17 | |
| *** eghobo has quit IRC | 05:17 | |
| *** eghobo has joined #openstack-dev | 05:19 | |
| *** eghobo_ has quit IRC | 05:21 | |
| *** kacieee has quit IRC | 05:21 | |
| *** RoozbehShafiee has quit IRC | 05:25 | |
| *** alanf-mc has quit IRC | 05:26 | |
| *** toey has joined #openstack-dev | 05:27 | |
| *** neeti has joined #openstack-dev | 05:27 | |
| *** Qiming__ has joined #openstack-dev | 05:28 | |
| *** neeti has quit IRC | 05:28 | |
| *** yatin has quit IRC | 05:28 | |
| *** natorious has joined #openstack-dev | 05:29 | |
| *** neeti has joined #openstack-dev | 05:30 | |
| *** toey has quit IRC | 05:30 | |
| *** vCloudernBeer has quit IRC | 05:31 | |
| *** Qiming_ has quit IRC | 05:31 | |
| *** RoozbehShafiee has joined #openstack-dev | 05:33 | |
| *** tnovacik has joined #openstack-dev | 05:33 | |
| *** alanf-mc has joined #openstack-dev | 05:34 | |
| *** alanf-mc has quit IRC | 05:35 | |
| *** gem has joined #openstack-dev | 05:37 | |
| *** bharath has joined #openstack-dev | 05:38 | |
| *** bharath_ has joined #openstack-dev | 05:40 | |
| *** gem has quit IRC | 05:40 | |
| *** gem has joined #openstack-dev | 05:40 | |
| *** yatin has joined #openstack-dev | 05:41 | |
| *** bharath has quit IRC | 05:42 | |
| *** hdd has joined #openstack-dev | 05:43 | |
| *** bharath_ has quit IRC | 05:44 | |
| *** numan has joined #openstack-dev | 05:44 | |
| *** xingchao has joined #openstack-dev | 05:45 | |
| *** asettle has joined #openstack-dev | 05:46 | |
| *** ajayaa has joined #openstack-dev | 05:46 | |
| *** itzikb has joined #openstack-dev | 05:47 | |
| *** SumitNaiksatam has quit IRC | 05:48 | |
| *** SumitNaiksatam has joined #openstack-dev | 05:49 | |
| *** xingchao has quit IRC | 05:52 | |
| *** Qiming__ has quit IRC | 05:52 | |
| *** Qiming__ has joined #openstack-dev | 05:52 | |
| *** alanf-mc has joined #openstack-dev | 05:53 | |
| *** mikeoutland has joined #openstack-dev | 05:53 | |
| *** mikeoutland has quit IRC | 05:54 | |
| *** spandhe has joined #openstack-dev | 05:54 | |
| *** gem has quit IRC | 05:55 | |
| *** bharath has joined #openstack-dev | 05:55 | |
| *** gem_ has joined #openstack-dev | 05:55 | |
| *** bharath has quit IRC | 05:56 | |
| *** bharath_ has joined #openstack-dev | 05:56 | |
| *** spandhe_ has joined #openstack-dev | 05:57 | |
| *** benjamingr has quit IRC | 05:57 | |
| *** gokrokve_ has joined #openstack-dev | 05:58 | |
| *** spandhe has quit IRC | 05:58 | |
| *** spandhe_ is now known as spandhe | 05:58 | |
| *** bharath_ has quit IRC | 05:59 | |
| *** Longgeek has joined #openstack-dev | 05:59 | |
| *** bharath has joined #openstack-dev | 05:59 | |
| *** mrunge has joined #openstack-dev | 05:59 | |
| *** itzikb_ has joined #openstack-dev | 06:00 | |
| *** gokrokve has quit IRC | 06:01 | |
| *** natorious is now known as zz_natorious | 06:02 | |
| *** gokrokve_ has quit IRC | 06:02 | |
| *** kushal has joined #openstack-dev | 06:02 | |
| *** itzikb has quit IRC | 06:03 | |
| *** yatin has quit IRC | 06:04 | |
| *** alanf-mc has quit IRC | 06:05 | |
| *** ngoracke has quit IRC | 06:05 | |
| *** maishsk has joined #openstack-dev | 06:06 | |
| *** asettle has quit IRC | 06:06 | |
| *** mkletzan has joined #openstack-dev | 06:07 | |
| *** bharath has quit IRC | 06:07 | |
| *** oro has joined #openstack-dev | 06:07 | |
| *** doron_afk has joined #openstack-dev | 06:07 | |
| *** bharath has joined #openstack-dev | 06:08 | |
| *** pkoniszewski has joined #openstack-dev | 06:09 | |
| *** gem_ has quit IRC | 06:10 | |
| *** gem_ has joined #openstack-dev | 06:10 | |
| *** otter768 has joined #openstack-dev | 06:10 | |
| *** ijw has joined #openstack-dev | 06:12 | |
| *** bharath has quit IRC | 06:13 | |
| *** zz_natorious is now known as natorious | 06:14 | |
| *** otter768 has quit IRC | 06:14 | |
| *** Longgeek_ has joined #openstack-dev | 06:14 | |
| *** bharath has joined #openstack-dev | 06:15 | |
| *** Longgeek has quit IRC | 06:17 | |
| *** _cjones_ has joined #openstack-dev | 06:17 | |
| *** masayukig has quit IRC | 06:19 | |
| *** masayukig has joined #openstack-dev | 06:19 | |
| *** natorious is now known as zz_natorious | 06:22 | |
| *** sahid has joined #openstack-dev | 06:22 | |
| *** _cjones_ has quit IRC | 06:22 | |
| *** Qiming__ is now known as Qiming | 06:22 | |
| *** itzikb_ is now known as itzikb | 06:23 | |
| *** kudryashova has quit IRC | 06:24 | |
| *** vCloudernBeer has joined #openstack-dev | 06:24 | |
| *** kudryashova has joined #openstack-dev | 06:25 | |
| *** gem_ has quit IRC | 06:25 | |
| *** gem_ has joined #openstack-dev | 06:25 | |
| vCloudernBeer | How do I re-submit my change after fixing what automation test suggested | 06:25 |
|---|---|---|
| *** ijw has quit IRC | 06:27 | |
| *** scheuran has joined #openstack-dev | 06:27 | |
| *** bharath has quit IRC | 06:27 | |
| *** bharath has joined #openstack-dev | 06:28 | |
| *** ijw has joined #openstack-dev | 06:28 | |
| *** gamado has quit IRC | 06:29 | |
| *** bharath has quit IRC | 06:30 | |
| *** bharath has joined #openstack-dev | 06:31 | |
| *** oro has quit IRC | 06:31 | |
| *** emagana has quit IRC | 06:32 | |
| *** mrunge has quit IRC | 06:33 | |
| *** emagana has joined #openstack-dev | 06:33 | |
| *** vCloudernBeer has quit IRC | 06:35 | |
| *** bharath has quit IRC | 06:35 | |
| *** mrunge has joined #openstack-dev | 06:37 | |
| *** emagana has quit IRC | 06:38 | |
| *** ukalifon1 has joined #openstack-dev | 06:38 | |
| *** kushal has quit IRC | 06:38 | |
| *** kushal has joined #openstack-dev | 06:40 | |
| *** safchain has joined #openstack-dev | 06:41 | |
| *** comay has quit IRC | 06:41 | |
| *** [1]evgenyf has joined #openstack-dev | 06:41 | |
| *** xgsa has joined #openstack-dev | 06:43 | |
| *** jprovazn has joined #openstack-dev | 06:44 | |
| *** yatin has joined #openstack-dev | 06:44 | |
| *** dulek has joined #openstack-dev | 06:48 | |
| *** jcoufal has joined #openstack-dev | 06:50 | |
| *** bharath has joined #openstack-dev | 06:51 | |
| *** lpetrut has quit IRC | 06:54 | |
| *** bharath has quit IRC | 06:59 | |
| *** ijw has quit IRC | 07:01 | |
| *** yeylon has joined #openstack-dev | 07:04 | |
| *** SridharG has joined #openstack-dev | 07:05 | |
| *** athomas has joined #openstack-dev | 07:06 | |
| *** gamado has joined #openstack-dev | 07:09 | |
| *** roeyc has joined #openstack-dev | 07:10 | |
| *** sandywalsh has quit IRC | 07:11 | |
| *** eghobo has quit IRC | 07:12 | |
| *** sandywalsh has joined #openstack-dev | 07:12 | |
| *** markus_z has joined #openstack-dev | 07:12 | |
| *** matrohon has joined #openstack-dev | 07:13 | |
| *** yamahata has quit IRC | 07:14 | |
| *** mmagr has joined #openstack-dev | 07:15 | |
| *** achanda has quit IRC | 07:16 | |
| *** achanda has joined #openstack-dev | 07:22 | |
| *** stevemar has quit IRC | 07:22 | |
| *** achanda has quit IRC | 07:24 | |
| *** lpetrut has joined #openstack-dev | 07:24 | |
| *** yassine has joined #openstack-dev | 07:24 | |
| *** egallen has joined #openstack-dev | 07:25 | |
| *** mmagr is now known as mmagr|afk | 07:28 | |
| *** maishsk has quit IRC | 07:30 | |
| *** maishsk has joined #openstack-dev | 07:32 | |
| *** doron_afk is now known as doron | 07:33 | |
| *** mmagr|afk is now known as mmagr | 07:35 | |
| *** k4n0 has joined #openstack-dev | 07:35 | |
| *** yassine has quit IRC | 07:37 | |
| *** dulek has quit IRC | 07:43 | |
| *** dulek has joined #openstack-dev | 07:44 | |
| *** maishsk has quit IRC | 07:44 | |
| *** maishsk has joined #openstack-dev | 07:46 | |
| *** oro has joined #openstack-dev | 07:47 | |
| *** hdd_ has joined #openstack-dev | 07:48 | |
| *** yatin has quit IRC | 07:48 | |
| *** gfidente has joined #openstack-dev | 07:49 | |
| *** gfidente has quit IRC | 07:49 | |
| *** gfidente has joined #openstack-dev | 07:49 | |
| *** hdd has quit IRC | 07:51 | |
| *** ndipanov has joined #openstack-dev | 07:52 | |
| *** dulek has quit IRC | 07:55 | |
| *** dulek has joined #openstack-dev | 07:55 | |
| *** aloga has quit IRC | 07:57 | |
| *** doron is now known as doron_afk | 07:57 | |
| *** yassine has joined #openstack-dev | 08:01 | |
| *** dulek has quit IRC | 08:02 | |
| *** yatin has joined #openstack-dev | 08:03 | |
| *** liam_ has joined #openstack-dev | 08:04 | |
| *** liam_ is now known as Guest74585 | 08:05 | |
| *** _cjones_ has joined #openstack-dev | 08:06 | |
| *** oro has quit IRC | 08:06 | |
| *** haomaiw__ has joined #openstack-dev | 08:08 | |
| *** Guest74585 has quit IRC | 08:09 | |
| *** jordanP has joined #openstack-dev | 08:10 | |
| *** haomaiwang has quit IRC | 08:10 | |
| *** hashar has joined #openstack-dev | 08:11 | |
| *** hashar has quit IRC | 08:11 | |
| *** hashar has joined #openstack-dev | 08:11 | |
| *** otter768 has joined #openstack-dev | 08:11 | |
| *** _cjones_ has quit IRC | 08:12 | |
| *** ygbo has joined #openstack-dev | 08:12 | |
| *** otter768 has quit IRC | 08:16 | |
| *** yuukigoodman has quit IRC | 08:20 | |
| *** xingchao has joined #openstack-dev | 08:21 | |
| *** derekh has joined #openstack-dev | 08:23 | |
| *** dulek has joined #openstack-dev | 08:23 | |
| *** xingchao has quit IRC | 08:25 | |
| *** MattMan has joined #openstack-dev | 08:33 | |
| *** ssurana has quit IRC | 08:34 | |
| *** MaxV has joined #openstack-dev | 08:34 | |
| *** yatin has quit IRC | 08:42 | |
| *** mmagr is now known as mmagr|afk | 08:43 | |
| *** SumitNaiksatam has quit IRC | 08:44 | |
| *** yatin has joined #openstack-dev | 08:44 | |
| *** SumitNaiksatam has joined #openstack-dev | 08:46 | |
| *** kushal has quit IRC | 08:49 | |
| *** dguitarbite has joined #openstack-dev | 08:49 | |
| *** pnavarro has joined #openstack-dev | 08:53 | |
| *** MaxV has quit IRC | 08:53 | |
| *** yatin has quit IRC | 08:55 | |
| *** mmagr|afk is now known as mmagr | 08:56 | |
| *** gem_ has quit IRC | 08:56 | |
| *** gem_ has joined #openstack-dev | 08:56 | |
| *** jistr has joined #openstack-dev | 08:56 | |
| *** yatin has joined #openstack-dev | 08:58 | |
| *** aix has quit IRC | 08:59 | |
| *** deepakcs has joined #openstack-dev | 08:59 | |
| *** pelix has joined #openstack-dev | 08:59 | |
| *** dulek has quit IRC | 09:00 | |
| *** scheuran has quit IRC | 09:02 | |
| *** Krin has joined #openstack-dev | 09:04 | |
| *** djinni has quit IRC | 09:12 | |
| *** jtomasek has joined #openstack-dev | 09:13 | |
| *** shardy_ has joined #openstack-dev | 09:13 | |
| *** shardy has quit IRC | 09:14 | |
| *** gszasz has joined #openstack-dev | 09:14 | |
| *** scheuran has joined #openstack-dev | 09:16 | |
| *** cdent has joined #openstack-dev | 09:17 | |
| *** hdd__ has joined #openstack-dev | 09:18 | |
| *** shardy_ has quit IRC | 09:19 | |
| *** ho has quit IRC | 09:19 | |
| *** shardy has joined #openstack-dev | 09:19 | |
| *** hdd_ has quit IRC | 09:21 | |
| *** jyuso1 has quit IRC | 09:21 | |
| *** pixelbeat has joined #openstack-dev | 09:21 | |
| *** ncoghlan has quit IRC | 09:22 | |
| *** akrivoka has joined #openstack-dev | 09:22 | |
| *** mmagr is now known as mmagr|afk | 09:22 | |
| *** [1]evgenyf has quit IRC | 09:24 | |
| *** alexpilotti has joined #openstack-dev | 09:25 | |
| *** maishsk has quit IRC | 09:26 | |
| *** gem_ has quit IRC | 09:26 | |
| *** gem__ has joined #openstack-dev | 09:26 | |
| *** AndroUser has quit IRC | 09:26 | |
| *** AndroUser has joined #openstack-dev | 09:26 | |
| *** maishsk has joined #openstack-dev | 09:26 | |
| *** aix has joined #openstack-dev | 09:27 | |
| *** maishsk has quit IRC | 09:32 | |
| *** maishsk has joined #openstack-dev | 09:32 | |
| *** gilllliard is now known as gilliard_afk | 09:33 | |
| *** tdruiva has quit IRC | 09:43 | |
| *** mblaszczak has joined #openstack-dev | 09:47 | |
| *** mblaszczak is now known as pasqd | 09:47 | |
| *** markvoelker has quit IRC | 09:48 | |
| *** yeylon has quit IRC | 09:49 | |
| *** yeylon has joined #openstack-dev | 09:49 | |
| *** roeyc has quit IRC | 09:49 | |
| *** yeylon has quit IRC | 09:49 | |
| *** Qiming has quit IRC | 09:50 | |
| *** benjamingr has joined #openstack-dev | 09:50 | |
| *** djinni has joined #openstack-dev | 09:51 | |
| *** zz_dimtruck is now known as dimtruck | 09:54 | |
| *** _cjones_ has joined #openstack-dev | 09:56 | |
| *** scheuran has quit IRC | 09:57 | |
| *** f13o has joined #openstack-dev | 09:58 | |
| *** zhiwei has quit IRC | 09:59 | |
| *** bharath has joined #openstack-dev | 09:59 | |
| *** _cjones_ has quit IRC | 10:01 | |
| *** [1]evgenyf has joined #openstack-dev | 10:02 | |
| *** djinni has quit IRC | 10:03 | |
| *** bharath has quit IRC | 10:04 | |
| *** [2]evgenyf has joined #openstack-dev | 10:06 | |
| *** maishsk has quit IRC | 10:07 | |
| *** maishsk has joined #openstack-dev | 10:07 | |
| *** [1]evgenyf has quit IRC | 10:08 | |
| *** dimtruck is now known as zz_dimtruck | 10:10 | |
| *** aix has quit IRC | 10:12 | |
| *** otter768 has joined #openstack-dev | 10:12 | |
| *** djinni has joined #openstack-dev | 10:15 | |
| *** f13o has quit IRC | 10:15 | |
| *** otter768 has quit IRC | 10:16 | |
| *** RoozbehShafiee has quit IRC | 10:17 | |
| *** tobe has joined #openstack-dev | 10:18 | |
| *** kbyrne has quit IRC | 10:18 | |
| *** Guest50818 has quit IRC | 10:19 | |
| *** nekolus has joined #openstack-dev | 10:19 | |
| *** dulek has joined #openstack-dev | 10:19 | |
| nekolus | when ssl is enabled in localrc of devstack, i get this error message 2015-05-05 08:55:21.816 | ++ iniset /etc/boto.cfg Boto ca_certificates_file /opt/stack/data/ca-bundle.pem 2015-05-05 08:55:21.831 | /home/nicholas/devstack/inc/ini-config: line 148: /etc/boto.cfg: Permission denied | 10:19 |
| *** dims has joined #openstack-dev | 10:20 | |
| *** wznoinsk has joined #openstack-dev | 10:20 | |
| *** kbyrne has joined #openstack-dev | 10:20 | |
| *** roeyc has joined #openstack-dev | 10:22 | |
| *** dims_ has joined #openstack-dev | 10:22 | |
| jordanP | nekolus, only when ssl is enabled ? | 10:23 |
| *** btulchinsky has quit IRC | 10:24 | |
| *** kiran-r has joined #openstack-dev | 10:24 | |
| *** aix has joined #openstack-dev | 10:25 | |
| *** mmagr|afk is now known as mmagr | 10:26 | |
| *** dims has quit IRC | 10:26 | |
| *** nekolus has quit IRC | 10:28 | |
| *** maishsk is now known as maishsk_afk | 10:29 | |
| *** wznoinsk has quit IRC | 10:29 | |
| *** claudiub has joined #openstack-dev | 10:29 | |
| *** wznoinsk has joined #openstack-dev | 10:30 | |
| *** armax has joined #openstack-dev | 10:34 | |
| *** armax has quit IRC | 10:42 | |
| *** sahid has quit IRC | 10:44 | |
| *** egallen has quit IRC | 10:44 | |
| *** pc_m has joined #openstack-dev | 10:44 | |
| *** doron_afk is now known as doron | 10:46 | |
| *** scheuran has joined #openstack-dev | 10:54 | |
| *** armax has joined #openstack-dev | 10:59 | |
| *** rtheis has joined #openstack-dev | 10:59 | |
| *** djinni has quit IRC | 11:04 | |
| *** tdruiva has joined #openstack-dev | 11:07 | |
| *** stanzgy has quit IRC | 11:11 | |
| *** sahid has joined #openstack-dev | 11:12 | |
| *** sandywalsh has quit IRC | 11:12 | |
| *** Krin has quit IRC | 11:12 | |
| *** achampion has quit IRC | 11:13 | |
| *** k4n0 has quit IRC | 11:13 | |
| *** doron is now known as doron_afk | 11:14 | |
| *** sandywalsh has joined #openstack-dev | 11:14 | |
| *** xingchao has joined #openstack-dev | 11:16 | |
| *** djinni has joined #openstack-dev | 11:19 | |
| *** deepthi has quit IRC | 11:21 | |
| *** btulchinsky has joined #openstack-dev | 11:27 | |
| *** egallen has joined #openstack-dev | 11:28 | |
| *** ndk has quit IRC | 11:32 | |
| *** FunnyLookinHat has joined #openstack-dev | 11:33 | |
| *** SridharG has quit IRC | 11:34 | |
| *** numan has quit IRC | 11:35 | |
| *** Krin has joined #openstack-dev | 11:37 | |
| *** Qiming has joined #openstack-dev | 11:38 | |
| *** thrash|g0ne is now known as thrash | 11:38 | |
| *** ajayaa has quit IRC | 11:40 | |
| *** egallen has quit IRC | 11:42 | |
| *** ctina_ has joined #openstack-dev | 11:44 | |
| *** _cjones_ has joined #openstack-dev | 11:45 | |
| *** igordcard_ has quit IRC | 11:45 | |
| *** igordcard_ has joined #openstack-dev | 11:46 | |
| *** igordcard_ has quit IRC | 11:46 | |
| *** igordcard_ has joined #openstack-dev | 11:46 | |
| *** numan has joined #openstack-dev | 11:48 | |
| *** _cjones_ has quit IRC | 11:50 | |
| *** SridharG has joined #openstack-dev | 11:51 | |
| *** markvoelker has joined #openstack-dev | 11:51 | |
| *** Tross has joined #openstack-dev | 11:52 | |
| *** egallen has joined #openstack-dev | 11:53 | |
| *** djinni has quit IRC | 11:53 | |
| *** dprince has joined #openstack-dev | 11:54 | |
| *** Tross1 has quit IRC | 11:54 | |
| *** djinni has joined #openstack-dev | 11:55 | |
| *** Tross has quit IRC | 11:56 | |
| *** Tross1 has joined #openstack-dev | 11:56 | |
| *** tdruiva has quit IRC | 11:58 | |
| *** yeylon has joined #openstack-dev | 11:58 | |
| *** igordcard_ has quit IRC | 11:58 | |
| *** galstrom_zzz is now known as galstrom | 11:59 | |
| *** armax has quit IRC | 12:00 | |
| *** ajayaa has joined #openstack-dev | 12:01 | |
| *** gmoro has quit IRC | 12:02 | |
| *** egallen has quit IRC | 12:03 | |
| *** igordcard_ has joined #openstack-dev | 12:03 | |
| *** maishsk_afk has quit IRC | 12:04 | |
| *** pc_m has quit IRC | 12:07 | |
| *** raildo has joined #openstack-dev | 12:07 | |
| *** k4n0 has joined #openstack-dev | 12:08 | |
| *** thrash has quit IRC | 12:09 | |
| *** gmoro has joined #openstack-dev | 12:09 | |
| *** achampion has joined #openstack-dev | 12:09 | |
| *** thrash has joined #openstack-dev | 12:11 | |
| *** AndroUser has quit IRC | 12:12 | |
| *** zhiwei has joined #openstack-dev | 12:12 | |
| *** ajayaa has quit IRC | 12:13 | |
| *** otter768 has joined #openstack-dev | 12:13 | |
| *** AndroUser has joined #openstack-dev | 12:13 | |
| *** kgiusti has joined #openstack-dev | 12:13 | |
| *** achampion has quit IRC | 12:14 | |
| *** deepakcs has quit IRC | 12:15 | |
| *** mwagner_lap has quit IRC | 12:16 | |
| *** dkranz has joined #openstack-dev | 12:16 | |
| *** igordcard_ has quit IRC | 12:17 | |
| *** igordcard_ has joined #openstack-dev | 12:17 | |
| *** otter768 has quit IRC | 12:18 | |
| *** egallen has joined #openstack-dev | 12:18 | |
| *** trown|outttypeww is now known as trown | 12:19 | |
| sdague | boris-42: I like the inventive use of the starredby: queries in the rally dashboard, going to have to use that other places :) | 12:20 |
| *** zhiwei has left #openstack-dev | 12:21 | |
| *** egallen has quit IRC | 12:22 | |
| *** doron_afk is now known as doron | 12:24 | |
| *** egallen has joined #openstack-dev | 12:24 | |
| *** dkranz has quit IRC | 12:26 | |
| *** achampion has joined #openstack-dev | 12:27 | |
| *** gamado has quit IRC | 12:29 | |
| *** achampion has quit IRC | 12:32 | |
| jd__ | flaper87: psssst https://review.openstack.org/#/c/163940/ | 12:32 |
| *** mwagner_lap has joined #openstack-dev | 12:33 | |
| *** ukalifon1 has quit IRC | 12:36 | |
| *** Qiming_ has joined #openstack-dev | 12:37 | |
| *** morazi has joined #openstack-dev | 12:38 | |
| *** rfolco has joined #openstack-dev | 12:38 | |
| *** dkranz has joined #openstack-dev | 12:38 | |
| *** gyee has joined #openstack-dev | 12:39 | |
| *** baoli has joined #openstack-dev | 12:40 | |
| *** aysyd has joined #openstack-dev | 12:40 | |
| *** Qiming has quit IRC | 12:40 | |
| *** Networkn3rd has joined #openstack-dev | 12:41 | |
| *** dims_ has quit IRC | 12:41 | |
| *** dims has joined #openstack-dev | 12:41 | |
| *** FunnyLookinHat has quit IRC | 12:42 | |
| *** hashar is now known as hasharAway | 12:43 | |
| *** maishsk has joined #openstack-dev | 12:45 | |
| *** RoozbehShafiee has joined #openstack-dev | 12:45 | |
| *** Networkn3rd has quit IRC | 12:45 | |
| *** pixelbeat has quit IRC | 12:46 | |
| *** notnownikki has joined #openstack-dev | 12:48 | |
| *** mmagr is now known as mmagr|afk | 12:48 | |
| *** pixelbeat has joined #openstack-dev | 12:53 | |
| *** galstrom is now known as galstrom_zzz | 12:53 | |
| *** doron is now known as doron_afk | 12:54 | |
| *** vhoward has quit IRC | 13:00 | |
| *** NithyaG has joined #openstack-dev | 13:00 | |
| *** vhoward has joined #openstack-dev | 13:00 | |
| *** radez_g0n3 is now known as radez | 13:00 | |
| *** yatin has quit IRC | 13:01 | |
| *** slaweq has joined #openstack-dev | 13:02 | |
| *** ctina_ has quit IRC | 13:03 | |
| *** _elmiko is now known as elmiko | 13:04 | |
| *** tiswanso has joined #openstack-dev | 13:04 | |
| *** aloga has joined #openstack-dev | 13:05 | |
| *** armax has joined #openstack-dev | 13:05 | |
| *** eharney has joined #openstack-dev | 13:05 | |
| boris-42 | sdague: heh I don't know how I live before=) | 13:06 |
| *** ajayaa has joined #openstack-dev | 13:08 | |
| *** gem__ has quit IRC | 13:08 | |
| *** mmagr|afk is now known as mmagr | 13:08 | |
| *** changbl has quit IRC | 13:08 | |
| *** amuller has joined #openstack-dev | 13:09 | |
| *** egafford has joined #openstack-dev | 13:09 | |
| *** miqui has joined #openstack-dev | 13:10 | |
| *** rlrevell has joined #openstack-dev | 13:10 | |
| *** neelashah has joined #openstack-dev | 13:11 | |
| *** dustins has joined #openstack-dev | 13:11 | |
| *** sarob has joined #openstack-dev | 13:12 | |
| *** Ctina has joined #openstack-dev | 13:13 | |
| *** yassine has quit IRC | 13:13 | |
| *** RoozbehShafiee has quit IRC | 13:15 | |
| *** roeyc has quit IRC | 13:15 | |
| *** vladikr has joined #openstack-dev | 13:16 | |
| *** danielbruno has joined #openstack-dev | 13:17 | |
| jdandrea | Thanks sdague! (workflow for devstack change) | 13:17 |
| *** achampion has joined #openstack-dev | 13:17 | |
| *** BharatK has quit IRC | 13:17 | |
| sdague | jdandrea: you're welcome? (though I' | 13:18 |
| sdague | m not quite sure what it's in reference to) | 13:18 |
| jdandrea | Whoops - a link would help. :) https://review.openstack.org/#/c/178819/ | 13:18 |
| *** amakarov_away is now known as amakarov | 13:20 | |
| *** ngoracke has joined #openstack-dev | 13:20 | |
| sdague | oh, right, no prob :) | 13:21 |
| *** egallen has quit IRC | 13:22 | |
| *** gyee has quit IRC | 13:22 | |
| *** bswartz has quit IRC | 13:23 | |
| *** numan has quit IRC | 13:23 | |
| *** egallen has joined #openstack-dev | 13:25 | |
| *** kbringard has joined #openstack-dev | 13:25 | |
| *** dkranz has quit IRC | 13:26 | |
| *** gaughen has joined #openstack-dev | 13:26 | |
| *** bknudson has joined #openstack-dev | 13:26 | |
| *** gyee has joined #openstack-dev | 13:27 | |
| *** dkranz has joined #openstack-dev | 13:27 | |
| *** mikeit has joined #openstack-dev | 13:27 | |
| *** yassine has joined #openstack-dev | 13:28 | |
| *** athomas has quit IRC | 13:28 | |
| *** mkletzan has quit IRC | 13:30 | |
| *** roeyc has joined #openstack-dev | 13:30 | |
| *** Nikolay_St has joined #openstack-dev | 13:31 | |
| *** BharatK has joined #openstack-dev | 13:31 | |
| *** ajayaa has quit IRC | 13:32 | |
| *** signed8bit has joined #openstack-dev | 13:33 | |
| *** _cjones_ has joined #openstack-dev | 13:34 | |
| *** Guest15 has joined #openstack-dev | 13:35 | |
| *** athomas has joined #openstack-dev | 13:35 | |
| *** jaosorior has joined #openstack-dev | 13:35 | |
| *** pc_m has joined #openstack-dev | 13:36 | |
| *** BharatK has quit IRC | 13:36 | |
| *** tdruiva has joined #openstack-dev | 13:37 | |
| jprovazn | Hi, anyone who could +2/+A https://review.openstack.org/#/c/174233/ which adds tuskarclient into openstack requirements? It already has bunch of positive votes | 13:38 |
| *** mugsie_ is now known as mugsie | 13:38 | |
| *** _cjones_ has quit IRC | 13:39 | |
| *** signed8b_ has joined #openstack-dev | 13:39 | |
| *** tobe has quit IRC | 13:40 | |
| *** signed8bit has quit IRC | 13:40 | |
| *** tdruiva_ has joined #openstack-dev | 13:41 | |
| *** tdruiva has quit IRC | 13:42 | |
| *** Ala has quit IRC | 13:42 | |
| *** danielbruno has quit IRC | 13:45 | |
| *** cbouch has joined #openstack-dev | 13:45 | |
| *** egallen_ has joined #openstack-dev | 13:46 | |
| *** jbell8 has joined #openstack-dev | 13:46 | |
| *** FunnyLookinHat has joined #openstack-dev | 13:46 | |
| *** pkoniszewski has quit IRC | 13:47 | |
| *** egallen has quit IRC | 13:48 | |
| *** egallen_ is now known as egallen | 13:48 | |
| *** zzzeek has joined #openstack-dev | 13:48 | |
| *** Ala has joined #openstack-dev | 13:48 | |
| *** BharatK has joined #openstack-dev | 13:49 | |
| *** jckasper has joined #openstack-dev | 13:49 | |
| *** claudiub_ has joined #openstack-dev | 13:50 | |
| *** bdemers has joined #openstack-dev | 13:51 | |
| *** johnbelamaric has joined #openstack-dev | 13:51 | |
| *** mriedem has joined #openstack-dev | 13:51 | |
| *** claudiub has quit IRC | 13:51 | |
| *** ksobkowiak has joined #openstack-dev | 13:52 | |
| *** bswartz has joined #openstack-dev | 13:56 | |
| *** Sukhdev has joined #openstack-dev | 13:56 | |
| *** yamahata has joined #openstack-dev | 13:57 | |
| *** egallen has quit IRC | 13:58 | |
| *** carl_baldwin has joined #openstack-dev | 13:58 | |
| *** fawadkhaliq has joined #openstack-dev | 13:59 | |
| *** neelashah has quit IRC | 13:59 | |
| *** djinni has quit IRC | 14:00 | |
| *** danielbruno has joined #openstack-dev | 14:00 | |
| *** achampion has quit IRC | 14:01 | |
| *** yassine has quit IRC | 14:01 | |
| *** ajayaa has joined #openstack-dev | 14:01 | |
| *** egallen has joined #openstack-dev | 14:02 | |
| *** pcm_ has joined #openstack-dev | 14:02 | |
| *** ayoung has joined #openstack-dev | 14:03 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:03 | |
| *** pc_m has quit IRC | 14:06 | |
| *** zz_jgrimm is now known as jgrimm | 14:06 | |
| *** ajmiller has joined #openstack-dev | 14:07 | |
| *** cbouch has quit IRC | 14:08 | |
| *** numan has joined #openstack-dev | 14:10 | |
| *** kacieee has joined #openstack-dev | 14:10 | |
| *** gokrokve has joined #openstack-dev | 14:10 | |
| *** mdenny has joined #openstack-dev | 14:10 | |
| *** exploreshaifali has joined #openstack-dev | 14:11 | |
| *** djinni has joined #openstack-dev | 14:11 | |
| ZZelle | zzzeek, hi | 14:11 |
| *** mpavone has joined #openstack-dev | 14:13 | |
| *** tdruiva_ is now known as tdruiva | 14:13 | |
| *** otter768 has joined #openstack-dev | 14:14 | |
| *** jrist has quit IRC | 14:15 | |
| *** mmagr is now known as mmagr|afk | 14:15 | |
| *** achampion has joined #openstack-dev | 14:15 | |
| *** jprovazn has quit IRC | 14:16 | |
| *** mmagr|afk is now known as mmagr | 14:17 | |
| *** armax has quit IRC | 14:18 | |
| *** otter768 has quit IRC | 14:18 | |
| *** dulek has quit IRC | 14:19 | |
| *** xarses has quit IRC | 14:19 | |
| *** ajayaa has quit IRC | 14:19 | |
| *** cbouch has joined #openstack-dev | 14:20 | |
| *** lifeless has quit IRC | 14:20 | |
| *** yassine has joined #openstack-dev | 14:21 | |
| *** k4n0 has quit IRC | 14:21 | |
| *** yassine has quit IRC | 14:23 | |
| *** yassine has joined #openstack-dev | 14:23 | |
| *** Guest15 has quit IRC | 14:24 | |
| *** electrocucaracha has joined #openstack-dev | 14:24 | |
| *** jbell8 has quit IRC | 14:24 | |
| *** gargola has joined #openstack-dev | 14:25 | |
| *** itzikb has quit IRC | 14:25 | |
| *** itzikb has joined #openstack-dev | 14:26 | |
| *** k4n0 has joined #openstack-dev | 14:26 | |
| *** lifeless has joined #openstack-dev | 14:27 | |
| *** k4n0 has quit IRC | 14:27 | |
| *** k4n0 has joined #openstack-dev | 14:27 | |
| *** jrist has joined #openstack-dev | 14:28 | |
| *** annegentle has joined #openstack-dev | 14:29 | |
| *** numan has quit IRC | 14:29 | |
| *** mikeoutland has joined #openstack-dev | 14:30 | |
| *** armax has joined #openstack-dev | 14:31 | |
| *** maishsk is now known as maishsk_afk | 14:33 | |
| *** mikeoutland has quit IRC | 14:33 | |
| *** amitgandhinz has joined #openstack-dev | 14:33 | |
| *** dulek has joined #openstack-dev | 14:34 | |
| *** nelsnelson has joined #openstack-dev | 14:34 | |
| *** kiran-r has quit IRC | 14:36 | |
| *** kgriffs|afk is now known as kgriffs | 14:36 | |
| *** mattfarina has joined #openstack-dev | 14:36 | |
| *** Qiming_ has quit IRC | 14:39 | |
| *** Shrews has left #openstack-dev | 14:40 | |
| *** emagana has joined #openstack-dev | 14:40 | |
| *** numan has joined #openstack-dev | 14:41 | |
| *** jprovazn has joined #openstack-dev | 14:41 | |
| *** SridharG has quit IRC | 14:42 | |
| *** hichihara has joined #openstack-dev | 14:42 | |
| *** jrist has quit IRC | 14:42 | |
| *** gokrokve_ has joined #openstack-dev | 14:43 | |
| *** Networkn3rd has joined #openstack-dev | 14:44 | |
| *** gokrokve has quit IRC | 14:46 | |
| zzzeek | ZZelle: hi | 14:47 |
| *** hasharAway is now known as hashar | 14:47 | |
| *** sputnik13 has joined #openstack-dev | 14:48 | |
| *** coolsvap is now known as coolsvap|afk | 14:48 | |
| ZZelle | zzzeek, hi, about https://review.openstack.org/179955 | 14:48 |
| *** pnavarro has quit IRC | 14:49 | |
| *** zz_natorious is now known as natorious | 14:49 | |
| *** AndroUser has quit IRC | 14:50 | |
| ZZelle | zzzeek, i responded to your comment, i am not to see how we can use update-on-match as we don't update the row we lock but we use it to ensure an "atomic" section | 14:50 |
| *** sputnik13 has quit IRC | 14:50 | |
| *** itzikb has quit IRC | 14:50 | |
| zzzeek | ZZelle: is there an INSERT occurring as part of this routine? | 14:51 |
| *** egallen has quit IRC | 14:52 | |
| ZZelle | zzzeek, yes, we always perform an insert | 14:52 |
| ZZelle | zzzeek, except when no cidr matchs | 14:52 |
| zzzeek | ZZelle: so, is the idea, lock row X, then perform an INSERT of row Y which refers to X, the lock is to prevent X from being updated before the INSERT proceeds? | 14:53 |
| zzzeek | ZZelle: all that is needed to not use SELECT..FOR UPDATE is to make it so that one of the SQL statements fails in some way if concurrent changes have happened. then you retry | 14:53 |
| *** MattMan has quit IRC | 14:53 | |
| *** hdd__ has quit IRC | 14:54 | |
| zzzeek | ZZelle: with Galera, the row is not locked on other nodes | 14:54 |
| *** hdd__ has joined #openstack-dev | 14:54 | |
| zzzeek | ZZelle: it can still be updated by another process | 14:54 |
| *** mattfarina has quit IRC | 14:55 | |
| *** RabidCicada has joined #openstack-dev | 14:55 | |
| *** MattMan has joined #openstack-dev | 14:55 | |
| *** MattMan has left #openstack-dev | 14:55 | |
| *** notnownikki has quit IRC | 14:55 | |
| *** mattfarina has joined #openstack-dev | 14:55 | |
| ZZelle | zzzeek, the trouble is that the check is on subnet cidrs ... with current db model it seems impossible to express the constraint as an sql condition | 14:55 |
| zzzeek | ZZelle: first off, would the “constraint” be at the level of, INSERT a row that refers to another row that is already present? | 14:56 |
| *** jbell8 has joined #openstack-dev | 14:56 | |
| *** hdd__ has quit IRC | 14:56 | |
| zzzeek | ZZelle: it could be as easy as, do the INSERTs, then just do a second SELECT at the end of the transation to see if the “locked” row is still there, if not, retry | 14:56 |
| *** Marga_ has joined #openstack-dev | 14:57 | |
| *** Marga_ has quit IRC | 14:57 | |
| *** Marga_ has joined #openstack-dev | 14:58 | |
| *** abregman has quit IRC | 14:58 | |
| ZZelle | zzzeek, referring to an other row is only used to lock and provide a critical section | 14:58 |
| *** Sukhdev has quit IRC | 14:58 | |
| *** reed_ has joined #openstack-dev | 14:59 | |
| ZZelle | zzzeek, we have currently no way to check that the row we are inserting is in conflict with existent rows ... we must do it in python | 15:00 |
| zzzeek | ZZelle: can you add jaypipes as a reviewer. all im saying is, this lock does not lock | 15:00 |
| zzzeek | ZZelle: so i dont see its purpose | 15:00 |
| ZZelle | zzzeek, ok | 15:00 |
| zzzeek | ZZelle: jaypipes is way more into this issue than I am | 15:01 |
| *** SridharG has joined #openstack-dev | 15:01 | |
| *** achanda has joined #openstack-dev | 15:01 | |
| jaypipes | zzzeek: feel free to add me as a reviewer. | 15:01 |
| ZZelle | jaypipes, it's done :) | 15:02 |
| jaypipes | cool, thanks Cedric :) | 15:02 |
| zzzeek | jaypipes: am i correct that the basic idea is, “SELECT FOR UPDATE” doesnt really “lock” effectively | 15:02 |
| *** hichihara has quit IRC | 15:03 | |
| *** xarses has joined #openstack-dev | 15:03 | |
| *** vhoward has left #openstack-dev | 15:03 | |
| *** kushal has joined #openstack-dev | 15:03 | |
| *** kushal has quit IRC | 15:03 | |
| *** kushal has joined #openstack-dev | 15:03 | |
| jaypipes | zzzeek: correct. it won't prevent another physical Galera node from updating the locked rows, and when the node that locked that row with SELECT FOR UPDATE goes to actually update the "locked" rows, a deadlock error will occur when the UPDATE reaches another node that has already updated the record. | 15:04 |
| *** mpavone has quit IRC | 15:04 | |
| zzzeek | jaypipes: Ok but in this case they are not actually emitting UPDATE for the row | 15:04 |
| *** galstrom_zzz is now known as galstrom | 15:04 | |
| ZZelle | zzzeek, jaypipes you are, but the aim of the change is to replace a non-working lock with a lock working at least when you don't use galera | 15:04 |
| jaypipes | zzzeek: ? then why bother with SELECT FOR UPDATE? :) | 15:04 |
| zzzeek | jaypipes: they are locking it becuase somehow some other rows to be INSERTed are dependent on it and for some reason they cannot cretate SQL constraints from the secondary rows to the primary | 15:04 |
| zzzeek | jaypipes: yes that was my thought :) | 15:04 |
| *** _cjones_ has joined #openstack-dev | 15:05 | |
| jaypipes | ok, well I will do a review and leave comments on the patch itself... | 15:05 |
| *** achanda has quit IRC | 15:06 | |
| *** epopt37 has quit IRC | 15:06 | |
| *** fallenpegasus has joined #openstack-dev | 15:08 | |
| *** epopt37 has joined #openstack-dev | 15:08 | |
| *** cbouch has quit IRC | 15:08 | |
| *** dasp has left #openstack-dev | 15:09 | |
| *** _cjones_ has quit IRC | 15:09 | |
| *** mrunge has quit IRC | 15:09 | |
| *** jobewan has joined #openstack-dev | 15:10 | |
| *** tdruiva has quit IRC | 15:12 | |
| *** jaypipes has quit IRC | 15:13 | |
| *** gyee has quit IRC | 15:13 | |
| *** abregman has joined #openstack-dev | 15:13 | |
| *** lpetrut has quit IRC | 15:14 | |
| *** gyee has joined #openstack-dev | 15:14 | |
| *** pradeep has joined #openstack-dev | 15:15 | |
| *** yeylon has quit IRC | 15:18 | |
| *** neeti has quit IRC | 15:18 | |
| *** ndk has joined #openstack-dev | 15:18 | |
| *** bpokorny has joined #openstack-dev | 15:19 | |
| *** neeti has joined #openstack-dev | 15:20 | |
| *** claudiub_ has quit IRC | 15:21 | |
| *** sridhar has joined #openstack-dev | 15:21 | |
| *** sambol has joined #openstack-dev | 15:23 | |
| *** Ala has quit IRC | 15:24 | |
| *** mpaolino has joined #openstack-dev | 15:26 | |
| *** Marga_ has quit IRC | 15:26 | |
| *** nelsnelson has quit IRC | 15:26 | |
| *** jdurgin1 has joined #openstack-dev | 15:27 | |
| *** pradeep has quit IRC | 15:27 | |
| *** dasp has joined #openstack-dev | 15:27 | |
| *** abregman has quit IRC | 15:28 | |
| *** mchestr has joined #openstack-dev | 15:29 | |
| *** Swami has joined #openstack-dev | 15:29 | |
| *** yamahata has quit IRC | 15:30 | |
| *** doron_afk has quit IRC | 15:30 | |
| *** fallenpegasus has quit IRC | 15:30 | |
| *** dims_ has joined #openstack-dev | 15:32 | |
| *** mpaolino has quit IRC | 15:34 | |
| *** dims has quit IRC | 15:34 | |
| *** schwicht has joined #openstack-dev | 15:35 | |
| *** jrist has joined #openstack-dev | 15:35 | |
| *** jrist has quit IRC | 15:35 | |
| *** jrist has joined #openstack-dev | 15:35 | |
| *** maishsk_afk is now known as maishsk | 15:37 | |
| *** dasp has left #openstack-dev | 15:37 | |
| *** nelsnelson has joined #openstack-dev | 15:37 | |
| *** jrist has quit IRC | 15:37 | |
| *** ssurana has joined #openstack-dev | 15:38 | |
| *** jobewan has quit IRC | 15:39 | |
| *** tsekiyama has joined #openstack-dev | 15:40 | |
| *** mwagner_lap has quit IRC | 15:40 | |
| *** jrist has joined #openstack-dev | 15:41 | |
| *** ngoracke has quit IRC | 15:41 | |
| *** abregman has joined #openstack-dev | 15:41 | |
| *** jobewan has joined #openstack-dev | 15:42 | |
| *** shakamunyi has joined #openstack-dev | 15:43 | |
| *** Ctina is now known as ctina | 15:43 | |
| *** reed_ has quit IRC | 15:45 | |
| *** changbl has joined #openstack-dev | 15:45 | |
| *** Networkn3rd has quit IRC | 15:46 | |
| *** hemnafk is now known as hemna | 15:48 | |
| *** jcoufal has quit IRC | 15:48 | |
| *** smccully has joined #openstack-dev | 15:50 | |
| *** yassine has quit IRC | 15:50 | |
| *** yamahata has joined #openstack-dev | 15:51 | |
| *** spandhe has quit IRC | 15:51 | |
| *** maishsk has quit IRC | 15:53 | |
| *** xingchao has quit IRC | 15:53 | |
| *** armax has quit IRC | 15:54 | |
| *** ngoracke has joined #openstack-dev | 15:55 | |
| *** sridhar has quit IRC | 15:56 | |
| *** jbell8 has quit IRC | 15:57 | |
| *** neeti has quit IRC | 15:57 | |
| *** jbell8 has joined #openstack-dev | 15:57 | |
| *** SridharG has left #openstack-dev | 15:57 | |
| *** electrocucaracha has quit IRC | 15:57 | |
| *** bhunter71 has joined #openstack-dev | 15:58 | |
| *** scheuran has quit IRC | 15:58 | |
| *** btulchinsky has quit IRC | 15:58 | |
| *** k4n0 has quit IRC | 15:58 | |
| *** MarkAtwood has joined #openstack-dev | 15:58 | |
| *** btulchinsky has joined #openstack-dev | 15:59 | |
| *** mmagr is now known as mmagr|afk | 16:00 | |
| *** btulchinsky has quit IRC | 16:01 | |
| *** eghobo has joined #openstack-dev | 16:01 | |
| *** Marga_ has joined #openstack-dev | 16:01 | |
| *** btulchinsky has joined #openstack-dev | 16:01 | |
| *** numan has quit IRC | 16:01 | |
| *** matrohon has quit IRC | 16:01 | |
| *** mikeit has quit IRC | 16:02 | |
| *** fallenpegasus has joined #openstack-dev | 16:02 | |
| *** mmagr|afk is now known as mmagr | 16:02 | |
| *** _cjones_ has joined #openstack-dev | 16:04 | |
| *** hashar is now known as hasharMeeting | 16:04 | |
| *** electrocucaracha has joined #openstack-dev | 16:06 | |
| *** Networkn3rd has joined #openstack-dev | 16:06 | |
| *** jbell8 has quit IRC | 16:11 | |
| *** arnaud____ has joined #openstack-dev | 16:13 | |
| *** unicell has quit IRC | 16:13 | |
| *** dguitarbite has quit IRC | 16:14 | |
| *** bknudson has quit IRC | 16:14 | |
| *** otter768 has joined #openstack-dev | 16:15 | |
| *** [2]evgenyf has quit IRC | 16:16 | |
| *** jtomasek has quit IRC | 16:16 | |
| *** pradeep has joined #openstack-dev | 16:18 | |
| *** otter768 has quit IRC | 16:20 | |
| *** _cjones_ has quit IRC | 16:20 | |
| *** slaweq has quit IRC | 16:21 | |
| rbradfor | Could anybody shed some historical light on why pbr 0.7.0 is specifically not valid for openstack projects (i.e. requirement.txt pbr>=0.6,!=0.7,<1.0) | 16:21 |
| *** daniel3_ has joined #openstack-dev | 16:21 | |
| *** Nikolay_St has quit IRC | 16:21 | |
| *** annegentle has quit IRC | 16:23 | |
| *** maishsk has joined #openstack-dev | 16:23 | |
| *** sputnik13 has joined #openstack-dev | 16:23 | |
| *** annegentle has joined #openstack-dev | 16:25 | |
| *** _cjones_ has joined #openstack-dev | 16:25 | |
| *** jistr has quit IRC | 16:26 | |
| *** maishsk has quit IRC | 16:26 | |
| *** salv-orlando has quit IRC | 16:27 | |
| *** salv-orlando has joined #openstack-dev | 16:27 | |
| *** annegent_ has joined #openstack-dev | 16:27 | |
| *** annegentle has quit IRC | 16:28 | |
| *** tdruiva has joined #openstack-dev | 16:28 | |
| *** baoli has quit IRC | 16:29 | |
| *** Krin has quit IRC | 16:29 | |
| *** tiswanso has quit IRC | 16:30 | |
| *** tiswanso has joined #openstack-dev | 16:31 | |
| *** maishsk has joined #openstack-dev | 16:31 | |
| *** baoli_ has joined #openstack-dev | 16:31 | |
| *** tiswanso_ has joined #openstack-dev | 16:31 | |
| *** ctina has quit IRC | 16:32 | |
| *** ctina has joined #openstack-dev | 16:32 | |
| *** annegent_ has quit IRC | 16:33 | |
| *** tdruiva has quit IRC | 16:33 | |
| *** jbell8 has joined #openstack-dev | 16:33 | |
| *** dulek has quit IRC | 16:34 | |
| *** slupers has joined #openstack-dev | 16:34 | |
| *** slupers has joined #openstack-dev | 16:35 | |
| *** tiswanso has quit IRC | 16:35 | |
| *** leeantho has joined #openstack-dev | 16:37 | |
| *** arnaud____ has quit IRC | 16:37 | |
| *** jaypipes has joined #openstack-dev | 16:38 | |
| *** safchain has quit IRC | 16:38 | |
| *** roeyc has quit IRC | 16:39 | |
| *** gokrokve_ has quit IRC | 16:39 | |
| *** sahid has quit IRC | 16:40 | |
| *** Krin has joined #openstack-dev | 16:40 | |
| *** neelashah has joined #openstack-dev | 16:42 | |
| *** egallen has joined #openstack-dev | 16:42 | |
| *** ygbo has quit IRC | 16:42 | |
| *** Krin has quit IRC | 16:43 | |
| jgriffith | mtreinish: FYI, your subunit db module is proving to be quite handy | 16:43 |
| jgriffith | mtreinish: and works like a champ | 16:43 |
| *** SumitNaiksatam has quit IRC | 16:45 | |
| *** unicell has joined #openstack-dev | 16:45 | |
| *** jordanP has quit IRC | 16:47 | |
| *** ajayaa has joined #openstack-dev | 16:48 | |
| reed | bauzas, regarding your post re:Nova trivial bugs: can those bugs be used for Upstream Training? | 16:49 |
| *** gszasz has quit IRC | 16:49 | |
| * reed is looking for more trivial bugs to assign to newcomers at Upstream Training Vancouver in two weeks tag:low-hanging-fruit | 16:49 | |
| *** jobewan has quit IRC | 16:52 | |
| *** spzala has joined #openstack-dev | 16:52 | |
| *** maishsk_ has joined #openstack-dev | 16:52 | |
| *** doude has quit IRC | 16:52 | |
| *** maishsk has quit IRC | 16:52 | |
| *** maishsk_ is now known as maishsk | 16:52 | |
| *** doude has joined #openstack-dev | 16:52 | |
| *** exploreshaifali has quit IRC | 16:53 | |
| *** slupers has quit IRC | 16:54 | |
| bauzas | reed: the issue is that we're finding changes, not bugs :( | 16:55 |
| bauzas | reed: ie. someone posted a solution to the bug | 16:55 |
| *** mikeoutland has joined #openstack-dev | 16:55 | |
| *** markus_z has quit IRC | 16:56 | |
| bauzas | reed: by doing that initiative, we're just encouraging people to help the core reviewers to find quick-wins | 16:56 |
| *** Marga_ has quit IRC | 16:56 | |
| reed | bauzas, thanks, I understand that now | 16:56 |
| *** dustins has quit IRC | 16:56 | |
| *** Marga_ has joined #openstack-dev | 16:57 | |
| *** btulchinsky has quit IRC | 16:57 | |
| bauzas | reed: but I can speak with johnthetubaguy and see what we can do for identifying bugs which are waiting for a solution | 16:57 |
| *** mattfarina has quit IRC | 16:57 | |
| reed | i am about to send an email encouraging people to tag bugs as low-hanging-fruit | 16:57 |
| *** nmagnezi has joined #openstack-dev | 16:57 | |
| *** benjamingr has quit IRC | 16:57 | |
| bauzas | reed: at least those which are triaged are self-explanatory | 16:57 |
| *** mattfarina has joined #openstack-dev | 16:57 | |
| *** mattfarina has quit IRC | 16:57 | |
| *** BharatK has quit IRC | 16:57 | |
| bauzas | johnthetubaguy: you around ? | 16:57 |
| *** tdruiva has joined #openstack-dev | 16:58 | |
| johnthetubaguy | bauzas: I am | 16:59 |
| bauzas | johnthetubaguy: cool, was just discussing with reed | 16:59 |
| *** deepakcs has joined #openstack-dev | 16:59 | |
| bauzas | johnthetubaguy: we'll be running for another round the UpstreamTraining in Vancouver | 16:59 |
| bauzas | johnthetubaguy: https://wiki.openstack.org/wiki/OpenStack_Upstream_Training | 16:59 |
| johnthetubaguy | bauzas: ah, you need something to work on I guess? | 16:59 |
| *** jschwarz has joined #openstack-dev | 17:00 | |
| *** patrickeast has joined #openstack-dev | 17:00 | |
| bauzas | johnthetubaguy: the thing is, we'll basically give pointers to help people understand how all of that works | 17:00 |
| reed | johnthetubaguy, that's the idea | 17:00 |
| bauzas | johnthetubaguy: but in order to be efficient, our attendees will work on a bug | 17:00 |
| bauzas | johnthetubaguy: not during the training, they will come back and work | 17:01 |
| reed | https://bugs.launchpad.net/openstack/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=low-hanging-fruit&field.tags_combinator=ANY&field.ha | 17:01 |
| reed | s_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on | 17:01 |
| *** gfidente is now known as gfidente|afk | 17:01 | |
| reed | that's the URL we'll use to find low-hanging-fruit bugs | 17:01 |
| bauzas | johnthetubaguy: so we're encouraging them to find a specific bug to identify, come at the training to discuss on how they should manage it, and then, back at home, work on the fix | 17:01 |
| bauzas | johnthetubaguy: the first round is to look at the low-hanging-fruits, but you know how we're good at triaging bugs | 17:02 |
| *** dustins_ has joined #openstack-dev | 17:02 | |
| reed | there aren't many to choose from if you exclude the CI ones (which may be obsolete, since infra doesn't use launchpad anymore) | 17:02 |
| *** pixelbeat has quit IRC | 17:02 | |
| bauzas | johnthetubaguy: so I was wondering how we could just Do Something | 17:03 |
| mtreinish | jgriffith: awesome, glad it's working for you | 17:03 |
| *** maishsk has quit IRC | 17:03 | |
| *** dustins_ is now known as dustins | 17:03 | |
| mtreinish | jgriffith: if you're interested I've got a few more analysis tooling patches up: https://review.openstack.org/#/q/status:open+project:openstack-infra/subunit2sql+branch:master+topic:176114,n,z | 17:03 |
| mtreinish | so you can make pretty pictures :) | 17:03 |
| -openstackstatus- NOTICE: zuul has been restarted to troubleshoot an issue, gerrit events between 15:00-17:00 utc were lost and changes updated or approved during that time will need to be rechecked or have their approval votes readded to trigger testing | 17:03 | |
| *** mwang2 has joined #openstack-dev | 17:04 | |
| bauzas | johnthetubaguy: I pointed reed to look at the triaged bugs, but perhaps you have another idea ? | 17:04 |
| *** dustins has quit IRC | 17:04 | |
| reed | that url is not good, it shows also bugs that have assignee | 17:05 |
| bauzas | johnthetubaguy: I can kick-off a new mail thread asking for volunteers to triage the triaged and set the low-hanging-fruits but I also wonder if that's something we could raise up the topic during the next Nova meeting | 17:05 |
| bauzas | (speaking of Nova specifically of course) | 17:06 |
| bauzas | reed: the URL sucked, because it was too long for my client, could you please minimize it ? | 17:06 |
| johnthetubaguy | bauzas: bauzas: yeah, I mean that tag is there to help for exactly that | 17:06 |
| *** jdurgin1 has quit IRC | 17:06 | |
| reed | bauzas, I think raising the issue during meetings is good | 17:06 |
| *** bobmel_ has joined #openstack-dev | 17:06 | |
| *** gokrokve has joined #openstack-dev | 17:07 | |
| reed | bauzas, what's a good shortener? | 17:07 |
| deepakcs | mtreinish, Hi | 17:07 |
| *** gokrokve has quit IRC | 17:07 | |
| bauzas | reed: I usually take goo.gl | 17:07 |
| reed | https://goo.gl/HX9Nxa | 17:07 |
| reed | all your base are belong to google anyway | 17:08 |
| *** bobmel has quit IRC | 17:08 | |
| mtreinish | deepakcs: hello | 17:08 |
| deepakcs | mtreinish, need your attention on my comment in https://review.openstack.org/#/c/172841/ | 17:08 |
| *** harlowja has quit IRC | 17:08 | |
| johnthetubaguy | reed: bauzas: an email to ask for some folks to help build a new list is cool, I guess, can't think of anything better I am afraid, I guess its just not a tag we have added recently, so good to advertise it again | 17:09 |
| *** harlowja has joined #openstack-dev | 17:09 | |
| deepakcs | mtreinish, there seems to be some difference of opinion between you and ianw on how to approach the "skipping tempest test for specific protocol" | 17:09 |
| deepakcs | mtreinish, we can continue in infra channel since thats more appropriate | 17:09 |
| bauzas | johnthetubaguy: fair enough | 17:10 |
| mtreinish | deepakcs: this channel is fine | 17:10 |
| bauzas | reed: I see, 141 low-hanging-fruits right ? | 17:10 |
| deepakcs | mtreinish, sure, you might want to read the discussion that happened in https://review.openstack.org/#/c/173408/11 | 17:10 |
| reed | bauzas, yes but some are already assigned to someone so the query is not right | 17:10 |
| reed | bauzas, also it queries OpenStack CI which moved to Storyboard and those bugs are not kept in sync | 17:11 |
| mtreinish | deepakcs: adding a skip inline to tempest for a driver specific issue is not something that will be accepted. Tempest is supposed to be cloud agnostic and asking users to know specifics about backend is something we try not to do | 17:11 |
| deepakcs | mtreinish, in case its confusing, let me know, i can quickly help summarise you | 17:11 |
| *** gokrokve has joined #openstack-dev | 17:12 | |
| mtreinish | deepakcs: you do realize that the simple solution, is just to set the snapshot volume feature flag to false in the tempest.config | 17:12 |
| *** gokrokve has quit IRC | 17:12 | |
| mtreinish | that's kinda of what the flag is for to say your env doesn't support volume snapshots | 17:12 |
| deepakcs | mtreinish, ianw 's point is also valid IMHO. i.e. we skip using devstack-gate running tempest outside infra could still hit the issue | 17:12 |
| *** alanf-mc has joined #openstack-dev | 17:12 | |
| bauzas | reed: well, I think the problem is just categorizing the bugs we have per project | 17:12 |
| *** ybathia has joined #openstack-dev | 17:13 | |
| deepakcs | mtreinish, 'not supporting voluem snap' is something NOT true for glusterfs, since it does support. Its just that it doesn't support the scnerio of boot_volume_pattern | 17:13 |
| mtreinish | deepakcs: it's incomplete because it doesn't behave the same way | 17:13 |
| deepakcs | mtreinish, and that too only because libvirt doesn't yet support the offline snap deletion | 17:13 |
| *** cference has joined #openstack-dev | 17:13 | |
| mtreinish | deepakcs: I'm not going to argue with you about that bug, it's there and setting that flag in the tempest conf is the easiest way to skip the test | 17:13 |
| deepakcs | mtreinish, but there are many other scnearios and tests related to snaps which work for glusterfs, it won't be ideal to turn all of them OFF just bcos of 1 scenarios | 17:14 |
| *** gyee has quit IRC | 17:14 | |
| bauzas | reed: so it requires *someone* from each project to look at the existing bugs and tag them | 17:14 |
| mtreinish | deepakcs: then skip using the test runner, adding per backend workarounds to skip or do something else isn't something which we do in tempest | 17:14 |
| mtreinish | there are far too many drivers and permutations to consider doing that | 17:14 |
| *** SumitNaiksatam has joined #openstack-dev | 17:15 | |
| reed | bauzas, I agree, that's the best approac | 17:15 |
| *** vladikr has quit IRC | 17:15 | |
| bauzas | reed: I can handle that for Nova but I will only look at the new and triaged | 17:15 |
| mtreinish | the config file is there to tell tempest how it should expect things to behave and how to perform certain operations | 17:15 |
| deepakcs | mtreinish, by test runner u mean skip using tempest regex or somethign else ? | 17:15 |
| reed | I'll write a call for action for the mailing list today, tag [all] to ask for collaboration | 17:15 |
| bauzas | reed: just because confirmed bugs are so huuuge | 17:15 |
| mtreinish | and you leverage the test runner to select which specific set of tests you want to run | 17:15 |
| reed | i'll mention it quickly at today's xproject meeting too | 17:15 |
| bauzas | reed: I think it needs to be addressed per project IMHO | 17:15 |
| mtreinish | deepakcs: yes, that tempest regex eventually get's passed to testr as the selection regex for running tests | 17:15 |
| *** topol has joined #openstack-dev | 17:16 | |
| *** achanda has joined #openstack-dev | 17:16 | |
| mtreinish | deepakcs: ie test run --parallel $TEMPEST_REGEX | 17:16 |
| mtreinish | s/test/testr | 17:16 |
| *** mmagr has quit IRC | 17:16 | |
| bauzas | reed: so I'll try to take the action for Nova, but you should find other peers for other projects :) | 17:16 |
| *** ssurana has quit IRC | 17:16 | |
| *** vladikr has joined #openstack-dev | 17:16 | |
| deepakcs | mtreinish, thats what we tried to do in https://review.openstack.org/#/c/173408/11 by creating a new CI job and skipping inside that (since sdague mentioned not to mess around with devstack plugin generic template job) | 17:16 |
| bauzas | reed: and I still remember I need to review the slides :) | 17:16 |
| reed | bauzas, adding you to the reviews I pushed | 17:17 |
| bauzas | reed: cool | 17:17 |
| deepakcs | mtreinish, to which ianw gave a -1 and instead said we should skip this in tempest itself (which we already tried and you shot it down in https://review.openstack.org/#/c/172841/) | 17:17 |
| bauzas | reed: I'm just trying to find 28 hours in a day :) | 17:17 |
| *** jrist has quit IRC | 17:17 | |
| reed | i've applied for the upgrade, I'll let you know if it's approved | 17:17 |
| mtreinish | deepakcs: well, I think in this case ianw just wasn't considering the longstanding tempest policy on this kind of thing | 17:17 |
| bauzas | reed: need to step down a little bit, dinner time | 17:18 |
| bauzas | reed: will ttyl | 17:18 |
| reed | ttyl | 17:18 |
| deepakcs | mtreinish, so can you pls respond to ianw in https://review.openstack.org/#/c/173408/11 so that he can revert his -1 and we can contineu with the tempest regex in our own CI job ? | 17:18 |
| deepakcs | mtreinish, the diff of opinion between the 2 approaches has inhibited us to proceed further | 17:19 |
| *** gyee has joined #openstack-dev | 17:21 | |
| *** liam_ has joined #openstack-dev | 17:21 | |
| *** liam_ is now known as Guest95660 | 17:21 | |
| *** baoli_ has quit IRC | 17:22 | |
| *** AnilV4 has quit IRC | 17:25 | |
| deepakcs | mtreinish, there ? | 17:25 |
| *** comay has joined #openstack-dev | 17:27 | |
| *** tnovacik has quit IRC | 17:27 | |
| *** neelashah has quit IRC | 17:28 | |
| *** trown is now known as trown|lunch | 17:29 | |
| *** baoli has joined #openstack-dev | 17:30 | |
| *** AbyssOne has quit IRC | 17:31 | |
| *** tiswanso_ has quit IRC | 17:31 | |
| *** xgsa has quit IRC | 17:32 | |
| *** tdruiva has quit IRC | 17:32 | |
| *** tiswanso has joined #openstack-dev | 17:32 | |
| *** oro has joined #openstack-dev | 17:33 | |
| *** tdruiva has joined #openstack-dev | 17:33 | |
| *** dims_ has quit IRC | 17:33 | |
| *** mwagner_lap has joined #openstack-dev | 17:34 | |
| *** dims has joined #openstack-dev | 17:34 | |
| *** mwang2 has quit IRC | 17:34 | |
| *** Sukhdev has joined #openstack-dev | 17:35 | |
| *** Guest95660 has quit IRC | 17:35 | |
| *** jschwarz has quit IRC | 17:35 | |
| *** athomas has quit IRC | 17:37 | |
| *** slupers has joined #openstack-dev | 17:38 | |
| *** kushal has quit IRC | 17:38 | |
| *** maishsk has joined #openstack-dev | 17:39 | |
| *** xingchao has joined #openstack-dev | 17:39 | |
| *** nmagnezi has quit IRC | 17:40 | |
| *** AbyssOne has joined #openstack-dev | 17:40 | |
| *** lodgenbd has joined #openstack-dev | 17:41 | |
| *** melwitt_ has joined #openstack-dev | 17:42 | |
| *** melwitt_ has quit IRC | 17:42 | |
| deepakcs | mtreinish, i need to logoff now, sent a mail to you and ianw with the hope of resolving this. thanks | 17:44 |
| mtreinish | deepakcs: I left a comment on the review | 17:45 |
| *** dustins has joined #openstack-dev | 17:45 | |
| *** bharath has joined #openstack-dev | 17:46 | |
| *** abregman has quit IRC | 17:46 | |
| *** Networkn3rd has quit IRC | 17:46 | |
| *** oro has quit IRC | 17:47 | |
| deepakcs | mtreinish, thanks, didn't see ur comment before. that was a big comment :) So your -1 is so that we add a echo stating we are skipping this test ? Let me know if i misunderstood | 17:48 |
| mtreinish | deepakcs: the -1 is because the regex is wrong | 17:48 |
| mtreinish | I'm +1 on the approach | 17:48 |
| mtreinish | deepakcs: it's also worth noting you can test the regex by running 'testr list-tests $REGEX' in the tempest repo | 17:49 |
| *** mikeoutland has quit IRC | 17:49 | |
| mtreinish | it'll show everything that will try to be run | 17:49 |
| deepakcs | mtreinish, ok, didn't see ur inline comment, my bad, could you suggest the right regex pls ? not clear to me.. idempotent id instead of test id ? | 17:50 |
| *** mikeoutland has joined #openstack-dev | 17:50 | |
| deepakcs | mtreinish, ok thats good to know. But i believe bharat did use the regex in the patch and it worked is what i was told | 17:50 |
| deepakcs | mtreinish, ah ok, IIUC u r saying we need to skip the test we intend to skip + the slow tests as well ? | 17:51 |
| deepakcs | mtreinish, the current regex doesn't do the latter , right ? | 17:51 |
| mtreinish | deepakcs: yes | 17:51 |
| *** packet has joined #openstack-dev | 17:51 | |
| *** maishsk is now known as maishsk_afk | 17:52 | |
| deepakcs | mtreinish, if possible pls suggest the right regex :) would help us save addnl iterations | 17:52 |
| mtreinish | the second half was just a suggestion you can use: '557cd2c2-4eb8-4dce-98be-f86765ff311b' as the exculde for all test_volume_boot_pattern tests instead of tempest.scenario* | 17:52 |
| *** ZZelle has quit IRC | 17:53 | |
| *** ZZelle has joined #openstack-dev | 17:53 | |
| *** mikeoutland has quit IRC | 17:54 | |
| deepakcs | mtreinish, and would there be an id to skipp all the slow tests as well ? pardon my ignorace, haven't gotten deep into tempest yet | 17:54 |
| *** ajayaa has quit IRC | 17:55 | |
| mtreinish | deepakcs: no, slow is an attr which tags the tests which gets put on tests which would take too long in the gate | 17:55 |
| mtreinish | while that uuid is a tracking id per: http://specs.openstack.org/openstack/qa-specs/specs/meta-data-and-uuid-for-tests.html | 17:56 |
| mtreinish | the attrs are appended to the test id so you can use them as part of the regex for matching or excluding | 17:56 |
| deepakcs | mtreinish, Ok, thanks for your time. Ignore the mail I sent (with the hope that ianw agrees to your comment :) ) Good day. We will send the next patchset in the series soon | 17:57 |
| *** pixelbeat has joined #openstack-dev | 17:57 | |
| *** tiswanso has quit IRC | 17:57 | |
| *** bpokorny_ has joined #openstack-dev | 17:58 | |
| *** xingchao has quit IRC | 17:59 | |
| *** deepakcs has quit IRC | 17:59 | |
| *** kgriffs is now known as kgriffs|afk | 18:00 | |
| *** bknudson has joined #openstack-dev | 18:00 | |
| *** baoli has quit IRC | 18:00 | |
| *** markmc has joined #openstack-dev | 18:01 | |
| *** bpokorny has quit IRC | 18:01 | |
| *** baoli_ has joined #openstack-dev | 18:02 | |
| *** tiswanso_ has joined #openstack-dev | 18:03 | |
| *** bdemers has quit IRC | 18:04 | |
| *** yeylon has joined #openstack-dev | 18:08 | |
| *** gokrokve has joined #openstack-dev | 18:09 | |
| *** patrickeast_ has joined #openstack-dev | 18:09 | |
| jdandrea | Something's going on with zuul. I know it was bounced ... and a bunch of gate tests appear to be failing. Eek. | 18:09 |
| *** hdd has joined #openstack-dev | 18:10 | |
| *** patrickeast has quit IRC | 18:11 | |
| *** patrickeast_ is now known as patrickeast | 18:11 | |
| *** barra204 has joined #openstack-dev | 18:12 | |
| *** gokrokve has quit IRC | 18:12 | |
| *** packet has quit IRC | 18:14 | |
| *** otter768 has joined #openstack-dev | 18:15 | |
| *** penick has joined #openstack-dev | 18:18 | |
| *** Nikolay_St has joined #openstack-dev | 18:19 | |
| *** otter768 has quit IRC | 18:20 | |
| *** jtomasek has joined #openstack-dev | 18:21 | |
| *** kiran-r has joined #openstack-dev | 18:24 | |
| *** bpokorny has joined #openstack-dev | 18:24 | |
| *** gokrokve has joined #openstack-dev | 18:24 | |
| *** arnaud____ has joined #openstack-dev | 18:25 | |
| *** bharath has quit IRC | 18:27 | |
| *** barra204 has quit IRC | 18:27 | |
| *** oro has joined #openstack-dev | 18:27 | |
| *** bpokorny_ has quit IRC | 18:27 | |
| *** hdd has quit IRC | 18:28 | |
| *** logan2 has quit IRC | 18:28 | |
| *** gokrokve has quit IRC | 18:29 | |
| *** Ala has joined #openstack-dev | 18:29 | |
| *** natorious is now known as zz_natorious | 18:30 | |
| *** fawadkhaliq has quit IRC | 18:30 | |
| *** arnaud____ has quit IRC | 18:30 | |
| *** jdandrea has quit IRC | 18:30 | |
| *** logan2 has joined #openstack-dev | 18:31 | |
| *** shardy_ has joined #openstack-dev | 18:33 | |
| *** gokrokve has joined #openstack-dev | 18:34 | |
| *** shardy has quit IRC | 18:34 | |
| *** hdd has joined #openstack-dev | 18:35 | |
| *** Tross has joined #openstack-dev | 18:36 | |
| *** dasp has joined #openstack-dev | 18:37 | |
| *** Tross1 has quit IRC | 18:37 | |
| *** jobewan has joined #openstack-dev | 18:38 | |
| *** shardy_ has quit IRC | 18:38 | |
| *** packet has joined #openstack-dev | 18:39 | |
| *** shardy has joined #openstack-dev | 18:39 | |
| *** rlrevell has quit IRC | 18:40 | |
| *** Marga_ has quit IRC | 18:41 | |
| *** Marga_ has joined #openstack-dev | 18:42 | |
| *** barra204 has joined #openstack-dev | 18:42 | |
| *** fawadkhaliq has joined #openstack-dev | 18:43 | |
| *** jobewan has quit IRC | 18:44 | |
| *** dasp has left #openstack-dev | 18:45 | |
| *** jobewan has joined #openstack-dev | 18:45 | |
| *** egallen has quit IRC | 18:46 | |
| *** jobewan has quit IRC | 18:47 | |
| *** maishsk_afk has quit IRC | 18:48 | |
| *** jobewan has joined #openstack-dev | 18:48 | |
| *** maishsk has joined #openstack-dev | 18:49 | |
| *** jrist has joined #openstack-dev | 18:49 | |
| *** bharath has joined #openstack-dev | 18:49 | |
| *** zaitcev has joined #openstack-dev | 18:50 | |
| *** jdandrea has joined #openstack-dev | 18:52 | |
| *** tdruiva has quit IRC | 18:53 | |
| *** tnovacik has joined #openstack-dev | 18:53 | |
| *** tdruiva has joined #openstack-dev | 18:53 | |
| *** penick has quit IRC | 18:55 | |
| *** gfidente|afk is now known as gfidente | 18:56 | |
| *** trown|lunch is now known as trown | 18:57 | |
| *** tdruiva_ has joined #openstack-dev | 18:57 | |
| *** kgriffs|afk is now known as kgriffs | 18:58 | |
| *** annegentle has joined #openstack-dev | 18:59 | |
| *** penick has joined #openstack-dev | 19:00 | |
| *** bknudson has quit IRC | 19:00 | |
| *** jobewan has quit IRC | 19:00 | |
| *** mmedvede has joined #openstack-dev | 19:00 | |
| *** jobewan has joined #openstack-dev | 19:00 | |
| *** kiran-r has quit IRC | 19:00 | |
| *** lpetrut has joined #openstack-dev | 19:01 | |
| *** tdruiva has quit IRC | 19:01 | |
| *** gokrokve has quit IRC | 19:01 | |
| *** gokrokve has joined #openstack-dev | 19:01 | |
| *** jobewan has quit IRC | 19:02 | |
| *** Longgeek_ has quit IRC | 19:02 | |
| *** gfidente^2nd has joined #openstack-dev | 19:02 | |
| *** electrocucaracha has quit IRC | 19:02 | |
| *** rlrevell has joined #openstack-dev | 19:02 | |
| *** mmagr has joined #openstack-dev | 19:02 | |
| *** gfidente is now known as Guest23590 | 19:02 | |
| *** Guest23590 has quit IRC | 19:03 | |
| *** hdd has quit IRC | 19:04 | |
| *** Networkn3rd has joined #openstack-dev | 19:05 | |
| *** Networkn3rd has quit IRC | 19:06 | |
| *** baoli_ has quit IRC | 19:06 | |
| *** jistr has joined #openstack-dev | 19:08 | |
| *** lpetrut has left #openstack-dev | 19:09 | |
| *** ZZelle_ has joined #openstack-dev | 19:09 | |
| *** novas0x2a|laptop has joined #openstack-dev | 19:10 | |
| *** lodgenbd has quit IRC | 19:11 | |
| *** spandhe has joined #openstack-dev | 19:12 | |
| *** gfidente^2nd is now known as gfidente | 19:12 | |
| *** gfidente has quit IRC | 19:12 | |
| *** gfidente has joined #openstack-dev | 19:12 | |
| *** jobewan has joined #openstack-dev | 19:13 | |
| *** gyee has quit IRC | 19:13 | |
| *** tiswanso_ has quit IRC | 19:14 | |
| *** ayoung has quit IRC | 19:14 | |
| *** EmilienM is now known as EmilienM|afk | 19:15 | |
| *** gyee has joined #openstack-dev | 19:15 | |
| *** ijw has joined #openstack-dev | 19:17 | |
| *** jdennis has quit IRC | 19:18 | |
| *** baoli has joined #openstack-dev | 19:18 | |
| *** achampion has quit IRC | 19:22 | |
| *** kbrierly has joined #openstack-dev | 19:23 | |
| clayg | mriedem: i'm not sure patch 180276 addresses whatever is going on with lp bug #1451992 | 19:23 |
| openstack | Launchpad bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] https://launchpad.net/bugs/1451992 | 19:23 |
| mriedem | clayg: open to suggestions | 19:23 |
| mriedem | clayg: i didn't see any library updates that would cause htis | 19:24 |
| *** ssurana has joined #openstack-dev | 19:24 | |
| mriedem | logstash shows this exploding in the last 24 hours | 19:24 |
| mriedem | it's xattr related | 19:24 |
| clayg | mriedem: well if the node is blowing up on "import xattr" something with the installation build dependencies seems most likely? | 19:24 |
| mriedem | clayg: would https://review.openstack.org/#/c/180276/ be tickling that though? | 19:24 |
| clayg | mriedem: i don't think so? | 19:25 |
| mriedem | clayg: lifeless pointed this out http://morepypy.blogspot.ch/2015/05/cffi-10-beta-1.html | 19:25 |
| clayg | mriedem: we've had the dependecy on xattr a long time - and from the traceback the and bug you can see it's in the object server - which that middleware change didn't effect? | 19:25 |
| mriedem | in this case, cffi==0.9.2 | 19:25 |
| lifeless | yes, so - cffi has been a headache for ages, from just this sort of thing | 19:26 |
| clayg | seems worth looking into - i guess we'll need to try a clean build and see what version of cffi comes in when it tries to build/install xattr? | 19:26 |
| lifeless | note that the c file it references is in __pycache__ | 19:26 |
| clayg | it looks like the trigger should just be importing it | 19:26 |
| lifeless | 0.9.2 is the one we should be using for now | 19:26 |
| mriedem | clayg: there is this https://github.com/openstack/swift/commit/03536dbb55c219b94389e635babec2df2e2759fa - but that was merged a week ago | 19:26 |
| *** arnaud____ has joined #openstack-dev | 19:27 | |
| *** subscope_ has joined #openstack-dev | 19:27 | |
| *** tiswanso has joined #openstack-dev | 19:27 | |
| *** amakarov is now known as amakarov_away | 19:27 | |
| clayg | mriedem: also seems innocuous - os.fsync works with fileobject or fd - but the weird fctrl hack (freebsd?) seems to prefer the real fileobject.fileno() | 19:28 |
| clayg | still seems mostly unrelated to "import xattr" blowing up tho? | 19:28 |
| *** annegentle has quit IRC | 19:28 | |
| *** penick has quit IRC | 19:30 | |
| clayg | yeah idk, i just noted that I had python-xattr ubuntu package installed on my dev machine - when I uninstalled it and let pip bring in stuff from pypi it's totally reproducable for me :\ | 19:30 |
| clayg | cffi==0.9.2 xattr==0.7.5 | 19:31 |
| clayg | <0.7.5 seems to work | 19:32 |
| clayg | what's in global requirements? | 19:32 |
| mriedem | sdague: ^ | 19:32 |
| *** arnaud____ has quit IRC | 19:32 | |
| *** Networkn3rd has joined #openstack-dev | 19:32 | |
| mriedem | https://github.com/openstack/requirements/blob/master/global-requirements.txt#L171 | 19:32 |
| mriedem | https://github.com/openstack/requirements/blob/master/global-requirements.txt#L13 | 19:32 |
| mriedem | basically uncapped | 19:32 |
| clayg | well xattr 0.7.4 seems fine with cffi 0.9.2 - so I guess it's 0.7.5 - wonder when that came out? | 19:33 |
| sdague | time to fire up what-broke.py | 19:33 |
| *** ijw has quit IRC | 19:33 | |
| mriedem | sdague: none of those changed | 19:33 |
| mriedem | sdague: fungi was commenting in -qa | 19:33 |
| mriedem | xattr 0.7.5 was 3/23 | 19:34 |
| clayg | yeah everyhting looks oldish :P | 19:34 |
| *** penick has joined #openstack-dev | 19:34 | |
| fungi | yeah, sorry, i was troubleshooting it in infra and then moved to qa about an hour ago because we have infra firefights going on i'm juggling in the other channel | 19:34 |
| jogo | fungi: how many fires are you fighting right now? | 19:34 |
| *** Ala has quit IRC | 19:35 | |
| fungi | also, infra meeting for another half hour | 19:35 |
| *** Sukhdev has quit IRC | 19:35 | |
| mriedem | ha | 19:35 |
| mriedem | fungi: well bug is https://bugs.launchpad.net/swift/+bug/1451992 | 19:35 |
| mriedem | if you have notes | 19:35 |
| openstack | Launchpad bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] | 19:35 |
| *** Nikolay_St has quit IRC | 19:35 | |
| sdague | it also failed super early | 19:35 |
| sdague | so how did the patch merge? | 19:35 |
| *** doug-fish has quit IRC | 19:36 | |
| clayg | it wasn't a change in swift that caused the issue? | 19:36 |
| clayg | oh weird, after installing xattr 0.7.4, then cffi 0.92, then xattr 0.7.5 - now newest versions works - guess something got cached! | 19:37 |
| sdague | http://logs.openstack.org/28/178528/1/check/check-tempest-dsvm-full/dd77f9e/logs/pip-freeze.txt.gz - is the tests from the merge | 19:37 |
| sdague | so, any theories except cffi is a disaster? | 19:38 |
| *** zz_natorious is now known as natorious | 19:39 | |
| *** pradeep has quit IRC | 19:40 | |
| clayg | sdague: I think pinning xattr to 0.7.4 might work? | 19:41 |
| sdague | clayg: why? | 19:41 |
| sdague | because this all worked on an xattr 0.7.5 test run to land this code | 19:41 |
| clayg | cause it doesn't get a permission error in __pycache__ when i install it and then try to import it :P | 19:41 |
| clayg | sdague: fwiw i can get 0.7.5 to WOMM with cffi 0.9.2 - but only if I install xattr 0.7.4 first? | 19:42 |
| *** tdruiva_ has quit IRC | 19:42 | |
| *** emagana has quit IRC | 19:42 | |
| clayg | hrmm... well... that may not be 100% accurate :P | 19:42 |
| *** emagana has joined #openstack-dev | 19:42 | |
| *** tdruiva has joined #openstack-dev | 19:42 | |
| sdague | mriedem: are all your fails on rax nodes? | 19:43 |
| clayg | I'm probably disadvantaged by the fact that I have no idea how cffi works :D | 19:43 |
| sdague | the tests that passed are an hp cloud ndoe | 19:43 |
| jogo | sdague: isn't dist packages apt-get installed? | 19:43 |
| mtreinish | sdague: ugh, are you thinking another mirror issue? | 19:43 |
| *** xingchao has joined #openstack-dev | 19:43 | |
| mriedem | sdague: no | 19:43 |
| sdague | mtreinish: it's not a mirror isue | 19:43 |
| mriedem | hpcloud too | 19:43 |
| *** dasp has joined #openstack-dev | 19:43 | |
| sdague | cffi magically auto generates c bindings on the fly | 19:43 |
| sdague | it's ... super awesome developer magic, and complete freaking crazy pants for anyone trying to deploy things in a stable way | 19:44 |
| clayg | fresh install of ciff 0.9.1 and xattr 0.7.4 works - it something about combining the two | 19:47 |
| *** mestery has quit IRC | 19:48 | |
| clayg | also once you get them working you can upgrade just fine | 19:48 |
| *** vhoward has joined #openstack-dev | 19:48 | |
| jogo | we haven't used xattr 0.7.4 for at least 12 hours | 19:48 |
| *** electrocucaracha has joined #openstack-dev | 19:48 | |
| *** gfidente has quit IRC | 19:48 | |
| clayg | jogo: 0.7.5 was released like last year - it wouldn't surpise me if we've had 0.7.5 even longer than that? | 19:49 |
| *** jistr has quit IRC | 19:49 | |
| jogo | clayg: yeah I am only looking at past 12 hours | 19:49 |
| *** jprovazn has quit IRC | 19:49 | |
| *** markmc has quit IRC | 19:49 | |
| jogo | same for cffi 0.9.1 those didn't change | 19:49 |
| jogo | is this happening on rax ? | 19:50 |
| jogo | ahh yes it is :( | 19:50 |
| clayg | jogo: is there any chance looking back at those 12 hours that some of those machines had versions of cffi or xattr installed on them that were *upgraded* to those versions - or do you recycle the test nodes every 24 hours or something? | 19:50 |
| lifeless | clayg: installing 0.7.4 first suggests that its the building of the cache files 0 which won't be retrigged when you upgrade cffi - thats the issue | 19:50 |
| *** xingchao has quit IRC | 19:51 | |
| fungi | as i mentioned in #-qa, logstash says this started right around the time today's updated nodepool images would have just started coming into use | 19:52 |
| jogo | fungi: ahhhh that makes sense | 19:52 |
| * jogo checks apt-get logs | 19:52 | |
| fungi | so, yeah, it's possible there's a package update or something within the previous 24 hour timeframe which might be triggering this | 19:52 |
| *** promulo_ has quit IRC | 19:52 | |
| *** promulo_ has joined #openstack-dev | 19:53 | |
| sdague | fungi: yeh, that would be awesome. If we could diff package lists from yesterday to today | 19:53 |
| jogo | sdague: turns out we can | 19:54 |
| jogo | diff good bad: http://paste.openstack.org/show/215057 | 19:55 |
| *** Networkn_ has joined #openstack-dev | 19:55 | |
| jogo | hmm I should check if those are the both the same cloud | 19:55 |
| jogo | yup they are | 19:56 |
| *** Networkn3rd has quit IRC | 19:56 | |
| sdague | setuptools ? | 19:56 |
| sdague | from package? | 19:56 |
| *** Networkn3rd has joined #openstack-dev | 19:57 | |
| *** etoews has joined #openstack-dev | 19:57 | |
| clayg | wow, if I install cffi 0.9.2 *first* (not part of xattr depends) it works fine with xattr 0.7.5! (?) | 19:57 |
| sdague | yeh, this is all this magic caching stuff where order is important | 19:58 |
| sdague | we are also building things into wheels early, which I'm not sure how that impacts this or not | 19:58 |
| jogo | diff good bad for pip: http://paste.openstack.org/show/215059 | 19:59 |
| jogo | sdague: the new image doesn't have apt-get installed setuptools | 19:59 |
| *** maishsk has quit IRC | 19:59 | |
| *** nmagnezi has joined #openstack-dev | 19:59 | |
| *** maishsk has joined #openstack-dev | 20:00 | |
| *** Networkn_ has quit IRC | 20:00 | |
| *** Networkn_ has joined #openstack-dev | 20:01 | |
| *** Networkn3rd has quit IRC | 20:01 | |
| *** bharath_ has joined #openstack-dev | 20:02 | |
| *** bharath has quit IRC | 20:02 | |
| sigmavirus24 | jogo: requests 2.7.0 :thumbsup: | 20:02 |
| *** mriedem has quit IRC | 20:02 | |
| clayg | weird, when you run the xattr install without cffi installed - pip says "Installed /tmp/pip_build_root/xattr/cffi-1.0.0b1-py2.7-linux-x86_64.egg" <- which seems oldish? | 20:03 |
| sdague | that seems like a lot of changes for an overnight, fungi was there a new process here? | 20:03 |
| jogo | wow that is a bug jump in the requests version | 20:04 |
| *** bdemers has joined #openstack-dev | 20:04 | |
| clayg | idk, just upgraded pip and it works | 20:05 |
| *** jtomasek has quit IRC | 20:05 | |
| jogo | sdague: so that wasn't an overnight diff that was the weekend ... | 20:05 |
| *** annegentle has joined #openstack-dev | 20:05 | |
| fungi | sdague: it's possible... but given that we're seeing this in rackspace too, and we're using two separate build methods in hpcloud vs rackspace it's not likely related to any of the dib work | 20:05 |
| fungi | jogo: was this a rax or hpcloud log you diffed? | 20:06 |
| sdague | jogo: so that's because cloud-init was deleted, so python-requests was deleted | 20:06 |
| jogo | sdague: both rax | 20:06 |
| sdague | so then we got stuff from pip | 20:06 |
| *** doug-fish has joined #openstack-dev | 20:06 | |
| fungi | the churn there doesn't look too unusual for a day's worth of ubuntu updates | 20:07 |
| fungi | most of it seems to have been a kernel update | 20:07 |
| *** mmagr has quit IRC | 20:07 | |
| clayg | jogo: can you make the code that generates pip-freeze.txt.gz use pip list instead so you can see the version of pip installed? | 20:09 |
| *** tdruiva has quit IRC | 20:10 | |
| *** yeylon_ has joined #openstack-dev | 20:10 | |
| jogo | good is pip 6.1.1 | 20:10 |
| jogo | bad is as well | 20:10 |
| *** kgiusti has quit IRC | 20:10 | |
| *** mchestr has quit IRC | 20:11 | |
| *** achanda has quit IRC | 20:11 | |
| *** rushiagr_away is now known as rushiagr | 20:12 | |
| jogo | revised overnight diffs | 20:12 |
| jogo | http://paste.openstack.org/show/215061 | 20:12 |
| jogo | http://paste.openstack.org/show/215062 | 20:12 |
| *** yeylon has quit IRC | 20:12 | |
| jogo | good: http://logs.openstack.org/58/173658/1/check/check-tempest-dsvm-full/b052732/logs/ | 20:12 |
| jogo | bad: http://logs.openstack.org/88/110888/11/gate/gate-tempest-dsvm-full/f61c679/logs/devstacklog.txt.gz | 20:12 |
| *** ctina_ has joined #openstack-dev | 20:14 | |
| *** otter768 has joined #openstack-dev | 20:16 | |
| *** matrohon has joined #openstack-dev | 20:16 | |
| *** maishsk has quit IRC | 20:17 | |
| *** ctina has quit IRC | 20:18 | |
| jogo | so everything is failing how do we unwedge? | 20:18 |
| *** ctina_ has quit IRC | 20:19 | |
| fungi | do we know yet what the trigger was and what we should patch to fix it? | 20:19 |
| jogo | fungi: AFAIK no | 20:19 |
| fungi | then that's a prerequisite to even discussing working around the wedge to get the fix(es) merged | 20:20 |
| *** jckasper has quit IRC | 20:20 | |
| *** akrivoka has quit IRC | 20:20 | |
| fungi | i can delete today's devstack-trusty images in nodepool and fall back on yesterday's to help | 20:21 |
| fungi | but we're going to need to figure out the cause or we're likely to be right back to broken again tomorrow | 20:21 |
| jogo | don't know if that will help even | 20:21 |
| jogo | anyway we can test that out? | 20:21 |
| *** otter768 has quit IRC | 20:21 | |
| *** _gryf is now known as _gryf|afk | 20:22 | |
| fungi | with the current infra fires we're fighting with nodepool/jenkins i don't think we're going to have a lot of time to work out a way to test whether deleting the most recent images is going to help | 20:23 |
| fungi | the timing suggests that's when the bug first surfaced | 20:23 |
| clayg | is there any chance that we didn't always do a wheel install of xattr? | 20:23 |
| *** spandhe has quit IRC | 20:23 | |
| fungi | should be easy to see from a working log | 20:23 |
| *** maishsk has joined #openstack-dev | 20:23 | |
| *** matrohon has quit IRC | 20:23 | |
| *** jkremer has joined #openstack-dev | 20:24 | |
| jogo | fungi: ohh I see gerrit is backed up too :( | 20:24 |
| fungi | jogo: what's the reference known-good run log you're working from? | 20:24 |
| jogo | fungi: http://logs.openstack.org/58/173658/1/check/check-tempest-dsvm-full/b052732/logs/ | 20:24 |
| jogo | clayg: in the good run above ^ the wheel stuff looked the same to me | 20:24 |
| *** mwang2 has joined #openstack-dev | 20:25 | |
| fungi | "Running setup.py bdist_wheel for xattr" | 20:25 |
| fungi | yeah | 20:25 |
| clayg | jogo yeah it looks exactly the same | 20:26 |
| jogo | clayg: do swift unit tests work? | 20:26 |
| clayg | jogo: yeah of course | 20:26 |
| jogo | clayg: as in does this bug hit them or not | 20:26 |
| clayg | jogo: as long as import xattr works :) | 20:27 |
| jogo | clayg: gotcha | 20:27 |
| jogo | clayg: is there even a 1% chance that https://review.openstack.org/#/c/180276/ would fix it? | 20:27 |
| *** abregman has joined #openstack-dev | 20:27 | |
| *** vladikr has quit IRC | 20:27 | |
| *** gyee has quit IRC | 20:28 | |
| clayg | jogo: i honestly can't see how - sorry :\ | 20:28 |
| jogo | clayg: yeah I see now, doesn't even import xatr | 20:28 |
| jogo | so I guess we are still back in square 1 | 20:28 |
| clayg | jogo: like I said - i'm not really having any issue reproducing the build/install failure with xattr 0.7.5 and cffi 0.9.2 | 20:28 |
| clayg | but I had a system with all kinda of old stuff - pip 1.5.4 | 20:29 |
| clayg | as I've upgraded things it keeps getting more and more reliable | 20:29 |
| fungi | not having any issue as in it's easy to reproduce for you? | 20:29 |
| clayg | fungi: yeah it was easy to repro | 20:30 |
| fungi | okay, that's what i thought you meant | 20:30 |
| jogo | fungi: if its easy to remove the latest image and the risk of doing so is low | 20:30 |
| jogo | fungi: maybe its worth doing even if we don't know if that will help or not for sure | 20:31 |
| *** achampion has joined #openstack-dev | 20:31 | |
| *** tdruiva has joined #openstack-dev | 20:32 | |
| fungi | yeah, that's my thinking as well. it seems like the cause for this is potentially systemic. that and the timing suggest it's a very likely short-term solution | 20:32 |
| fungi | and we did successfully run these jobs on nodes built from yesterday's images | 20:34 |
| *** tdruiva_ has joined #openstack-dev | 20:34 | |
| *** tiswanso has quit IRC | 20:34 | |
| clayg | lol - can you make the machines "sudo python -c 'import xattr'" before you start devstack :P | 20:34 |
| *** tiswanso has joined #openstack-dev | 20:35 | |
| *** achanda has joined #openstack-dev | 20:35 | |
| clayg | I wonder if the __pycache__ is supposed to be built in the wheel - or not until it gets installed? | 20:35 |
| jogo | fungi: so lets do it, that will at least buy us some time | 20:35 |
| *** miqui has quit IRC | 20:36 | |
| *** tdruiva has quit IRC | 20:37 | |
| jogo | clayg: isn't __pycache__ python3? | 20:37 |
| *** barra204 has quit IRC | 20:37 | |
| elmiko | anyone know if there is an oslo specific channel, or is this it? | 20:37 |
| jogo | elmiko: #openstack-oslo AFAIK | 20:38 |
| clayg | jogo: nah it's a cffi thing | 20:38 |
| elmiko | jogo: thanks! | 20:38 |
| jogo | clayg: oh | 20:38 |
| *** aix has quit IRC | 20:38 | |
| *** mehdi has quit IRC | 20:38 | |
| *** Guest15 has joined #openstack-dev | 20:39 | |
| *** Trozz has quit IRC | 20:39 | |
| *** rlrevell has quit IRC | 20:39 | |
| *** rlrevell1 has joined #openstack-dev | 20:39 | |
| *** Trozz has joined #openstack-dev | 20:39 | |
| *** natorious has quit IRC | 20:39 | |
| *** persia has quit IRC | 20:40 | |
| *** jehb has quit IRC | 20:40 | |
| *** rushiagr is now known as rushiagr_away | 20:40 | |
| *** mehdi has joined #openstack-dev | 20:40 | |
| *** d34dh0r53 has quit IRC | 20:41 | |
| *** fungi has quit IRC | 20:41 | |
| *** raorn has quit IRC | 20:41 | |
| *** Hien has quit IRC | 20:41 | |
| *** persia has joined #openstack-dev | 20:41 | |
| *** persia has quit IRC | 20:41 | |
| *** persia has joined #openstack-dev | 20:41 | |
| *** raorn has joined #openstack-dev | 20:42 | |
| *** rlrevell1 has quit IRC | 20:42 | |
| jogo | so its built the first time cffi is run? | 20:42 |
| *** mwagner_lap has quit IRC | 20:42 | |
| clayg | jogo: yeah the first time you import xattr if it's not already there | 20:43 |
| jogo | clayg: because maybe one of the pip changes is causing ordering of things to subtly change? | 20:43 |
| *** rlrevell has joined #openstack-dev | 20:43 | |
| clayg | jogo: yeah i don't know - when I install it from pypi i get __pycache__ during install I think, when I install from wheel I don't? hard to say... i keep boucning my pip version around | 20:43 |
| *** natorious has joined #openstack-dev | 20:44 | |
| *** d34dh0r53 has joined #openstack-dev | 20:45 | |
| *** amitgandhinz has quit IRC | 20:46 | |
| *** kacieee has quit IRC | 20:46 | |
| *** jehb has joined #openstack-dev | 20:46 | |
| *** fungi has joined #openstack-dev | 20:47 | |
| fungi | jogo: not sure how much you saw before freenode fell over | 20:48 |
| fungi | i deleted the newer images some minutes ago, so hopefully we start getting usable machines again shortly | 20:49 |
| jogo | fungi: ? | 20:49 |
| *** Hien has joined #openstack-dev | 20:49 | |
| jogo | fungi: cool | 20:49 |
| fungi | i need to disappear for a couple hours, but will return and check in on things | 20:49 |
| jogo | could this be related to the coud-init change? | 20:49 |
| *** raildo is now known as raildo_away | 20:50 | |
| *** Lee1092 has quit IRC | 20:51 | |
| *** rlrevell has quit IRC | 20:51 | |
| *** fawadkhaliq has quit IRC | 20:51 | |
| *** patrickeast has quit IRC | 20:52 | |
| *** aysyd has quit IRC | 20:52 | |
| *** patrickeast has joined #openstack-dev | 20:53 | |
| clarkb | what cloud init change? | 20:54 |
| jogo | how is xattr getting into dist-packages | 20:54 |
| jogo | clarkb: sdague said something in backlog | 20:54 |
| clarkb | xattr is a dep of swift iirc | 20:54 |
| *** gyee has joined #openstack-dev | 20:54 | |
| jogo | so that's because cloud-init was deleted, so python-requests was deleted | 20:54 |
| *** amitgandhinz has joined #openstack-dev | 20:54 | |
| clarkb | so that is on e way it gets in | 20:54 |
| jogo | is the quote | 20:54 |
| clarkb | uh I am unaware of deleting cloud init | 20:54 |
| jogo | clarkb: hmm kk | 20:55 |
| jogo | see https://bugs.launchpad.net/swift/+bug/1451992 | 20:55 |
| openstack | Launchpad bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] | 20:55 |
| sdague | clarkb: I thought that was the whole point of recent dib changes | 20:56 |
| clarkb | sdague none of that is in use | 20:57 |
| jogo | hmm everything goes to dist-packages in the gate | 20:57 |
| clarkb | jogo thats where pip install goes yes | 20:57 |
| sdague | clarkb: hmmm... how did we end up losing cloud-init in the image then? (though it's not really relevant to this problem I think) | 20:57 |
| clarkb | sdague I am pretty sure we havent rax and hpcloud both currently depend on it | 20:58 |
| clarkb | we wont boot correctly without it | 20:58 |
| *** frobware_ has joined #openstack-dev | 20:58 | |
| sdague | clarkb: hmmm... I guess I'm confused why it disappeared from the package list that jogo created then | 20:59 |
| *** spandhe has joined #openstack-dev | 20:59 | |
| jogo | sdague: it didn't change | 20:59 |
| sdague | oh, ok | 20:59 |
| jogo | http://paste.openstack.org/show/215061/ http://paste.openstack.org/show/215062/ | 20:59 |
| *** cference has quit IRC | 20:59 | |
| *** nelsnelson has quit IRC | 21:00 | |
| *** tdruiva_ is now known as tdruiva | 21:00 | |
| *** noslzzp has quit IRC | 21:00 | |
| *** bpokorny_ has joined #openstack-dev | 21:01 | |
| *** bpokorny has quit IRC | 21:01 | |
| clarkb | so if xattr is having trouble it may be possible something installs it at image build time then we break the C bindings for it when devstack updates stuff? | 21:01 |
| clarkb | I thought cffi was supposed to rebuild on every invocation to avoid that though | 21:02 |
| *** ybathia has quit IRC | 21:02 | |
| jogo | clarkb: possibly? | 21:02 |
| jogo | clarkb: that is as good a guess as any at this point | 21:02 |
| *** hdd has joined #openstack-dev | 21:02 | |
| clarkb | however I would expect old images to break if that was the case | 21:03 |
| clarkb | do we have data from old images today yet? | 21:03 |
| jogo | clarkb: soon I think, fungi deleted the new image | 21:03 |
| *** slupers has quit IRC | 21:04 | |
| clarkb | cool will keep an eye on that | 21:04 |
| *** slupers has joined #openstack-dev | 21:05 | |
| *** electrocucaracha has quit IRC | 21:05 | |
| *** Networkn_ has quit IRC | 21:06 | |
| *** maishsk has quit IRC | 21:06 | |
| *** radez is now known as radez_g0n3 | 21:06 | |
| sdague | clarkb: even if we wheel it? | 21:06 |
| greghaynes | clarkb: the traceback looks like its trying to do the runtime compilation | 21:06 |
| sdague | that's what I'm getting suspicious of | 21:06 |
| sdague | greghaynes: yeh, cffi does that | 21:07 |
| sdague | it's crazy pants :) | 21:07 |
| greghaynes | yep | 21:07 |
| *** electrocucaracha has joined #openstack-dev | 21:07 | |
| greghaynes | so that doesnt seem odd though | 21:07 |
| *** Networkn3rd has joined #openstack-dev | 21:07 | |
| clarkb | no I don't think its doing the compilation I think it assumes its done and in the cache then it fails for whatever reason | 21:08 |
| clarkb | sdague: ya because if the thing the wheel is bound against changes then you lose | 21:08 |
| *** hdd has quit IRC | 21:08 | |
| *** nmagnezi has quit IRC | 21:08 | |
| clarkb | sdague: so maybe its even an order of ops where we wheel then update system packages breaking C indings | 21:08 |
| sdague | which... doesn't look like it should be the case here | 21:08 |
| clarkb | but I think we do system first otherwise things can't link at all | 21:08 |
| sdague | yeh, so short term, we drop xattr out of wheel building? | 21:09 |
| clayg | when the wheel is built and .so is created and added to the wheel | 21:09 |
| sdague | anyway, I have to deal with dinner, I'll cycle around when my daughter is going to bed if there are patches I can shepard through | 21:09 |
| clarkb | clayg: yup, but if we then change the underlying thing that it linked against we can see this sort of trouble | 21:10 |
| clarkb | clayg: cffi supposedly rebuilds on every run for this reason | 21:10 |
| clayg | but cffi will sometimes think the "_cffi__x282d9483x6ee6f75.so" isn't valid for the c code the python module is feeding to the library (it's some sort of crc thing going on) and it will helpfully recomiple it for you - in __pycache__ | 21:10 |
| *** annegentle has quit IRC | 21:11 | |
| clarkb | oh I wonder if making the wheel doesn't install that file | 21:11 |
| * clarkb runs some local tests | 21:11 | |
| *** annegentle has joined #openstack-dev | 21:12 | |
| clayg | clarkb: it seems to drop the .so out with the rest of the libs | 21:12 |
| clayg | er... lib.py | 21:12 |
| clayg | in dist-packages/xattr | 21:12 |
| *** bswartz has quit IRC | 21:12 | |
| greghaynes | clarkb: reading docs - looks like cffi caches the bindings in __pycache__ and detects for changes | 21:13 |
| clayg | greghaynes: *nod* seems like it only needs __pycache__ *if* it detects a change - then it wants some scratch to rebuild the .so | 21:13 |
| *** amuller has quit IRC | 21:14 | |
| clarkb | I get no __pycache__ at all fwiw | 21:14 |
| clarkb | using an sdist install | 21:14 |
| greghaynes | yea, I think its only on change | 21:15 |
| greghaynes | so if you make a new def/veify run with different definitions | 21:15 |
| clarkb | ya but look at the stacktrace | 21:15 |
| clarkb | is opening that file for writing, the reason it fails is the dir doesn't exist | 21:16 |
| *** arnaud____ has joined #openstack-dev | 21:16 | |
| clarkb | if the file didn't exist it would just make it | 21:16 |
| clarkb | but if dir doesn't exist then you get that error | 21:16 |
| clayg | clarkb: idk, when i sudo python -c 'import xattr' in this state it will make the dir create the file and everything carries on like nothing happened | 21:17 |
| clarkb | when I do import xattr it does not create the dir/file for me | 21:17 |
| *** ybathia has joined #openstack-dev | 21:17 | |
| clayg | like maybe it will catch enoent and try to mkdir, then if that raises eperm it will re-raise or something - idk | 21:17 |
| clarkb | but it also doesn't explode so I may not be tickling the issue | 21:17 |
| clayg | clarkb: yeah you're cffi.so matches the installed code - so you're all good | 21:17 |
| *** kbrierly has quit IRC | 21:18 | |
| *** dkranz has quit IRC | 21:18 | |
| clayg | clarkb: this seems to happen that way for me most of the time if i install cffi first rather than with the xattr build | 21:18 |
| *** dprince has quit IRC | 21:18 | |
| *** xingchao has joined #openstack-dev | 21:18 | |
| *** kbrierly has joined #openstack-dev | 21:18 | |
| greghaynes | oh! are we using a different cffi version than what the wheel was made with? | 21:19 |
| jogo | greghaynes: ? | 21:19 |
| clayg | greghaynes: that's what i was thinking - but I don't see any evidence of the cffi version changing - it's 0.9.2 all the way down? unless it's 0.9.1 sometimes | 21:19 |
| clayg | doesn't the wheel get rebuilt every go? | 21:19 |
| greghaynes | kind of grasping at straws with that.. | 21:19 |
| greghaynes | unsure | 21:19 |
| jogo | we build the wheel in devstack | 21:19 |
| jogo | until greghaynes fixes that :) | 21:20 |
| *** claudiub has joined #openstack-dev | 21:20 | |
| lifeless | we could blacklist that wheel | 21:20 |
| clarkb | clayg: greghaynes it could be libffi that is changing | 21:20 |
| clarkb | which makes cffi invalid | 21:20 |
| greghaynes | yea, im thinking something down that path is very plausible | 21:20 |
| jogo | clarkb: doing a package diff didn't show any changes for libffi | 21:20 |
| clarkb | jogo: ok | 21:21 |
| jogo | clarkb: but that is only run at the end | 21:21 |
| *** arnaud____ has quit IRC | 21:21 | |
| *** tiswanso has quit IRC | 21:21 | |
| *** baoli has quit IRC | 21:23 | |
| jogo | https://review.openstack.org/180330 | 21:24 |
| jogo | will at least confirm or deny the wheel theory | 21:24 |
| *** baoli_ has joined #openstack-dev | 21:24 | |
| *** Nikolay_St has joined #openstack-dev | 21:24 | |
| jogo | any other theories we can test? | 21:26 |
| *** gyee has quit IRC | 21:27 | |
| clayg | jogo: would be nice to get a find /usr/local/lib/python2.7/dist-packages/xattr towards the end, and maybe 'sudo python -c "import xattr"' and another snapshot | 21:28 |
| *** daniel3_ has quit IRC | 21:28 | |
| *** boris-42 has quit IRC | 21:28 | |
| *** johnbelamaric has quit IRC | 21:28 | |
| jogo | clayg: want to push a devstack patch up to test that out? | 21:29 |
| *** danielbruno has quit IRC | 21:29 | |
| *** doug-fish has quit IRC | 21:30 | |
| clayg | jogo: oohhoohhhh - i *should* do that | 21:30 |
| *** alanf-mc has quit IRC | 21:31 | |
| jogo | clayg: well you have a clear idea of what it looks like in your head ;) | 21:31 |
| *** shardy is now known as shardy_z | 21:31 | |
| jogo | I can do it if you really want though | 21:31 |
| *** mgagne has left #openstack-dev | 21:31 | |
| *** alanf-mc has joined #openstack-dev | 21:31 | |
| *** mgagne has joined #openstack-dev | 21:31 | |
| *** Networkn3rd has quit IRC | 21:32 | |
| clayg | heh | 21:32 |
| clayg | which log was the *good* log? | 21:33 |
| *** dguerri is now known as _dguerri | 21:33 | |
| clayg | jogo: http://logs.openstack.org/58/173658/1/check/check-tempest-dsvm-full/b052732/logs/devstacklog.txt.gz i guess | 21:33 |
| *** _dguerri is now known as dguerri | 21:33 | |
| *** dustins has quit IRC | 21:33 | |
| jogo | clayg: yeah that is where output from devstack's stack.sh goes | 21:33 |
| *** trown is now known as trown|outttypeww | 21:34 | |
| greghaynes | do we have some kind of pip config to make logs quieter? http://logs.openstack.org/27/169827/10/check/check-tempest-dsvm-full/506c93a/logs/devstacklog.txt.gz#_2015-05-05_17_12_11_725 | 21:35 |
| greghaynes | that isnt what I get when I pip wheel xattr | 21:35 |
| *** topol has quit IRC | 21:36 | |
| greghaynes | I see the file were erroring on being made | 21:36 |
| *** Nikolay_St has quit IRC | 21:37 | |
| *** jbell8 has quit IRC | 21:37 | |
| clayg | greghaynes: it's quiet for me like that with pip 6.1.1 | 21:37 |
| greghaynes | aye, new pips | 21:38 |
| *** dguerri is now known as _dguerri | 21:38 | |
| *** _dguerri is now known as dguerri | 21:38 | |
| *** FunnyLookinHat has quit IRC | 21:39 | |
| clayg | god this crc thing is annoying! why isn't the cffi.so in the wheel the same as the one it thinks it wants after install? | 21:39 |
| greghaynes | ok, now im seeing the same | 21:39 |
| jogo | greghaynes: what about if you add a '-v' to pip wheel | 21:39 |
| greghaynes | I dont think its the cffi.so its checking, I think its a crc of the cffi definitions I think | 21:39 |
| greghaynes | jogo: one sec | 21:40 |
| *** bdemers has quit IRC | 21:40 | |
| clayg | greghaynes: the docs make it sound like it's the crc of the source you're feeding into cffi - then that gets encoded into the _cffi_<magichex>.so that goes in the wheel | 21:40 |
| greghaynes | yes | 21:41 |
| clayg | the auto build kicks in when it thinks the source getting feed into cffi doesn't match the <magichex> that's already built | 21:41 |
| clayg | if it's a match (and it should be, and most often *is*) then there's no __pycache__ needed - which is good - because most of the time trying to write into /usr/local/lib is going to blow up on you ;) | 21:41 |
| greghaynes | so its a matter of finding module_foo${crc(source)}.c | 21:42 |
| clarkb | clayg: well in a virtualenv it will write to a path relative to the virtualenv instead | 21:42 |
| greghaynes | jogo: yea, that shows me all the goodness | 21:42 |
| clayg | clarkb: so you're saying I was using "most of the time" too liberly? | 21:42 |
| clarkb | clayg: yes | 21:42 |
| clayg | heh | 21:43 |
| * clayg takes his this is supposed to work places besides dev hat off | 21:43 | |
| *** angdraug has joined #openstack-dev | 21:43 | |
| clarkb | well don't get me started on cffi, its now been the cause of ~3 issues in the last 2 weeks? its great fun | 21:43 |
| jogo | crm114 thinks this is the different logs: http://logstash.openstack.org/#eyJmaWVsZHMiOiBbXSwgInNlYXJjaCI6ICJidWlsZF91dWlkOiBcImUzNWRhYzFmOTA2NjRkMTVhMmI0YjFlNmVmZGZjYjk5XCIgQU5EIGVycm9yX3ByOltcIi0xMDAwLjBcIiBUTyBcIi0xMC4wXCJdICIsICJ0aW1lZnJhbWUiOiAiODY0MDAwIiwgImdyYXBobW9kZSI6ICJjb3VudCIsICJvZmZzZXQiOiAwfQ== | 21:44 |
| *** yeylon_ has quit IRC | 21:45 | |
| clarkb | when we build the wheel is cffi the same version? | 21:46 |
| *** slupers has quit IRC | 21:46 | |
| *** stevemar has joined #openstack-dev | 21:47 | |
| clarkb | I wonder if we have a stale cffi, build xattr, then upgrade cffi | 21:47 |
| *** SumitNaiksatam has quit IRC | 21:47 | |
| *** slupers has joined #openstack-dev | 21:47 | |
| greghaynes | yea, I looked, its 0.9.2 everywhere | 21:47 |
| greghaynes | also tested that case and didnt repro | 21:47 |
| clayg | clarkb: i get a different crc in the whl for xattr when i wheel xattr with cffi installed vs. wheel xattr with no cffi installed | 21:47 |
| greghaynes | waaa | 21:48 |
| clarkb | interesting | 21:48 |
| jogo | I may have it | 21:48 |
| jogo | we now install pyOpenSSL-0.15.1 for keystone | 21:48 |
| jogo | which uses cffi | 21:48 |
| *** doug-fish has joined #openstack-dev | 21:48 | |
| jogo | or I think that is what CRM114 is telling me | 21:48 |
| clarkb | jogo: I thought we were forcing pyopenssl to be old because the cffi version was OOMing | 21:49 |
| clayg | jogo: neat! what version! | 21:49 |
| clarkb | jogo: but if that is getting installed it could be the case | 21:49 |
| clarkb | jogo: also cool that crm114 points that out | 21:49 |
| jogo | pyOpenSSL-0.15.1 | 21:49 |
| jogo | clarkb: very cool yeah | 21:49 |
| *** Guest15 has quit IRC | 21:50 | |
| clarkb | jogo: so maybe if we do a wheel of cffi, then wheel of the other two we avoid this? | 21:50 |
| jogo | so I lied | 21:50 |
| jogo | wrong way | 21:50 |
| jogo | we dropped pyopenssl | 21:50 |
| jogo | we used to install it now we don't | 21:50 |
| greghaynes | we do pre-build cffi because we make a wheel for cryptography | 21:51 |
| clayg | so we lost cffi, so the wheel building is happening w/o cffi installed - which is where I see the broken <magiccrc> that results in needing a rebuilt | 21:51 |
| greghaynes | but other than that it seems normal | 21:51 |
| clarkb | right ok, that lines up with what I had thought, the reason we stopped was they switched to cryptography and it exploded memory usage and led to OOMing | 21:51 |
| *** mriedem has joined #openstack-dev | 21:51 | |
| greghaynes | clayg: we lost cffi? | 21:51 |
| clayg | greghaynes: i understood what jogo was saying about "other way" meaning we *used* to have cffi installed with packages depends - then we didn't | 21:52 |
| greghaynes | clayg: I am seeing the same thing as you about the crc's being different in wheel vs install then import | 21:53 |
| clarkb | ya, unless something else pulls it in I wouldn't be suprised if its skipped | 21:53 |
| clarkb | since the pyopenssl we are using should be pre cffi usage | 21:54 |
| *** pcm_ has quit IRC | 21:54 | |
| clarkb | greghaynes: clayg what are the reproduction steps? | 21:54 |
| jogo | clayg: a few days ago pyopenssl was installed with keystone | 21:54 |
| jogo | now it is not | 21:54 |
| greghaynes | I dont get a traceback because when I import xattr it just recompiles as it should | 21:54 |
| jogo | err 12 hours ago or so | 21:54 |
| clayg | greghaynes: pip 6.1.1 no cffi or xattr installed - wheel xattr - install cffi.whl install xattr.whl - import xattr | 21:54 |
| *** igordcard has joined #openstack-dev | 21:54 | |
| greghaynes | I just am looking at pip wheel -v xattr and then pip install xattr;import xattr and its a different crc | 21:55 |
| greghaynes | could be a red herring though | 21:55 |
| clayg | greghaynes: sorry, ^ clarkb | 21:55 |
| clarkb | thanks | 21:55 |
| *** ptoohill is now known as pothole | 21:55 | |
| *** annegent_ has joined #openstack-dev | 21:55 | |
| *** TrevorV is now known as trevhole | 21:56 | |
| greghaynes | so yea, thats the issue actually | 21:57 |
| greghaynes | it doesnt have perms to write the file? | 21:57 |
| *** cdent has quit IRC | 21:57 | |
| greghaynes | either that, or their ensure_dir method is broken | 21:57 |
| greghaynes | I bet both | 21:57 |
| clarkb | oh! I see what you mean about writing to /usr/ | 21:58 |
| greghaynes | yes | 21:58 |
| clarkb | because it happens at run time as a different user | 21:58 |
| clarkb | ARG | 21:58 |
| clayg | greghaynes: I don't think it's supposed to be trying to compile it | 21:58 |
| greghaynes | yes | 21:58 |
| greghaynes | theres basically 3 bugs | 21:58 |
| clarkb | so a I think that entire cache system is bonghits | 21:58 |
| *** asettle has joined #openstack-dev | 21:58 | |
| greghaynes | it shouldnt recompile | 21:58 |
| greghaynes | it should fail when it tries to ensure the dir | 21:58 |
| *** asettle has quit IRC | 21:58 | |
| clayg | greghaynes: the .so is in the wheel - if the crc would match it wouldn't do that and it would need these perms | 21:58 |
| *** asettle has joined #openstack-dev | 21:58 | |
| *** SumitNaiksatam has joined #openstack-dev | 21:58 | |
| greghaynes | clayg: yes | 21:58 |
| *** _neoXsys_ has quit IRC | 21:58 | |
| greghaynes | I am vehemently agreeing :) | 21:59 |
| clarkb | clayg: greghaynes except that the crc may not match at some point later in the future when you update libffi right? | 21:59 |
| clarkb | and you would get the same sort of failure | 21:59 |
| greghaynes | clarkb: In theory, yes, in practice im not sure - they say they crc the definitions | 21:59 |
| clarkb | in that case its a different failure mode for what I describe | 21:59 |
| greghaynes | yes | 21:59 |
| clarkb | your bindings just won't work | 21:59 |
| clayg | eh | 21:59 |
| clarkb | (potentially) | 21:59 |
| greghaynes | yea, im not sure whether they are smarter than their docs say or not ;) | 22:00 |
| *** annegentle has quit IRC | 22:00 | |
| greghaynes | anyhow, a fix is to sudo python -c "import xattr" ? | 22:00 |
| clayg | ^ would totally work | 22:00 |
| clarkb | yes as that should have perms to update the file thats such bonghits | 22:00 |
| greghaynes | yep | 22:00 |
| greghaynes | welcome to the jungle | 22:01 |
| jogo | but what broke? | 22:01 |
| clarkb | jogo: cffi | 22:01 |
| clarkb | their cache model is derpy | 22:01 |
| *** penick has quit IRC | 22:01 | |
| greghaynes | jogo: theres a bug where we need to rebuild bindings, unsure why | 22:01 |
| greghaynes | but because its a system package they are being built somewhere that requires root | 22:02 |
| jogo | clarkb: cffi hasn't changed in a long while | 22:02 |
| greghaynes | but we run as !root | 22:02 |
| greghaynes | oh, youre asking what changed | 22:02 |
| jogo | greghaynes: yeah how did we go from working to not working? | 22:02 |
| greghaynes | I wonder if its pip | 22:02 |
| clarkb | possibly wheel or pip | 22:02 |
| clarkb | since new pip auto wheels now iirc | 22:02 |
| clarkb | though maybe that isn't released yet | 22:02 |
| *** pixelbeat has quit IRC | 22:02 | |
| clarkb | greghaynes: but ya I agree its something in the build/install toolchain likely | 22:03 |
| *** Swami has quit IRC | 22:03 | |
| *** SumitNaiksatam has quit IRC | 22:03 | |
| *** oro has quit IRC | 22:03 | |
| *** changbl has quit IRC | 22:04 | |
| jogo | pip and wheel haven't changed recently | 22:05 |
| clarkb | setuptools? | 22:05 |
| *** bnemec has quit IRC | 22:05 | |
| jogo | clarkb: possibly we don't use setuptools from ubuntu anymore | 22:06 |
| jogo | http://paste.openstack.org/show/215061/ | 22:06 |
| clayg | i like the idea of cffi being installed on the system during wheel building for good times, and cffi not being installed for bad times | 22:06 |
| *** penick has joined #openstack-dev | 22:06 | |
| clayg | doesn't even seem to matter if it was the same version - just having in the system is somehow making the crc good - where not having it making it - what's the word? bonghits? | 22:07 |
| jogo | setuptools-15.2 is what we use now | 22:07 |
| jogo | before that 3.3-1ubuntu1 was installed | 22:07 |
| *** bnemec has joined #openstack-dev | 22:08 | |
| greghaynes | jogo: any idea where a good place to put this sudo python -c import xattr is in devstack | 22:08 |
| * greghaynes needs to learn some devstack fu | 22:09 | |
| jogo | greghaynes: right after we install it? | 22:09 |
| greghaynes | thats the stack_install_service swift right? | 22:09 |
| jogo | greghaynes: maybe install it early in in devstack | 22:10 |
| jogo | after we build the wheel or something | 22:10 |
| jogo | just in case other things use this | 22:10 |
| jogo | we can always shuffle it later | 22:10 |
| lifeless | does xattr install-require cffi ? | 22:10 |
| *** FunnyLookinHat has joined #openstack-dev | 22:10 | |
| *** thrash is now known as thrash|g0ne | 22:11 | |
| *** bpokorny has joined #openstack-dev | 22:11 | |
| *** packet has quit IRC | 22:12 | |
| jogo | clarkb: where is setuptools installed? | 22:12 |
| jogo | it seems to be before devstack starts | 22:12 |
| *** amitgandhinz has quit IRC | 22:12 | |
| clarkb | jogo: its installed by devstack right after intsalling pip | 22:13 |
| jogo | clarkb: that doesn't do anything though http://logs.openstack.org/27/169827/10/check/check-tempest-dsvm-full/506c93a/logs/devstacklog.txt.gz#_2015-05-05_17_11_10_776 | 22:13 |
| clarkb | jogo: then likely image build | 22:14 |
| *** bpokorny_ has quit IRC | 22:14 | |
| jogo | clarkb: ahh yup | 22:14 |
| jogo | http://nodepool.openstack.org/hpcloud-b1.bare-trusty.log | 22:14 |
| jogo | so before we had pip installed setuptools and apt-get installed | 22:17 |
| jogo | I assume apt-get overridded | 22:17 |
| clarkb | lifeless: yes https://github.com/xattr/xattr/blob/master/setup.py#L63 | 22:17 |
| *** otter768 has joined #openstack-dev | 22:17 | |
| *** kbringard has quit IRC | 22:19 | |
| *** electrocucaracha has quit IRC | 22:19 | |
| *** jdennis has joined #openstack-dev | 22:21 | |
| jogo | clarkb: when was the latest image deleted? | 22:21 |
| clarkb | jogo: whenever fungi said he did it | 22:21 |
| jogo | and when can we expect the old image to start kicking in | 22:21 |
| jogo | 20:49 UTC | 22:22 |
| *** galstrom is now known as galstrom_zzz | 22:22 | |
| clarkb | jogo: the oldimage kicks in immediately for any new image builds | 22:22 |
| clarkb | er node builds | 22:22 |
| *** otter768 has quit IRC | 22:22 | |
| jogo | clarkb: so https://jenkins04.openstack.org/job/check-tempest-dsvm-full/100/console | 22:22 |
| clayg | wow, when I just python setup.py bdist_wheel from a xattr checkout i get *both* _cffi_<magichash>.so in the .whl | 22:22 |
| *** slupers has quit IRC | 22:23 | |
| greghaynes | hah | 22:23 |
| *** slupers has joined #openstack-dev | 22:23 | |
| *** johnbelamaric has joined #openstack-dev | 22:23 | |
| *** stevemar has quit IRC | 22:24 | |
| jogo | clarkb: so based on the kernel there http://logs.openstack.org/77/180177/1/check/check-tempest-dsvm-full/e1fe71d//logs/dpkg-l.txt.gz that is the old image | 22:25 |
| clayg | oh k | 22:25 |
| jogo | hmm but python-setuptools isn't there | 22:25 |
| jogo | either way that job failed and it started well after 20:49 | 22:25 |
| clayg | so it's one of those tricks where you get a different version of something depending on if it's setuptools/setup.py doing the installing or if it's pip | 22:25 |
| clarkb | jogo: it won't be python-setuptools if pip says its up to date when pip installs setuptools | 22:26 |
| clayg | the only reason I was getting both was because I wasn't cleaning up my build dir between bdist_wheel's | 22:26 |
| clayg | pip wheel -v doesn't show you the output of the bdist_wheel part - it just says running command | 22:26 |
| lifeless | clayg: well, no - you're getting both because theres a bug :) | 22:26 |
| *** ramishra_ has joined #openstack-dev | 22:26 | |
| jogo | clarkb: is there a good way to check the image build time? | 22:26 |
| clayg | lifeless: well maybe it's making two different magic hashes because it's from two different versions of cffi | 22:26 |
| clarkb | jogo: for a slave? I can check nodepool logs | 22:26 |
| *** haypo has joined #openstack-dev | 22:26 | |
| jogo | haypo: we cannot figure out what the change that caused the issue is | 22:27 |
| *** anthonyper_ has joined #openstack-dev | 22:27 | |
| jogo | clarkb: http://logs.openstack.org/77/180177/1/check/check-tempest-dsvm-full/e1fe71d//console.html | 22:27 |
| clayg | the _cffi__x282d9483x6ee6f75 hash you get form 0.9 - well basically anything - the _cffi__x2f484baxaad6dc56 came from like cffi 1. something | 22:27 |
| jogo | how do I check if that is the new image or old? | 22:27 |
| *** jcooley_ has joined #openstack-dev | 22:27 | |
| *** achampio1 has joined #openstack-dev | 22:27 | |
| *** baoli_ has quit IRC | 22:27 | |
| *** cgoncalv1s has joined #openstack-dev | 22:28 | |
| clarkb | jogo: you probably can't (get keystoneclient to log cleanly and you can though) | 22:28 |
| *** jgrimm_ has joined #openstack-dev | 22:28 | |
| clarkb | jogo: I hae to check nodepool logs to see what image that node booted off of | 22:28 |
| clayg | oh... no it's not old - it's new! | 22:28 |
| haypo | jogo, if you need a quickfix, we told me to use >sudo python -c "import xattr"< on #pypy (IRC channel used for cffi dev) | 22:28 |
| *** Ephur has quit IRC | 22:28 | |
| clayg | 0.9.2 vs 1.0.0b1 | 22:28 |
| greghaynes | haypo: https://review.openstack.org/#/c/180341/1 :) | 22:28 |
| haypo | (temporary workaround) | 22:28 |
| clayg | is that like a release canditate or something? | 22:28 |
| jogo | haypo: so what changed that triggered this? | 22:29 |
| *** electrocucaracha has joined #openstack-dev | 22:29 | |
| *** doude_ has joined #openstack-dev | 22:29 | |
| *** sweston_ has joined #openstack-dev | 22:29 | |
| clarkb | clayg: according to pep440 that is a pre release version so ya | 22:29 |
| *** rbradfor has quit IRC | 22:29 | |
| haypo | jogo, i'm investing the issue, i don't know what changed devstack behaviour | 22:29 |
| lifeless | clayg: we shouldn't have 1.0.0.b1 installed though, unless something is explictly bringing that in | 22:29 |
| *** darvon_ has joined #openstack-dev | 22:29 | |
| jogo | lifeless: pip freeze shows its not there | 22:29 |
| clayg | lifeless: yeah bdist_wheel is doing it because it's install_requires | 22:29 |
| clayg | you got it ;) | 22:29 |
| *** ramishra has quit IRC | 22:29 | |
| *** anthonyper has quit IRC | 22:30 | |
| *** jcooley has quit IRC | 22:30 | |
| *** achampion has quit IRC | 22:30 | |
| *** jgrimm has quit IRC | 22:30 | |
| *** cgoncalves has quit IRC | 22:30 | |
| *** darvon has quit IRC | 22:30 | |
| *** tteggel has quit IRC | 22:30 | |
| *** rwsu_ has quit IRC | 22:30 | |
| *** kbrierly has quit IRC | 22:30 | |
| *** doude has quit IRC | 22:30 | |
| *** sweston has quit IRC | 22:30 | |
| *** morgabra has quit IRC | 22:30 | |
| *** ndipanov has quit IRC | 22:30 | |
| *** kbrierly has joined #openstack-dev | 22:30 | |
| *** ndipanov has joined #openstack-dev | 22:30 | |
| *** dmellado has quit IRC | 22:30 | |
| haypo | jogo, it's maybe an upgrade of an external library like wheel | 22:30 |
| *** jcooley_ is now known as jcooley | 22:30 | |
| *** anthonyper_ is now known as anthonyper | 22:30 | |
| *** morgabra has joined #openstack-dev | 22:30 | |
| *** darvon_ is now known as darvon | 22:30 | |
| *** electrocucaracha has quit IRC | 22:30 | |
| *** tteggel has joined #openstack-dev | 22:30 | |
| jogo | haypo: wheel hasn't changed recently | 22:30 |
| *** ramishra_ is now known as ramishra | 22:30 | |
| greghaynes | yea, we suspected wheel or pip but wasnt either | 22:30 |
| haypo | jogo, i just gave an example ;) | 22:30 |
| clarkb | jogo: that node was created at 2015-05-05 21:53:31,748 so should be new image | 22:30 |
| clayg | jogo: when did 1.0.0b1 get published | 22:30 |
| jogo | haypo: diff from good and bad https://bugs.launchpad.net/swift/+bug/1451992/comments/4 | 22:30 |
| openstack | Launchpad bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] | 22:30 |
| greghaynes | jogo: actually, did you check what-broke.py? | 22:30 |
| greghaynes | ;) | 22:30 |
| *** rwsu_ has joined #openstack-dev | 22:30 | |
| clayg | 2015-05-05 | 22:30 |
| jogo | greghaynes: yes nothing stood out | 22:31 |
| greghaynes | aw | 22:31 |
| clayg | bdist_wheel is so stupid | 22:31 |
| *** electrocucaracha has joined #openstack-dev | 22:31 | |
| *** dmellado has joined #openstack-dev | 22:31 | |
| *** sweston_ is now known as sweston | 22:31 | |
| lifeless | clarkb: since mordred seems awol - can you spare a +2 for https://review.openstack.org/#/c/180265/ ? | 22:31 |
| clayg | pooor setup.py doesn't know that's not a for-realzy version yet | 22:31 |
| jogo | clarkb: that timestamp is after fungi deleted the new image though right? | 22:31 |
| *** unicell1 has joined #openstack-dev | 22:31 | |
| *** unicell has quit IRC | 22:31 | |
| mordred | lifeless: what did I do? | 22:31 |
| lifeless | mordred: nothing | 22:32 |
| mordred | lifeless: sweet | 22:32 |
| jogo | clayg: are you saying something is actually reading the 1.0.0.b1 version? | 22:32 |
| *** johnbelamaric has quit IRC | 22:32 | |
| *** johnbelamaric has joined #openstack-dev | 22:32 | |
| lifeless | mordred: [I pinged you ~3 hrs back about a critical fix for pbr, breaks egg_info on nova kilo | 22:32 |
| lifeless | mordred: so I wanted you to do something :) | 22:32 |
| clarkb | lifeless: is the problem that they somehow merged non utf8? | 22:33 |
| clarkb | because if its pre merge I say -1 | 22:33 |
| lifeless | clarkb: see the bug | 22:33 |
| lifeless | clarkb: [yes] | 22:33 |
| lifeless | clarkb: we can't control what history someone has, particular existing projects that might start using pbr | 22:34 |
| *** subscope_ has quit IRC | 22:34 | |
| haypo | greghaynes, +1 https://review.openstack.org/#/c/180341/ | 22:34 |
| clarkb | lifeless: the bug doesn't indicate if that is part of merged and accepted history | 22:34 |
| clarkb | lifeless: only that things fail when in the changeog which I think is fine for unmerged history | 22:35 |
| lifeless | clarkb: it doesn't call it out, but its referring to a revision in stable/kilo which implies that yes, its merged and accepted history | 22:35 |
| clarkb | which is weird because it should've failed and not merged I think | 22:35 |
| * clarkb goes digging | 22:35 | |
| lifeless | no | 22:35 |
| lifeless | older pbr didn't process commit messages | 22:36 |
| lifeless | only oneline summaries | 22:36 |
| clarkb | ah | 22:36 |
| clarkb | ok | 22:36 |
| *** Sukhdev has joined #openstack-dev | 22:36 | |
| lifeless | even if we had | 22:36 |
| clayg | jogo: yeah bdist_wheel does | 22:36 |
| clayg | jogo: it doesn't know better | 22:36 |
| clayg | setup.py is stupid :) | 22:36 |
| lifeless | we don't control CI for things outside of openstack | 22:36 |
| lifeless | but affiliated folk may well want to use pbr | 22:36 |
| jogo | clayg: wow | 22:36 |
| greghaynes | jogo: clarkb clayg Another question in addition to "what changed" is "why dont we hit the same bug for cryptography?" | 22:36 |
| clarkb | lifeless: sure, but why would you not use utf8? | 22:36 |
| jogo | clayg: epic fail, so that is what changed | 22:36 |
| clayg | if you have cffi 0.9.2 *installed* when you bdist_wheel it works - now i know why | 22:36 |
| lifeless | clarkb: I dunno, ask vishy | 22:36 |
| greghaynes | oh? | 22:36 |
| clayg | greghaynes: maybe cffi is already installed by the time the wheel is built? | 22:37 |
| lifeless | clarkb: its a poor idea for a tool in pbr's position to break because someone doesn't though. | 22:37 |
| clarkb | looks like mordred approved before I could so we should be set | 22:37 |
| *** btulchinsky has joined #openstack-dev | 22:37 | |
| lifeless | cool | 22:37 |
| lifeless | so if we want to lint and prevent non-utf8 in commits | 22:37 |
| clarkb | lifeless: I dunno, commit messages should always be readable by humans, utf8 is how you do that | 22:37 |
| lifeless | we should do that as a gate check | 22:37 |
| lifeless | and I support doing that | 22:37 |
| lifeless | pbr is just the wrong tool for it | 22:38 |
| greghaynes | clayg: I dont think so, looks like we build the cffi wheel immediately before xattr | 22:38 |
| greghaynes | er, sorry | 22:38 |
| clarkb | lifeless: sure, I was just trying to figure out how we threaded that through | 22:38 |
| lifeless | clarkb: ack | 22:38 |
| greghaynes | clayg: the cryptography wheel immediately before xattr | 22:38 |
| *** ngoracke has quit IRC | 22:39 | |
| jogo | greghaynes: can you test out your change locally? | 22:39 |
| greghaynes | I dont have a devstack deal setup anywhere, no | 22:40 |
| greghaynes | jogo: is the infra CI still not moving or something? | 22:41 |
| clarkb | it is still pretty sad | 22:41 |
| jogo | greghaynes: and this is blocking everything as well | 22:42 |
| jogo | so time is of the essence | 22:42 |
| greghaynes | splendid | 22:42 |
| clarkb | jogo: not really, everything is still likely failing on the other stuff too | 22:42 |
| jogo | http://jogo.github.io/gate/ | 22:42 |
| clarkb | so while fixing it would be nice, doing so before the other fixes isn't really going to help | 22:42 |
| jogo | clarkb: what else is stuck? | 22:42 |
| clarkb | jogo: jenkins jobs | 22:42 |
| clarkb | for all the reasons, we have fixed 1/3 so far | 22:43 |
| jogo | ahh | 22:43 |
| greghaynes | well, let me see if my hpcloud quota bump actually went through... maybe I can make a node to run this on | 22:43 |
| *** salv-orlando has quit IRC | 22:43 | |
| *** penick has quit IRC | 22:45 | |
| clarkb | also we may consider not making a wheel for these items (this certainly makes wheels far less attractive as a tool) | 22:46 |
| clarkb | if you consider that wheels are handy to reduce compile time but you can't use them with package that rely on compiled bindings... well its time to have a sad | 22:46 |
| lifeless | you should be able to | 22:47 |
| lifeless | but this is why pip 7 has a blacklist facility | 22:47 |
| lifeless | I haven't followed all the detail, is there a summary etherpad or something I can look at? Or can someone give me the low-down? | 22:47 |
| clarkb | lifeless: https://bugs.launchpad.net/swift/+bug/1451992/comments/7 | 22:47 |
| openstack | Launchpad bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] | 22:47 |
| jogo | greghaynes: ok trying your patch | 22:48 |
| *** egafford has quit IRC | 22:48 | |
| greghaynes | jogo: ok | 22:48 |
| jogo | greghaynes: you should get a devstack env :) | 22:49 |
| *** alex_xu has quit IRC | 22:49 | |
| greghaynes | heh | 22:49 |
| greghaynes | I still dev in a tripleo-like env | 22:50 |
| lifeless | ok so its just Putting the compile options into the package code instead of the build script (setup.py) means that developers have to invent their own method of shuttling those configs into the package at install time. | 22:50 |
| lifeless | I'm pretty sure i've run into another problem but I can't think of it off the top of my head. | 22:50 |
| lifeless | bah | 22:50 |
| lifeless | https://bitbucket.org/cffi/cffi/issue/109/enable-sane-packaging-for-cffi | 22:50 |
| *** jerematic has quit IRC | 22:50 | |
| *** alex_xu has joined #openstack-dev | 22:51 | |
| *** jerematic has joined #openstack-dev | 22:51 | |
| *** llu_office has quit IRC | 22:51 | |
| lifeless | clarkb: ok so - compiled bindings work fine in wheels | 22:51 |
| clarkb | lifeless: yes I know that | 22:51 |
| lifeless | cool | 22:51 |
| clarkb | lifeless: I am saying cffi in particular does not seem to work because of cffi things | 22:51 |
| lifeless | I read your comment earlier as being broader than you perhaps meant | 22:51 |
| *** EmilienM|afk is now known as EmilienM | 22:52 | |
| greghaynes | if cffi just didnt work here cryptography should be exploding in the same way? | 22:52 |
| *** llu_office has joined #openstack-dev | 22:52 | |
| lifeless | cffi is just stupid | 22:52 |
| clarkb | greghaynes: not necessarily since its a cffi issue in the package itself | 22:52 |
| clarkb | greghaynes: so one package could work fine and the other could be broken | 22:52 |
| *** jeremati_ has joined #openstack-dev | 22:52 | |
| lifeless | greghaynes: there's a couple of reasons cryptography might work | 22:52 |
| *** jerematic has quit IRC | 22:52 | |
| lifeless | greghaynes: it might fail to build a wheel | 22:52 |
| lifeless | greghaynes: and import its local cffi module during 'python setup.py install' | 22:53 |
| lifeless | greghaynes: that combination would work | 22:53 |
| *** sarob has quit IRC | 22:53 | |
| greghaynes | looks like the cryptography weel build works and seems like its installed from wheel | 22:54 |
| fungi | so much with the scrollback | 22:54 |
| greghaynes | the differing crc's for the .so are very ominious, and make me think xattr is doing something wierd in its cffi use | 22:55 |
| clarkb | greghaynes: lifeless the cffi.so goes in the wheel | 22:55 |
| lifeless | clarkb: yes, I see that | 22:56 |
| clarkb | greghaynes: lifeless cryptography may make that file copy properly and xattr does not | 22:56 |
| lifeless | clarkb: in the wrong path | 22:56 |
| clarkb | lifeless: yup, so I think simplest and most likely reason cryptography works is they use the correct path | 22:56 |
| lifeless | ok so | 22:56 |
| lifeless | this seems to be in good hands | 22:56 |
| * clarkb makes a cryptography wheel | 22:56 | |
| lifeless | I'm going to go get some yummy fatty proteiny lunch | 22:57 |
| *** penick has joined #openstack-dev | 22:57 | |
| lifeless | and then dive dive dive into pip again. | 22:57 |
| greghaynes | heh, fun | 22:57 |
| lifeless | well | 22:57 |
| lifeless | I nearly have the install_req cache in place | 22:57 |
| lifeless | that will let me do setup_requires sanely | 22:57 |
| lifeless | and also a resolver | 22:57 |
| *** emagana has quit IRC | 22:57 | |
| lifeless | been slow progress for the last week, world of interrupts | 22:57 |
| *** Swami has joined #openstack-dev | 22:57 | |
| *** ZZelle_ has quit IRC | 22:58 | |
| *** emagana has joined #openstack-dev | 22:58 | |
| clarkb | hrm cryptography does not do __pycache__ either | 22:59 |
| jogo | hmm I am not sure if I am installing swift locally woops | 23:00 |
| clarkb | also I get 3 differeny cryptography .so's | 23:01 |
| *** penick has quit IRC | 23:01 | |
| *** bswartz has joined #openstack-dev | 23:02 | |
| *** ijw has joined #openstack-dev | 23:02 | |
| clayg | clarkb: make sure you clean your build dri before you bdist_wheel | 23:02 |
| *** annegent_ has quit IRC | 23:02 | |
| *** emagana has quit IRC | 23:02 | |
| clarkb | clayg: is that the source of all the .so files? | 23:02 |
| *** gargola has quit IRC | 23:03 | |
| clayg | clarkb: was on my machine when i was trying bdist_wheel on xattr w and w/o cffi installed | 23:03 |
| clayg | clarkb: oh, well nm, cryptography may just make more .so's | 23:04 |
| *** alanf-mc_ has joined #openstack-dev | 23:04 | |
| *** arnaud____ has joined #openstack-dev | 23:05 | |
| *** bogdando has joined #openstack-dev | 23:05 | |
| *** rtheis has quit IRC | 23:05 | |
| *** topol has joined #openstack-dev | 23:05 | |
| jogo | wow installing pyECLIb is slow | 23:07 |
| *** alanf-mc has quit IRC | 23:07 | |
| *** jobewan has quit IRC | 23:08 | |
| *** arnaud____ has quit IRC | 23:09 | |
| *** emagana has joined #openstack-dev | 23:09 | |
| *** alex_xu has quit IRC | 23:10 | |
| *** slupers has quit IRC | 23:10 | |
| * jogo watches openstackclient be slow | 23:10 | |
| *** alex_xu has joined #openstack-dev | 23:11 | |
| * jogo waves at dtroyer | 23:11 | |
| *** slupers has joined #openstack-dev | 23:11 | |
| clayg | jogo: you in a hurry today or something? | 23:12 |
| jogo | clayg: yes | 23:12 |
| jogo | clayg: afternoon snack :) | 23:12 |
| jogo | clayg: and watching devstack take its time is no fun | 23:12 |
| jogo | ah here we go | 23:12 |
| jogo | swift-proxy just launched | 23:12 |
| jogo | greghaynes: looks like your fix works! | 23:13 |
| *** emagana has quit IRC | 23:14 | |
| jogo | greghaynes: now we need to get 2 devstack cores | 23:14 |
| greghaynes | jogo: w00t | 23:14 |
| clayg | jogo: heh, what's the change? | 23:14 |
| jogo | mtreinish: I think your one right? | 23:14 |
| jogo | https://review.openstack.org/#/c/180341/2 | 23:14 |
| clayg | sudo python -c 'import xattr' | 23:14 |
| clayg | i'd be real surpirsed if the sudo import is needed - pip install'ing it (instead of using the bad bits in the wheel) should be good enough | 23:16 |
| *** jbell8 has joined #openstack-dev | 23:16 | |
| jogo | anyway we have a fix, just need to figure out if it needs tweaking or not etc and get the right people to review it | 23:17 |
| jogo | heading out for a bit | 23:17 |
| greghaynes | clayg: the issue is that the __pycache__ dir is in /usr/... since were installing it system wide | 23:18 |
| greghaynes | so yes, the sudo is needed | 23:18 |
| clayg | greghaynes: I think installing via pip will use the 0.9.2 cffi and the crc will match once it's installed - i'd guess no __pycache__ is created with that import | 23:19 |
| greghaynes | that is what we currently do | 23:20 |
| clayg | greghaynes: no I don't think so - it's currently pip installing from the wheel - which is build while 0.9.2 cffi is not installed - so bdist_wheel pulls in cffi 1.0.0b1 via install_requires and builds with a crc that doesn't match what is eventually required for 0.9.2 - which then tries to rebuild triggering the __pycache__ permissins issue | 23:21 |
| clarkb | OH! is that because install_requires is easy_install and it doesn't grok versions? | 23:22 |
| greghaynes | yes, that is another option, we could sudo pip install and not from wheel. The change I posted is still installing from wheel. | 23:22 |
| clarkb | so that may actually be what changed here | 23:22 |
| clayg | clarkb: yeah | 23:22 |
| clarkb | how new is 1.0.0b1? | 23:22 |
| haypo | clayg, cffi only tries to write the source when it fails to find the .so | 23:22 |
| clayg | clarkb: yeah 1.0.0b1 got released | 23:22 |
| clayg | today | 23:22 |
| clarkb | clayg: awesome ok now we know why it just started | 23:22 |
| *** pixelbeat has joined #openstack-dev | 23:22 | |
| clayg | haypo: or when the .so it finds doesn't match the crc? | 23:22 |
| clarkb | or I do and you all knew long before me and I am catching up :) | 23:23 |
| haypo | clayg, yes, it probably fails to find the .so because xattr was build with a different cffi than the installed cffi | 23:23 |
| clayg | haypo: yeah | 23:23 |
| *** abregman has quit IRC | 23:23 | |
| clayg | haypo: i don't really know why bdist_wheel from the cryptography package - which also pulls in 1.0.0b1 - doesn't end up in the same state? | 23:23 |
| greghaynes | based on the logs the wheel is built with cffi 0.9.2 installed | 23:23 |
| clayg | but the cryptography package is filled with comments about "working around cffi issue #XX" | 23:23 |
| *** ksobkowiak has quit IRC | 23:24 | |
| haypo | clayg, in fact, the __pycache__ doesn't matter, cffi is able to locate the .so in the xattr directory | 23:24 |
| *** signed8b_ is now known as signed8bit_ZZZzz | 23:24 | |
| clayg | so I'm guessing there *is* something xattr could do to *also* work with cffi 1.0.0b1 - but I think it'd be better if pip wheel <pakcage> would just throw "install_requires" out the window and use the wheels it build before calling bdist_wheel | 23:24 |
| clarkb | haypo: it matters because it tries to write there without permissions to do so | 23:25 |
| clarkb | haypo: but beyond that seems fine | 23:25 |
| clayg | haypo: __pycache__ doesn't matter *if* cffi is able to locate the .so in the xattr dir that matches the crc it's looking for | 23:25 |
| clarkb | clayg: ya that | 23:25 |
| clayg | clarkb: cryptograph does this neat monkey patch trick to cffi to prevent this automatic build trick and raises a sane error about "why are you trying to rebuild an installed package?" | 23:26 |
| haypo | clarkb, i reproduce the issue with cffi 0.9.2 already installed on my system. does devstack build the wheel package in a venv disconnected of the system? | 23:26 |
| fungi | clarkb: yep, the beta landed on our mirrors minutes before the issue arose. unfortunately i saw no evidence in pip freeze that we were actually using it so wrote it off as a spooky coincidence | 23:26 |
| greghaynes | haypo: yes, its in a venv | 23:26 |
| fungi | also our image updates were coming online at the same moment, which posed a potential explanation for something else changing | 23:26 |
| clayg | fungi: yeah it's never pip installed - it's only pulled down with install_requires during bdist_wheel step - so it never shows up in pip freeze | 23:27 |
| fungi | right | 23:27 |
| fungi | which i didn't consider at the time | 23:27 |
| fungi | because... sanity? | 23:27 |
| clayg | if you're in a vevn - even if you have cffi installed - will install_requires still think it needs to download and install cffi to bdist_wheel? | 23:27 |
| clarkb | clayg: ya I think so | 23:27 |
| clayg | or did you mean you had cffi installed in the venv? | 23:27 |
| haypo | clayg, clarkb greghaynes : would it be possible to ensure that cffi 0.9.2 is installed in the venv used to build the xattr wheel package? | 23:28 |
| *** jbell8 has quit IRC | 23:28 | |
| clayg | i think it'd be hard to "reproduce" the issue in a venv - unless you're just checking to see if __pycache__ gets created or not (it shouldn't if bdist_wheel did it's job) | 23:28 |
| greghaynes | http://logs.openstack.org/27/169827/10/check/check-tempest-dsvm-full/506c93a/logs/devstacklog.txt.gz#_2015-05-05_17_11_22_490 is where we grab cffi and then http://logs.openstack.org/27/169827/10/check/check-tempest-dsvm-full/506c93a/logs/devstacklog.txt.gz#_2015-05-05_17_12_11_424 we dont re-download it | 23:28 |
| clayg | haypo: I think pip wheel <package> should do just that | 23:28 |
| clayg | isn't like lifeless a pip maintainer now? :D | 23:28 |
| clarkb | dstufft: ^ able to fill us in? | 23:29 |
| haypo | clayg, "pip wheel" downloads dependencies if needed, i asked if it would be possible to ensure in devstack that cffi 0.9.2 is already installed before trying to build xattr | 23:29 |
| dstufft | wat | 23:29 |
| clayg | greghaynes: why would we? ime even with pip wheel <package> -v ... you don't really get to see the output from the bdist_wheel step (which is where the 1.0.0b1 is getting pulled in) | 23:29 |
| greghaynes | haypo: it is | 23:29 |
| clayg | haypo: yeah sure if we just wheel cffi and then pip install ciff.whl before we do anything else that would work for sure | 23:30 |
| sdague | back... | 23:30 |
| *** slupers has quit IRC | 23:30 | |
| sdague | so the issue is that building a wheel cache in advance with cffi is just fail-o-potamus? | 23:30 |
| haypo | greghaynes, i don't understand. the bug is that building the xattr wheel packages download cffi 1.0 | 23:30 |
| clayg | i just think if pip wheel <package> name is going to all this trouble to discover all the depends and makes wheels for all of them - maybe it should make sure those are the packages that are used as part of install_requires | 23:30 |
| dstufft | clarkb: can you summarize question so I don't have to read scrollback, knee deep in uinsurance stuff | 23:30 |
| lifeless | am back | 23:31 |
| clarkb | sdague: thats the short story, some packages work around it but generally yes | 23:31 |
| *** slupers has joined #openstack-dev | 23:31 | |
| *** penick has joined #openstack-dev | 23:31 | |
| greghaynes | haypo: ah, crap, youre right, thats a build not an install | 23:31 |
| *** shakamunyi has quit IRC | 23:31 | |
| sdague | dstufft: the short fail story. In devstack we started building a wheel cache to speed things up | 23:31 |
| sdague | for all the c compiled stuff | 23:31 |
| clarkb | dstufft: looks like xattr uses install_requires on cffi which pulls latest beta of cffi, then later when we install the wheel and use xattr we have cffi 0.9.2 which means that the .so we built is wrong | 23:31 |
| *** jbell8 has joined #openstack-dev | 23:31 | |
| greghaynes | so yes, we could preseed it | 23:31 |
| lifeless | clayg: that -v thing is a bug, i'm not sure if there is an issue open for it or not | 23:31 |
| sdague | which includes things like cffi | 23:31 |
| clarkb | dstufft: so then it decided it has to rebuild it | 23:31 |
| lifeless | dstufft: https://bugs.launchpad.net/swift/+bug/1451992 | 23:32 |
| openstack | Launchpad bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] | 23:32 |
| dstufft | um | 23:32 |
| sdague | clarkb: oh, so xattr is bypassing pip? | 23:32 |
| *** jaosorior has quit IRC | 23:32 | |
| clarkb | sdague: yes, bceause install_requires | 23:32 |
| lifeless | clarkb: oh, wat - two versions, yes thats a new form of breakage | 23:32 |
| clarkb | its using easy_install | 23:32 |
| clarkb | aiui | 23:32 |
| dstufft | easy_install doesn't filter out pre-releases | 23:32 |
| lifeless | clarkb: no, we query egg_info and then use that to resolve deps | 23:32 |
| clarkb | which means that you get the beta not latest real release for build time | 23:32 |
| dstufft | pip does | 23:32 |
| haypo | sdague, DevStack creates a venv to build the wheel package of xattr, this venv doesn't care of openstack requirements :) | 23:32 |
| clarkb | dstufft: yup exactly | 23:32 |
| lifeless | but whats causing easy-install to trigger | 23:33 |
| dstufft | that's probably going to break some things | 23:33 |
| lifeless | is it a setup_requires? | 23:33 |
| dstufft | setup_requires | 23:33 |
| greghaynes | clarkb: so looks like the previous question is answered of "what happens when cffi changes under the bindings" ;) | 23:33 |
| dstufft | when it compiles the cffi modules | 23:33 |
| clayg | for any devstack cores - jogo is pretty sure https://review.openstack.org/#/c/180341/2 would unstick the gate no matter what ya'll decide to fix later | 23:33 |
| clarkb | greghaynes: ya you basically break the world | 23:33 |
| *** jkremer has quit IRC | 23:33 | |
| sdague | clayg: I can fast approve that, let me look | 23:33 |
| clarkb | greghaynes: so new cffi release is going to e lol | 23:33 |
| lifeless | yup, there it is: setup_requires=["cffi>=0.4"], | 23:33 |
| clarkb | sorry setup_requires | 23:33 |
| dstufft | so it'll get a beta release of cffi in setup_requires when it compiles the .so, but a non beta release when it actually installs it | 23:33 |
| clarkb | whatever the thing is that runs easy_install | 23:33 |
| greghaynes | clarkb: once we make a wheel mirror.... | 23:33 |
| dstufft | cffi 1.0 is pretty awesome though | 23:33 |
| clarkb | dstufft: ya but all installs will break as soon as they upgrade to it | 23:34 |
| dstufft | it's just not ABI compat with cffi < 1.0 :) | 23:34 |
| lifeless | dstufft: whats jaracao's view on making easy-install pep-440 compatible? | 23:34 |
| * haypo tries to add cffi==0.9.2 to files/venv-requirements.txt | 23:34 | |
| lifeless | haypo: is that installed before anything else? if so it should bypass the headache yes | 23:34 |
| sdague | dstufft: .... so based on digging out of cffi build failures, I fail to believe that cffi is awesome :) | 23:34 |
| haypo | lifeless, i'm trying to put it at the beginning | 23:35 |
| lifeless | haypo: jus tadding cffi should be enough because pip is interpreting the requirements.txt files, and it will filter betas | 23:35 |
| dstufft | lifeless: he's OK with it, setuptools already uses the same library as pip for parsing/versioning and stuff, I just got frustrated trying to pass around the things we needed into the right places so I just hardcoded prereleases=True when I implemented pep 440 with the hopes I (or someone) would come back later and make it handle pre-releases | 23:35 |
| clarkb | lifeless: except we have to do that before we make wheels which is a bit chicken and egg but will work | 23:35 |
| dstufft | if you think pip's codebase is badly factored, setuptools is worse | 23:35 |
| lifeless | haypo: what we probably need is to match g-r though, so we don't get suprrised in future | 23:35 |
| lifeless | dstufft: believe me, I have no illusions there | 23:35 |
| dstufft | armin might just take cffi 1.0 off PyPI for now | 23:36 |
| dstufft | maybe just poke him first | 23:36 |
| lifeless | the alternative would be to make our dep be on onthe dev version | 23:36 |
| lifeless | so we don't downgrade | 23:36 |
| haypo | dstufft, and distutils is better than pip? :) | 23:36 |
| greghaynes | clarkb: so if we are using a wheel mirror, and a cffi release happens like this... the world goes boom until we rebuild the wheel mirror after upgrading cffi? | 23:36 |
| sdague | greghaynes: so, we've never seen a cffi release that didn't make things go boom | 23:37 |
| greghaynes | hah | 23:37 |
| sdague | 0.9.1 -> 0.9.2 did the same thing | 23:37 |
| clarkb | greghaynes: it depends on whether or not the versions match | 23:37 |
| lifeless | I think we probably want to blacklist cffi *and its transitive deps* | 23:37 |
| lifeless | from wheel building | 23:37 |
| dstufft | well | 23:37 |
| clayg | greghaynes: i'm not 100% sure about that - when I install cffi 0.9.1 and then xattr, then upgrade cffi to 0.9.2 it still seems cool with the pre-existing .so? | 23:37 |
| dstufft | that's because cffi did something kinda dumb | 23:37 |
| dstufft | originally | 23:37 |
| sdague | lifeless: that was the whole *point* of wheel building | 23:37 |
| dstufft | cffi 1.0 fixes that afaik | 23:37 |
| lifeless | sdague: for the wheels where it works, absolutely | 23:37 |
| sdague | to stop compiling over and over and over again | 23:37 |
| greghaynes | clayg: yea, itd just be for abi incompat releases | 23:37 |
| lifeless | dstufft: its opt-in | 23:37 |
| lifeless | dstufft: in 1.0, new api to use it | 23:37 |
| clarkb | dstufft: it doesn't fix anything if you have an older version installed | 23:38 |
| dstufft | (and indiiddual projects can fix it themselves anyways, cryptography didn't go boom from 0.9.1 to 0.9.2) | 23:38 |
| dstufft | clarkb: well they can't go back in time and fix it in older versions | 23:38 |
| haypo | lifeless, the annoying thing is that pip installs 1.0beta instead of stable 0.9.2 :) it would be nice to being able to not install beta | 23:38 |
| *** abregman has joined #openstack-dev | 23:38 | |
| sdague | dstufft: so... until cffi can do a release and not explode across all the boundaries, I will continue to believe it's not awesome | 23:38 |
| clayg | i wish I knew how cryptography managed to get it's bdist_egg to work even in the face of 1.0.0b1 | 23:38 |
| lifeless | haypo: s/pip/easy_install/ | 23:38 |
| clarkb | dstufft: no, but the could have a less bonghits process for updating cache | 23:38 |
| haypo | lifeless, can cffi maintainers workaround this issue? | 23:38 |
| lifeless | haypo: we need to fix setuptools to fix that | 23:38 |
| clarkb | dstufft: basically you can't assume you can update the package install after it is installed | 23:38 |
| clarkb | dstufft: you have to continue working out of the package install | 23:39 |
| *** jbell8 has quit IRC | 23:39 | |
| clarkb | dstufft: but by changing versions here it is assumed you can write to /usr apparently | 23:39 |
| fungi | greghaynes: sdague: i've enqueued and promoted 180341,2 to the front of the gate | 23:39 |
| sdague | fungi: thank you | 23:39 |
| greghaynes | fungi: \O/ ive been promoted | 23:39 |
| fungi | given the current tight test resources, it would likely have taken forever to merge otherwise | 23:39 |
| dstufft | clarkb: to be fair to them, it's a dev release on PyPI and pip won't install it by default so they just assumed that was a safe thing to do. It's not really reasonable to expect random folks to understand that ``pip install foo`` sometimes doesn't use pip | 23:40 |
| *** apmelton is now known as apmelton_away | 23:40 | |
| dstufft | I mean, that's just python packaging being crazy | 23:40 |
| clarkb | dstufft: so I get that but this is a problem from 0.9.2 -> 1.0.0 when that happens too | 23:40 |
| clarkb | dstufft: anyone with 0.9.2 built packages will hit this when 1.0.0 releases and is updated to locally | 23:40 |
| clarkb | dstufft: that is just bad for users | 23:40 |
| fungi | is also why we only uploaded beta releases as wheels | 23:40 |
| clarkb | we happen to see this early because setup_requires | 23:40 |
| *** abregman has quit IRC | 23:40 | |
| clarkb | but its still an issue once release happens | 23:41 |
| *** ajmiller has quit IRC | 23:41 | |
| fungi | pip new enough to grok wheels is new enough to not install beta releases unless asked, and easy_install doesn't wheel | 23:41 |
| greghaynes | yes, wed need to pin cffi everywhere and then do some kind of lockstep unpin and wheel rebuilding for our mirrors | 23:41 |
| sdague | clarkb: can you file an upstream bug? because this assumption of write access to /usr is kind of ... crazy pants | 23:41 |
| sdague | greghaynes: pinning doesn't help | 23:41 |
| lifeless | sdague: upstream on xattr or cffi | 23:41 |
| dstufft | fungi: well cffi has compiled code in it so you can't do that and get windows coverage :/ | 23:41 |
| sdague | because we don't control this part, as it bypasses pip | 23:41 |
| clarkb | it won't help bceause of setup_requires | 23:41 |
| sdague | upstream cffi | 23:42 |
| lifeless | sdague: its cffi that does it, and the bug already exists | 23:42 |
| greghaynes | aye | 23:42 |
| lifeless | sdague: I linked it earlier | 23:42 |
| dstufft | cffi 1.0 (if you use the new APIs) doesn't expect write access | 23:42 |
| *** btulchinsky has quit IRC | 23:42 | |
| sdague | lifeless: ok. and it's just *shrug*? | 23:42 |
| *** penick_ has joined #openstack-dev | 23:42 | |
| clarkb | dstufft: yes but you have to transition to 1.0 | 23:42 |
| lifeless | sdague: no, its in progress | 23:42 |
| clarkb | dstufft: my point is that transition is broken | 23:42 |
| *** penick has quit IRC | 23:42 | |
| *** penick_ is now known as penick | 23:42 | |
| clarkb | dstufft: the only way to do it is to reinstall the world | 23:42 |
| lifeless | sdague: https://bugs.launchpad.net/swift/+bug/1451992/comments/9 | 23:42 |
| openstack | Launchpad bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] | 23:42 |
| sdague | lifeless: that's the swift bug | 23:42 |
| sdague | oh, sorry, embedded link | 23:43 |
| lifeless | sdague: and prose and stuff :) | 23:43 |
| sdague | so, right, only 1.5 years old :) | 23:43 |
| *** salv-orlando has joined #openstack-dev | 23:44 | |
| dstufft | clarkb: when you're going from broken to not broken (or at leas tless broken) sometimes you have to transition? I'm not sure what you want them to do here, if you installed cffi 1.0.devwhatever instead of 0.9 when you did the install (not the build) it'd work fine, the problem is that you built against one ABI and installed against another. | 23:45 |
| lifeless | so the there are several combining issues | 23:46 |
| lifeless | easy-install and pip have different version selection algorithms | 23:46 |
| lifeless | cffi can't deal with different versions being installed without write access to __pycache__ in the installed dir | 23:46 |
| lifeless | python doesn't allow __pycache__ to be user-local when the thing being run is global installed | 23:46 |
| lifeless | cffi uploaded a beta to pypi | 23:47 |
| lifeless | pick any of them, that would prevent the damage today. | 23:47 |
| dstufft | (the second thing there is solved by cffi 1.0 assuming projects are using the new APIs, but not if they are using the old APIs) | 23:47 |
| *** annegentle has joined #openstack-dev | 23:47 | |
| sdague | dstufft: that seems like a big assumption | 23:47 |
| lifeless | Everyone is acting in good faith; yes cffi made naive choices early on - this is not unusual amongst upstreams. | 23:47 |
| *** annegentle has quit IRC | 23:47 | |
| lifeless | Not everyone is as scarred and bitter as we | 23:47 |
| haypo | ok, here is my proposed workaround for the bug: https://review.openstack.org/180354 | 23:48 |
| dstufft | sdague: So what do you want cffi to do that doesn't involve time traveling to never make the original mistake. | 23:48 |
| lifeless | we need a short term answer for today | 23:48 |
| sdague | lifeless: we have the short term fix | 23:48 |
| sdague | that's in the gate | 23:48 |
| lifeless | and we either need cffi to make the 0.9.2->1.0.0 transition better than with the beta [which might be possible: lets file a bug] | 23:49 |
| clarkb | dstufft: you make the transition easier | 23:49 |
| clarkb | dstufft: possibly by carrying the version of cffi that was used to build with you | 23:49 |
| lifeless | or we need to put some cross-check in place, somewhere, to detect this failure mode and correct it | 23:49 |
| lifeless | clarkb: 'you' ? | 23:49 |
| dstufft | clarkb: if you installed cffi 1.0 into the final environment instead of 0.9 it would have worked fine fwiw | 23:49 |
| clarkb | dstufft: yup, thats wh I am suggesting carrying that version used to build forward | 23:50 |
| dstufft | you only got explosions because you built against 1.0 but ran against 0.9.x | 23:50 |
| clarkb | if cffi can write an so it can write a cffi module out too | 23:50 |
| haypo | lifeless, clarkb, clayg, greghaynes : could you please review my workaround https://review.openstack.org/180354 which enforces cffi 0.9.2? | 23:50 |
| sdague | right, why don't built packages vendor their cffi | 23:50 |
| sdague | then it would never explode | 23:50 |
| clarkb | sdague: well and not just packages but cffi itself so make that easy, its already writing an so | 23:50 |
| * fungi has ceased trying to reason about non-native python libs. so much trouble | 23:51 | |
| haypo | oh, sdague accepts the other workaround (https://review.openstack.org/#/c/180341/) so mine is maybe not needed | 23:51 |
| lifeless | clarkb: there's also runtime support in cffi itself | 23:51 |
| lifeless | clarkb: AIUI, so sadly just the so isn't enough | 23:51 |
| dstufft | in cffi 1.0 there is still a CFFI .so that the cffi using project links against | 23:51 |
| clarkb | lifeless: correct | 23:51 |
| dstufft | in cffi there is no runtime python needed | 23:51 |
| clarkb | lifeless: you put that stuff in the package too | 23:51 |
| dstufft | cffi 1.0* | 23:51 |
| lifeless | dstufft: there is according to armin | 23:51 |
| *** salv-orlando has quit IRC | 23:51 | |
| clarkb | you basicall say "this so requires a particular version of cffi to work therefore we will include that with the .so" | 23:52 |
| lifeless | dstufft: (see the blog post - I may be misreading) | 23:52 |
| dstufft | lifeless: not in 1.0, in 1.0 you only have cffi-runtime which is just a .so (and is only needed on CPython, PyPy will have it built in) | 23:52 |
| lifeless | dstufft: except cffi-runtime doesn't exist yet so he says to use cffi still | 23:52 |
| clarkb | thats one way to tackle this | 23:52 |
| *** rfolco has quit IRC | 23:52 | |
| dstufft | lifeless: yea, currently the dev version doesn't have that, I mean that's the eventual goal, nobody was supposed to be using the dev versino for serious things yet :) that's jsut a mistake because they didn't realize easy_isntall would still install it | 23:53 |
| lifeless | dstufft: I almost wish pypi would reject betas with a warning, and require an opt-in parameter to upload them | 23:53 |
| *** chutwig has joined #openstack-dev | 23:54 | |
| clarkb | another option would be to fallback on writing to a different location, add that to path and move along from there (maybe with a warning) | 23:54 |
| dstufft | we're going the other way, we're trying to make it possible to upload them, just things will ingore them by default - we're just not all the way there yet | 23:54 |
| lifeless | dstufft: yes, I know, but... | 23:54 |
| lifeless | dstufft: not allowing the uploads until the ecosystem is fixed, would avoid firedrills like this | 23:54 |
| dstufft | (actually, given nobody is yelling at cryptography yet, I don't even think cffi broke ABI, I bet this is entirely about the stupid module name thing in cffi < 1.0) | 23:55 |
| haypo | dstufft, "pip install foo sometimes doesn't use pip", oh i didn't know that | 23:55 |
| clarkb | dstufft: aiui cryptography works around this particular issue | 23:55 |
| dstufft | (wich cryptography works around by hardcoding the modulename) | 23:55 |
| dstufft | clarkb: right, if cffi 1.0 actually broke ABI in the .so, then this should effect cryptography | 23:56 |
| clarkb | another option would be to use a completely new package name for ABI compat breaks rather than versions | 23:56 |
| dstufft | if this is just the "cffi used to make it real fragile when upgrading versions because it hardcoded version number info into the auto generated name" problem, then it should not affect crpyotgraphy | 23:56 |
| clarkb | dstufft: the underlying problem is you cannot assume you have write perms to the install location | 23:57 |
| clarkb | dstufft: so any case where you need to rebuild the so can fail | 23:57 |
| lifeless | we're going in circles | 23:57 |
| lifeless | everyone gets that that is bad (dstufft even says so himself in the upstream bug) | 23:58 |
| greghaynes | dstufft: I think our cryptography wheel must be built against 0.9.2, it doesnt attempt to rebuild extensions | 23:58 |
| dstufft | clarkb: sure, which is why cffi exists, to add new APIs to get rid of their original mistake | 23:58 |
| dstufft | greghaynes: cryptography works around the problem that is causing rebuilds. | 23:58 |
| greghaynes | well then | 23:58 |
| dstufft | xattr apparently doesn't | 23:58 |
| clarkb | dstufft: yes and rebuilds are the problem because you can't write to /usr as not root | 23:58 |
| lifeless | haypo: I think we should switch that loop to doing a single big call | 23:58 |
| lifeless | haypo: (separately) | 23:59 |
| sdague | right, so what about actually checking for writability of the cache, and if not, going to tmpdir like clarkb posted | 23:59 |
| sdague | that seems like a 0.9.x fixable thing | 23:59 |
| lifeless | I suspect it would go into 1.0.0 too as another improvement | 23:59 |
| sdague | it's ugly, but at least provides transition | 23:59 |
| clarkb | and emit a warning | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!