Thursday, 2015-04-02

*** ryanpetrello has joined #openstack-infra		00:00
*** achanda has quit IRC		00:00
*** sarob_ is now known as sarob		00:00
clarkb	jogo: I have no idea what that is about	00:03
clarkb	jogo: are you talking about VMs sharing addresses? hypervisors sharing addresses?	00:03
*** achanda has joined #openstack-infra		00:06
*** unicell has joined #openstack-infra		00:06
clarkb	jogo: https://bugs.launchpad.net/nova/+bug/1092347 related to that bug possibly?	00:09
openstack	Launchpad bug 1092347 in OpenStack Compute (nova) "DHCP fails after live migration when using network multi host" [Medium,Incomplete]	00:09
jogo	clarkb: looking into http://logs.openstack.org/73/169873/1/check/check-tempest-dsvm-aiopcpu-full/ea5db7b/console.html#_2015-04-01_19_35_42_279	00:09
jogo	clarkb: its not a a live migration test	00:09
clarkb	yes I see the logs my problem is I don't parse half of what you and vishy said	00:09
clarkb	but that bug explains share_dhcp_address so I think I understand now	00:10
*** sarob has quit IRC		00:11
clarkb	jogo: why did you zero in on dhcp? ssh failed to hit an address which could be any number of things	00:11
jogo	clarkb: vishy did	00:11
clarkb	yes but how? I am not convinced	00:11
jogo	http://logs.openstack.org/73/169873/1/check/check-tempest-dsvm-aiopcpu-full/ea5db7b/console.html#_2015-04-01_19_35_50_739	00:11
*** spzala has quit IRC		00:12
clarkb	ah ok	00:12
*** mtanino has quit IRC		00:12
jogo	anyway tomorrow it may be nice to take one of the aiopcpu nodes offline and investigate it in more detail	00:12
clarkb	sure	00:12
jogo	vishy: would you be around tomorrow?	00:12
vishy	i can be sure	00:12
vishy	should have more time for openstack stuff now :)	00:13
clarkb	so dhcp is going to send an l2 broadcast	00:13
jogo	vishy: oh?	00:14
clarkb	then any listening dhcp server should respond with an offer.	00:14
vishy	jogo: you missed the news today?	00:14
clarkb	so either the broadcast is filtered or the response	00:14
vishy	jogo: https://www.nebula.com/	00:14
clarkb	or there was no response	00:14
*** Swami has quit IRC		00:14
jogo	is this an April fools thing?	00:15
dtroyer	unfortunately no	00:15
*** aduarte has quit IRC		00:15
jogo	:(	00:15
dstufft	the worst date to announce that :( Although I doubt any company would do an april fools of "we're shutting down"	00:15
lifeless	Its 2nd april now :)	00:16
vishy	yeah terrible timing	00:16
vishy	we really hoped to announce yesterday	00:16
*** yamamoto has quit IRC		00:16
dstufft	lifeless: not where Nebula's HQ is ;P	00:16
vishy	but we couldn’t get it organized in time	00:16
lifeless	dstufft: so many folk living in the past	00:16
dstufft	vishy: I'm sad about it, Nebula was my first real start up ;(	00:16
vishy	yeah it is sad	00:17
vishy	startups are hard :)	00:17
*** dannywilson has quit IRC		00:18
clarkb	jogo: one thing to start with is likely `tcpdump -n -e -i any port 67 or port 68` on both hosts and just write that to a log file that you collect	00:20
clarkb	jogo: I don't expect those files to get too gigantic	00:20
clarkb	jogo: though it will catch hpcloud dhcp traffic	00:22
*** ghostpl_ has quit IRC		00:22
openstackgerrit	Timo Tijhof proposed openstack-infra/zuul: Status: Use $DEST_DIR instead of $BASE_DIR/public_html https://review.openstack.org/169955	00:24
openstackgerrit	Timo Tijhof proposed openstack-infra/zuul: Status: Move external dependencies into public_html/lib https://review.openstack.org/169956	00:24
openstackgerrit	Timo Tijhof proposed openstack-infra/zuul: Status: Fix JSHint violations and commit settings https://review.openstack.org/169957	00:24
mordred	vishy: oh!	00:24
*** garyh has quit IRC		00:25
clarkb	mordred: `sudo ansible logstash-worker* -m shell -a 'grep fb2e0ad /var/log/logstash/log-worker-*-debug.log'` is one of my favorite things now	00:25
mordred	clarkb: ++	00:28
*** sputnik13 has quit IRC		00:29
*** rbradfor has quit IRC		00:30
*** erlon has quit IRC		00:31
*** SumitNaiksatam has joined #openstack-infra		00:31
*** ZZelle has quit IRC		00:32
*** sdake has quit IRC		00:32
*** hdd has quit IRC		00:36
*** hdd has joined #openstack-infra		00:36
*** wenlock has quit IRC		00:37
*** annegentle has joined #openstack-infra		00:38
clarkb	trove functional tests take a LONG time	00:39
clarkb	waiting on one job before I can restart 07 then all the jenkinses are done	00:40
clarkb	jhesketh: you still around?	00:41
clarkb	jhesketh: https://review.openstack.org/#/c/167158/ can you double check the -1 there? jeblair and I came to that conclusion a bit earlier today. And it would be good to move forward on that if possible	00:42
clarkb	(was mostly me catching up to what jeblair had said all along)	00:43
*** annegentle has quit IRC		00:43
*** salv-orlando has quit IRC		00:44
*** tiswanso has joined #openstack-infra		00:45
jogo	clarkb: if you think just collecting tcpdump works that sounds good to me	00:45
clarkb	jogo: well its an easy thing we can start with	00:46
*** hichihara has joined #openstack-infra		00:46
clarkb	jogo: and since this doesn't happen 100% of the time might be a good place to start until we can narrow the problem down to something a bit more concrete	00:46
jogo	clarkb: agreed, you want to push the patch up?	00:46
clarkb	jogo: no, I am busy babysitting jenkinses	00:46
jogo	as it sounds like you have a clear idea of what you want	00:46
clarkb	then I have to go put together a bassinet	00:46
jogo	clarkb: is that practice for the real thing?	00:47
clarkb	jogo: preparation	00:47
*** tiswanso has quit IRC		00:48
*** tiswanso has joined #openstack-infra		00:49
*** hdd has quit IRC		00:50
clarkb	restarting jenkins 07 now	00:50
jhesketh	clarkb: replied	00:51
jhesketh	jeblair: replied to https://review.openstack.org/#/c/167158/	00:51
jogo	clarkb: the babysitting jenkinses part too?	00:52
jogo	clarkb: anyway thanks, I'll pick this up tomrrow	00:52
clarkb	jhesketh: right, but in that change you also copy them via scp	00:53
clarkb	jhesketh: which is what the confusion is about. So I think we just need to remove the scp copies	00:53
jhesketh	clarkb: ah I see, yes you are right	00:55
jhesketh	clarkb: so the index_footer.html still appears in the index and is directly accessible when uploaded by swift... do we want it to also be accessible when uploaded by scp	00:56
jhesketh	which also means we end up with a copy of the footer at the time of the logs	00:56
jhesketh	even though it's not the one apache serves	00:56
clarkb	jhesketh: I don't think so	00:56
clarkb	jhesketh: we should keep the old logs working the same and new logs do whatever they need to do	00:57
jhesketh	yep, that was my feeling, just checking	00:57
jhesketh	will fix then :-)	00:57
clarkb	thanks	00:57
*** ryanpetrello has quit IRC		00:57
*** ryanpetrello has joined #openstack-infra		00:58
openstackgerrit	Joshua Hesketh proposed openstack-infra/project-config: Upload devstack-gate documentation with logs https://review.openstack.org/167158	00:59
openstackgerrit	Joshua Hesketh proposed openstack-infra/project-config: Update devstack log publisher to use new macro https://review.openstack.org/167842	00:59
openstackgerrit	Joshua Hesketh proposed openstack-infra/project-config: zuul-swift-devstack-logs includes console https://review.openstack.org/167378	00:59
*** marcusvrn1 has joined #openstack-infra		00:59
jhesketh	clarkb, jeblair: ^	00:59
*** marun has joined #openstack-infra		01:01
clarkb	jhesketh: thanks	01:02
*** marcusvrn has quit IRC		01:02
clarkb	also 07 is happily back up again so all jenkins masters with dynamic slaves have been restarted today	01:02
clarkb	jhesketh: I will bug jeblair to rereview tomorrow (to make sure everything lines up for him) then I can babysit that stack	01:05
*** asettle has quit IRC		01:05
clarkb	jhesketh: we are so close to getting this done, very exciting	01:05
*** marun has quit IRC		01:06
HenryG	I lost the "Toggle CI" button in gerrit. This happened to me before but I can't remember why?	01:06
*** asettle has joined #openstack-infra		01:06
clarkb	HenryG: its there for me after a hard refresh	01:06
clarkb	HenryG: maybe try that? I suppose its possible that your cache'd js is funny	01:07
HenryG	clarkb: hard refresh did it, thanks!	01:07
*** esker has joined #openstack-infra		01:07
HenryG	I have been ad-blocking clippy all day, that probably messed things up	01:08
clarkb	ah ya that could do it	01:08
clarkb	they are both served by the same block of html + js	01:08
HenryG	Next year I am taking April 1 off. :D	01:09
clarkb	also clippy is awesome	01:09
*** harlowja is now known as harlowja_away		01:14
*** dboik has joined #openstack-infra		01:14
*** dboik_ has joined #openstack-infra		01:15
*** stevemar has joined #openstack-infra		01:16
jhesketh	clarkb: awesome, thanks!	01:17
*** dboik has quit IRC		01:19
*** mattfarina has joined #openstack-infra		01:19
*** crc32 has quit IRC		01:25
*** alexpilotti has quit IRC		01:29
*** kaisers has joined #openstack-infra		01:30
*** asselin has joined #openstack-infra		01:31
*** mattfarina has quit IRC		01:31
*** kaisers1 has quit IRC		01:31
*** sarob has joined #openstack-infra		01:32
*** sarob has quit IRC		01:33
*** sarob has joined #openstack-infra		01:33
*** sarob has quit IRC		01:35
*** sarob has joined #openstack-infra		01:35
*** weshay has quit IRC		01:35
*** tqtran is now known as tqtran_afk		01:35
*** Longgeek has quit IRC		01:36
*** otter768 has joined #openstack-infra		01:36
*** Longgeek has joined #openstack-infra		01:37
*** annegentle has joined #openstack-infra		01:39
*** sarob has quit IRC		01:41
*** Longgeek has quit IRC		01:43
mordred	clippy ++	01:43
*** annegentle has quit IRC		01:44
*** Longgeek has joined #openstack-infra		01:44
*** salv-orlando has joined #openstack-infra		01:44
*** baoli has quit IRC		01:45
*** tnovacik has quit IRC		01:48
*** salv-orlando has quit IRC		01:49
*** imcsk8 has quit IRC		01:52
*** imcsk8 has joined #openstack-infra		01:52
*** dims_ has joined #openstack-infra		01:52
*** Longgeek_ has joined #openstack-infra		01:57
*** xyang1 has quit IRC		01:58
*** Longgeek has quit IRC		02:00
*** mmedvede has quit IRC		02:01
*** dims_ has quit IRC		02:03
*** patrickeast has quit IRC		02:04
*** gulic has joined #openstack-infra		02:04
*** macjack has quit IRC		02:04
*** dims_ has joined #openstack-infra		02:04
*** julim has joined #openstack-infra		02:05
fungi	HenryG: i plan to start considering it a personal holiday	02:06
fungi	different people have holidays dedicated to their religions, so why shouldn't i?	02:06
*** dims_ has quit IRC		02:08
*** sarob has joined #openstack-infra		02:13
*** sdake has joined #openstack-infra		02:16
*** sdake has quit IRC		02:17
*** unicell has quit IRC		02:17
Krinkle	404 Not Found http://zuul.openstack.org/jquery-visibility.js - http://zuul.openstack.org/	02:18
*** Somay has quit IRC		02:19
*** macjack has joined #openstack-infra		02:19
*** sarob has quit IRC		02:21
*** sdake has joined #openstack-infra		02:21
*** harlowja_at_home has joined #openstack-infra		02:21
fungi	Krinkle: i think we don't deploy that file, but i'll check	02:22
Krinkle	maybe forgot to run fetch-dependencies.sh when updating the install on that domain	02:22
Krinkle	other dependencies are deployed properly	02:22
fungi	no, looks like it should get deployed... http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/manifests/init.pp#n256	02:22
Krinkle	-min	02:23
Krinkle	someone renamed it a couple weeks back	02:23
fungi	yep, so it seems	02:23
Krinkle	https://review.openstack.org/#/c/169956/ might help	02:23
*** sarob has joined #openstack-infra		02:24
Krinkle	Oh interesting	02:24
Krinkle	it's not deployed using the shell script	02:24
Krinkle	it fetches upstream master directly	02:24
fungi	yeah, but looks like we install -min there	02:24
Krinkle	that seems a bit risky? Wouldn't one normally use a submodule, or tag, or some other versioning?	02:24
*** yamamoto has joined #openstack-infra		02:25
fungi	we do continuous deployment on lots of things. at the moment the http://zuul.openstack.org/ page flies under the radar a bit since it's not our primary status page for zuul	02:26
Krinkle	yeah	02:26
*** baoli has joined #openstack-infra		02:27
fungi	looks like that exec should probably just be replaced with a file resource	02:28
*** tkelsey has joined #openstack-infra		02:30
*** yamahata has quit IRC		02:30
Krinkle	fungi: I'm debugging a cache issue with the main Zuul status.json entry point. It seems in the past few weeks in the refactoring by openstack, the ajax() call was change to no longer provide a cache buster query string. However now, using our install, it is hitting a never-ending cache.	02:34
*** tkelsey has quit IRC		02:34
Krinkle	It seems the 304 Not Modified header is always returned.	02:34
openstackgerrit	Jeremy Stanley proposed openstack-infra/puppet-zuul: Link non-minified jquery-visibility https://review.openstack.org/169972	02:34
fungi	Krinkle: ^ that's probably the fix for zuul.openstack.org	02:34
Krinkle	I guess the Zuul install for openstack is no longer behind apache or some other caching layer. So it always gets a fresh 200 OK	02:34
*** otter768 has quit IRC		02:35
fungi	Krinkle: it's still got apache in place... checking to see what our vhost config looks like currently	02:36
Krinkle	Cool	02:36
Krinkle	E.g. using http://zuul.openstack.org/?source_url=https://integration.wikimedia.org/zuul/status.json you can observe it's hitting 304 constantly after the first request	02:36
*** bhunter71 has joined #openstack-infra		02:36
Krinkle	in the Network panel, status.json 304, status.json 304	02:36
fungi	Krinkle: this is the puppet erb template for our vhost on zuul.o.o http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/templates/zuul.vhost.erb	02:36
Krinkle	Interesting	02:38
Krinkle	https://github.com/openstack-infra/puppet-zuul/commit/3c21c650af9470d73c3362e8878bcfdb99ae2fe2	02:38
Krinkle	Well, I guess it's not working.	02:40
Krinkle	hitting http://zuul.openstack.org/status.json repeatedly always gives me 200 OK with a fresh Last-Modified.	02:40
Krinkle	even within 5 seconds	02:40
*** annegentle has joined #openstack-infra		02:40
fungi	i'm not surprised. a derivative of "if it's not tested it's broken" is "if nobody uses it, it's broken"	02:41
* fungi checks to see if mod_mem_cache is present		02:41
fungi	yep, it's enabled	02:42
fungi	both mod_cache and mod_mem_cache	02:42
Krinkle	And I assume mod_cache as well?	02:42
Krinkle	Hm..	02:42
Krinkle	intuitively though, this kind of caching should not be the definition, only the optimisation. E.g. Zuul itself already sends 304 headers that communicate with the browser about what and when to cache.	02:43
Krinkle	the proxy helps so that different users share a common cache as additional caching	02:43
fungi	cache.load, mem_cache.conf and mem_cache.load are all linked within the /etc/apache2/mods-enabled directory, so it's presumably not the conditional disabling it	02:43
Krinkle	I suspect Zuul might be senders the headers wrongly and as such Apache is complying with not caching those requests.	02:43
Krinkle	and Varnish (at Wikimedia) is interpreting them as allowing caching and doing that, but to the point that it never expires.	02:44
fungi	Krinkle: that sounds plausible. it's getting late for me, but jhesketh i think has a good grasp of this topic and it's his daytime	02:44
Krinkle	awesome :)	02:44
*** annegentle has quit IRC		02:45
Krinkle	So https://integration.wikimedia.org/zuul/status.json (zuul+varnish) caches too much, and http://zuul.openstack.org/status.json caches too little.	02:45
fungi	be also has a separate zuul instance running in production, so probably can verify things as well	02:45
*** yamamoto has quit IRC		02:47
*** Somay has joined #openstack-infra		02:48
*** yamamoto has joined #openstack-infra		02:48
*** yamamoto has quit IRC		02:50
jhesketh	Krinkle: heya	02:51
*** otter768 has joined #openstack-infra		02:51
mordred	I blame jhesketh	02:52
mordred	but that's mostly because he's awesome and can take it	02:52
jhesketh	mordred: you think that now... ;-)	02:53
jhesketh	Krinkle: so zuul only supplies a 'last modified' as best as I can tell.. any caching from there is up to apache/varnish/etc	02:55
*** crc32 has joined #openstack-infra		02:56
Krinkle	jhesketh: Relevant information: https://phabricator.wikimedia.org/T94796	02:57
Krinkle	jhesketh: Yeah, it should provide more headers, right?	02:58
jhesketh	Krinkle: what header(s) would you expect?	03:00
Krinkle	jhesketh: Well, I've worked a lot with caching for Wikipedia's caching layer two years ago. Afaik it is not valid HTTP to only send Last-Modified.	03:01
Krinkle	To expect caching, it needs more information	03:01
*** otter768 has quit IRC		03:01
*** asselin has quit IRC		03:01
Krinkle	Typically Expires is involved	03:04
Krinkle	And a Cache-control: public, max-age=XXX, s-maxage=XXX	03:04
Krinkle	Cache-control:public, max-age=2592000, s-maxage=2592000	03:05
Krinkle	Last-Modified:Fri, 27 Mar 2015 18:08:30 GMT	03:05
Krinkle	Expires:Sat, 02 May 2015 03:04:07 GMT	03:05
Krinkle	Date:Thu, 02 Apr 2015 03:05:13 GMT	03:05
Krinkle	That's an example of a resource last refreshed a couple days ago and expires in a month.	03:05
Krinkle	where browsers honour max-age, and proxies s-maxage (optional)	03:06
jhesketh	hmm, okay	03:07
jhesketh	I don't know much about caching, but I don't see any reason why zuul couldn't provide those details	03:07
*** sdake has quit IRC		03:10
*** baoli has quit IRC		03:10
jhesketh	Krinkle: have you tried adding them to the webapp?	03:10
Krinkle	It's a lot to install locally.	03:11
Krinkle	I'll give it a shot in a bit.	03:11
*** vinkman_ has quit IRC		03:11
Krinkle	jhesketh: Yeah, totally. I'll make a patch that adds them to Zuul and defaults to e.g. 1 minute caching	03:11
jhesketh	awesome :-)	03:11
jhesketh	curious to know if that helps	03:12
Krinkle	any dedicated proxies or configs like wikimedia and openstack's can extend that via the proxy if they want to cache longer	03:12
*** yamahata has joined #openstack-infra		03:12
*** vinkman has joined #openstack-infra		03:18
*** yamamoto has joined #openstack-infra		03:21
*** yamamoto has quit IRC		03:21
*** melwitt has quit IRC		03:21
*** btully has quit IRC		03:21
*** ghostpl_ has joined #openstack-infra		03:23
*** tiswanso has quit IRC		03:24
*** baoli has joined #openstack-infra		03:27
*** ghostpl_ has quit IRC		03:28
*** yamamoto has joined #openstack-infra		03:29
*** yamamoto has quit IRC		03:30
*** sarob has quit IRC		03:30
*** sarob has joined #openstack-infra		03:30
*** sarob has quit IRC		03:30
*** rm_work\|away is now known as rm_work		03:31
*** macjack1 has joined #openstack-infra		03:32
*** sarob has joined #openstack-infra		03:32
*** samueldmq has quit IRC		03:36
*** sarob has quit IRC		03:37
*** SumitNaiksatam has quit IRC		03:38
*** thingee has quit IRC		03:38
*** achanda has quit IRC		03:38
*** SumitNaiksatam has joined #openstack-infra		03:38
*** macjack1 has quit IRC		03:39
*** macjack has left #openstack-infra		03:39
*** SumitNaiksatam has quit IRC		03:40
*** annegentle has joined #openstack-infra		03:41
*** SumitNaiksatam has joined #openstack-infra		03:42
*** bhunter71 has quit IRC		03:43
*** vinkman has quit IRC		03:45
*** vinkman has joined #openstack-infra		03:46
*** annegentle has quit IRC		03:50
*** jamielennox is now known as jamielennox\|away		03:51
*** sputnik13 has joined #openstack-infra		03:52
*** jcoufal has joined #openstack-infra		03:57
*** yamamoto has joined #openstack-infra		03:57
*** woodster_ has quit IRC		04:00
*** sputnik13 has quit IRC		04:02
*** annegentle has joined #openstack-infra		04:02
*** harlowja_at_home has quit IRC		04:05
*** baoli has quit IRC		04:06
*** SumitNaiksatam has quit IRC		04:07
*** SumitNaiksatam has joined #openstack-infra		04:07
*** links has joined #openstack-infra		04:11
*** Bsony has joined #openstack-infra		04:13
*** salv-orlando has joined #openstack-infra		04:16
openstackgerrit	Timothy R. Chavez proposed openstack-infra/shade: Fix naming inconsistencies in rebuild_server tests https://review.openstack.org/169991	04:18
*** salv-orlando has quit IRC		04:20
*** ibiris_away is now known as ibiris		04:21
*** sputnik13 has joined #openstack-infra		04:24
*** sputnik13 has quit IRC		04:27
*** sputnik13 has joined #openstack-infra		04:28
*** SumitNaiksatam has quit IRC		04:30
*** tkelsey has joined #openstack-infra		04:31
*** rm_work is now known as rm_work\|away		04:32
*** dhritishikhar_ has joined #openstack-infra		04:32
*** prmtl has joined #openstack-infra		04:32
*** achanda has joined #openstack-infra		04:32
*** SumitNaiksatam has joined #openstack-infra		04:34
*** woodster_ has joined #openstack-infra		04:35
*** tkelsey has quit IRC		04:35
*** ibiris is now known as ibiris_away		04:36
*** Somay has quit IRC		04:40
*** sputnik13 has quit IRC		04:41
*** koolhead17 has joined #openstack-infra		04:43
*** sputnik13 has joined #openstack-infra		04:45
*** BharatK has joined #openstack-infra		04:48
*** sputnik13 has quit IRC		04:52
*** ibiris_away is now known as ibiris		04:53
*** dhritishikhar_ has quit IRC		04:53
*** yfried has joined #openstack-infra		04:58
*** sputnik13 has joined #openstack-infra		05:00
*** otter768 has joined #openstack-infra		05:02
*** btully has joined #openstack-infra		05:06
*** otter768 has quit IRC		05:06
*** Bsony has quit IRC		05:11
*** sputnik13 has quit IRC		05:19
*** prmtl has quit IRC		05:24
*** armax has quit IRC		05:33
*** hichihara has quit IRC		05:34
*** annegentle has quit IRC		05:39
*** jcoufal has quit IRC		05:39
*** annegentle has joined #openstack-infra		05:40
*** annegentle has quit IRC		05:45
*** unicell has joined #openstack-infra		05:54
*** achanda has quit IRC		05:58
*** dims_ has joined #openstack-infra		06:00
openstackgerrit	OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/169624	06:01
*** tnovacik has joined #openstack-infra		06:02
*** mrmartin has joined #openstack-infra		06:04
*** achanda has joined #openstack-infra		06:06
*** dims_ has quit IRC		06:06
*** ildikov_ has quit IRC		06:12
*** prmtl has joined #openstack-infra		06:17
*** jcoufal has joined #openstack-infra		06:20
*** ianw has quit IRC		06:24
*** ianw has joined #openstack-infra		06:24
*** crc32 has quit IRC		06:28
openstackgerrit	Timothy R. Chavez proposed openstack-infra/shade: Add some unit test for create_server https://review.openstack.org/170019	06:38
*** woodster_ has quit IRC		06:40
*** scheuran has joined #openstack-infra		06:40
*** annegentle has joined #openstack-infra		06:41
*** tkelsey has joined #openstack-infra		06:41
*** sergsh has joined #openstack-infra		06:42
*** tkelsey has quit IRC		06:42
*** achanda has quit IRC		06:43
*** annegentle has quit IRC		06:46
*** achanda has joined #openstack-infra		06:46
*** rm_work\|away is now known as rm_work		06:51
*** afazekas has joined #openstack-infra		06:54
*** chlong has quit IRC		06:55
*** mrda is now known as mrda-away		06:57
*** mhu has quit IRC		06:58
*** jlibosva has joined #openstack-infra		06:58
*** doude has joined #openstack-infra		06:59
*** ildikov_ has joined #openstack-infra		07:04
*** achanda has quit IRC		07:07
*** arxcruz has joined #openstack-infra		07:08
*** dtantsur\|afk is now known as dtantsur		07:09
*** Bsony has joined #openstack-infra		07:10
*** koolhead17 has quit IRC		07:12
*** jistr has joined #openstack-infra		07:13
*** salv-orlando has joined #openstack-infra		07:13
*** koolhead_ has joined #openstack-infra		07:14
*** Longgeek_ has quit IRC		07:17
*** Longgeek has joined #openstack-infra		07:17
*** boris-42 has quit IRC		07:18
*** koolhead_ has quit IRC		07:18
*** salv-orlando has quit IRC		07:21
*** markus_z has joined #openstack-infra		07:22
*** koolhead17 has joined #openstack-infra		07:27
*** achanda has joined #openstack-infra		07:29
*** stevemar has quit IRC		07:30
*** medieval1 has quit IRC		07:33
*** mhu has joined #openstack-infra		07:35
*** salv-orlando has joined #openstack-infra		07:41
*** koolhead17 has quit IRC		07:42
*** annegentle has joined #openstack-infra		07:42
*** tnovacik has quit IRC		07:44
*** tqtran_afk has quit IRC		07:46
*** annegentle has quit IRC		07:47
*** Hal has joined #openstack-infra		07:50
*** Hal is now known as Guest6393		07:51
*** HeOS has quit IRC		07:57
*** MaxV has joined #openstack-infra		07:58
*** Guest6393 has quit IRC		07:58
*** Bsony has quit IRC		07:58
*** mpaolino has joined #openstack-infra		07:59
*** mpaolino has quit IRC		08:00
*** mrmartin has quit IRC		08:01
*** btully has quit IRC		08:01
*** mpaolino has joined #openstack-infra		08:02
*** mrmartin has joined #openstack-infra		08:02
*** ildikov_ has quit IRC		08:02
*** devvesa has joined #openstack-infra		08:04
*** hashar has joined #openstack-infra		08:06
*** achanda has quit IRC		08:14
*** andrey-mp has joined #openstack-infra		08:15
andrey-mp	Can anyone see to https://review.openstack.org/#/c/166648/ ?	08:15
*** ildikov_ has joined #openstack-infra		08:16
*** Hal has joined #openstack-infra		08:16
*** wicope has joined #openstack-infra		08:17
*** Hal is now known as Guest20173		08:17
*** markvoelker has quit IRC		08:18
*** e0ne has joined #openstack-infra		08:18
*** e0ne has quit IRC		08:20
*** andreykurilin_ has joined #openstack-infra		08:21
*** ssam2 has joined #openstack-infra		08:25
*** tnovacik has joined #openstack-infra		08:26
*** pblaho has joined #openstack-infra		08:26
*** andreykurilin_ has quit IRC		08:27
*** andreykurilin_ has joined #openstack-infra		08:27
*** achanda has joined #openstack-infra		08:28
*** andrey-mp has quit IRC		08:29
*** tnovacik has quit IRC		08:31
*** medieval1 has joined #openstack-infra		08:34
*** zz_johnthetubagu is now known as johnthetubaguy		08:35
*** yamahata has quit IRC		08:37
*** medieval1 has quit IRC		08:39
*** mhu has quit IRC		08:40
*** achanda has quit IRC		08:40
*** yolanda has quit IRC		08:40
*** _shaps_ has joined #openstack-infra		08:41
*** aswadr has joined #openstack-infra		08:41
*** achanda has joined #openstack-infra		08:41
*** pelix has joined #openstack-infra		08:43
*** annegentle has joined #openstack-infra		08:43
*** annegentle has quit IRC		08:48
*** markvoelker has joined #openstack-infra		08:49
*** dims_ has joined #openstack-infra		08:50
*** doude has quit IRC		08:52
*** andreykurilin_ has quit IRC		08:52
*** andreykurilin_ has joined #openstack-infra		08:53
*** yolanda has joined #openstack-infra		08:53
*** yolanda_ has joined #openstack-infra		08:53
*** yolanda_ has quit IRC		08:54
*** markvoelker has quit IRC		08:55
*** dims_ has quit IRC		08:55
*** andreykurilin_ has quit IRC		08:58
*** Bsony has joined #openstack-infra		08:58
*** dtantsur is now known as dtantsur\|brb		08:58
*** masoz has joined #openstack-infra		08:59
*** andreykurilin_ has joined #openstack-infra		08:59
*** doude has joined #openstack-infra		08:59
*** shardy has joined #openstack-infra		09:00
*** mhu has joined #openstack-infra		09:00
*** Bsony has quit IRC		09:03
*** andreykurilin_ has quit IRC		09:04
*** otter768 has joined #openstack-infra		09:04
*** amotoki has joined #openstack-infra		09:05
*** derekh has joined #openstack-infra		09:05
*** amotoki has quit IRC		09:05
*** amotoki has joined #openstack-infra		09:06
*** otter768 has quit IRC		09:09
*** HeOS has joined #openstack-infra		09:10
*** YorikSar has quit IRC		09:11
*** achanda has quit IRC		09:12
*** e0ne has joined #openstack-infra		09:13
*** ibiris is now known as ibiris_away		09:20
*** jistr has quit IRC		09:26
*** ihrachyshka has joined #openstack-infra		09:27
*** e0ne is now known as e0ne_		09:30
*** amotoki has quit IRC		09:31
*** achanda has joined #openstack-infra		09:32
*** BharatK has quit IRC		09:33
*** e0ne_ is now known as e0ne		09:35
*** medieval1 has joined #openstack-infra		09:35
openstackgerrit	Timo Tijhof proposed openstack-infra/zuul: Status: Support older versions not having item.live https://review.openstack.org/170057	09:37
openstackgerrit	Timo Tijhof proposed openstack-infra/zuul: Status: Use vertical-align for progress bar instead of margin https://review.openstack.org/170058	09:37
*** ibiris_away is now known as ibiris		09:38
openstackgerrit	Timo Tijhof proposed openstack-infra/zuul: Status: Use vertical-align for progress bar instead of margin https://review.openstack.org/170058	09:38
openstackgerrit	Timo Tijhof proposed openstack-infra/zuul: Status: Support older versions not having item.live https://review.openstack.org/170057	09:38
openstackgerrit	Timo Tijhof proposed openstack-infra/zuul: Status: Increase width of change-progress-row-left https://review.openstack.org/170059	09:38
*** medieval1 has quit IRC		09:40
*** achanda has quit IRC		09:41
*** ghostpl_ has joined #openstack-infra		09:42
*** skraynev has quit IRC		09:42
*** skraynev has joined #openstack-infra		09:43
*** annegentle has joined #openstack-infra		09:43
*** jistr has joined #openstack-infra		09:44
Krinkle	I'm pretty sure that gate-zuul-python26 is a false failure. It seems to be failing on all my patches.	09:44
Krinkle	I'm not touching any backend files.	09:44
*** YorikSar has joined #openstack-infra		09:45
*** annegentle has quit IRC		09:48
*** ildikov_ has quit IRC		09:50
*** amotoki has joined #openstack-infra		09:52
*** dims_ has joined #openstack-infra		09:52
*** dims_ has quit IRC		09:52
*** tnovacik has joined #openstack-infra		09:56
*** slagle has quit IRC		09:56
*** dims__ has joined #openstack-infra		09:56
*** slagle has joined #openstack-infra		09:57
*** sarob has joined #openstack-infra		10:00
*** _nadya_ has joined #openstack-infra		10:03
*** dims__ has quit IRC		10:03
*** yfried is now known as yfried\|afk		10:03
*** jesusaurus has quit IRC		10:03
*** ptoohill has quit IRC		10:03
*** ptoohill has joined #openstack-infra		10:04
*** jesusaurus has joined #openstack-infra		10:04
*** sarob has quit IRC		10:05
*** ildikov_ has joined #openstack-infra		10:05
*** dims has joined #openstack-infra		10:09
*** amotoki has quit IRC		10:12
*** wicope has quit IRC		10:13
*** alaski has quit IRC		10:13
*** alaski has joined #openstack-infra		10:14
*** koolhead17 has joined #openstack-infra		10:17
*** koolhead17 has joined #openstack-infra		10:17
*** wicope has joined #openstack-infra		10:17
*** yfried\|afk is now known as yfried		10:18
*** yamamoto has quit IRC		10:19
*** btully has joined #openstack-infra		10:21
*** dtantsur\|brb is now known as dtantsur		10:22
*** btully has quit IRC		10:26
*** e0ne is now known as e0ne_		10:29
*** cdent has joined #openstack-infra		10:31
*** devvesa has quit IRC		10:32
*** marun has joined #openstack-infra		10:33
*** devvesa has joined #openstack-infra		10:35
*** medieval1 has joined #openstack-infra		10:36
*** pc_m has joined #openstack-infra		10:37
*** e0ne_ is now known as e0ne		10:37
*** marun has quit IRC		10:37
*** katyafervent is now known as katyafervent_awa		10:39
*** medieval1 has quit IRC		10:40
*** katyafervent_awa is now known as katyafervent		10:43
*** annegentle has joined #openstack-infra		10:44
*** tnovacik has quit IRC		10:47
openstackgerrit	Sergey Kraynev proposed openstack/requirements: Bump last version of ceilometer-client https://review.openstack.org/167527	10:47
*** annegentle has quit IRC		10:49
openstackgerrit	Sergey Kraynev proposed openstack/requirements: Bump last version of ceilometer-client https://review.openstack.org/167527	10:51
*** markvoelker has joined #openstack-infra		10:52
*** johnthetubaguy is now known as zz_johnthetubagu		10:54
pelix	is there a way to have the console logs for all the jobs run to check a patch appear at http://logs.openstack.org/46/164446/4/check/? It's missing the output of the pep8, docs, and coverage builds for JJB, however they do exist and I can find the docs build output http://logs.openstack.org/46/164446/4/check/gate-jenkins-job-builder-docs/71a6a98 but it's pretty inconvenient to have to hand construct a url to check the build output as it also	10:56
lifeless	man, I'm over these please answer my questions for my phd mails.	10:57
*** markvoelker has quit IRC		10:57
grue_pm	pelix you got truncated - there's a character limit in irc	10:58
*** samueldmq has joined #openstack-infra		10:58
pelix	oops. ok looking at http://logs.openstack.org/46/164446/4/check/ for a given patch I can only see links for the output of three of the jobs	10:58
pelix	It's missing the output of the pep8, docs, and coverage builds for JJB, however they do exist and I can hand construct urls to find them	10:59
pelix	such as http://logs.openstack.org/46/164446/4/check/gate-jenkins-job-builder-docs/71a6a98 but it's pretty inconvenient	10:59
pelix	any way to have all the logs that are actually present under http://logs.openstack.org/46/164446/4/check/ be listed there?	10:59
pelix	Since we have a link checker running as part of the docs build it's useful to be able to check the output	11:02
*** otter768 has joined #openstack-infra		11:05
*** otter768 has quit IRC		11:09
openstackgerrit	Timo Tijhof proposed openstack-infra/zuul: web: Allow clients (browsers, proxies) to cache status.json https://review.openstack.org/170081	11:11
*** marcusvrn has joined #openstack-infra		11:12
*** _nadya_ has quit IRC		11:14
*** marcusvrn1 has quit IRC		11:15
*** e0ne is now known as e0ne_		11:16
*** tnovacik has joined #openstack-infra		11:17
*** yfried is now known as yfried\|afk		11:19
*** aswadr has quit IRC		11:21
*** dhritishikhar_ has joined #openstack-infra		11:22
*** e0ne_ has quit IRC		11:27
*** ujuc has joined #openstack-infra		11:27
*** medieval1 has joined #openstack-infra		11:37
pleia2	clarkb: re: nested kvm, apparently it's gotten much more stable since I looked at it in late 2013, but given our throughput I don't think "more stable" is sufficient to make it a wise decision still, even if our cloud hosts let us do it - which would be a tough sell, since the kernel panics are on the host, messy	11:37
* pleia2 back to vacation		11:37
*** koolhead17 has quit IRC		11:37
*** ldnunes has joined #openstack-infra		11:40
*** yfried\|afk has quit IRC		11:41
*** yfried\|afk has joined #openstack-infra		11:41
*** medieval1 has quit IRC		11:42
*** alexpilotti has joined #openstack-infra		11:43
*** annegentle has joined #openstack-infra		11:45
*** sarob has joined #openstack-infra		11:48
*** annegentle has quit IRC		11:50
*** kir_ has quit IRC		11:51
*** sarob has quit IRC		11:53
*** weshay has joined #openstack-infra		11:53
*** markvoelker has joined #openstack-infra		11:53
*** sputnik13 has joined #openstack-infra		11:55
openstackgerrit	Sean Dague proposed openstack-infra/devstack-gate: Temporarily add trove plugin configuration https://review.openstack.org/167412	11:58
*** markvoelker has quit IRC		11:58
*** subscope has quit IRC		11:59
*** ildikov_ has quit IRC		12:00
*** boris-42 has joined #openstack-infra		12:00
*** ildikov_ has joined #openstack-infra		12:01
*** dprince has joined #openstack-infra		12:02
*** kgiusti has joined #openstack-infra		12:03
*** dkranz has quit IRC		12:05
*** kgiusti has left #openstack-infra		12:05
*** sputnik13 has quit IRC		12:07
*** mpaolino has quit IRC		12:07
*** subscope has joined #openstack-infra		12:15
*** sdake has joined #openstack-infra		12:15
*** bswartz has quit IRC		12:16
*** rfolco has joined #openstack-infra		12:16
openstackgerrit	Cyril Roelandt proposed openstack-infra/project-config: saharaclient: enable Python3 jobs https://review.openstack.org/168899	12:16
*** dprince has quit IRC		12:19
*** dprince has joined #openstack-infra		12:19
*** dhritishikhar_ has quit IRC		12:21
*** claudiub has joined #openstack-infra		12:23
*** doug-fish has left #openstack-infra		12:23
*** dkliban_afk is now known as dkliban		12:24
*** sdake_ has joined #openstack-infra		12:25
*** tnovacik has quit IRC		12:25
*** dims has quit IRC		12:25
*** yamamoto has joined #openstack-infra		12:26
*** tnovacik has joined #openstack-infra		12:26
*** dims has joined #openstack-infra		12:26
*** aysyd has joined #openstack-infra		12:27
*** woodster_ has joined #openstack-infra		12:27
*** sdake has quit IRC		12:28
*** markvoelker has joined #openstack-infra		12:29
*** sputnik13 has joined #openstack-infra		12:30
*** tnovacik has quit IRC		12:30
*** bknudson has quit IRC		12:32
*** annegentle has joined #openstack-infra		12:32
*** gordc has joined #openstack-infra		12:33
*** ssam2 has quit IRC		12:33
*** annegentle has quit IRC		12:33
*** zz_johnthetubagu is now known as johnthetubaguy		12:35
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Fix docs nit - make it clear the arg is a string https://review.openstack.org/170105	12:37
*** medieval1 has joined #openstack-infra		12:37
*** smccully has joined #openstack-infra		12:38
*** annegentle has joined #openstack-infra		12:39
sdague	jeblair: so... Depends-On is breaking down with dependent patches being updated. The latest version of the trove patch is still checking against version #2 of the devstack-gate patch - http://logs.openstack.org/34/168134/3/check/gate-tempest-dsvm-trove/da59a83/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-04-02_12_08_42_662	12:40
sdague	version #5 is needed	12:40
mordred	sdague: hrm. I never would have expected that behavior	12:41
sdague	which also is troubling for using it cross project	12:41
sdague	yep	12:41
*** e0ne has joined #openstack-infra		12:41
sdague	I don't have any idea why	12:41
sdague	but I've been banging my head on why the localrc line wasn't showing up	12:41
sdague	and then realized it wasn't using the right patch revision	12:42
*** medieval1 has quit IRC		12:42
*** mpaolino has joined #openstack-infra		12:42
*** esker has quit IRC		12:43
*** esker has joined #openstack-infra		12:43
*** esker has quit IRC		12:43
*** esker has joined #openstack-infra		12:44
openstackgerrit	Kiall Mac Innes proposed openstack-infra/project-config: Add a PgSQL Designate DSVM Job https://review.openstack.org/170109	12:44
*** peristeri has joined #openstack-infra		12:45
mordred	Kiall: ewww	12:45
Kiall	mordred: I know	12:45
openstackgerrit	Merged openstack-infra/shade: Fix naming inconsistencies in rebuild_server tests https://review.openstack.org/169991	12:45
Kiall	but pgSQL keeps breaking, and we have a review up that finally fixes it again	12:45
mordred	Kiall: you could go the other way and just stop supporting it ...	12:45
sdague	Kiall: or test at a lower level	12:45
*** mjturek1 has joined #openstack-infra		12:46
Kiall	sdague: yea, I have every intention of dropping SQLite support and moving the py27 jobs to use mysql/postgres next cycle.. but just didn't have the time to do this this go around :(	12:46
Kiall	mordred: re dropping postgres? We could, but if there's no real need to other than we need write better migrations, it seems fairly harmless to support it	12:47
*** ssam2 has joined #openstack-infra		12:48
mordred	Kiall: indeed. it's more my general "openstack should stop trying to support multiple databases" bandwagon - but I have not yet gotten enough traction on that	12:48
Kiall	:)	12:49
*** esker has quit IRC		12:49
sdague	Kiall: if it's just migrations, those are already up and running for unit tests	12:51
sdague	we test all the migrations on both dbs in nova during unit test runs	12:51
sdague	running a full devstack is way more than you should do for validating that	12:52
Kiall	sdague: yea, you mentioned that in Sunnyvalue from memory .. We just didn't get time to go figure it out this cycle, and I wanted to have something that would keep pgSQL working for stable/kilo once cut, and drop it from master after we get it going in the tests	12:52
*** ldnunes has quit IRC		12:52
*** mwagner_lap has quit IRC		12:53
Kiall	Sunnyvalue? lol.. Sunnyvale	12:53
sdague	Kiall: yeh, my experience is if you put that test in there, you'll never get rid of it	12:53
sdague	seeg: nova	12:53
seeg	?	12:53
sdague	sorry	12:53
Kiall	I think he meant, "see: nova" ;)	12:53
sdague	seeg:	12:53
seeg	:)	12:54
sdague	yeh, apparently xchat won't let me type 'see :'	12:54
sdague	without the space	12:54
seeg	bugger :)	12:54
sdague	without auto correcting	12:54
seeg	perils of autocompletion :)	12:55
mordred	sdague: maybe we should just accept that as a new spelling	12:55
sdague	heh	12:55
Kiall	Here's a possible way to avoid that (the never get rid of it, not the autocorrect) - cut rc1, apply the new job only to {stable,proposed/kilo} - that way, it never lands on master in the first place?	12:56
*** e0ne is now known as e0ne_		12:58
openstackgerrit	Merged openstack-infra/shade: Allow for passing cache class in as a parameter https://review.openstack.org/162148	12:59
*** bswartz has joined #openstack-infra		12:59
*** jerryz has joined #openstack-infra		13:00
*** dkranz has joined #openstack-infra		13:01
openstackgerrit	Merged openstack-infra/shade: Namespace caching per cloud https://review.openstack.org/168615	13:01
openstackgerrit	Merged openstack-infra/shade: Add task management framework https://review.openstack.org/168664	13:01
*** chlong has joined #openstack-infra		13:01
*** lpetrut has joined #openstack-infra		13:02
*** ssam2 has quit IRC		13:03
*** masoz has quit IRC		13:03
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Migrate API calls to task management https://review.openstack.org/168665	13:04
*** dkranz has quit IRC		13:05
*** otter768 has joined #openstack-infra		13:06
*** scheuran has quit IRC		13:06
*** e0ne_ has quit IRC		13:08
*** eharney has quit IRC		13:08
*** annegentle has quit IRC		13:09
*** ldnunes has joined #openstack-infra		13:10
*** otter768 has quit IRC		13:11
*** koolhead17 has joined #openstack-infra		13:12
*** bknudson has joined #openstack-infra		13:13
*** tonytan4ever has joined #openstack-infra		13:14
*** ssam2 has joined #openstack-infra		13:15
*** subscope has quit IRC		13:16
*** yamamoto has quit IRC		13:16
*** prmtl has quit IRC		13:16
*** subscope has joined #openstack-infra		13:16
*** dustins has joined #openstack-infra		13:17
*** dkranz has joined #openstack-infra		13:17
*** yamamoto has joined #openstack-infra		13:19
*** dboik_ has quit IRC		13:19
*** annegentle has joined #openstack-infra		13:20
sdague	mordred: http://dl.dropbox.com/u/6514884/screenshot_222.png yeh definitely picking 167412,2	13:21
openstackgerrit	Merged openstack-infra/shade: Create swift container if it does not exist https://review.openstack.org/167823	13:21
sdague	so that's in no way confusing	13:21
*** _nadya_ has joined #openstack-infra		13:22
openstackgerrit	Timothy R. Chavez proposed openstack-infra/shade: Add some unit test for create_server https://review.openstack.org/170019	13:22
sdague	I guess I'll leave this broken until jeblair comes around so he can figure out why	13:22
*** annegentle has quit IRC		13:23
*** annegentle has joined #openstack-infra		13:23
*** kgiusti has joined #openstack-infra		13:24
*** bhunter71 has joined #openstack-infra		13:26
*** mriedem_away is now known as mriedem		13:28
*** tiswanso has joined #openstack-infra		13:28
*** vlaza has joined #openstack-infra		13:30
*** Krinkle is now known as Krinkle\|detached		13:30
*** miqui has joined #openstack-infra		13:31
*** ZZelle__ has quit IRC		13:31
*** e0ne has joined #openstack-infra		13:32
*** baoli has joined #openstack-infra		13:33
*** mattfarina has joined #openstack-infra		13:34
*** sarob has joined #openstack-infra		13:37
*** aswadr has joined #openstack-infra		13:37
*** zigo has quit IRC		13:38
*** dboik has joined #openstack-infra		13:38
*** medieval1 has joined #openstack-infra		13:38
*** ChuckC has quit IRC		13:38
*** koolhead17 has quit IRC		13:39
*** sdake has joined #openstack-infra		13:39
*** ldnunes has quit IRC		13:40
*** zigo_ is now known as zigo		13:41
openstackgerrit	Russell Bryant proposed openstack-infra/reviewstats: Document installation from repository https://review.openstack.org/169728	13:41
*** koolhead17 has joined #openstack-infra		13:41
*** sarob has quit IRC		13:42
*** erlon has joined #openstack-infra		13:42
*** sdake_ has quit IRC		13:43
*** medieval1 has quit IRC		13:43
*** signed8bit has joined #openstack-infra		13:44
*** wenlock has joined #openstack-infra		13:45
*** zz_dimtruck is now known as dimtruck		13:46
*** doug-fish has joined #openstack-infra		13:46
openstackgerrit	Merged openstack-infra/shade: Migrate API calls to task management https://review.openstack.org/168665	13:46
*** chlong has quit IRC		13:47
*** ildikov_ has quit IRC		13:47
*** eharney has joined #openstack-infra		13:47
*** koolhead_ has joined #openstack-infra		13:47
*** koolhead17 has quit IRC		13:48
*** gordc has quit IRC		13:48
*** radez_g0n3 is now known as radez		13:49
*** yamamoto has quit IRC		13:49
openstackgerrit	Jeremy Stanley proposed openstack-infra/system-config: Rename summit server to design-summit-prep https://review.openstack.org/170132	13:51
*** sdake_ has joined #openstack-infra		13:51
openstackgerrit	Thierry Carrez proposed openstack-infra/system-config: Update ttx public SSH key https://review.openstack.org/170136	13:53
*** sdake has quit IRC		13:55
*** amitgandhinz has joined #openstack-infra		13:55
*** sabeen has joined #openstack-infra		13:57
*** Guest20173 has quit IRC		13:57
*** subscope has quit IRC		13:57
*** ildikov_ has joined #openstack-infra		13:59
*** sigmavirus24_awa is now known as sigmavirus24		14:01
fungi	sdague: jeblair: mordred: i have a feeling it's related to https://review.openstack.org/168146 having an outdated git dependency on 167412,2	14:02
*** sputnik13 has quit IRC		14:03
sdague	fungi: except that's not in the change path	14:03
*** Bsony has joined #openstack-infra		14:04
*** gordc has joined #openstack-infra		14:04
*** rbradfor has joined #openstack-infra		14:05
fungi	sdague: i know it's not, but it's the only connection i find to that older patchset. when zuul loads a change it also loads any changes for the same repo which depend on it, and i wonder if the bug is somewhere in that code path	14:05
sdague	fungi: could be	14:05
*** btully has joined #openstack-infra		14:05
*** mwagner_lap has joined #openstack-infra		14:07
fungi	if i get a sec i'll dig up the relevant debug logs	14:07
*** koolhead_ has quit IRC		14:07
openstackgerrit	Merged openstack-infra/reviewstats: Document installation from repository https://review.openstack.org/169728	14:09
*** yfried\|afk is now known as yfried		14:10
*** garyh has joined #openstack-infra		14:10
openstackgerrit	Merged openstack-infra/reviewstats: Sync Swift core group https://review.openstack.org/167549	14:10
openstackgerrit	Merged openstack-infra/reviewstats: Add --server option to openapproved script https://review.openstack.org/168067	14:11
*** Bsony has quit IRC		14:11
openstackgerrit	Merged openstack-infra/reviewstats: Add GERRIT_KEY and GERRIT_SERVER options in scripts https://review.openstack.org/168073	14:12
anteaya	clarkb: on https://review.openstack.org/#/c/167243/ I was waiting on the governance patch to merge, I personally don't like to +A patches until their dependencies have merged because you can't tell from the dependency that there is a patch that depends on it, so if the dependency is edited prior to it merging there is no way to trace back to ensure the patch that depends on it is also accurate, and	14:12
anteaya	you might not catch that until after both have merged, and in project creation that is more work than just a revert	14:12
anteaya	jeblair: thanks for evaluating the etherpad	14:13
openstackgerrit	Merged openstack-infra/reviewstats: Update docs and security-docs teams https://review.openstack.org/168290	14:13
anteaya	jeblair: yes I have concerns that the new direction makes my work harder	14:13
*** ChuckC has joined #openstack-infra		14:13
anteaya	jeblair: mostly in that I have zero idea what to do now	14:13
*** subscope has joined #openstack-infra		14:15
openstackgerrit	Sean Dague proposed openstack-infra/project-config: Adjust regression exceptions for Nova Cells V1 job https://review.openstack.org/166396	14:16
*** scheuran has joined #openstack-infra		14:16
*** xyang1 has joined #openstack-infra		14:17
fungi	anteaya: also with the current behavior around depends-on relationships, pre-approving the depending change won't trigger anything in zuul, so you still end up having to revisit it to reset or add an approval anyway before it will get picked up for gating	14:17
fungi	so i agree there's not a lot of point to approving them when the change they depend on in another repo hasn't been approved yet	14:18
anteaya	true, and thanks for stating you share my perspective	14:18
anteaya	though I belive zaro is working on some code to fix this situation so that approved patches with depends-on get picked up and merged automatically	14:19
anteaya	I have no idea on the status of that work	14:19
*** xyang1 has quit IRC		14:19
anteaya	oh and I'm currently not -1'ing much anyway right now	14:20
*** yfried is now known as yfried\|afk		14:20
*** medieval1 has joined #openstack-infra		14:21
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Skip passing in timeout to glance if it's not set https://review.openstack.org/169803	14:21
*** koolhead17 has joined #openstack-infra		14:23
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Add delete_image call https://review.openstack.org/169783	14:25
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Poll on the actual image showing up https://review.openstack.org/169804	14:25
*** dvorak is now known as clayton		14:25
*** eharney has quit IRC		14:26
*** links has quit IRC		14:26
*** stevemar has joined #openstack-infra		14:26
*** sarob has joined #openstack-infra		14:28
mpaolino	sdague, I'm planning to release a new, plug-in based, version of https://review.openstack.org/#/c/108714/ as you suggested	14:30
*** Bsony has joined #openstack-infra		14:30
mpaolino	but I'm quite new to external plugins, can you (or anybody else) point me to an example of such a plugin?	14:31
*** koolhead17 has quit IRC		14:31
mpaolino	I would have a few questions: shall I create a new change-id?	14:31
mpaolino	sorry, I messed up with copy paste :P	14:31
*** kaisers has quit IRC		14:34
*** medieval1 has quit IRC		14:34
*** Adri2000 has quit IRC		14:36
*** yfried\|afk is now known as yfried		14:37
*** ajo has joined #openstack-infra		14:37
openstackgerrit	Jeremy Stanley proposed openstack-infra/system-config: Rename summit server to design-summit-prep https://review.openstack.org/170132	14:38
*** carl_baldwin has joined #openstack-infra		14:40
*** sarob has quit IRC		14:40
*** sarob has joined #openstack-infra		14:41
*** Adri2000 has joined #openstack-infra		14:42
*** ZZelle has joined #openstack-infra		14:43
*** ldnunes has joined #openstack-infra		14:43
*** adalbas has joined #openstack-infra		14:43
*** eharney has joined #openstack-infra		14:44
*** scheuran has quit IRC		14:45
*** armax has joined #openstack-infra		14:49
*** deepakcs has joined #openstack-infra		14:49
sdague	mpaolino: see - https://github.com/stackforge/devstack-plugin-glusterfs	14:50
mpaolino	thanks	14:51
*** ildikov_ has quit IRC		14:53
*** vlaza has left #openstack-infra		14:54
mpaolino	so basically the external plugin has to be located to https://github.com/stackforge/plugin_name?	14:54
openstackgerrit	James E. Blair proposed openstack-infra/zuul: Add commit needed-by https://review.openstack.org/170155	14:54
mpaolino	or can it be located to a whatever github repository?	14:54
*** Bsony has quit IRC		14:55
jeblair	fungi, sdague, mordred: https://review.openstack.org/170155 i think that should fix 2 problems at once. needs testing.	14:55
mordred	jeblair: looking	14:55
fungi	jeblair: yep--that looks like it	14:56
jeblair	anteaya: ^ that also fixes the thing you were just talking about.	14:56
jeblair	and i did not know zaro was working on that	14:56
*** ildikov_ has joined #openstack-infra		14:56
*** Bsony has joined #openstack-infra		14:57
*** xyang1 has joined #openstack-infra		14:58
*** signed8bit is now known as signed8bit_ZZZzz		14:58
*** BobH has joined #openstack-infra		14:59
*** doug-fish has left #openstack-infra		14:59
jeblair	sdague: to muddle through now, i think we will need to push updated patchsets of all the changes in the series in order to get zuul to refresh the cache	14:59
*** dangers_away is now known as dangers		14:59
fungi	jeblair: see a formatting issue, comment inline	15:00
fungi	oh, it also failed jenkins jobs because of that	15:00
*** signed8bit_ZZZzz is now known as signed8bit		15:00
*** baoli has quit IRC		15:01
sdague	jeblair: ok, if you want to, that would be cool, I'm chasing a couple of other things	15:01
sdague	I think functionally the patch series is right now	15:01
*** baoli has joined #openstack-infra		15:02
*** dboik_ has joined #openstack-infra		15:03
jeblair	sdague: i'll probably focus on getting the zuul fix in then since i think this is a fairly big violation of what we expect from it	15:03
*** dkranz has quit IRC		15:04
*** MarkAtwood has joined #openstack-infra		15:04
openstackgerrit	James E. Blair proposed openstack-infra/zuul: Add commit needed-by https://review.openstack.org/170155	15:05
sdague	jeblair: coolio	15:05
*** dboik has quit IRC		15:06
*** otter768 has joined #openstack-infra		15:07
*** nelsnelson has joined #openstack-infra		15:08
openstackgerrit	Merged openstack-infra/shade: Skip passing in timeout to glance if it's not set https://review.openstack.org/169803	15:08
*** xyang1 has quit IRC		15:11
*** otter768 has quit IRC		15:11
*** yamamoto has joined #openstack-infra		15:12
*** Bsony has quit IRC		15:13
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Fix docs nit - make it clear the arg is a string https://review.openstack.org/170105	15:14
*** marun has joined #openstack-infra		15:14
openstackgerrit	Merged openstack-infra/shade: Add delete_image call https://review.openstack.org/169783	15:14
openstackgerrit	Dan Smith proposed openstack-infra/devstack-gate: Do not merge me - I am for testing. https://review.openstack.org/169907	15:14
*** jaypipes has joined #openstack-infra		15:16
*** ajo has quit IRC		15:16
openstackgerrit	Merged openstack-infra/shade: Poll on the actual image showing up https://review.openstack.org/169804	15:17
*** dkranz has joined #openstack-infra		15:17
*** radez is now known as radez_g0n3		15:17
*** yamahata has joined #openstack-infra		15:17
*** deepakcs has quit IRC		15:18
*** alexpilotti_ has joined #openstack-infra		15:19
*** hashar has quit IRC		15:22
*** alexpilotti has quit IRC		15:22
*** alexpilotti_ is now known as alexpilotti		15:22
*** sabeen has quit IRC		15:22
*** jlibosva has quit IRC		15:23
*** sabeen has joined #openstack-infra		15:23
*** Hal has joined #openstack-infra		15:24
*** thedodd has joined #openstack-infra		15:24
*** Hal is now known as Guest63783		15:24
*** dannywilson has joined #openstack-infra		15:24
*** ajmiller has joined #openstack-infra		15:24
*** jlibosva has joined #openstack-infra		15:24
*** dannywil_ has joined #openstack-infra		15:29
*** dannywilson has quit IRC		15:29
*** markus_z has quit IRC		15:30
*** garyh has quit IRC		15:33
*** sputnik13 has joined #openstack-infra		15:35
*** medieval1 has joined #openstack-infra		15:35
*** SumitNaiksatam has quit IRC		15:37
*** garyh has joined #openstack-infra		15:37
*** sdake has joined #openstack-infra		15:38
*** medieval1 has quit IRC		15:40
*** esker has joined #openstack-infra		15:40
*** medieval1 has joined #openstack-infra		15:40
*** MaxV has quit IRC		15:41
*** sdake_ has quit IRC		15:42
*** medieval1 has quit IRC		15:43
*** spredzy is now known as spredzy\|afk		15:47
*** yfried is now known as yfried\|afk		15:49
*** sergsh has quit IRC		15:51
*** aswadr has quit IRC		15:52
*** e0ne is now known as e0ne_		15:52
*** packet has joined #openstack-infra		15:52
mordred	jeblair: so - sometimes uploads to swift fail. I'm thinking about adding an iterate_timeout interface so that shade keeps retrying the put up until the timeout value. what do you think about that?	15:53
*** pblaho has quit IRC		15:53
*** emagana has joined #openstack-infra		15:53
mordred	oh!	15:53
mordred	it's not an image upload that's failing	15:53
mordred	it's a failure on setting the associated metadata	15:54
jeblair	mordred: mixed feeling -- sometimes create servers fails, but not really, so it would be awkward to retry that behind the scenes	15:54
*** prmtl has joined #openstack-infra		15:54
jeblair	mordred: is that because some task is not finished yet (even thought it says it is?)	15:54
mordred	jeblair: maybe	15:54
mordred	jeblair: I will investigate further- I thought it was a different problem	15:55
*** unicell has quit IRC		15:56
*** nelsnelson has quit IRC		15:56
*** unicell has joined #openstack-infra		15:56
*** yamamoto has quit IRC		15:57
*** tkelsey has joined #openstack-infra		15:57
openstackgerrit	Jeremy Stanley proposed openstack-infra/bindep: Support for older dpkg https://review.openstack.org/170189	15:57
*** nelsnelson has joined #openstack-infra		15:58
*** enikanorov__ has joined #openstack-infra		15:58
*** e0ne_ is now known as e0ne		15:59
*** eharney has quit IRC		16:00
*** yfried\|afk is now known as yfried		16:00
*** signed8bit has quit IRC		16:01
*** enikanorov has quit IRC		16:01
jaypipes	mordred: am I missing something? did the "toggle CI votes" button thing disappear from the Gerrit web UI?	16:02
*** hdd has joined #openstack-infra		16:02
*** derekh has quit IRC		16:02
*** ihrachyshka has quit IRC		16:02
*** mriedem has quit IRC		16:02
mordred	jaypipes: i still have it	16:03
*** nelsnelson has quit IRC		16:03
*** xyang1 has joined #openstack-infra		16:03
*** nelsnelson has joined #openstack-infra		16:04
*** SumitNaiksatam has joined #openstack-infra		16:04
*** yamamoto has joined #openstack-infra		16:04
*** signed8bit has joined #openstack-infra		16:04
zaro	morning	16:05
jaypipes	mordred: weird. if you go here: https://review.openstack.org/#/c/139758/15, where do you see it?	16:05
jaypipes	mordred: I see the old web UI screen...	16:05
zaro	jeblair: here's the upstream change the thing anteaya was referring to: https://gerrit-review.googlesource.com/#/c/65530/	16:05
mordred	jaypipes: I agree! I do not see it on that screen	16:06
jaypipes	heh, weird.	16:06
*** xyang1 has quit IRC		16:06
*** kir has joined #openstack-infra		16:06
mordred	jaypipes: I did see it exist for a minute while the page was loading	16:07
*** mriedem has joined #openstack-infra		16:07
zaro	fungi: you happen to know which version of mysql gerrit is using? i'm wondering if i should test on similar version or does not even matter?	16:07
mordred	zaro: Server version: 5.1.73-1 (Debian)	16:08
mordred	also WOW	16:08
mordred	that's old	16:08
*** carl_baldwin has quit IRC		16:08
*** dprince has quit IRC		16:09
sdague	jaypipes: it's getting set to display: none	16:09
sdague	hmmmm	16:09
zaro	mordred: you think that's older than one provided by precise?	16:09
jeblair	zaro: oh okay, yeah. that's a different problem than the zuul problem i worked on this morning. carry on! :)	16:09
zaro	ohh actually precise has ver 5.5, so yeah that is really old	16:09
zaro	fungi, mordred : well then maybe i just won't bother trying to use the same version.	16:11
*** cdent has quit IRC		16:11
mordred	jeblair, zaro: we may want to think about a database migration to a new instance at some point	16:11
*** yfried is now known as yfried\|afk		16:11
openstackgerrit	Merged openstack-infra/shade: Fix docs nit - make it clear the arg is a string https://review.openstack.org/170105	16:11
*** adalbas has quit IRC		16:13
sdague	jaypipes: the regex is failing with the patchset version included	16:14
*** hodos has joined #openstack-infra		16:15
*** afazekas has quit IRC		16:16
*** xyang1 has joined #openstack-infra		16:16
jaypipes	sdague: hah, indeed, that's right :) nice work, Holmes :)	16:18
sdague	ok, fix inbound	16:18
*** ldnunes has quit IRC		16:18
*** dtantsur is now known as dtantsur\|afk		16:18
*** unicell has quit IRC		16:18
*** jcoufal has quit IRC		16:19
*** Bsony has joined #openstack-infra		16:19
*** yfried\|afk has quit IRC		16:20
openstackgerrit	Sean Dague proposed openstack-infra/system-config: fix hideci regex to work on /#/c/XXXXX/YY https://review.openstack.org/170200	16:21
sdague	jeblair: ^^^	16:21
*** aswadr has joined #openstack-infra		16:21
*** yfried\|afk has joined #openstack-infra		16:22
*** devvesa has quit IRC		16:22
*** tonytan4ever has quit IRC		16:22
fungi	zaro: yeah, it was one of our first trove instances, so we're running whatever version of mysql rackspace was deploying at the time	16:23
*** jistr has quit IRC		16:23
fungi	we could consider an extended outage to extract/load into a newer instance too	16:23
fungi	zaro: also i believe the mysql version in the trove instance for review-dev matches if you want to just take gerrit offline there and test it out	16:24
mordred	although if we're going to do that - I think we should think about our database design and strategy	16:24
*** Bsony has quit IRC		16:24
mordred	that we would have to do an extended outage to roll to a new database server is kinda not cool	16:24
fungi	perhaps trove people know whether there's an in-place upgrade option i'm overlooking	16:25
*** baoli has quit IRC		16:25
mordred	I'm more talking about the fact that we can't do replication	16:26
*** ghostpl_ has quit IRC		16:27
mordred	which is the way I'd normally deal with this - set up slave, let it catch up, perform failover, rebuild old master, reattach to new master as slave	16:27
fungi	yep, understood	16:27
*** arxcruz has quit IRC		16:27
mordred	also - the facebook team says that the lossless semi-sync in 5.7 is very good	16:28
*** ghostpl_ has joined #openstack-infra		16:28
*** SumitNaiksatam has left #openstack-infra		16:28
jeblair	mordred: do we really want to go down the mysql replication route at this point? the app itself is single-homed	16:29
mordred	jeblair: not saying we do - saying we should consider whether we should	16:29
*** sdake has quit IRC		16:29
jeblair	mordred: and we take downtime all the time just to maintain the app, we can probably handle a db switch	16:29
jeblair	mordred: yeah	16:29
*** SumitNaiksatam has joined #openstack-infra		16:30
clarkb	also I think trove does allow for replication	16:30
clarkb	it just may not do it for you	16:30
*** sdake has joined #openstack-infra		16:30
mordred	I thought it did not allow for it at all	16:30
mordred	last time I checked	16:30
mordred	but I could be wrong	16:30
*** signed8bit has quit IRC		16:31
* mordred can play with it later		16:31
clarkb	https://blueprints.launchpad.net/trove/+spec/replication-v1	16:31
*** mpaolino has quit IRC		16:31
mordred	neat	16:32
*** yamahata has quit IRC		16:32
*** _shaps_ has quit IRC		16:33
openstackgerrit	Matthew Treinish proposed openstack-infra/devstack-gate: Do not merge me - I am for testing. https://review.openstack.org/169907	16:33
*** jlibosva has quit IRC		16:33
fungi	i too am for testing	16:34
fungi	but first, i am for lunch	16:34
fungi	back in a bit	16:34
mtreinish	fungi: oh, you know that's a good idea. I should do the same	16:35
*** yamamoto has quit IRC		16:36
*** nelsnelson has quit IRC		16:36
*** hodos has quit IRC		16:37
clarkb	jeblair: 167158 had the scp thing addressed	16:39
clarkb	jeblair: I am happy to babysit that change if you approve it	16:39
clarkb	(or if anyone else approves)	16:40
*** tonytan4ever has joined #openstack-infra		16:40
*** wenlock has quit IRC		16:41
*** Bsony has joined #openstack-infra		16:41
*** Swami has joined #openstack-infra		16:42
*** harlowja_away is now known as harlowja		16:42
*** MaxV has joined #openstack-infra		16:45
*** baoli has joined #openstack-infra		16:46
*** melwitt has joined #openstack-infra		16:46
*** sdake_ has joined #openstack-infra		16:46
*** e0ne is now known as e0ne_		16:46
*** unicell has joined #openstack-infra		16:49
*** sdake has quit IRC		16:50
*** Longgeek has quit IRC		16:50
*** tonytan4ever has quit IRC		16:51
*** carl_baldwin has joined #openstack-infra		16:51
*** tonytan4ever has joined #openstack-infra		16:52
*** yamahata has joined #openstack-infra		16:53
*** ildikov_ has quit IRC		16:54
*** e0ne_ is now known as e0ne		16:55
openstackgerrit	Monty Taylor proposed openstack-infra/project-config: Add elements for Infra servers https://review.openstack.org/140840	16:55
*** tqtran_afk has joined #openstack-infra		16:56
*** dustins_ has joined #openstack-infra		16:58
*** nelsnelson has joined #openstack-infra		16:59
*** dustins has quit IRC		16:59
*** dboik_ has quit IRC		17:00
mordred	clarkb, fungi: ci.openstack.org ...	17:00
*** dboik has joined #openstack-infra		17:00
*** bswartz has quit IRC		17:01
*** EmilienM is now known as EmilienM\|afk		17:01
openstackgerrit	Clark Boylan proposed openstack-infra/nodepool: Add test for multi format image https://review.openstack.org/170210	17:02
clarkb	jeblair: ^ that adds a test like you suggseted, but I did it in a follow up change to avoid losing your +2	17:02
clarkb	mordred: ?	17:02
mordred	clarkb, fungi: I cannot ssh in to it	17:03
mordred	what is it?	17:03
clarkb	the cloud docs site looks like	17:03
mordred	but we scp things to it	17:03
*** annegentle has quit IRC		17:03
mordred	docs.o.o is a cloud site thing	17:03
*** bswartz has joined #openstack-infra		17:03
clarkb	ya I think we may have a user set up to do that?	17:04
mordred	clarkb: so you can also not ssh to it?	17:04
*** wenlock has joined #openstack-infra		17:04
clarkb	I can	17:04
mordred	ok. I think it just has a stale key for me	17:04
mordred	from way back in the day	17:05
clarkb	mordred: I can manually update your key with the current public key	17:05
clarkb	give me a couple minutes	17:05
mordred	clarkb: thanks! (I'm investigating why the shade docs don't publish)	17:05
*** medieval1 has joined #openstack-infra		17:05
clarkb	mordred: I will let you remove the old public key as desired	17:06
*** Guest63783 has quit IRC		17:06
clarkb	mordred: try now	17:07
*** otter768 has joined #openstack-infra		17:07
mordred	clarkb: I'm in	17:07
mordred	clarkb: wow. ... is this actually the _original_ un-puppeted machine?	17:08
clarkb	mordred: looks like we use the hudson user to scp to that host	17:08
mordred	perhaps we want to finish moving off of this one ...	17:08
jeblair	clarkb: 167158 aprvd	17:08
clarkb	ya the swift stuff is moving along ^	17:08
clarkb	jeblair: awesome thank you	17:08
clarkb	and once we have swift for logs next step is swift for docs iirc	17:08
clarkb	or at least that is an option we have	17:08
mordred	yah. but this isn't related to docs.o.o	17:08
clarkb	mordred: thats true we could host ci.o.o independently without much fuss	17:09
mordred	yah. I mean, we can also just wait	17:09
jeblair	i think we want to move everything on ci.o.o to docs.o.o/infra	17:09
jeblair	at least, we've been putting new stuff there, like specs and the infra manual	17:09
*** medieval1 has quit IRC		17:09
jeblair	i figured we'd move the ci.o.o content there too, and then keep a redirect around for a while	17:10
jeblair	(we can move the redirect to static when everything else is moved)	17:10
*** tqtran_afk is now known as tqtran		17:11
*** ujuc has quit IRC		17:11
jeblair	clarkb: that looks like just the test i imagined! :)	17:11
jeblair	i'm going to go into zuul land and work on 170155 for a while. back later.	17:12
*** otter768 has quit IRC		17:12
clarkb	have fun	17:12
*** SumitNaiksatam has quit IRC		17:12
*** signed8bit has joined #openstack-infra		17:12
*** prmtl has quit IRC		17:13
*** pelix has quit IRC		17:16
*** SumitNaiksatam has joined #openstack-infra		17:17
*** ssam2 has quit IRC		17:17
clarkb	I have approved the 2 changes after 167158 as that will actually get that log footer stuff on the dg specific dsvm jobs	17:18
*** Hefeweizen has quit IRC		17:19
clarkb	looking at nodepool coverage reports I have discovered http://logs.openstack.org/47/126747/12/check/nodepool-coverage/700b32b/cover/keybd_closed.png is a bug in os_loganalyze	17:19
*** _nadya_ has quit IRC		17:20
clarkb	I will try to sort that out today as well	17:20
*** hodos has joined #openstack-infra		17:20
pabelanger	Does nodepool only keep so many images for base snapshots on the cloud?	17:20
pabelanger	for example, max of 2?	17:20
clarkb	pabelanger: it keeps 2 yup	17:20
clarkb	a current image and the previous image	17:20
pabelanger	clarkb, do you know where that is check? is that a perodicjob?	17:20
clarkb	pabelanger: ya its the periodic cleanup internal cron job	17:21
*** MaxV has quit IRC		17:21
openstackgerrit	Merged openstack-infra/project-config: Upload devstack-gate documentation with logs https://review.openstack.org/167158	17:21
clarkb	which runs every 15 minutes iirc	17:21
pabelanger	okay, thanks	17:21
pabelanger	looks like cleanupOneImage is the function I want	17:21
Shrews	mordred: this ... _tasks.NetworkCreate(body=dict({'network': network}))	17:23
Shrews	mordred: that's not supposed to be a dict within a dict, is it?	17:23
mordred	Shrews: probably not	17:24
Shrews	mordred: let me test it....	17:24
mordred	Shrews: I would say that's almost certainly a bug	17:24
clarkb	Shrews: are dicts the new turtle?	17:24
mordred	clarkb: they are in neutronclient	17:24
openstackgerrit	Matthew Treinish proposed openstack-infra/devstack-gate: Do not merge me - I am for testing. https://review.openstack.org/169907	17:24
*** dustins_ has quit IRC		17:26
*** MaxV has joined #openstack-infra		17:26
clarkb	mtreinish: dansmith re ^ as a user of qcow2 images and glance and nova, I think we should probably just use the .img by default?	17:27
clarkb	definitely get the test working that way first though :)	17:27
openstackgerrit	Merged openstack-infra/project-config: zuul-swift-devstack-logs includes console https://review.openstack.org/167378	17:27
dansmith	clarkb: yes, I think we should be doing that by default	17:27
mtreinish	clarkb: yeah the tempest test/code using that is dumb	17:27
dansmith	clarkb: but given where we are, probably don't want to switch that right this moment	17:28
mtreinish	I need to fix it, but until I figure out the best way to do that devstack should be able to make it work with the wonky config options	17:28
clarkb	dansmith: ya no rush from here. Just trying to put my user hat on and improve tests if possible	17:28
dansmith	yeah, definitely	17:28
*** radez_g0n3 is now known as radez		17:28
openstackgerrit	Merged openstack-infra/project-config: Update devstack log publisher to use new macro https://review.openstack.org/167842	17:28
*** tonytan4ever has quit IRC		17:28
dansmith	clarkb: there are different code paths in cinder and nova when UECs are involved, though, so it seems like maybe having _some_ run that exercises that code is still good	17:29
dansmith	clarkb: even if just a background one	17:29
*** tonytan4ever has joined #openstack-infra		17:29
*** patrickeast has joined #openstack-infra		17:29
*** aswadr has quit IRC		17:30
clarkb	dansmith: in other news dtroyer +2'd my devstack change for mangling the cpu map. progress!	17:30
*** alexpilotti_ has joined #openstack-infra		17:30
dansmith	clarkb: like, yay :)	17:30
mtreinish	clarkb: do you have a link?	17:31
*** Hal has joined #openstack-infra		17:31
*** ldnunes has joined #openstack-infra		17:31
*** unicell has quit IRC		17:31
*** unicell1 has joined #openstack-infra		17:31
*** Hal is now known as Guest15579		17:31
*** alexpilotti has quit IRC		17:32
clarkb	mtreinish: https://review.openstack.org/#/c/168407/ probably good to get sdague's feedback too because I know he was thinking about how to do this as well	17:32
clarkb	mtreinish: and he may have come up with even better plans	17:32
*** signed8bit is now known as signed8bit_ZZZzz		17:33
*** alexpilotti_ is now known as alexpilotti		17:33
greghaynes	clarkb: your nodepool multi format patch is going to break my monkeypatch patch :(	17:33
*** e0ne has quit IRC		17:33
clarkb	greghaynes: awesome	17:33
clarkb	greghaynes: how badly?	17:33
Shrews	mordred: very surprisingly, it works that way AND without the {} AND without the dict()	17:34
greghaynes	not that badly, im still wrapping my head around how this works in nodepool - but basically were changing the -o param to dib to not have the file extension with your patch, right?	17:34
clarkb	Shrews: it really is turtles the whole way down	17:34
clarkb	greghaynes: correct	17:34
greghaynes	clarkb: yea, so the fake-dib thing relies on the old format	17:35
clarkb	greghaynes: because dib was removing it anyway and doing the right thing. So we are making nodepool a bit more accomodating of dib behavior	17:35
*** andreykurilin_ has joined #openstack-infra		17:35
Shrews	clarkb: python is much smarter than I	17:35
greghaynes	clarkb: yea, you have to if youre doing multiformat	17:35
greghaynes	clarkb: basically the fake needs to fake the -t behavior	17:35
clarkb	greghaynes: gotcha	17:36
greghaynes	clarkb: another question about it - do we detect a dib failure based on exit code or what files it produces?	17:36
clarkb	greghaynes: exit code	17:37
greghaynes	so, if dib makes a qcow and fails to make the vhd...	17:37
greghaynes	ah, ok, theyll both fail	17:37
greghaynes	probably a good thing to test :)	17:37
clarkb	greghaynes: yup which is the behavior we want because we want t have the same image in use across the board	17:37
greghaynes	yes	17:37
openstackgerrit	James E. Blair proposed openstack-infra/zuul: Add commit needed-by https://review.openstack.org/170155	17:37
jeblair	that passes tests locally ^ (has test fixes); now to add a new test	17:38
mordred	Shrews: WOW	17:38
*** YorikSar has quit IRC		17:40
sdague	clarkb: +A	17:40
openstackgerrit	David Shrewsbury proposed openstack-infra/shade: Add API method create_subnet() https://review.openstack.org/170217	17:40
*** dustins has joined #openstack-infra		17:41
*** signed8bit_ZZZzz is now known as signed8bit		17:41
*** eharney has joined #openstack-infra		17:42
*** emagana has quit IRC		17:42
*** alexpilotti has quit IRC		17:42
*** sdake has joined #openstack-infra		17:43
*** alexpilotti has joined #openstack-infra		17:44
*** SumitNaiksatam_ has joined #openstack-infra		17:45
*** sdake_ has quit IRC		17:46
*** sarob has quit IRC		17:47
*** SumitNaiksatam has quit IRC		17:47
*** SumitNaiksatam_ is now known as SumitNaiksatam		17:47
*** sarob has joined #openstack-infra		17:47
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Add API auto-generation based on docstrings https://review.openstack.org/170220	17:48
mordred	Shrews: ^^	17:48
mordred	Shrews: just for yo	17:48
*** andreykurilin_ has quit IRC		17:49
Shrews	mordred: aw shucks	17:49
*** andreykurilin_ has joined #openstack-infra		17:49
Shrews	mordred: now I can see what my docs from https://review.openstack.org/170217 look like!	17:49
Shrews	which is basically a novella	17:50
openstackgerrit	Monty Taylor proposed openstack-infra/shade: Add API auto-generation based on docstrings https://review.openstack.org/170220	17:51
*** bcourt has quit IRC		17:52
*** ildikov_ has joined #openstack-infra		17:52
*** sarob has quit IRC		17:53
*** achanda has joined #openstack-infra		17:53
openstackgerrit	Monty Taylor proposed openstack-infra/project-config: Publish shade docs to docs.openstack.org/infra https://review.openstack.org/170223	17:53
mordred	clarkb, jeblair: ^^ also, if teh plan is to publish to the new location, might as well do it for shade	17:54
*** ghostpl_ has quit IRC		17:54
*** russellb has quit IRC		17:55
JayF	We're in desperate need of a devstack change landing so we can enable Cleaning in Ironic by default for K; I've asked for reviews in -qa a lot, but asking here now: https://review.openstack.org/#/c/168175/ just bumping a timeout on a tempest test for ironic-only (since deletes take longer) and adding an Ironic setting to avoid shredding in devstack	17:55
*** marcusvrn1 has joined #openstack-infra		17:56
JayF	+1s from lots of Ironic folks, and the Ironic patch that depends on it is passing all tests	17:56
*** johnthetubaguy is now known as zz_johnthetubagu		17:56
*** marcusvrn has quit IRC		17:56
clarkb	ok I see the issue with binary data and os_loganalyze. With the on disk logs we only run .txt(.gz)? and .html(.gz)? through htmlify but with swift we run everything through it	17:57
openstackgerrit	James E. Blair proposed openstack-infra/zuul: Add commit needed-by https://review.openstack.org/170155	17:57
*** hdd has quit IRC		17:57
clarkb	sdague: jhesketh ^ Iam going to work on having the swift object getter return the raw data if it doesn't match those above rules	17:57
*** annegentle has joined #openstack-infra		17:58
*** russellb has joined #openstack-infra		17:58
*** HeOS has quit IRC		17:59
openstackgerrit	James E. Blair proposed openstack-infra/zuul: Add commit needed-by https://review.openstack.org/170155	18:00
*** tkelsey has quit IRC		18:00
jeblair	clarkb, mordred, fungi, sdague: ^ https://review.openstack.org/170155 is ready for real review; it has tests that replicate both problems it tries to fix and passes tests locally	18:00
*** ivar-lazzaro has joined #openstack-infra		18:01
clarkb	jeblair: cool will take a look shortly	18:01
*** dustins_ has joined #openstack-infra		18:03
*** doug-fish has joined #openstack-infra		18:03
sdague	jeblair: ok, I'm not sure i know enough zuul to provide helpful review on it	18:04
sdague	jeblair: https://review.openstack.org/#/c/170200/ will fix the Toggle CI button missing on a bunch of pages that jaypipes reported earlier	18:04
*** YorikSar has joined #openstack-infra		18:05
jeblair	sdague: no problem	18:06
*** medieval1 has joined #openstack-infra		18:06
*** dustins has quit IRC		18:06
*** ddieterly has joined #openstack-infra		18:08
*** yamamoto has joined #openstack-infra		18:08
*** marun has quit IRC		18:10
*** medieval1 has quit IRC		18:11
*** bcourt has joined #openstack-infra		18:12
*** ZZelle_ has joined #openstack-infra		18:12
jeblair	sdague: +2 with a suggestion for another incremental improvement. i can +A than now if you want.	18:12
sdague	let me look, I can respin now	18:14
*** e0ne has joined #openstack-infra		18:14
*** davideag_ has joined #openstack-infra		18:15
*** russellb has quit IRC		18:15
sdague	jeblair: I'll respin, one sec	18:15
*** thedodd has quit IRC		18:15
*** unicell1 has quit IRC		18:16
*** unicell has joined #openstack-infra		18:16
openstackgerrit	Sean Dague proposed openstack-infra/system-config: fix hideci regex to work on /#/c/XXXXX/YY https://review.openstack.org/170200	18:16
sdague	jeblair: fixed	18:17
fungi	jeblair: now that i'm back from lunch, i'll see your https://review.openstack.org/170155 and raise you http://fivethirtyeight.com/features/a-statistical-analysis-of-the-work-of-bob-ross/	18:17
fungi	(for science!)	18:18
*** davideagnello has quit IRC		18:18
*** yamamoto has quit IRC		18:18
*** sdake_ has joined #openstack-infra		18:19
*** russellb has joined #openstack-infra		18:19
*** mriedem has quit IRC		18:19
*** annegentle has quit IRC		18:20
Shrews	fungi: i loved watching bob ross back in the day. but i can no longer view him the same way after having watched this: https://www.youtube.com/watch?v=x1JKAyxk9cs	18:21
jeblair	sdague: +2 fungi: do we _have_ happy clouds?	18:22
jeblair	Shrews: i don't know what that is but i refuse to watch it if it will alter my perception of bob ross	18:22
mordred	jeblair: Family Guy	18:23
*** sdake has quit IRC		18:23
fungi	seth has no idea what he doth mock	18:23
fungi	i'm sure i saw every episode of it several times as a child. my grandmother was an artist and painter by trade, and the joy of painting was a favorite of hers	18:24
fungi	"Ross’s 36 ocean paintings were also more likely to feature cliffs, clouds and rocks than the average painting."	18:25
fungi	statistics is awesome	18:25
*** Swami has quit IRC		18:25
*** masayukig has quit IRC		18:26
mordred	fungi: "What is the probability, given that Ross painted a happy tree, that he then painted a friend for that tree?"	18:26
fungi	93%	18:27
fungi	yes, that's was a great quote. also the comparison of cumulus vs cirrus clouds is amusing	18:27
*** nelsnelson has quit IRC		18:27
fungi	(and almost topical here!)	18:28
*** tkelsey has joined #openstack-infra		18:29
*** ibiris is now known as ibiris_away		18:29
*** MaxV has quit IRC		18:29
dstufft	https://www.youtube.com/watch?v=YLO7tCdBVrA bob ross remix	18:29
*** JayJ_ has joined #openstack-infra		18:30
asselin_	can we get these pushed through? needs another core: https://review.openstack.org/#/c/165588/ https://review.openstack.org/#/c/167444/	18:30
*** EmilienM\|afk is now known as EmilienM		18:30
jeblair	"The majority of people who watch Bob Ross have no interest in painting," she said. "Mostly it's his calming voice."	18:31
jeblair	i really need the voice of bob ross reading the -dev list	18:31
*** dustins has joined #openstack-infra		18:31
fungi	also their lithium prescriptions	18:31
*** tonytan4ever has quit IRC		18:32
*** masayukig has joined #openstack-infra		18:32
openstackgerrit	Monty Taylor proposed openstack-infra/project-config: Publish shade docs to docs.openstack.org/infra https://review.openstack.org/170223	18:32
mordred	jeblair: what if we could make a computer thing that would narrate the emails of -dev in bob ross's voice?	18:32
*** tiswanso has quit IRC		18:33
openstackgerrit	Clark Boylan proposed openstack-infra/os-loganalyze: Have swift pass through non log data unoutched https://review.openstack.org/170234	18:33
clarkb	sdague: ^ tahts really ugly and needs testing, but your early input would probably be valuable.	18:33
*** tiswanso has joined #openstack-infra		18:33
fungi	no doubt there's a festival plugin in the works	18:33
*** tiswanso has quit IRC		18:33
*** ddieterly has quit IRC		18:33
dstufft	The early life section of https://en.wikipedia.org/wiki/Bob_Ross is pretty cool	18:33
*** dustins_ has quit IRC		18:34
*** tkelsey has quit IRC		18:34
sdague	clarkb: looking	18:34
*** tiswanso has joined #openstack-infra		18:34
clarkb	sdague: I think it might be better to completely rewrite a good chunk of that code to make the error handling and generator retrieval a bit less obtuse	18:34
fungi	asselin_: what's the plan with 167444? do you intend to flesh out puppet-openstackci's dependencies and then subtract those from system-config's modules.env?	18:35
asselin_	fungi, ??	18:36
fungi	asselin_: mostly wondering what things look like once we start reconsuming puppet-openstackci from within system-config	18:36
sdague	clarkb: do the objects in swift have public urls ?	18:36
clarkb	sdague: they do not	18:36
sdague	hmph	18:36
clarkb	sdague: we could do a redirect instead if we ste that up though	18:36
clarkb	sdague: but we haven't set that up	18:36
sdague	yeh, that I think would be much better if it wasn't too hard	18:36
asselin_	fungi, e.g. the depends-on change: https://review.openstack.org/#/c/167426/	18:36
clarkb	sdague: actually thats probably the best way to do it	18:36
fungi	asselin_: since there's no reason for system-config's modules.env to load puppet-openstackci unless system-config is going to start making use of that module	18:36
clarkb	sdague: let me look into what will need to be done to do this with rax swift	18:37
*** rhe00 has left #openstack-infra		18:37
sdague	yeh, otherwise we're loading this stuff into apache memory fully, and then flushing it out on the wire	18:37
sdague	which seems, weird.	18:37
clarkb	sdague: agreed	18:37
asselin_	fungi, right...the depends on change is the plan.	18:37
sdague	ok, so how about I hold really looking at that patch for a day to figure out if we can instead go a 302 route instead	18:37
fungi	asselin_: yeah, just started looking at that. it's a great example--thanks!	18:38
clarkb	sdague: sure sounds good	18:38
*** baoli has quit IRC		18:38
sdague	clarkb: ok, let me know what you find on making public urls of these things	18:38
clarkb	sdague: from memory we have to use rax's cdn and you get an hmac'd dns record to use	18:38
sdague	clarkb: ok, I only half understand that last part	18:39
fungi	asselin_: i would have expected a little more mention of why in the commit message, but in the grand scheme of things it's a one-line change and people looking at the change in context later will have no problem figuring out the reason so lgtm	18:39
asselin_	fungi, yeah I can mention the spec	18:39
clarkb	sdague: basicall its a ridiculous dns record that I am not sure we can generate independently, so it would have to be a config option to os_loganalyze	18:39
asselin_	in the commit message	18:39
fungi	asselin_: no need, it's approved	18:39
clarkb	sdague: but it allows swift to negotiate with the cdn without auth iirc	18:39
*** marun has joined #openstack-infra		18:40
asselin_	fungi, thanks	18:40
sdague	clarkb: but it's stable? doesn't change over time?	18:40
*** blitzrage has quit IRC		18:40
clarkb	sdague: it only changes if your container changes, so mostly stable	18:40
notmyname	clarkb: cdn domain names with rax?	18:40
clarkb	notmyname: ya	18:40
sdague	yeh, that seems ok for putting it into the config then	18:40
clarkb	notmyname: for public swift object access	18:40
sdague	it's roughly the same as the swift creds	18:40
*** Swami has joined #openstack-infra		18:41
notmyname	clarkb: I'm looking in the code to find the method that creates them	18:42
*** russellb has quit IRC		18:42
clarkb	notmyname: I thought there was a shared secret involved so that rax and cdn could generate the same key and trust each other?	18:42
clarkb	notmyname: and since we don't have that we can't generate it independently but I am probably misremembering	18:42
openstackgerrit	Merged openstack-infra/puppet-openstackci: Initial files for the puppet-openstackci repo https://review.openstack.org/165588	18:43
notmyname	clarkb: ah, yes	18:45
*** russellb has joined #openstack-infra		18:47
*** mriedem has joined #openstack-infra		18:48
*** sarob has joined #openstack-infra		18:49
openstackgerrit	Merged openstack-infra/system-config: Add puppet-openstackci to included modules https://review.openstack.org/167444	18:50
*** achanda has quit IRC		18:50
*** yamamoto has joined #openstack-infra		18:50
*** leifmadsen has joined #openstack-infra		18:51
*** annegentle has joined #openstack-infra		18:53
*** sarob has quit IRC		18:54
timrc	mordred, When will nodepool restarted next so that the task metrics start getting pushed into graphite?	18:54
timrc	be restarted*	18:55
*** signed8bit is now known as signed8bit_ZZZzz		18:56
*** bcourt has quit IRC		18:56
mordred	timrc: not sure ... jeblair, clarkb, fungi: ^^ thoughts?	18:56
jeblair	clarkb, sdague: i thought we worked out streaming the swift stuff through apache, so we weren't loading everything fully into memory	18:56
clarkb	jeblair: I have commented on 170155 I think it can merge as is but figured I would get a chance to talk bout those comments before approving	18:56
fungi	timrc: we're really not particularly underwater, so i can restart nodepoold right now if nobody objects	18:57
clarkb	jeblair: it uses fileinput iterators and the swiftclient iterator implementation	18:57
fungi	the additional metrics may help us characterize what we're seeing from a provider performance standpoint anyway	18:57
clarkb	jeblair: so it will load as much into memory as those do (I think the file input iterator will do the right thing, unsure about swiftclient)	18:57
notmyname	clarkb: if it does, it's a bug	18:58
*** lpetrut has quit IRC		18:58
*** signed8bit_ZZZzz is now known as signed8bit		18:58
openstackgerrit	Ramy Asselin proposed openstack-infra/puppet-openstackci: Create a log server like http://logs.openstack.org https://review.openstack.org/167425	18:58
*** lpetrut has joined #openstack-infra		18:58
openstackgerrit	Merged openstack-infra/system-config: fix hideci regex to work on /#/c/XXXXX/YY https://review.openstack.org/170200	18:58
*** lpetrut has quit IRC		18:59
*** annegentle has quit IRC		18:59
clarkb	jeblair: do you have a perference for proxying it all through apache and not complicating setup with rax cdn?	18:59
timrc	fungi, That'd be great. I have the afternoon to spend on playing around with visualizing those numbers.	18:59
*** bswartz has quit IRC		18:59
*** annegentle has joined #openstack-infra		18:59
openstackgerrit	Ramy Asselin proposed openstack-infra/system-config: Use common log server implementation https://review.openstack.org/167426	18:59
*** yamamoto has quit IRC		19:00
fungi	timrc: well, the numbers from this afternoon will be pretty dull, but it's a start!	19:00
clarkb	notmyname: thanks, I think it uses requests under the covers which is probably what really implements it?	19:00
fungi	clarkb: jeblair: mordred: objections to me performing a controlled nodepoold restart now?	19:00
clarkb	notmyname: but in any case good to know that if this doesn't work as expected yall would consider it a bug	19:01
clarkb	fungi: none here	19:01
jeblair	clarkb: a slight one preference for not involving the cdn, i think. not a strong one.	19:01
timrc	fungi, Well... I'm going to be playing around with cubism and graphene, so that won't much matter to me at the moment. Just having a data source will be a Good Thing (tm).	19:01
jeblair	fungi: no objection	19:01
mordred	fungi: nope. go for it!	19:01
*** baoli has joined #openstack-infra		19:01
clarkb	btw https://jenkins06.openstack.org/job/check-dg-tempest-dsvm-full/255/console should use the new log footer swift stuff	19:01
fungi	restarting now	19:01
*** Ryan52 has quit IRC		19:02
clarkb	jeblair: I think I have an idea of how to rewrite that code to make proxying it with the wsgi much simpler	19:02
*** YorikSar has quit IRC		19:02
*** baoli has quit IRC		19:03
*** baoli has joined #openstack-infra		19:04
*** ivar-lazzaro has quit IRC		19:06
*** BobH has quit IRC		19:07
*** medieval1 has joined #openstack-infra		19:07
*** ivar-lazzaro has joined #openstack-infra		19:07
*** andreykurilin_ has quit IRC		19:07
*** achanda has joined #openstack-infra		19:08
*** otter768 has joined #openstack-infra		19:08
*** cdent has joined #openstack-infra		19:09
clarkb	"At this time, there is no way to publicly access the objects stored in Cloud Files unless that container is published to CDN." and http://www.rackspace.com/knowledge_center/article/serve-static-content-for-websites-by-using-cloud-files	19:10
clarkb	so we would have to do what I describe	19:10
*** medieval1 has quit IRC		19:11
clarkb	jeblair: considering your preference I will give a bit more time to the passthrough option	19:11
clarkb	sdague: ^	19:11
openstackgerrit	Jeremy Stanley proposed openstack-infra/bindep: Accept old dpkg-query errors https://review.openstack.org/170244	19:12
*** bcourt has joined #openstack-infra		19:13
*** otter768 has quit IRC		19:13
*** nelsnelson has joined #openstack-infra		19:14
*** nelsnelson has quit IRC		19:15
krotscheck	Can someone help me out with a pep error? I'm getting a "multi line docstring summary not separated with an empty line" error on this: http://paste.openstack.org/show/197973/	19:15
*** e0ne has quit IRC		19:15
krotscheck	And, well, I have zero idea what it's trying to tell me.	19:16
*** nelsnelson has joined #openstack-infra		19:16
clarkb	krotscheck: like a git commit your docstring should have a single subject/title line then additional content	19:16
fungi	krotscheck: it expects a title line which fits within a single line, not wrapped	19:16
krotscheck	Ahhh	19:16
krotscheck	That's.	19:16
* krotscheck signs		19:16
* krotscheck sighs		19:16
clarkb	the reason for it is help(foo) rendering	19:16
*** thedodd has joined #openstack-infra		19:16
clarkb	its actually a not ridiculous rule	19:16
krotscheck	clarkb: Your opinion may vary from mine.	19:17
clarkb	krotscheck: do you use help()?	19:17
krotscheck	clarkb: Nope.	19:17
clarkb	you should give it a try :)	19:17
fungi	right, when in invoke context documentation within the interactive interpreter, i'm quite thankful for people providing well-formatted docstrings	19:17
fungi	er, when i	19:17
fungi	it's also beneficial for things like sphinx autodoc	19:18
krotscheck	Perhaps, however enforcing that in Unit Tests seems a little overkill	19:18
tchaypo	jeblair: I think I need to start digging into what’s in my gertty queue some more. I went to bed with 3000 items and woke up with 27730	19:18
fungi	krotscheck: look at it as an annoying requirement which will result in you needing to write less actual documentation separate from your source code	19:18
*** ghostpl_ has joined #openstack-infra		19:19
fungi	tchaypo: how many projects do you subscribe to?	19:19
*** __TheDodd__ has joined #openstack-infra		19:19
tchaypo	~80	19:20
fungi	i would already be seeking medication if my queue was 3000, much less nearly 10x that	19:20
*** sdake has joined #openstack-infra		19:21
*** thedodd has quit IRC		19:21
mtreinish	clarkb, fungi: do you have any thoughts on: https://review.openstack.org/#/c/169933/ if you don't want os-testr reviews here I can respin the patch to drop -infra	19:21
*** HeOS has joined #openstack-infra		19:22
tchaypo	Most of these projects I’ve got just in case I want to look at one or two reviews occasionally	19:23
tchaypo	maybe I should forget about using gertty as my main way to acccess gerrit and just list a few projects I care a lot about	19:23
*** baoli has quit IRC		19:24
*** sdake_ has quit IRC		19:24
fungi	tchaypo: in that case some of jeblair's patches form this week will probably interest you (sync changes you own, support for starring changes and syncing those by default)	19:24
fungi	er, from	19:25
fungi	mtreinish: i'm on the fence... it feels like it might fall into the same realm as devstack, which while of potential infra interest only reports changes in #-qa	19:25
tchaypo	yeah, that chain does look interesting	19:26
mtreinish	fungi: heh, yeah I was on the fence too	19:27
*** enikanorov__ has quit IRC		19:27
fungi	could make it easier to not bother syncing those projects... just ctrl-o to plug in the change number and then star it so you can continue to track it without tracking all changes for that repo	19:27
*** sarob has joined #openstack-infra		19:27
*** ddieterly has joined #openstack-infra		19:28
tchaypo	I’ve already got 168181 included	19:28
fungi	i still need to take a minute to try those out, but am excited by the prospect	19:28
tchaypo	I think maybe I’d like to be able to subscribe to a search rather than a project	19:29
*** hdd has joined #openstack-infra		19:29
*** Guest15579 has quit IRC		19:29
fungi	that's an intriguing idea	19:29
*** packet has quit IRC		19:30
tchaypo	status:open branch:stable/juno	19:30
mordred	fungi: you restarted the nodepool, yeah?	19:31
*** yamamoto has joined #openstack-infra		19:31
*** Bsony has quit IRC		19:31
tchaypo	for someone trying to keep on track of open reviews in all the stable branches, that’s a much smaller set of changes to sync than subbing to everything that has a stable branch	19:31
mordred	fungi: I don't see any new counters in graphite - wondering if I should debug more	19:31
fungi	mordred: i did. i was about to go fishing in graphite to check for stats	19:31
tchaypo	I wonder if I could try that out by having something scrape gerrit, then poke gertty’s DB to star all those changes	19:32
fungi	nodepool.task.x.y	19:32
mordred	fungi: yah - I don't see task in the nodepool namespace at all	19:32
tchaypo	although really I think this calls for a new status rather than re-using starred	19:32
fungi	mordred: nor do i. so either we're both going crazy or something's not quite right there (or both!)	19:33
fungi	tchaypo: star has the benefit that it's a feature present in gerrit	19:34
timrc	Blah cubism seems so much more simplistic than graphene but it requires a CLA to contribute... who in their right mind would require one of those... oh wait :)	19:34
mordred	timrc: who indeed?	19:34
fungi	timrc: fascists, that's who	19:34
*** YorikSar has joined #openstack-infra		19:35
timrc	:-)	19:35
*** spzala has joined #openstack-infra		19:35
fungi	mordred: /opt/nodepool is current, but pip list says the git sha is from march 18. i'm going to check puppet logs but i suspect pip failed to upgrade nodepool	19:36
mordred	fungi: yay!	19:36
clarkb	http://logs.openstack.org/08/168408/5/check/check-dg-tempest-dsvm-full-reexec/be452b5/index.html?source=swift it works!	19:37
clarkb	http://logs.openstack.org/08/168408/5/check/check-dg-tempest-dsvm-full-reexec/be452b5/logs/index.html?source=swift also	19:38
clarkb	jhesketh: ^ thanks a bunch	19:38
clarkb	was that the last step before wide rollout? I think so (though I should figure out binary data and swift and os_loganalyze soon)	19:38
tchaypo	fungi: agree. this would be more equivalent to a bookmarked search url, or maybe a dashboard	19:39
openstackgerrit	Matt Riedemann proposed openstack-infra/elastic-recheck: Remove query for bug 1431724 https://review.openstack.org/170253	19:39
openstack	bug 1431724 in tempest "test_preserve_preexisting_port failing against icehouse" [High,Fix released] https://launchpad.net/bugs/1431724 - Assigned to Andrea Frittoli (andrea-frittoli)	19:39
fungi	tchaypo: ahh, sync all changes matching a configured dashboard sounds like a reasonable feature	19:39
clarkb	sdague: ^ check out those log footers	19:39
tchaypo	When I do a search like "status:open branch:stable/juno” in gertty, is it submitting that to gerrit, or just returning matches in the local db?	19:40
* tchaypo thinks out loud while checking source		19:40
*** Krinkle\|detached is now known as Krinkle		19:40
openstackgerrit	Dexter Fryar proposed openstack-infra/git-review: Fixed a dead link to the git-review docs https://review.openstack.org/170256	19:41
*** yamamoto has quit IRC		19:42
fungi	here's something i find worrisome...	19:42
fungi	digging in the syslog on nodepool.o.o	19:42
*** crc32 has joined #openstack-infra		19:43
*** tiswanso has quit IRC		19:43
fungi	_YUM_ is installing packages	19:43
fungi	wtf?!?	19:43
tchaypo	what distro is this machine?	19:43
fungi	ubuntu	19:43
greghaynes	fungi: dib?	19:43
clarkb	yes that would be dib	19:43
mordred	fungi: log snippet? (sounds like a dib thing)	19:43
*** tiswanso_ has joined #openstack-infra		19:43
fungi	greghaynes: oh, maybe this is dib activity in a chroot ending up in the logger	19:43
greghaynes	that would be my guess	19:44
mordred	yes. I think that's what it is	19:44
*** packet has joined #openstack-infra		19:44
fungi	the timestamps look right for image updates	19:44
greghaynes	fungi: although when we switch to fedora-minimal itll be yum outside the chroot	19:44
fungi	okay, less worried now	19:44
greghaynes	:)	19:44
*** jerryz has quit IRC		19:44
fungi	it does make the syslog on this machine interesting to search through	19:44
*** lpetrut has joined #openstack-infra		19:46
fungi	so here's the breakdown apparently.	19:46
tchaypo	okay, it looks like these searches are parsed and run locally on gertty’s db. So it sounds like this would require a bit of a change, in that the dashboard queries would have to be sent off to gerrit to get a list of results. but that shouldn’t be any different to asking for a project’s open changes.	19:47
greghaynes	clarkb: what do you think of merging the monkeypatch stuff first, and then letting me patch it up in your multi format change? That way we can actually test it	19:47
fungi	when pip tried to upgrade nodepool it wanted to update to a new mysql-python	19:47
fungi	but couldn't compile the c extensions because mysql_config is not installed	19:47
greghaynes	otherwise were just doing the "This works, really I promise"	19:47
openstackgerrit	Merged openstack-infra/elastic-recheck: Remove query for bug 1431724 https://review.openstack.org/170253	19:47
openstack	bug 1431724 in tempest "test_preserve_preexisting_port failing against icehouse" [High,Fix released] https://launchpad.net/bugs/1431724 - Assigned to Andrea Frittoli (andrea-frittoli)	19:47
mordred	fungi: stab	19:48
fungi	would nodepool work fine with pymysql? ;)	19:48
greghaynes	hah	19:48
fungi	i'm tempted to "fix" it that way	19:48
greghaynes	I bet	19:48
mordred	probably. although we might want to just pin the mysql-python depend for now	19:48
greghaynes	The only differences I have found are random encoding stuff in error messages	19:49
greghaynes	which I really hope you dont use	19:49
clarkb	greghaynes: either way is fine. my change has been hanging around for forever, another few days won't hurt it	19:50
*** baoli has joined #openstack-infra		19:50
clarkb	yes I would expect pymysql to just work (tm)	19:50
greghaynes	clarkb: sweet, if someone would like to review https://review.openstack.org/#/c/165682/ I can then make the multi dib format patch not break after it merges	19:51
*** baoli has quit IRC		19:51
clarkb	but lunchtime, I need to run errands and things	19:51
*** yamamoto has joined #openstack-infra		19:52
*** yamamoto has quit IRC		19:52
*** baoli has joined #openstack-infra		19:52
clarkb	will try to pick os_loganalyze things back up when I get back	19:52
mtreinish	clarkb: in my experience using pymysql instead of the other one just works	19:52
mtreinish	although that's mostly just with subunit2sql stuffs	19:52
fungi	yeah, we already switched something else over to it as well i think, just can't recall what	19:54
mordred	fungi: it's worth a shot for sure	19:54
mordred	fungi: obvs would have to change the db url	19:55
fungi	yep	19:55
mtreinish	fungi: is there something else which uses it during a nodepool build? because that's why we used it with subunit2sql	19:55
*** achanda has quit IRC		19:56
*** smoser has joined #openstack-infra		19:58
smoser	hey. cloud-init using stackforge.	19:59
smoser	https://github.com/stackforge/cloud-init	19:59
mordred	smoser: welcome to the jungle	19:59
smoser	says "Cross-platform instance initialization http://openstack.org"	19:59
smoser	any way to not have it say openstack.org ?	19:59
smoser	i'm fine with it saying 'https://github.com/stackforge' or something.	20:00
*** AJaeger has joined #openstack-infra		20:00
smoser	as a "cost" of free infrastructure.	20:00
*** sarob has quit IRC		20:00
harlowja	hmmm, pretty sure enough people pay $$ for the free part; lol	20:00
mordred	hrm. I'm not 100% sure where that comes from	20:00
harlowja	including yahoo...	20:00
*** mrmartin has quit IRC		20:00
smoser	but i dont want confusion on inclusion of it as part of openstack.	20:00
smoser	well, its free to me, harlowja	20:01
harlowja	smoser fair enough :)	20:01
harlowja	freeish	20:01
mordred	yah. understood - similarly, I do think it's weird to have openstack.org on stackforge projects in the other direction either	20:01
*** pc_m has quit IRC		20:01
mordred	I think it may just be an unplanned thing	20:01
mordred	one sec	20:01
harlowja	mordred wants to use more of cloudinit i hear smoser	20:02
harlowja	;)	20:02
fungi	mordred: maybe that's our default homepage url in projects.yaml?	20:02
mordred	yup	20:03
mordred	and the project entry does not specify a homepage in it	20:03
*** ajo has joined #openstack-infra		20:04
*** ivar-laz_ has joined #openstack-infra		20:04
fungi	smoser: harlowja: you want a "homepage" parameter in openstack-infra/project-config:gerrit/projects.yaml (see pecan for an example)	20:04
mordred	fungi: I don't think jeepyb will update the homepage entry with changes to projects.yaml right?	20:04
fungi	i believe so	20:04
fungi	er, well, it used to	20:04
fungi	maybe we stopped updating on not-creation to cut down on github api failures?	20:05
fungi	now i don't remember	20:05
mordred	I think we did	20:05
mordred	I removed it for now	20:05
*** marcusvrn1 has quit IRC		20:06
harlowja	mordred while we are having a pow wow how can smoser and i help u with https://review.openstack.org/#/c/165914/ :)	20:06
harlowja	hugs not included	20:07
harlowja	lol	20:07
smoser	what do you use there?	20:08
smoser	instead of cloud-init	20:08
*** ivar-lazzaro has quit IRC		20:08
*** medieval1 has joined #openstack-infra		20:08
*** ghostpl_ has quit IRC		20:09
*** medieval1 has quit IRC		20:13
*** sputnik13 has quit IRC		20:13
fungi	smoser: harlowja: rackspace does some funky things with a special secret sauce agent daemon to handle network setup, which we're trying to work around. distro packages of cloud-init (which seem to come by default on a lot of "cloud" images) drag in distro-packaged python library dependencies which also have the potential to conflict with things on our job workers	20:13
fungi	smoser: harlowja: though i think there is hope to work around those things at some point	20:13
fungi	smoser: harlowja: in the meantime we're relying on dhcp where available (seriously? protocols written decades ago already supported network auto-configuration? who knew?) and config drive metadata where not	20:14
*** sputnik13 has joined #openstack-infra		20:14
harlowja	imagine that, dhcp , ha	20:15
harlowja	sounds magical	20:15
mordred	yah	20:15
mordred	what fungi said	20:15
fungi	it is so totally magical	20:15
greghaynes	I wonder what the h and c stand for	20:15
fungi	you know, unless you've ever taken the time to run a dhcp server or hundreds. then it's not so magical	20:15
harlowja	i try to avoid that and let other people do it, ha	20:15
mordred	smoser: I wrote a very small python script to just pull network info and ssh keys out of config-drive if config drive exists	20:16
fungi	greghaynes: the h and c stand for "it does more than networks"	20:16
mordred	smoser: because it's just config drive, it doesnt' need any other depends, so the impact on the dependency chains of he machiens is minimal	20:16
mordred	smoser: I'd love, as you start working on v2 of cloud-init	20:16
mordred	smoser: to chat about how a minimal install of cloud-init that doesn't need extra things for situations like ours could be accomplished	20:16
harlowja	how mimimal do u want mordred ?	20:17
mordred	smoser: if we got to a place where we deleted our simple-init thing and just used a minimal install of cloud-init (that could read the rax vendor data) that would be neat	20:17
harlowja	no deps at all?	20:17
mordred	harlowja: I want zero deps	20:17
mordred	harlowja: for config drive, you need nothing but json, which is in the standard lib	20:17
harlowja	right	20:17
fungi	pure python no deps would be awesome, but we could also install in and run from a virtualenv if we have to	20:17
JayF	Oh man, cloud-init with pure python no deps would be pretty amazing	20:18
harlowja	^ i think is doable; urllib3 and such avoid me from crying when i have to use urllib2	20:18
mordred	we could ... but I'd like to start from teh conversation of what would a pure minimal install look like and how awful would it make harlowja and smoser's life	20:18
JayF	It's a pain right now to ship non-distro-packaged cloud-init as a hoster as well	20:18
smoser	i'm kind of confused on . you could remove cloud-init and its dependencies, but a distro's choice of packages included in their images is always going to potentially conflict with what you're using.	20:19
fungi	you hosters and your crazy ideas about getting to sleep at night	20:19
mordred	smoser: we're not using disto's cloud images	20:19
openstackgerrit	Matthew Treinish proposed openstack/requirements: Add os-testr to global requirements list https://review.openstack.org/170268	20:19
mordred	smoser: we're building our own	20:19
harlowja	http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/requirements.txt doesn't exactly have alot of non-python-deps (i guess yaml might try to pull in a c module)	20:19
JayF	fungi: more like I don't want my customers to have to wake up if they unpin my cloud-init, "upgrade" to a crappy distro version, and their machine break on next reboot	20:19
fungi	smoser: yep, which is why we're building our own base images now to omit things like that	20:19
harlowja	pyserial also (but those are optional)	20:19
*** ddieterly has quit IRC		20:19
*** ddieterly has joined #openstack-infra		20:20
*** sdake_ has joined #openstack-infra		20:20
mordred	smoser: which also has less to do with any given distro choices and more to do with cloud vendors _Also_ modifying images	20:20
fungi	JayF: i understand. also as a customer i want the freedom to do just that and keep both pieces	20:20
mordred	yes	20:20
mordred	this is why DHCP is the better answer	20:20
mordred	because it doesn't require running anything special	20:20
mordred	but, you know, meh	20:20
dansmith	I don't think this dhcp thing is going to catch on	20:20
harlowja	depends on your infrastrucutre layout...	20:20
fungi	dansmith: it has an rfc (several in fact!) and everything. you never know	20:21
JayF	dansmith: DHCP doesn't work for all cases; such as Ironic bare metal in secured networks	20:21
mordred	harlowja: sure. I'm just stating my preference as a rather large cloud consumer	20:21
*** ivar-laz_ has quit IRC		20:21
JayF	dansmith: although I greatly appreciate the sentiment :) I wish I didn't have to use cloud-init to assemble my bonds and hook up vlans and could handle it all server side :/	20:21
smoser	JayF, well we'd for sure like to fix cloud-init for your use case.	20:22
*** ivar-lazzaro has joined #openstack-infra		20:22
JayF	smoser: I'm talking about the deps stuff, more or less	20:22
harlowja	mordred sure; rather large cloud consumer yahoo still using config-drive (dhcp in places); so i'm biased as well (but idk all the reasons, forgot them, lol)	20:22
smoser	but your statmeent of "i don twant my customers to have to wake up if <THEY DO SOMETHING WRONG>" seems a bit impossible to solve .	20:22
JayF	smoser: even if my stuff gets upstream, I still have to inject an updated cloud-init into >50% of my supported images	20:22
mordred	smoser: that's not something wrong	20:22
mordred	smoser: that's a user installing software in their machine	20:22
*** ryanpetrello has quit IRC		20:22
*** markmcclain has quit IRC		20:22
smoser	it is in JayF 's statement.	20:22
smoser	as it will break their system.	20:22
jroll	smoser: JayF had a bad problem statement :P	20:23
JayF	smoser: my statement is the problem still exists even if I require a non-downstream-patched, but newer than $distro_package cloud-init	20:23
mordred	right. but a user shouldn't need to have a patched version of cloud-init for their machien to boot onto the network	20:23
jroll	smoser: what if I upgrade a cloud-init dependency and it breaks cloud-init?	20:23
lifeless	the OS dependency metadata should prevent that	20:24
*** sdake has quit IRC		20:24
jroll	the what now	20:24
lifeless	dpkg/rpm.	20:24
*** ajo has quit IRC		20:24
smoser	well, i think generally you're ruunning into the problems of the way distributions are done.	20:24
JayF	jroll: our pain is because we're using non-distro cloud-init, and distros want to uninstall our version in favor of the distro version	20:24
jeblair	tchaypo: i'd really like to get gertty working for you; it should work fine subscribed to > 100 projects.	20:24
mordred	so - there are two different things here ...	20:24
mordred	one is JayF's pain	20:24
jroll	lifeless: sure, so now they can't install python-xyz which depends on a newer version of a cloud-init dependency, which is even sadder	20:24
mordred	one is ours	20:25
*** markmcclain has joined #openstack-infra		20:25
mordred	they are actually opposite pain	20:25
jeblair	tchaypo: from what you describe, i feel fairly certain there's a bug there.	20:25
fungi	having spent years in ops and customer support at a hosting provider, i'm inclined to believe that whatever breaks inside the bounds of the guest os is up to the customer to fix. and yes i've absolutely been brought in on special calls where i had to fix an 'rm -rf /' a customer did by accident	20:25
lifeless	fungi: +1	20:25
jroll	fungi: even when you embed hacks in their image to make their guest boot at all?	20:25
*** garyh has quit IRC		20:25
fungi	but ultimately, we told them that we would try to help (if they paid us to) but no guarantees it would be recoverable	20:25
jroll	(you as the provider)	20:25
*** gulic has quit IRC		20:25
lifeless	jroll: well, both redhat and canonical have realised that user workloads shouldn't be in the same namespace as platform workloads	20:26
fungi	jroll: right, one of the reasons we tried not to embed hacks in their systems to get them to boot in our environment	20:26
*** ryanpetrello has joined #openstack-infra		20:26
jeblair	tchaypo: feel free to paste some debug logs at me	20:26
tchaypo	jeblair: I have 432Mb of debug log from the last run of gertty	20:26
lifeless	jroll: so as a user if you want python-xyz version NEW, that should be in a new container/venv/click-package/software-collection anyway	20:26
tchaypo	may be a little too large to paste :)	20:26
mordred	lifeless: OR	20:26
jeblair	tchaypo: what's it compress to?	20:26
jroll	lifeless: I agree with not using distro packages, but that's not everyone's world view, see also openstack	20:27
*** ivar-lazzaro has quit IRC		20:27
lifeless	jroll: well, its more nuanced than that - see the new distro packaging formats that are just namespaced off	20:27
tchaypo	54Mb	20:27
jroll	fungi: yeah, that's a larger problem I can't reasonably solve :/	20:28
fungi	jroll: in our case, the main problem with distro-packaged python libs on our job workers is that we have some projects which need a very few select python libs which can't install into a virtualenv (for example python-libvirt until very recently) and so have to operate their text venvs with system site packages enabled	20:28
mordred	yes.	20:28
fungi	er, their test venvs	20:28
lifeless	oh which reminds me, does python-libvirt work with pypi?	20:28
clarkb	lifeless yes	20:29
lifeless	like, if I went batshit insane and wanted to add pypi jobs....	20:29
clarkb	you need libvirt >0.9.8 iirc	20:29
fungi	so minimizing python libraries installed into the global system namespace is something we try to achieve	20:29
lifeless	[see the twitters a few days ago]	20:29
mordred	lifeless: could you expand on that?	20:29
clarkb	pypi jobs?	20:29
mordred	lifeless: I'm probably not going to see the twitters from a few days ago :)	20:29
fungi	lifeless: i don't follow the tweeter, but related at all to your posts on the distutils-sig ml?	20:29
lifeless	fungi: no, entirely different	20:30
fungi	fun!	20:30
jeblair	tchaypo: i will pm you a way to send me the file	20:30
tchaypo	jeblair: it was running for ~3 days, but lost connectivity many times during those 3 days. this time around it’s only been going for an hour or so, but it has touched neutron and nova and hasn’t blown up - currently sitting around 1200 in queue	20:30
lifeless	thread here. https://twitter.com/dreid/status/581108305046704128	20:31
lifeless	bah, I meant pypy. morning holiday brain.	20:31
lifeless	tchaypo: you're poking at the gerrit perf regression again ?	20:31
mordred	lifeless: that is mindless trolling	20:31
fungi	pypy not pypi. no wonder i was confused	20:32
mordred	lifeless: "if openstack wants to obsessively use python, why not invest in making python better?" WE DO	20:32
mordred	what a jerk	20:32
tchaypo	lifeless: only in the sense that I seem to have triggered it again and captured logs	20:32
clarkb	uhm we pypy...	20:33
fungi	indeed. we have jobs	20:33
clarkb	its all there to be used if you want it	20:33
mordred	yeah. someone was cranky and decided to post anti-openstack things	20:33
lifeless	mordred: I know, and I answered :)	20:33
mordred	lifeless: stronger man that I	20:33
mordred	lifeless: I'd kickban that crap	20:33
fungi	we even maintained special worker images until we got trusty where reasonable recent pypy happens naturally	20:33
lifeless	clarkb: oh, we have pypy jobs already ? cool. I'm out of date.	20:33
clarkb	we have had them for a long time now	20:34
fungi	for a year or two anyway	20:34
mordred	lifeless: we've had them for something like 1.5 - 2 years	20:34
clarkb	and as fungi points out we did extra work just for them	20:34
lifeless	libs and cli only?	20:34
mordred	I believe all of swift gates on it	20:34
lifeless	oh awesome	20:34
*** bswartz has joined #openstack-infra		20:34
jroll	lifeless: it's available, just not widely used	20:34
lifeless	I think I'll not add to the thread :)	20:34
mordred	lifeless: :)	20:34
fungi	it's mostly cli jobs which were initially added, but alex was also pretty involved in getting it going for swift yeah	20:34
clarkb	so if people are interested have at it nothibg stops you	20:34
lifeless	instead I'm going to go fly pretent spaceships in 4K	20:34
lifeless	pretend	20:34
clarkb	and this is why I dont twitter	20:34
fungi	we had pypy movers and shakers directly involved in getting it running and up to the present point	20:35
jroll	lifeless: up until recently some eventlet/oslo things like loopingcall didn't work on pypy because shenanigans, we had pypy running on ironic-python-agent for some time	20:35
jroll	but then bugs	20:35
*** unicell has quit IRC		20:35
fungi	so whoever commented that there was no openstack interest in pypy is apparently not actually involved in pypy development in general	20:35
*** unicell has joined #openstack-infra		20:35
fungi	and is instead a bridge troll	20:36
lifeless	fungi: well, I might have said that	20:36
fungi	lifeless: well, you are also a known troll ;)	20:36
*** markmcclain has quit IRC		20:36
jroll	dreid is pretty involved in pypy, fwiw	20:36
lifeless	I didn't think as a project we had a specific interest in it	20:36
jroll	s/pretty/fairly	20:36
*** ryanpetrello has quit IRC		20:36
lifeless	I'm glad we have some jobs	20:36
fungi	though i haven't seen alex in here in recent months	20:37
mordred	lifeless: well, as a project, we don't particularly care, because it doesn't particularly matter	20:37
mordred	lifeless: however, we're pretty open and inviting	20:37
mordred	so when people show up and care	20:37
*** dkranz has quit IRC		20:37
*** mwagner_lap has quit IRC		20:37
lifeless	sure, and thats what I said	20:37
mordred	we do what we can to enable them	20:37
mordred	I'm pretty sure NOBODY cares about pypy	20:37
mordred	it's not just us	20:37
jroll	fungi: he's no longer at rackspace, probably part of it	20:37
*** Bsony has joined #openstack-infra		20:37
jroll	mordred: I would love to run my cloud on pypy	20:37
mordred	jroll: awesome!	20:38
fungi	jroll: oh, well if you run into him let him know we miss him!	20:38
jroll	whether it actually makes an impact is tbd	20:38
jroll	fungi: he's still on freenode :P	20:38
mordred	jroll: (when I say nobody, I mean broadly, not that literally no humans care :) - I think pypy is neat)	20:38
*** sabeen2 has joined #openstack-infra		20:38
openstackgerrit	Jay Faulkner proposed openstack-infra/project-config: Add docs build, requirements check to IPA https://review.openstack.org/170273	20:38
fungi	mordred: you're a robot	20:38
fungi	we have evidence and all	20:38
*** sabeen has quit IRC		20:39
jroll	mordred: ok, that's slightly more reasonable -- however I would love to see folks invest time on it	20:39
jroll	but there's generally too much to do to prioritize it, I'm sure	20:39
mordred	sure!	20:39
clarkb	jroll anyone can :)	20:39
clarkb	we arent stopping you	20:40
jroll	clarkb: oh, I'm aware	20:40
fungi	neat, so alex is working for the us dept. of veterans affairs now	20:40
jroll	like I said, priorities	20:40
lifeless	also whether it will help	20:41
lifeless	if there are bugs that are only fixed by using pypy	20:41
lifeless	I think we'd have a fairly substantial cross-project decision to come to	20:42
clarkb	one issue I do have with pypy is cffi	20:42
lifeless	[to whit: to stop supporting cPython for servers]	20:42
clarkb	ianw has found at least one lolzy problem with that	20:42
lifeless	the install one?	20:42
lifeless	[which is hilarious]	20:42
clarkb	the explode memory use one	20:42
*** unicell has quit IRC		20:43
clarkb	but the install one is another	20:43
lifeless	oh, I haven't seen that one	20:43
*** unicell has joined #openstack-infra		20:43
lifeless	does it have a scrypt implementation internally or something?	20:43
clarkb	lifeless the cparser apparebtly doesnt gc well	20:43
fungi	lifeless: https://github.com/eliben/pycparser/issues/72	20:43
*** AJaeger has quit IRC		20:44
fungi	clarkb: it's like your jenkins bug... maintainer asks for a better bug report immediately, but goes silent as soon as one is provided	20:44
openstackgerrit	Jay Faulkner proposed openstack-infra/project-config: Add docs build, requirements check to IPA https://review.openstack.org/170273	20:44
clarkb	fungi yup	20:44
openstackgerrit	Jay Faulkner proposed openstack-infra/project-config: Add docs build, requirements check to IPA https://review.openstack.org/170273	20:44
lifeless	well	20:45
lifeless	its not really a better report	20:45
lifeless	'uses X memory to parse' isn't a bug per se.	20:45
*** dkliban is now known as dkliban_afk		20:46
clarkb	uses an order magnitude more is...	20:46
fungi	it's certainly fuzzy as to whether the cryptography devs should be digging into it first	20:46
*** markmcclain has joined #openstack-infra		20:46
fungi	though when he opened https://github.com/pyca/pyopenssl/issues/202 they basically said go talk to the cparser folks	20:47
*** ryanpetrello has joined #openstack-infra		20:47
*** signed8bit is now known as signed8bit_ZZZzz		20:47
lifeless	clarkb: apples and oranges	20:47
lifeless	clarkb: the old bindings were a C module - already compiled. The new bindings are ffi based, so import time resolving	20:48
lifeless	clarkb: the most obvious space issue to me is that the AST is being kept around at all, not that the AST is big.	20:48
*** irdkehn is now known as dkehn		20:48
clarkb	thats not apples and oranges. we use pyopenssl it uses way more memory now for same thing. that is a bug	20:49
lifeless	that is	20:49
lifeless	this bug report is not that bug report	20:49
*** achanda has joined #openstack-infra		20:49
lifeless	this bug report is that pycparser is a large chunk of ram inprocess of a thing using pyopenssl	20:50
fungi	i think it's just going to resort in us pinning to cryptography<=0.13 because 0.14 introduces a larger memory requirement	20:50
*** signed8bit_ZZZzz is now known as signed8bit		20:50
lifeless	fungi: I think thats wise	20:50
fungi	and letting the cryptography folks know that if they want to see projects using 0.14+ they should look into making it less memory-hungry	20:50
*** annegentle has quit IRC		20:51
*** sabeen2 has quit IRC		20:51
greghaynes	dont you need to pin pyopenssl, not cryptography?	20:51
greghaynes	cryptography has had the bindings for a while AIUI	20:51
*** sabeen1 has joined #openstack-infra		20:51
greghaynes	its just that pyopenssl switched to using them	20:51
clarkb	yes	20:52
fungi	er, right pyopenssl<=0.13	20:52
fungi	pyopenssl 0.14 started using the cryptography library, which is the source of the difference in behavior	20:53
greghaynes	yerp	20:53
*** lpetrut has quit IRC		20:53
*** annegentle has joined #openstack-infra		20:54
*** bswartz has quit IRC		20:55
*** annegentle has quit IRC		20:55
*** ddieterly has quit IRC		20:57
*** packet has quit IRC		20:58
*** rfolco has quit IRC		20:58
*** peristeri has quit IRC		20:58
*** shashankhegde has joined #openstack-infra		20:59
dtroyer	mordred: so os-client-config doesn't have an LP project…should I just create it or is there other process for that?	21:00
*** dannywilson has joined #openstack-infra		21:00
dtroyer	clarkb, fungi: ^^^ also	21:01
*** patrickeast_ has joined #openstack-infra		21:01
*** ajo has joined #openstack-infra		21:01
sdague	clarkb: so, one last though on the pass through thing before I run to make dinner	21:01
*** annegentle has joined #openstack-infra		21:01
*** nelsnelson has quit IRC		21:02
sdague	can we just make our own apache pass through separate from os-loganalyze?	21:02
sdague	so we 302 to that	21:02
*** patrickeast has quit IRC		21:02
*** patrickeast_ is now known as patrickeast		21:02
*** melwitt has quit IRC		21:02
*** melwitt_ has joined #openstack-infra		21:02
*** dannywil_ has quit IRC		21:02
*** sdake_ is now known as sdake		21:02
sdague	or do we actually need python code to read out of swift in all cases here?	21:02
mordred	dtroyer: uh - oh - yeah - I guess it should have one. it's been a while since I've made one of those ...	21:02
*** melwitt_ has quit IRC		21:03
fungi	dtroyer: there's a writeup dhellmann did in the infra manual... just a sec	21:03
clarkb	sdague we could	21:03
fungi	dtroyer: http://docs.openstack.org/infra/manual/creators.html#set-up-launchpad	21:03
jlvillal	Did the "Toggle CI" button disappear for anyone else?	21:03
*** melwitt_ has joined #openstack-infra		21:03
clarkb	sdague its all mod rewritten in there and we could add a similar rule	21:04
clarkb	to what exists	21:04
clarkb	jeblair ^	21:04
morganfainberg	did the "toggle CI" thing disappear from gerrit?	21:04
*** Krinkle is now known as Krinkle\|detached		21:04
jlvillal	That's what I asked morganfainberg ! :)	21:04
*** packet has joined #openstack-infra		21:04
dtroyer	fungi: thx	21:04
sdague	clarkb: it would simplify the thinking about os-loganalyze if it wasn't a generic layer, just the layer we hit when we cared about the logs adds.	21:04
morganfainberg	jlvillal hah! :)	21:04
sdague	anyway, will chat tomorrow about it	21:05
fungi	morganfainberg: jvrbanac: i think we may have added a recent commit which modified them and sometimes gerrit seems to not notice when we tickle the file it watches to know to reload them. i'll give it a nudge	21:05
jlvillal	morganfainberg: It disappeared for me	21:05
fungi	see if it comes back now	21:05
fungi	er, jlvillal (sorry jvrbanac!)	21:05
morganfainberg	fungi, yep back now	21:05
morganfainberg	fungi, thnx	21:05
fungi	np. sorry for the annoyance	21:05
*** melwitt_ is now known as melwitt		21:06
jlvillal	fungi: It is back. Thanks! :)	21:06
morganfainberg	fungi, hah not an annoyance at all.	21:06
morganfainberg	fungi, just suddenly looked different	21:06
*** packet has quit IRC		21:07
clarkb	sdague thats a good point about it constraining os-loganalyzes feature set	21:07
fungi	i think our puppet attempt at touching ~gerrit2/review_site/etc/GerritSiteHeader.html when modifying javascript isn't always sufficient. i've noticed that if i manually modify the javascript and then touch that file immediately without delaying a few seconds, gerrit doesn't refresh the changes	21:07
*** emagana has joined #openstack-infra		21:07
*** davideag_ has quit IRC		21:08
*** MaxV has joined #openstack-infra		21:08
sdague	clarkb: or if we need it in python, put it in a separate wsgi app in tree	21:08
sdague	again, to simplify roles	21:08
*** medieval1 has joined #openstack-infra		21:09
sdague	because the break out point for pass through is weird given the current generator stack	21:09
clarkb	ya I have a plan to break it out and simplify the generator dtack	21:09
*** otter768 has joined #openstack-infra		21:09
sdague	anyway, random thoughts. But I need to deal with dinner.	21:09
*** emagana has quit IRC		21:09
clarkb	I may write that and the vhost change both so we can have options	21:09
sdague	clarkb: ok, before you do that, lets sync up, because there are some other things I want to add in that the current stack makes easy	21:10
*** emagana has joined #openstack-infra		21:10
sdague	maybe tomorrow once you get on	21:10
fungi	aha, it's storyboard that i was trying to remember is using pymysql	21:10
clarkb	sdague ok	21:10
*** ajmiller_ has joined #openstack-infra		21:10
*** aysyd has quit IRC		21:11
*** dustins has quit IRC		21:11
mtreinish	fungi: heh, yeah I wouldn't have guessed that :)	21:12
*** doude has quit IRC		21:13
*** medieval1 has quit IRC		21:13
*** tiswanso_ has quit IRC		21:13
*** otter768 has quit IRC		21:14
*** ajmiller has quit IRC		21:14
*** ldnunes has quit IRC		21:14
*** emagana has quit IRC		21:14
JayF	Hey, someone have a moment to help me figure out something with a failed test? http://logs.openstack.org/73/170273/3/check/gate-project-config-layout/a631334/console.html is indicating ironic-python-agent-docs does not exist (which is correct) but Ironic has ironic docs jobs that don't appear to be defined in jenkins/jobs/ironic.yaml	21:15
*** bcourt has quit IRC		21:15
JayF	So wondering what I'm missing from https://review.openstack.org/#/c/170273/ to make the job work	21:15
*** MaxV has quit IRC		21:15
*** emagana has joined #openstack-infra		21:16
greghaynes	fungi: yep, and that mostly 'just worked' except for one bug that I really hope doesnt apply to nodepool	21:17
greghaynes	which basically comes down to the way oslo.db does ForeignKeyConstraintError is nasty	21:18
*** sdake_ has joined #openstack-infra		21:18
*** xyang1 has quit IRC		21:18
*** davideagnello has joined #openstack-infra		21:18
clarkb	no oslo.db in nodepool	21:18
greghaynes	another crisis averted ;)	21:19
openstackgerrit	Jeremy Stanley proposed openstack-infra/nodepool: Switch from MySQL-python to PyMySQL https://review.openstack.org/170289	21:19
fungi	greghaynes: clarkb; ^ guess we'll find out	21:19
greghaynes	it also especially doesnt apply because no py3 for nodepool and the encoding issues only show up there	21:19
*** Bsony has quit IRC		21:21
*** sdake has quit IRC		21:22
*** dkranz has joined #openstack-infra		21:22
*** ajo has quit IRC		21:23
*** sdake has joined #openstack-infra		21:23
*** Krinkle\|detached is now known as Krinkle		21:23
fungi	assuming that works out, i'll submit the accompanying puppet-nodepool (and system-config?) change(s)	21:24
* fungi will brb		21:26
*** sdake_ has quit IRC		21:27
*** davideagnello has quit IRC		21:28
*** ayoung has quit IRC		21:30
*** sputnik13 has quit IRC		21:30
*** davideagnello has joined #openstack-infra		21:32
*** ociuhandu has quit IRC		21:32
*** sputnik13 has joined #openstack-infra		21:32
*** masayukig has quit IRC		21:33
*** sweston has quit IRC		21:33
*** mgagne has quit IRC		21:33
*** carl_baldwin has quit IRC		21:33
*** jlvillal has quit IRC		21:34
*** Ng has quit IRC		21:34
*** adam_g has quit IRC		21:34
*** tqtran_ has joined #openstack-infra		21:34
openstackgerrit	Ramy Asselin proposed openstack-infra/puppet-openstackci: Simple script to setup and test log server https://review.openstack.org/169117	21:34
*** taron has quit IRC		21:34
*** roaet_ has quit IRC		21:34
*** jbryce has quit IRC		21:34
*** dimtruck has quit IRC		21:34
*** zz_gondoi has quit IRC		21:35
*** dougwig has quit IRC		21:35
*** edwarnicke has quit IRC		21:35
*** cyeoh has quit IRC		21:35
*** ttx has quit IRC		21:35
*** freyes has quit IRC		21:35
*** wendar has quit IRC		21:35
*** hockeynut has quit IRC		21:35
*** wendar has joined #openstack-infra		21:35
*** dangers is now known as dangers_away		21:36
*** julim_ has joined #openstack-infra		21:36
*** sweston has joined #openstack-infra		21:36
*** crinkle_ has joined #openstack-infra		21:36
*** morganfainberg has quit IRC		21:37
*** Hunner has quit IRC		21:37
*** akislitsky has quit IRC		21:37
*** rm_work has quit IRC		21:37
*** morganfainberg has joined #openstack-infra		21:37
*** persia_ has joined #openstack-infra		21:37
*** persia_ has quit IRC		21:37
*** persia_ has joined #openstack-infra		21:37
*** crinkle has quit IRC		21:38
*** crinkle_ is now known as crinkle		21:38
*** emagana has quit IRC		21:38
*** edwarnicke has joined #openstack-infra		21:38
*** tchaypo_ has joined #openstack-infra		21:38
*** Ng has joined #openstack-infra		21:38
*** erikmwilson has quit IRC		21:38
*** tchaypo has quit IRC		21:38
*** rainya has quit IRC		21:38
*** krtaylor has quit IRC		21:38
*** hughsaunders has quit IRC		21:38
*** erw has quit IRC		21:38
*** lbragstad has quit IRC		21:38
*** radez has quit IRC		21:38
*** persia has quit IRC		21:38
*** comstud has quit IRC		21:38
*** jvrbanac has quit IRC		21:38
*** tqtran has quit IRC		21:38
*** julim has quit IRC		21:38
*** mfink has quit IRC		21:38
*** otherwiseguy has quit IRC		21:38
*** gus has quit IRC		21:38
*** mfink_ has joined #openstack-infra		21:38
*** otherwiseguy has joined #openstack-infra		21:39
*** baoli has quit IRC		21:39
*** gus has joined #openstack-infra		21:39
*** erikmwilson has joined #openstack-infra		21:39
*** jlvillal has joined #openstack-infra		21:39
*** comstud has joined #openstack-infra		21:39
*** lbragstad has joined #openstack-infra		21:39
*** ttx has joined #openstack-infra		21:39
*** radez has joined #openstack-infra		21:39
*** roaet_ has joined #openstack-infra		21:39
*** rainya has joined #openstack-infra		21:39
*** adam_g has joined #openstack-infra		21:39
*** gondoi has joined #openstack-infra		21:39
*** tchaypo_ is now known as tchaypo		21:39
*** hockeynut has joined #openstack-infra		21:39
*** adam_g has quit IRC		21:40
*** adam_g has joined #openstack-infra		21:40
*** mgagne has joined #openstack-infra		21:40
*** masayukig has joined #openstack-infra		21:40
*** zz_dimtruck has joined #openstack-infra		21:40
*** jbryce has joined #openstack-infra		21:40
*** jvrbanac has joined #openstack-infra		21:40
*** freyes has joined #openstack-infra		21:40
*** zz_dimtruck is now known as dimtruck		21:40
*** miqui has quit IRC		21:40
*** hughsaunders has joined #openstack-infra		21:40
*** Hunner has joined #openstack-infra		21:40
*** cyeoh has joined #openstack-infra		21:41
*** taron has joined #openstack-infra		21:41
*** rm_work has joined #openstack-infra		21:41
*** rm_work has quit IRC		21:41
*** rm_work has joined #openstack-infra		21:41
*** krtaylor has joined #openstack-infra		21:41
*** akislitsky has joined #openstack-infra		21:41
*** hdd has quit IRC		21:41
*** baoli has joined #openstack-infra		21:42
*** annegentle has quit IRC		21:44
*** dougwig has joined #openstack-infra		21:45
*** erw has joined #openstack-infra		21:45
clarkb	fungi +2 but didnt approve because I cant watch it	21:48
JayF	clarkb: do you have 5m to spare to help me with a project-config change? I think there's just one piece of the puzzle I'm missing/don't understand.	21:49
*** wicope has quit IRC		21:49
clarkb	JayF in a bit. Almost back home from mid day errands	21:50
JayF	clarkb: sweet! Thanks	21:50
*** doug-fish has left #openstack-infra		21:52
clarkb	ask.o.o 502ing for anyone else?	21:52
*** gordc has quit IRC		21:53
JayF	clarkb: WFM, at least just loading the homepage	21:54
clarkb	hrm	21:54
*** esker has quit IRC		21:54
*** gordc has joined #openstack-infra		21:54
JayF	clarkb: I checked both http and https for throughness, and it's working and loading at a speed I'd expect	21:55
clarkb	ya neither work from my phone	21:55
JayF	loads on chrome from my moto x (android 5.0)	21:56
openstackgerrit	James E. Blair proposed openstack-infra/zuul: Delay Gerrit events by 5s https://review.openstack.org/170300	21:58
*** dannywilson has quit IRC		21:58
*** dannywilson has joined #openstack-infra		21:59
jeblair	sdague, clarkb, fungi, mordred: ^ that's the "fix" to the other known issue from CRD. i think that's all of 'em.	21:59
*** alexpilotti has quit IRC		21:59
*** garyh has joined #openstack-infra		21:59
jeblair	clarkb: i agree with your comments, but I think i'm going to leave 170155 as-is.	21:59
*** signed8bit has quit IRC		22:00
clarkb	jeblair: wfm	22:02
clarkb	JayF: ok I am home	22:02
*** annegentle has joined #openstack-infra		22:02
JayF	clarkb: https://review.openstack.org/#/c/170273/ is the change in question -- the test is failing saying ironic-python-agent-docs doesn't exist, but I'm confused and can't find where the equivalant job is added for Ironic	22:03
*** gordc has quit IRC		22:04
*** masoz has joined #openstack-infra		22:04
clarkb	JayF: it comes from the openstack-publish-jobs.yaml file	22:05
clarkb	JayF: so in jenkins/jobs/projects.yaml under ipa you want to add openstack-publish-jobs to your job list	22:05
clarkb	JayF: ironic has this if yo uwant to see an example	22:06
*** hodos has quit IRC		22:06
clarkb	sdague: the zuul stuff in gerrit is shiny	22:07
*** melwitt has quit IRC		22:07
*** melwitt has joined #openstack-infra		22:07
clarkb	though doesn't seem to load each time? in any case \o/	22:07
JayF	clarkb: how does that jenkins/jobs/projects.yaml interact with jenkins/jobs/ironic-python-agent.yaml ... ipa has entries in both	22:07
*** cdent has quit IRC		22:08
*** melwitt has quit IRC		22:08
*** melwitt has joined #openstack-infra		22:09
*** melwitt has quit IRC		22:09
clarkb	JayF: projects.yaml is where we instantiate job templates. the other .yaml files either create jobs directly or define templates that must be instantiated with some arguments	22:09
*** medieval1 has joined #openstack-infra		22:10
JayF	clarkb: okay, that makes sense. I understand a bit more now, thanks. I'll push up that fix and then would appreciate a review on it if it looks good then :D	22:11
* JayF thrilled to finally have docs for IPA		22:11
clarkb	JayF: basically our jobs come in two flavors, the - job: and the - job-template. templates have to be explicitly instantiated for each occurence	22:11
clarkb	regular jobs are added directly to jenkins as is	22:11
*** annegentle has quit IRC		22:11
*** annegentle has joined #openstack-infra		22:12
openstackgerrit	Jay Faulkner proposed openstack-infra/project-config: Add docs build, requirements check to IPA https://review.openstack.org/170273	22:12
*** packet has joined #openstack-infra		22:12
*** SumitNaiksatam has quit IRC		22:12
*** sarob has joined #openstack-infra		22:13
*** medieval1 has quit IRC		22:14
*** oomichi_ has joined #openstack-infra		22:16
*** gulic has joined #openstack-infra		22:17
*** marcusvrn has joined #openstack-infra		22:18
*** andreykurilin_ has joined #openstack-infra		22:18
*** __TheDodd__ has quit IRC		22:19
*** marcusvrn has quit IRC		22:19
*** marcusvrn has joined #openstack-infra		22:19
*** crc32 has quit IRC		22:20
*** smccully has quit IRC		22:21
fungi	race in zuul's test_crd_check_git_depends test? http://logs.openstack.org/55/170155/5/gate/gate-zuul-python27/3500eb0/console.html	22:22
fungi	testtools.matchers._impl.MismatchError: u'1,1 2,1' != '1,1'	22:22
*** bknudson has quit IRC		22:22
*** marun has quit IRC		22:24
clarkb	I think I have a couple zuul changes to fix some of those	22:24
*** marun has joined #openstack-infra		22:24
*** JayJ_ has quit IRC		22:25
*** bruff has joined #openstack-infra		22:25
*** melwitt has joined #openstack-infra		22:25
fungi	looks like self.history ordering may not be deterministic there	22:25
bruff	ping for an infrastructure account question for a CI account	22:26
clarkb	fungi: it has to do with how we release the jobs, it can be made deterministic	22:27
*** annegentle has quit IRC		22:27
clarkb	fungi: which is what my changes do, they more carefully release things in order so that we get deterministic results	22:27
clarkb	bruff: ask away	22:27
fungi	yeah, self.history[0].changes has what the test was expecting in self.history[-1].changes instead	22:27
*** kgiusti has left #openstack-infra		22:27
*** sigmavirus24 is now known as sigmavirus24_awa		22:27
bruff	clarkb - I'm trying to associate an CI account with an OpenID. Our CI account was added manually as	22:27
bruff	13144 oracle-zfssa-ci Oracle ZFSSA CI	22:27
bruff	I was hoping to create an OpenId account and match up with this account, but now I've got another account, when I tried to login to review.openstack.org.	22:27
bruff	Account ID 15863	22:27
bruff	I would like to keep an account for review.openstack.org using.	22:28
bruff	userid: oracle-zfssa-ci	22:28
bruff	name: Oracle ZFSSA CI	22:28
*** unicell1 has joined #openstack-infra		22:28
*** unicell has quit IRC		22:28
fungi	clarkb: i'll find and review your changes, and recheck the one that failed to merge on this in the meantime	22:28
clarkb	bruff: we cannot merge accounts, we have just been having third party ci users use the newer openid based account as they move that direction	22:29
bruff	clarkb: so can I get the userid on the new account set to oracle-zfssa-ci ?	22:30
*** melwitt has quit IRC		22:31
fungi	clarkb: is it mainly just https://review.openstack.org/160491 ? i guess the other crd tests need something similar to what you did to the duplicates test there?	22:32
clarkb	bruff: no, they have to be unique, you will haveto use a new username	22:32
clarkb	bruff: but that should be fine since typically the only person that needs to know what that string is is the third party CI operator	22:32
*** amitgandhinz has quit IRC		22:32
clarkb	fungi: https://review.openstack.org/#/c/160491/ and https://review.openstack.org/#/c/160050/	22:33
*** weshay has quit IRC		22:33
bruff	clarkb: Is this the username field on the Settings Profile page on review.openstack.org?	22:33
*** packet has quit IRC		22:33
clarkb	bruff: yes	22:33
bruff	clarkb: Then that propagates through the various Infra servers from there?	22:34
bruff	clarkb: Also for CI user, do I need to agree to the New Contributor Agreement for that user?	22:35
*** tiswanso has joined #openstack-infra		22:36
clarkb	bruff: the only server that knows about that username is gerrit	22:39
*** garyh has quit IRC		22:39
clarkb	no CLA required to post CI votes	22:39
*** shardy has quit IRC		22:39
bruff	clarkb: OK, how do I associate it with the CI group?	22:40
asselin_	bruff, I can do that	22:40
fungi	clarkb: it's not immediately obvious to me how 160050 deals with nondeterminism in the tests. were you seeing build cancellation racing with status updates in the crd tests?	22:41
bruff	Full Name	22:41
bruff	oracle-zfssa-ci	22:41
bruff	Email Address	22:41
bruff	OPENSTCK-CI-TEST_US_GRP@oracle.com	22:41
bruff	Registered	22:41
bruff	Apr 2, 2015 1:33 PM	22:41
bruff	Account ID	22:41
bruff	15863	22:41
bruff	is the new CI account user	22:41
*** tiswanso has quit IRC		22:41
asselin_	bruff, you're here now: https://review.openstack.org/#/admin/groups/270,members	22:42
*** eharney has quit IRC		22:43
bruff	clarkb: asselin_: Many thanks	22:46
*** lhcheng has joined #openstack-infra		22:48
*** bhunter71 has quit IRC		22:50
*** wenlock has quit IRC		22:50
fungi	clarkb: hrm, and 160491 failed in the gate on a different problem in the zuul unit tests now	22:52
*** lhcheng has left #openstack-infra		22:53
fungi	'SUCCESS' != 'ABORTED': Build should have been aborted (in test_abandoned_gate)	22:53
openstackgerrit	Merged openstack-infra/zuul: Add commit needed-by https://review.openstack.org/170155	22:53
mtreinish	fungi: heh, on my gerritbot patch I just realized the same reasoning could also apply to subunit2sql, which reports in both :)	22:53
clarkb	fungi: I think thats another race, the job is running to completion before the thing that triggers abandonment occurs	22:55
*** mriedem is now known as mriedem_away		22:58
*** bruff has left #openstack-infra		22:58
*** baoli has quit IRC		22:59
fungi	agreed	23:02
jeblair	fungi: are you looking into that or do you want to hand it off to me?	23:02
* jeblair is back and catching up		23:03
*** dannywilson has quit IRC		23:03
fungi	jeblair: i was about to dig into finding a fix for the nondeterminism in test_crd_check_git_depends and got sidetracked by test_abandoned_gate, but am trying to stick with the first problem first	23:04
*** sabeen1 has quit IRC		23:04
jeblair	fungi: i'll take abandoned_gate then	23:05
fungi	jeblair: also if you're fine with https://review.openstack.org/170289 i think that may help with our current install/upgrade problems on nodepool.o.o	23:05
jeblair	clarkb: i double checked and i think i did not copy the problem you fixed in 160491 to any new tests	23:05
*** andreykurilin_ has quit IRC		23:05
*** dannywilson has joined #openstack-infra		23:05
*** achanda has quit IRC		23:08
openstackgerrit	Jeremy Stanley proposed openstack-infra/system-config: Switch nodepool from MySQL-python to PyMySQL https://review.openstack.org/170314	23:09
*** otter768 has joined #openstack-infra		23:10
*** medieval1 has joined #openstack-infra		23:10
*** unicell1 has quit IRC		23:11
*** unicell has joined #openstack-infra		23:11
clarkb	jeblair: cool	23:12
jeblair	fungi: +2d both pymysql changes	23:12
jeblair	did not aprv	23:12
jeblair	(i expect to head out in a few mins)	23:12
fungi	jeblair: no problem. i can take care of approving and restarting nodepool (and if absolutely necessary reverting and restarting it again)	23:13
fungi	thanks!	23:13
*** medieval1 has quit IRC		23:14
*** otter768 has quit IRC		23:15
*** sarob has quit IRC		23:16
*** mwagner_lap has joined #openstack-infra		23:16
mordred	fungi: woot!	23:21
*** hichihara has joined #openstack-infra		23:21
*** dboik_ has joined #openstack-infra		23:21
openstackgerrit	Merged openstack-infra/nodepool: Switch from MySQL-python to PyMySQL https://review.openstack.org/170289	23:22
*** sarob has joined #openstack-infra		23:23
*** dboik has quit IRC		23:25
*** patrickeast has quit IRC		23:25
*** dboik_ has quit IRC		23:26
openstackgerrit	Jeremy Stanley proposed openstack-infra/zuul: Fix race in CRD Git depends test https://review.openstack.org/170317	23:26
*** dannywilson has quit IRC		23:26
*** sdake_ has joined #openstack-infra		23:27
fungi	stab in the dark ^ based on clarkb's earlier fix, because something's horked up with my gcc at the moment so tox can't install stuff needing to build c extensions	23:27
*** sarob has quit IRC		23:28
*** rbradfor has quit IRC		23:30
*** sdake has quit IRC		23:30
*** ayoung has joined #openstack-infra		23:32
*** arosen has quit IRC		23:32
*** sdake has joined #openstack-infra		23:33
*** gulic has quit IRC		23:35
*** sdake_ has quit IRC		23:37
fungi	hrm, that didn't do what i expected	23:38
openstackgerrit	James E. Blair proposed openstack-infra/zuul: Fix checking all builds are waiting in tests https://review.openstack.org/170319	23:38
jeblair	clarkb, fungi: ^ so that was fun!	23:39
*** ZZelle_ has quit IRC		23:39
*** pritesh has joined #openstack-infra		23:39
jeblair	clarkb, fungi: i'm pretty sure the race condition in the test was an ACTUAL race condition in reality	23:39
fungi	jeblair: oh! that could be affecting more tests than just that one to	23:39
jeblair	so, um, "it only happens in tests" takes another knock :)	23:39
fungi	indeedy!	23:40
fungi	and "tests do find bugs" gets another notch	23:40
*** amitgandhinz has joined #openstack-infra		23:40
*** annegentle has joined #openstack-infra		23:41
*** MarkAtwood has quit IRC		23:45
*** rlucio has quit IRC		23:46
krotscheck	Where all do I have to define a job again? I thought I caught all places in https://review.openstack.org/#/c/169829/ , but clearly I missed one :/	23:47
mordred	krotscheck: you have defined a job-template but have not instantiated the template anywhere	23:49
*** FatBack has joined #openstack-infra		23:49
mordred	krotscheck: look in jenkins/jobs/projects.yaml for a horizon section	23:50
krotscheck	mordred: Thanks, checking	23:50
mordred	that's where you'll instantiate the job template for the horizon project	23:50
mordred	krotscheck: the multiple places thing is one of the things we're hoping to fix with zuulv3	23:50
*** erlon has quit IRC		23:51
openstackgerrit	Michael Krotscheck proposed openstack-infra/project-config: Add NPM Command Execution as a Build Target https://review.openstack.org/169829	23:52
krotscheck	mordred: Let's see if that does it, thanks :)	23:52
fungi	mordred: interestingly, when puppet upgraded nodepool from git just now, it set the package version to 0.0.1.post100	23:52
fungi	i'm guessing we got a new pbr in the bargain	23:53
mordred	fungi: yay!	23:53
*** bswartz has joined #openstack-infra		23:53
mordred	fungi: I wanna see new metrics!!!	23:53
*** jaypipes has quit IRC		23:54
fungi	well, i haven't restarted nodepoold just yet	23:54
mordred	:)	23:54
fungi	still checking the logs to make sure the pip install succeeded with the switch to pymysql	23:54
* mordred is weirdly overly excited to see time breakdowns by API call		23:54
fungi	i mean it _seems_ to have	23:54
fungi	yep, syslog lgtm	23:55
fungi	okay, restarting nodepoold now	23:55
mordred	the fun part is we should immediately get some counters, because it runs a bunch of flavor lists and stuff on restart to repopulate its internal caches and stuff	23:56
*** annegentle has quit IRC		23:57
jeblair	mordred: if you have a sec for 170319, it would be good to get it in before a zuul restart	23:58
jeblair	which i think i might to tomorrow	23:58
jeblair	mordred: oops, meant 170300	23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!