Tuesday, 2014-07-01

*** wenlock has quit IRC		00:00
anteaya	clarkb: okay	00:00
*** msabramo has joined #openstack-infra		00:01
pleia2	re: ansible support, I think we're stuck on this one: https://review.openstack.org/#/c/87821/17 (and I don't know the answer)	00:01
*** nati_uen_ has quit IRC		00:01
anteaya	clarkb: and I have commented back	00:05
*** anode has quit IRC		00:11
*** msabramo has quit IRC		00:12
*** zz_dimtruck is now known as dimtruck		00:12
*** r1chardj0n3s is now known as r1chard_afk		00:13
*** annegent_ has joined #openstack-infra		00:15
*** zigo has quit IRC		00:17
*** zigo has joined #openstack-infra		00:21
*** annegent_ has quit IRC		00:23
*** msabramo has joined #openstack-infra		00:23
*** markmcclain has quit IRC		00:24
*** gangil has quit IRC		00:24
*** markmcclain has joined #openstack-infra		00:26
*** gokrokve has quit IRC		00:26
*** markmcclain has quit IRC		00:27
*** markmcclain has joined #openstack-infra		00:27
*** praneshp has quit IRC		00:27
clarkb	anteaya: thank you	00:28
clarkb	anteaya: I don't think jeblair has changed his mind. So that lgtm	00:28
clarkb	mattoliverau: re comment on https://review.openstack.org/#/c/87821/ does my comment help clear it up?	00:31
anteaya	clarkb: k thanks	00:32
*** zigo has quit IRC		00:35
*** pradk has quit IRC		00:35
anteaya	clarkb: could you walk me through rebasing this? https://review.openstack.org/#/c/101361/	00:36
anteaya	using teh button on the gui failed	00:36
anteaya	I have updated the remotes	00:36
anteaya	and am in the branch	00:36
anteaya	I have rebased before and blown away my changes	00:36
clarkb	anteaya: next step is `git rebase -i master` assuming you updated your local master	00:36
anteaya	I'd like to avoid that this time	00:36
anteaya	I updated my local master yes	00:36
clarkb	this will open an editor with the list of changes we are rebasing. should be just one line with that change	00:37
clarkb	if it is just that one line, save the file and quit, this starts the rebase	00:37
anteaya	on master I had run `git remote update` and then `git pull --ff-only origin master`	00:37
clarkb	yup	00:37
anteaya	okay I will run git rebase -i master	00:37
clarkb	anteaya: run that command with your change checked out	00:37
jesusaurus	clarkb: if both foo and foo.gz exist, then that find command will ignore foo.gz but will try to compress foo to foo.gz (and prompt about overwriting foo.gz) aiui	00:37
clarkb	that tells git to rebase the current thing (your change) onto master	00:37
clarkb	jesusaurus: gotcha, so uh why not delete foo?	00:38
anteaya	yes I have done that, from the change branch I have run git rebase -i master	00:38
jesusaurus	clarkb: are we sure foo.gz is up-to-date?	00:38
anteaya	I am in an editor	00:38
clarkb	jesusaurus: in our case yes	00:38
clarkb	anteaya: cool, just double check it has one line at the top for your change and no othercommits then save and quit	00:39
*** sarob has quit IRC		00:39
clarkb	jesusaurus: these are logs in job run specific dirs	00:39
anteaya	this is what I see: http://paste.openstack.org/show/85191/	00:39
clarkb	jesusaurus: those logs should only ever be written once	00:39
jesusaurus	ah, true	00:39
clarkb	anteaya: yup that looks correct, go ahead and save and quit	00:39
jesusaurus	i was missing the context of what was being compressed	00:39
anteaya	clarkb: k	00:39
*** sarob has joined #openstack-infra		00:39
*** salv-orlando has quit IRC		00:40
jesusaurus	clarkb: then yeah, we can just delete foo	00:40
clarkb	anteaya: you will probably get a message that says blah blah conflicts	00:40
*** GheRivero is now known as GheRivero_afk		00:40
clarkb	anteaya: at this point do `git status` and it will show you the files that have conflicts	00:40
anteaya	clarkb: I was in vi and did :wq and got: http://paste.openstack.org/show/85192/	00:40
clarkb	anteaya: yup that is what I expect, next step is `git status` to see which files have conflicts	00:41
anteaya	kk	00:41
anteaya	http://paste.openstack.org/show/85193/	00:41
jesusaurus	clarkb: how long does it take for upstream to run nodepool image-update? does your initial puppet run also take forever and a day?	00:42
clarkb	anteaya: so for each file in the unmerged paths list. open that file. edit it to resolve conflicts. Basically make the file look like you want it to be in the resulting commit (don't forget to remove the git annotation tags like >>>>>>>>>>> HEAD or whatever)	00:42
anteaya	kk	00:42
clarkb	anteaya: once you have edited the files so that they look like you want them to. Do git add filename	00:42
clarkb	do that for each filename	00:42
anteaya	okay	00:42
clarkb	then once all of that is done do git rebase --continue	00:42
clarkb	jesusaurus: it takes about an hour iirc	00:43
jesusaurus	clarkb: thanks	00:43
clarkb	jesusaurus: initial puppet run for non devstack slaves can be slow	00:43
clarkb	because all the packages and db stuff	00:43
anteaya	clarkb: alright, I'll be a while	00:43
clarkb	anteaya: np	00:43
*** sarob has quit IRC		00:44
*** marcoemorais has quit IRC		00:45
*** marcoemorais has joined #openstack-infra		00:45
clarkb	anteaya: feel free to put up a file if you want an example of how I would resolve one. But once you figure out what git is telling you it should be straightforward	00:45
clarkb	one chunk is old, one chunk is new and you have to make a decision on what the next version should look like	00:45
*** zigo has joined #openstack-infra		00:47
anteaya	clarkb: I screwed up, because I forgot to bring down the latest patchset of that series (which came from dhelmann) before I began	00:47
anteaya	right, I am seeing that	00:47
anteaya	clarkb: how do I abort this rebase?	00:47
anteaya	or can I at this point?	00:47
*** marcoemorais has quit IRC		00:49
clarkb	you can	00:49
clarkb	anteaya: just do `git rebase --abort`	00:49
*** marcoemorais has joined #openstack-infra		00:49
anteaya	clarkb: k	00:49
clarkb	that will put you back to where you started	00:49
anteaya	great	00:49
clarkb	and you can update your branch and start again	00:49
anteaya	right	00:49
anteaya	now I do git review -d patchset,patchnumber	00:50
anteaya	but will that take me to a new branch?	00:50
clarkb	it might	00:51
clarkb	you can also leave off patchnumber and it will do the latest patch	00:51
mattoliverau	clarkb: thanks, I've changed my vote, but also gave some future suggestions if we start using ansible for more complicated tasks. Not for that change, but just from what I've learned from my somewhat limited experience with ansible :)	00:51
anteaya	kk	00:52
clarkb	mattoliverau: thanks	00:52
*** msabramo has quit IRC		00:52
anteaya	yeah, on a new branch have the latest patchset	00:53
anteaya	trying again	00:53
*** dprince has joined #openstack-infra		00:53
*** marcoemorais has quit IRC		00:55
*** r1chard_afk is now known as r1chardj0n3s		00:56
*** dkranz has joined #openstack-infra		00:56
*** markmcclain has quit IRC		00:56
*** HenryG has quit IRC		00:57
*** bdpayne has quit IRC		00:57
*** HenryG has joined #openstack-infra		00:59
*** msabramo has joined #openstack-infra		01:00
*** jaypipes has joined #openstack-infra		01:05
*** gokrokve has joined #openstack-infra		01:05
*** dstanek is now known as dstanek_zzz		01:06
*** gokrokve_ has joined #openstack-infra		01:07
*** gokrokve has quit IRC		01:10
ianw	does the infra puppet scripts support puppet 3?	01:12
*** msabramo has quit IRC		01:13
clarkb	ianw: the slave bits do	01:17
clarkb	ianw: the other bits probably don't	01:17
*** yamahata has joined #openstack-infra		01:18
*** thingee_ is now known as thingee		01:19
*** dimtruck is now known as zz_dimtruck		01:19
ianw	clarkb: ok, so i should just verify openstack_project::single_use_slave for centos ATM?	01:20
clarkb	ianw: yes	01:21
clarkb	and it should work as far as puppet3 is concerned	01:21
ianw	clarkb: ok. one thing i noticed that will bite is https://tickets.puppetlabs.com/browse/PUP-1082 ... puppet3 assumes that if on redhat, pip is called "pip-python" (the pkg name)	01:23
openstackgerrit	Anita Kuno proposed a change to openstack-infra/config: Bring some uniformity to olso acl files https://review.openstack.org/101361	01:23
clarkb	ianw: I think we handle that alredy	01:24
clarkb	ianw: iirc dprince added a symlink for that in our manifestse	01:24
*** nosnos has joined #openstack-infra		01:24
ianw	clarkb: hmm, ok, i was seeing that issue running site.pp on centos7, i'll dig further	01:24
anteaya	clarkb: thank you	01:26
anteaya	it was tedious	01:26
anteaya	I hope I got it right	01:26
anteaya	dhellmann: the last patchset is mine now, so you can review: https://review.openstack.org/#/c/101361/	01:26
dprince	ianw: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/pip/lib/puppet/provider/package/pip3.rb	01:27
dprince	ianw: not sure that will help you out (haven't seen your error) but that is what I did a while back	01:28
ianw	dprince: hmm, ok -- on centos7 i'm seeing puppet use /usr/share/ruby/vendor_ruby/puppet/provider/package/pip.rb	01:29
*** chuckC has quit IRC		01:29
dprince	ianw: with what puppet version?	01:30
*** HenryG has quit IRC		01:30
ianw	dprince: 3.6.2 as installed by https://review.openstack.org/#/c/103458/	01:30
*** HenryG has joined #openstack-infra		01:30
ianw	dprince: oh, hang out, that is pip3?	01:32
dprince	ianw: yes	01:33
ianw	dprince: so this is just "pip", but yeah, same-ish problem	01:33
dprince	ianw: Ah. So yeah. We aren't overwriting that provider so the default puppet 'pip' provider would get used.	01:34
dprince	ianw: honestly I'm a bit disappointed we haven't gotten an upstream puppet symlink/fix for this yet. But... you may just have to put a symlink in to make things happy	01:34
*** bdpayne has joined #openstack-infra		01:34
dprince	ianw: was that your take on this?	01:35
ianw	dprince: does osfamily "RedHat" not match fedora? (googling now :)	01:36
*** dstanek_zzz is now known as dstanek		01:36
dprince	ianw: osfamily should. Yes	01:37
dprince	ianw: distro would not though	01:37
dprince	ianw: See what we do for Fedora already in install_puppet.sh. We simply create the symlink in bash...	01:38
dprince	ianw: if you do the same on Centos 7 I think you may be set	01:38
ianw	dprince: and that answers my question of why this works on f20 and not centos7 :)	01:38
ianw	dprince: in terms of upstream https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/package/pip.rb#L38	01:39
ianw	dprince: do you think it's reasonable to propose a change that if 'pip' exists, someone must have gone to the effort of get-pip.py, so use it?	01:40
anteaya	yup, I called it	01:41
anteaya	4th night in a row for fireworks	01:41
dprince	ianw: I'm not sure that belongs in Puppet proper (now that I remember how all this works)	01:41
dprince	ianw: this is certainly a finer point to our setup... I mean ideally we could just use a packaged version of pip and then puppet would work. But we don't... so symlinking it is probably fine	01:42
*** moted has quit IRC		01:44
*** jyu has joined #openstack-infra		01:45
*** liusheng has quit IRC		01:45
*** gokrokve_ has quit IRC		01:45
ianw	dprince: so https://tickets.puppetlabs.com/browse/PUP-1082 is invalid?	01:47
*** timrc is now known as timrc-afk		01:47
nibalizer	clarkb: i tend to agree that the -not -name \*\[.-\]gz should keep already gziped files out	01:48
ianw	dprince: the symlink works, but it seems that we should either propose a fix or just say !packaged pip is always going to be an error (and probably enhance the err msg upstream to save people trying to figure out what the issue isse)	01:48
dprince	ianw: I don't think it is invalid. Improving things for the future would be great. Perhaps link that ticket into the install_puppet.sh script so we can loop back and check it again?	01:49
dprince	ianw: I haven't had the bandwidth to push puppet patches for all the wishlist items I find	01:50
dprince	ianw: but what you say would certainly be an improvement. The ability to explicitly pass in the location would be cool.	01:51
dprince	ianw: for that matter, we could inline our own pip.rb provider today and just use it. I don't see that as being as maintainable as having it upstream in puppet but we could certainly have our way with it.	01:52
dprince	ianw: if we did that probably best to rename it though to something unique.	01:52
dprince	ianw: our_pip, my_pip, me_pip? You'd have to change all the modules to use that instead of 'pip' too	01:53
ianw	dprince: i think the symlink is probably better than our own provider, now i understand. i'll expand the comment a bit. i doesn't seem like it would be hard to override that from an env variable, i'll see what i can do	01:53
ianw	i only speak pidgin ruby :)	01:54
*** gfa has quit IRC		01:54
dprince	ianw: :). Agree w/ you on the symlink. Good luck	01:54
*** gfa has joined #openstack-infra		01:55
ianw	dprince: thanks. i think rhel7/centos7 is in pretty good shape then, i'm just going to revert to fresh and try it all again	01:55
dprince	ianw: awesome, good work	01:56
*** devkulkarni has joined #openstack-infra		01:59
*** liusheng has joined #openstack-infra		02:01
*** wenlock_ has joined #openstack-infra		02:09
*** adalbas has quit IRC		02:10
*** yaguang has joined #openstack-infra		02:11
*** annegent_ has joined #openstack-infra		02:18
*** hdd_ has joined #openstack-infra		02:22
*** annegent_ has quit IRC		02:23
*** jyu has left #openstack-infra		02:23
*** jyu has joined #openstack-infra		02:23
*** bcrochet is now known as bcrochet\|g0ne		02:25
*** devkulkarni has quit IRC		02:30
*** mmaglana has joined #openstack-infra		02:30
*** nati_ueno has joined #openstack-infra		02:33
*** enikanorov has joined #openstack-infra		02:33
*** devkulkarni has joined #openstack-infra		02:35
*** enikanorov_ has quit IRC		02:38
*** zhiyan_ is now known as zhiyan		02:38
mordred	ianw: what is it we did?	02:42
mordred	dprince: also, no one should EVER use a packaged version of pip	02:42
mordred	dprince: EVER	02:42
mordred	they're all broken	02:42
mordred	ianw: ^^ you too	02:43
ianw	mordred: what do you mean by "we did"?	02:43
dprince	mordred: my point was more along the lines of that is what puppet is catering for	02:43
*** homeless has quit IRC		02:43
mordred	what is - packaged pip?	02:43
dprince	mordred: but sure, I understand your sentiment	02:43
mordred	nod	02:43
mordred	I mean, believe me - packaged python works 10000000* better than pip python	02:44
dprince	mordred: packaged pip is -> expload things into my system dirs	02:44
mordred	but if you're going to puppet install things using the pip provider, there is no reason to not get pip using pip instead of getting pip using packages	02:44
*** homeless has joined #openstack-infra		02:44
dprince	mordred: given your comments you are making me lean towards us having our own puppet provider for pip (like we do for pip3)	02:45
mordred	dprince: yup. I started on one of those	02:45
mordred	dprince: it's in use interally at hp, in fact, because it has to be because the puppet pip provider doesn't do proxies right	02:46
mordred	dprince: but when I tried to land it in infra it broke and I haven't gone back to figure it out yet	02:46
*** dims_ has quit IRC		02:46
dprince	mordred: ah	02:46
mordred	dprince: otoh - dstufft assures me that installing stuff system-wide with pip is a bug and should never be done	02:46
mordred	that you should either install by packages ...	02:46
mordred	or you should use a virtualenv	02:47
dstufft	well	02:47
mordred	dstufft: I'm speaking for you now :)	02:47
dstufft	that's true for the common case	02:47
*** crc32 has quit IRC		02:47
dstufft	installing stuff system wide with pip is one of those, if you break it you get to keep both pieces kind of things	02:47
dstufft	it can mess your system up, sometimes it painful ways	02:48
dprince	Yep. Agree for the common case. I wouldn't go so far as to say never though. In a pinch packaged pip might come in handy even if it does ruin your distro	02:48
mordred	dprince: have you seen: https://github.com/stankevich/puppet-python ? it's pretty good with puppetting virtualenv things	02:48
mordred	dprince: ++	02:48
* mordred agrees with dprince and dstufft		02:49
dstufft	dprince: yea I do it oftenish, mostly it's "if you don't know how to unbreak it, you probably don't want to do it"	02:49
dprince	mordred: I had not seen it. looks promising	02:49
mordred	dprince: we're using it for one thing ...	02:49
mordred	dprince: I want to enhance it to make it grok multi-pip and also to grok installing pip from get-pip (if requested)	02:49
dstufft	I wouldn't use a system packages pip, even though I contribute to helping them package it	02:50
*** zhiyan is now known as zhiyan_		02:50
dstufft	mostly because they do things like unbundle stuff which I don't think should be bundled :/	02:50
dstufft	should be unbundled*	02:50
dprince	mordred: cool	02:51
*** zhiyan_ is now known as zhiyan		02:51
mordred	dstufft: yah. well, also because it's invariably old	02:51
mordred	and I honestly think if you're willing to use pip to install software, you should be willing to use pip to install pip	02:51
dstufft	mordred: yea there's that too	02:52
dstufft	trusty is ok right now	02:52
mordred	yah	02:52
dstufft	but in 2 years it'll be lol'old	02:52
mordred	kinda like pip on precise	02:52
dstufft	oh god yes	02:52
dstufft	1.0	02:52
dstufft	it doesn't even TLS	02:53
mordred	yah	02:53
mordred	oh, speaking of ...	02:53
mordred	fungi: are you the only one who can do SSL certs for us any more?	02:53
*** dprince has quit IRC		02:53
*** msabramo has joined #openstack-infra		02:54
mordred	ZOMG. we have too many images	02:55
mordred	dstufft: oh - btw - the new bandersnatch mirror is live	02:55
*** marun is now known as marun_afk		02:56
ianw	mordred: can't we fix upstream rather that write our own provider? or is there history i don't know about?	02:56
mordred	ianw: well ... we can - but we're running puppet2 still	02:56
mordred	ianw: I'd love to fix upstream	02:56
mordred	ianw: and then migrate to puppet3	02:56
mordred	largely though - I think we just haven't gotten aroudn to it	02:56
ianw	mordred: well, not running 3 on the f20 (& centos7 when i get to it) nodes	02:57
* mordred hands ianw a goat and a case of beer		02:57
ianw	mordred: i mean, "are running" ^	02:57
mordred	++	02:57
r1chardj0n3s	mordred: I was going to comment on the bandersnatch review: I think you want to configure it to not delete packages	02:57
r1chardj0n3s	mordred: sorry I didn't get around to commenting - I've been mired in devstack and horizon	02:58
dstufft	mordred: awesome :)	02:58
mordred	r1chardj0n3s: no worries - I think we need it to respond to the deletes	02:58
dstufft	deleting packages is what you want if you want it to match what PyPI has	02:58
mrodden	bandersnatch == pypi mirror thing?	02:58
dstufft	mrodden: yes	02:58
mordred	mrodden: yes	02:58
mordred	http://pypi.openstack.org/simple	02:59
mordred	is the new mirror	02:59
dstufft	gotta love how slow /simple/ is	02:59
dstufft	even on PyPI itself where it's cached	02:59
mrodden	wow	02:59
r1chardj0n3s	yeah but it's effectively a private mirror so there's really no reason to delete; and you'll protect yourselves against a package owner deleting something you use	02:59
mrodden	is that ALL of pypi?	02:59
dstufft	mrodden: yes	02:59
mordred	r1chardj0n3s: that's actually ok though	02:59
r1chardj0n3s	yep :)	02:59
mrodden	wow	02:59
r1chardj0n3s	mordred: ok, just wanted to raise it as a thing :)	03:00
dstufft	probably they want to know if someone deletes a thing from PyPIif that's going to break them	03:00
mordred	r1chardj0n3s: because if that happens, it means everyone who uses opensatck is screwed with that package and we want to know abou tit	03:00
mordred	yah. what dstufft	03:00
mordred	r1chardj0n3s: but it's an excellent point	03:00
r1chardj0n3s	I just know I've had a lot of heat from poeople complaining when packages are deleted :P	03:00
mordred	mrodden: we're getting out of the business of gating by something not being in the mirror	03:00
mordred	r1chardj0n3s: well, people shouldn't delete packages	03:00
r1chardj0n3s	mordred: that's a fair call, yeah	03:00
mordred	:)	03:00
*** mriedem has quit IRC		03:00
r1chardj0n3s	mordred: well, that I can't control	03:00
*** zz_dimtruck is now known as dimtruck		03:00
mordred	SO many things I can't control	03:00
mordred	mrodden: next step is per-cloud-region mirrors	03:01
r1chardj0n3s	mordred: I have to respect someone saying that they just don't want their package out there any longer	03:01
mordred	mrodden: so that we're never pulling from outside of a cloud region for a given build	03:01
mrodden	how many packages does it pull locally?	03:01
mrodden	err	03:01
mrodden	how much package space/ disk usage	03:01
mordred	mrodden: the mirror is taking ~95G	03:01
mrodden	hmm thats not terrible	03:01
* mordred afks to store		03:02
*** r1chardj0n3s is now known as r1chard_afk		03:02
*** harlowja is now known as harlowja_away		03:02
*** jyu has quit IRC		03:02
*** jyu has joined #openstack-infra		03:03
*** jyu has quit IRC		03:04
mrodden	guess i should probably look into replacing my run_mirror.py with that eventually	03:04
*** arnaud has quit IRC		03:05
*** arnaud__ has quit IRC		03:05
*** msabramo has quit IRC		03:09
*** harlowja_away is now known as harlowja		03:09
*** praneshp has joined #openstack-infra		03:10
*** harlowja is now known as harlowja_away		03:10
*** harlowja_away is now known as harlowja		03:11
*** AaronGr has quit IRC		03:12
*** praneshp_ has joined #openstack-infra		03:13
*** AaronGr has joined #openstack-infra		03:14
*** ArxCruz has quit IRC		03:15
*** praneshp has quit IRC		03:16
*** praneshp_ is now known as praneshp		03:16
*** jdahlin has quit IRC		03:17
*** HenryG has quit IRC		03:17
*** annegent_ has joined #openstack-infra		03:18
*** HenryG has joined #openstack-infra		03:20
*** devkulkarni has quit IRC		03:24
*** annegent_ has quit IRC		03:24
*** UtahDave has joined #openstack-infra		03:27
*** yaguang has quit IRC		03:29
*** rlandy has joined #openstack-infra		03:30
*** sabeen1 has quit IRC		03:33
*** sabeen1 has joined #openstack-infra		03:33
*** r1chard_afk is now known as r1chardj0n3s		03:37
*** nosnos has quit IRC		03:38
*** annegent_ has joined #openstack-infra		03:38
*** annegent_ has quit IRC		03:41
*** annegentle has joined #openstack-infra		03:42
*** yaguang has joined #openstack-infra		03:42
*** niska has quit IRC		03:54
*** niska has joined #openstack-infra		03:59
*** chandan_kumar has joined #openstack-infra		03:59
*** gnuoy` has quit IRC		04:00
*** jamespage has quit IRC		04:00
*** gnuoy has joined #openstack-infra		04:00
*** jamespage has joined #openstack-infra		04:00
*** jamespage has joined #openstack-infra		04:00
*** devkulkarni has joined #openstack-infra		04:00
*** homeless_ has joined #openstack-infra		04:02
*** TheJulia has quit IRC		04:05
*** sileht has quit IRC		04:06
*** homeless has quit IRC		04:06
*** TheJulia has joined #openstack-infra		04:06
*** nati_ueno has quit IRC		04:06
*** sileht has joined #openstack-infra		04:08
*** craigbr has joined #openstack-infra		04:08
*** blamar has quit IRC		04:11
*** jogo_awa1 has quit IRC		04:11
*** fifieldt has quit IRC		04:11
*** bcrochet\|g0ne has quit IRC		04:11
*** gyee has quit IRC		04:11
*** ildikov has quit IRC		04:11
*** mfer has quit IRC		04:11
*** rfolco has quit IRC		04:11
*** pcm__ has quit IRC		04:11
*** annegentle has quit IRC		04:11
*** gema has quit IRC		04:11
*** grantbow has quit IRC		04:11
*** jpeeler has quit IRC		04:11
*** jamielennox has quit IRC		04:11
*** ekarlso has quit IRC		04:11
*** rakhmerov has quit IRC		04:11
*** bradm has quit IRC		04:11
*** wenlock_ has quit IRC		04:11
*** hdd has quit IRC		04:11
*** chmouel has quit IRC		04:11
*** dimtruck has quit IRC		04:11
*** gyee_ has quit IRC		04:11
*** pfalleno1 has quit IRC		04:11
*** proffalken has quit IRC		04:11
*** sc68cal has quit IRC		04:11
*** clarkb has quit IRC		04:11
*** devananda has quit IRC		04:11
*** jeblair has quit IRC		04:11
*** slagle has quit IRC		04:11
*** SlickNik has quit IRC		04:11
*** gothicmindfood has quit IRC		04:11
*** Kiall has quit IRC		04:11
*** jcooley has quit IRC		04:11
*** saper has quit IRC		04:11
*** jd__ has quit IRC		04:11
*** portante has quit IRC		04:11
*** briancline has quit IRC		04:11
*** tnurlygayanov has quit IRC		04:11
*** mrda has quit IRC		04:12
*** arnaud has joined #openstack-infra		04:13
*** arnaud__ has joined #openstack-infra		04:14
*** homeless_ has quit IRC		04:17
*** ArxCruz has joined #openstack-infra		04:21
*** ArxCruz has quit IRC		04:21
*** craigbr has quit IRC		04:21
*** mrda has joined #openstack-infra		04:22
*** annegentle has joined #openstack-infra		04:22
*** wenlock_ has joined #openstack-infra		04:22
*** blamar has joined #openstack-infra		04:22
*** jogo_awa1 has joined #openstack-infra		04:22
*** fifieldt has joined #openstack-infra		04:22
*** bcrochet\|g0ne has joined #openstack-infra		04:22
*** gyee has joined #openstack-infra		04:22
*** pcm__ has joined #openstack-infra		04:22
*** ildikov has joined #openstack-infra		04:22
*** mfer has joined #openstack-infra		04:22
*** rfolco has joined #openstack-infra		04:22
*** tnurlygayanov has joined #openstack-infra		04:22
*** gema has joined #openstack-infra		04:22
*** hdd has joined #openstack-infra		04:22
*** chmouel has joined #openstack-infra		04:22
*** dimtruck has joined #openstack-infra		04:22
*** gyee_ has joined #openstack-infra		04:22
*** pfalleno1 has joined #openstack-infra		04:22
*** proffalken has joined #openstack-infra		04:22
*** sc68cal has joined #openstack-infra		04:22
*** clarkb has joined #openstack-infra		04:22
*** devananda has joined #openstack-infra		04:22
*** grantbow has joined #openstack-infra		04:22
*** bradm has joined #openstack-infra		04:22
*** ekarlso has joined #openstack-infra		04:22
*** jamielennox has joined #openstack-infra		04:22
*** rakhmerov has joined #openstack-infra		04:22
*** jpeeler has joined #openstack-infra		04:22
*** jeblair has joined #openstack-infra		04:22
*** slagle has joined #openstack-infra		04:22
*** SlickNik has joined #openstack-infra		04:22
*** gothicmindfood has joined #openstack-infra		04:22
*** Kiall has joined #openstack-infra		04:22
*** jcooley has joined #openstack-infra		04:22
*** saper has joined #openstack-infra		04:22
*** jd__ has joined #openstack-infra		04:22
*** portante has joined #openstack-infra		04:22
*** briancline has joined #openstack-infra		04:22
*** Guest37445 is now known as zaro		04:23
*** msabramo has joined #openstack-infra		04:24
zaro	jhesketh: ping	04:24
jhesketh	zaro: pong	04:25
*** UtahDave has left #openstack-infra		04:26
*** willroberts has quit IRC		04:26
*** dstufft\|laptop has joined #openstack-infra		04:26
*** willroberts has joined #openstack-infra		04:26
*** gema_ has joined #openstack-infra		04:27
zaro	jhesketh: attempt to remove envinject failed. https://jenkins07.openstack.org/job/check-gerrit-buck/11/console	04:27
*** gema has quit IRC		04:27
zaro	jhesketh: are you able to view the workspace and see if the version.properties file exists?	04:28
jhesketh	zaro: nope, I don't have root sorry :-(	04:28
* jhesketh can only merge code that breaks the world.. . he cannot fix the world		04:28
jhesketh	clarkb: if you're around ^	04:28
zaro	jhesketh: any idea why source isn't sourcing the file?	04:28
jhesketh	looking..	04:30
clarkb	jhesketh: zaro the workspaces don't exist because we delete the salves	04:30
zaro	ohh right.	04:31
*** gyee has quit IRC		04:31
*** jamielennox has quit IRC		04:31
clarkb	the filename looks tyoped	04:31
*** ildikov has quit IRC		04:31
*** gyee has joined #openstack-infra		04:31
clarkb	yup thats it	04:31
*** jamielennox has joined #openstack-infra		04:32
clarkb	properites	04:32
*** dkehn has joined #openstack-infra		04:32
*** ildikov has joined #openstack-infra		04:32
*** jogo_awa1 has quit IRC		04:32
zaro	duhh!!!	04:32
*** jogo_away has joined #openstack-infra		04:32
*** grantbow has quit IRC		04:32
zaro	will put put a fix.	04:32
jhesketh	:-( sorry	04:32
jhesketh	zaro: I can if you prefer?	04:32
lifeless	clarkb: mmm salves	04:34
*** dkehnx has quit IRC		04:34
*** rlandy has quit IRC		04:35
openstackgerrit	Khai Do proposed a change to openstack-infra/config: Fix filename typo in gerrit jobs. https://review.openstack.org/103721	04:36
*** dimtruck is now known as zz_dimtruck		04:36
zaro	jhesketh, clarkb ^	04:36
*** grantbow has joined #openstack-infra		04:36
jhesketh	thanks z	04:36
jhesketh	*zaro	04:36
zaro	np. git grep for same typo but didn't find any other so it's isolated.	04:38
jhesketh	yeah ditto	04:39
openstackgerrit	A change was merged to openstack-infra/config: Fix filename typo in gerrit jobs. https://review.openstack.org/103721	04:42
*** gokrokve has joined #openstack-infra		04:45
*** nosnos has joined #openstack-infra		04:45
*** doude has quit IRC		04:46
*** talluri has joined #openstack-infra		04:48
*** ildikov has quit IRC		04:49
*** nosnos has quit IRC		04:51
*** nosnos has joined #openstack-infra		04:51
*** hdd_ has quit IRC		04:57
jhesketh	zaro: that seems to have fixed it : https://jenkins04.openstack.org/job/check-gerrit-buck/11/console	05:00
*** adrian_otto has quit IRC		05:00
*** devkulkarni has quit IRC		05:03
*** dstanek is now known as dstanek_zzz		05:05
*** harlowja is now known as harlowja_away		05:06
*** amitpp has joined #openstack-infra		05:09
*** om has joined #openstack-infra		05:12
*** dstanek_zzz is now known as dstanek		05:13
*** wenlock_ has quit IRC		05:21
*** pcrews has quit IRC		05:21
*** Longgeek has joined #openstack-infra		05:22
*** ildikov has joined #openstack-infra		05:22
*** wenlock_ has joined #openstack-infra		05:24
*** Longgeek has quit IRC		05:25
wenlock_	hi, in your puppet runs for infra for hpcloud have you been seeing puppet exceptions similar to the following? http://paste.openstack.org/show/85195/	05:26
wenlock_	i know this is a known issue... https://projects.puppetlabs.com/issues/17855 but it didnt seem to come up very often... this week it seems to be coming up alot	05:27
*** e0ne has joined #openstack-infra		05:28
*** dstanek is now known as dstanek_zzz		05:28
*** rcarrillocruz has joined #openstack-infra		05:31
*** basha has joined #openstack-infra		05:32
*** tkelsey has joined #openstack-infra		05:32
*** gokrokve has quit IRC		05:33
*** rcarrill` has quit IRC		05:34
*** Longgeek has joined #openstack-infra		05:35
*** reed has quit IRC		05:35
*** chandan_kumar has quit IRC		05:37
*** bdpayne has quit IRC		05:39
*** bdpayne has joined #openstack-infra		05:41
openstackgerrit	Ian Wienand proposed a change to openstack-infra/config: Initial RHEL7/Centos7 support https://review.openstack.org/103458	05:41
openstackgerrit	Ian Wienand proposed a change to openstack-infra/config: Add functions & cleanup install_puppet.sh https://review.openstack.org/103735	05:41
*** wenlock_ has quit IRC		05:42
*** e0ne has quit IRC		05:44
openstackgerrit	Ian Wienand proposed a change to openstack-infra/config: Add functions & cleanup install_puppet.sh https://review.openstack.org/103735	05:44
openstackgerrit	Ian Wienand proposed a change to openstack-infra/config: Initial RHEL7/Centos7 support https://review.openstack.org/103458	05:44
*** e0ne has joined #openstack-infra		05:44
*** e0ne has quit IRC		05:45
*** bdpayne has quit IRC		05:46
*** gyee has quit IRC		05:47
*** otherwiseguy has quit IRC		05:47
*** e0ne has joined #openstack-infra		05:50
openstackgerrit	Ph. Marek proposed a change to openstack/requirements: Add dbus library, needed for the DRBD cinder driver integration. https://review.openstack.org/99013	05:51
*** markmc has joined #openstack-infra		05:52
*** e0ne has quit IRC		05:55
*** bdpayne has joined #openstack-infra		05:55
*** e0ne has joined #openstack-infra		05:57
*** dkehn__ has joined #openstack-infra		06:00
*** nosnos has quit IRC		06:01
*** harlowja_at_home has joined #openstack-infra		06:02
*** dkehn has quit IRC		06:03
*** nosnos has joined #openstack-infra		06:03
*** bdpayne has quit IRC		06:06
*** e0ne has quit IRC		06:08
*** unicell1 has joined #openstack-infra		06:09
*** e0ne has joined #openstack-infra		06:10
*** unicell has quit IRC		06:10
*** dkehn has joined #openstack-infra		06:12
*** zigo has quit IRC		06:13
*** zigo has joined #openstack-infra		06:14
*** dkehn__ has quit IRC		06:15
*** Longgeek has quit IRC		06:16
*** abhishek has joined #openstack-infra		06:17
*** e0ne has quit IRC		06:18
*** Longgeek has joined #openstack-infra		06:19
*** nati_ueno has joined #openstack-infra		06:19
abhishek	https://review.openstack.org/#/c/102215/	06:20
abhishek	Hi all: can any one review this patch and help me to get this merged?	06:20
abhishek	Thank you	06:20
openstackgerrit	Mark McLoughlin proposed a change to openstack-infra/elastic-recheck: Add query for rare MySQL-Python install failure https://review.openstack.org/103746	06:20
*** msabramo has quit IRC		06:22
*** andreykurilin has joined #openstack-infra		06:22
*** e0ne has joined #openstack-infra		06:22
*** krtaylor has quit IRC		06:23
*** mkoderer has joined #openstack-infra		06:24
*** salv-orlando has joined #openstack-infra		06:29
*** e0ne has quit IRC		06:29
*** dstanek_zzz is now known as dstanek		06:29
*** e0ne has joined #openstack-infra		06:33
*** andreykurilin has quit IRC		06:34
*** krtaylor has joined #openstack-infra		06:35
*** e0ne has quit IRC		06:38
*** harlowja_at_home has quit IRC		06:40
*** unicell1 is now known as unicell		06:40
*** unicell has joined #openstack-infra		06:40
*** chandan_kumar has joined #openstack-infra		06:41
*** e0ne has joined #openstack-infra		06:41
*** e0ne has quit IRC		06:44
*** e0ne has joined #openstack-infra		06:47
*** sballe_ has joined #openstack-infra		06:49
*** e0ne has quit IRC		06:50
*** pblaho has joined #openstack-infra		06:52
*** sballe has quit IRC		06:53
*** yolanda has joined #openstack-infra		06:56
*** rcarrillocruz has quit IRC		06:56
*** camunoz is now known as camunoz_gone		06:58
*** rcarrillocruz has joined #openstack-infra		07:02
*** arnaud__ has quit IRC		07:06
*** arnaud has quit IRC		07:06
*** morganfainberg is now known as morganfainberg_Z		07:12
*** arnaud has joined #openstack-infra		07:14
*** jcoufal has joined #openstack-infra		07:14
*** mfer has quit IRC		07:15
*** arnaud__ has joined #openstack-infra		07:15
*** mrmartin has joined #openstack-infra		07:18
*** arnaud__ has quit IRC		07:20
*** arnaud has quit IRC		07:20
*** arnaud has joined #openstack-infra		07:20
*** arnaud__ has joined #openstack-infra		07:20
nibalizer	arstu/win 12	07:22
*** dstanek is now known as dstanek_zzz		07:24
*** skolekonov has joined #openstack-infra		07:25
*** mmaglana has quit IRC		07:26
*** jgallard has joined #openstack-infra		07:26
*** praneshp has quit IRC		07:27
*** lpaduano_freenod has joined #openstack-infra		07:28
*** eglynn-reg-offic is now known as eglynn-office		07:31
*** mkerrin has quit IRC		07:33
mattoliverau	Well I'm calling it a night, night all	07:35
*** rdopieralski has joined #openstack-infra		07:37
*** hashar has joined #openstack-infra		07:38
*** e0ne has joined #openstack-infra		07:39
*** ociuhandu has joined #openstack-infra		07:39
*** rdopieralski has quit IRC		07:40
*** dstufft\|laptop has quit IRC		07:40
*** dstufft\|laptop has joined #openstack-infra		07:42
*** rcarrill` has joined #openstack-infra		07:42
*** rcarrillocruz has quit IRC		07:43
*** Guest9812 has joined #openstack-infra		07:43
*** liusheng has quit IRC		07:44
*** salv-orlando has quit IRC		07:46
*** mrda is now known as mrda-away		07:49
openstackgerrit	A change was merged to openstack-infra/devstack-gate: Make one copy of grenade log files https://review.openstack.org/100131	07:52
*** vponomaryov has joined #openstack-infra		07:53
*** flaper87\|afk is now known as flaper87		07:55
*** jpich has joined #openstack-infra		07:55
*** morganfainberg_Z is now known as morganfainberg		07:55
*** HACKING-TWITTER has joined #openstack-infra		07:57
*** roeyc has joined #openstack-infra		08:04
*** e0ne has quit IRC		08:05
*** derekh_ has joined #openstack-infra		08:05
*** e0ne has joined #openstack-infra		08:05
*** nati_ueno has quit IRC		08:05
*** r1chardj0n3s is now known as r1chard_afk		08:05
*** matrohon has joined #openstack-infra		08:06
*** denis_makogon has joined #openstack-infra		08:06
*** ominakov has joined #openstack-infra		08:06
*** ihrachyshka has joined #openstack-infra		08:08
*** rcarrill` has quit IRC		08:12
*** ArxCruz has joined #openstack-infra		08:15
*** ArxCruz has quit IRC		08:15
*** dstanek_zzz is now known as dstanek		08:15
*** ihrachyshka has quit IRC		08:15
*** ihrachyshka has joined #openstack-infra		08:16
*** lcheng has joined #openstack-infra		08:17
*** HACKING-TWITTER has quit IRC		08:17
*** rcarrillocruz has joined #openstack-infra		08:17
*** HACKING-TWITTER has joined #openstack-infra		08:19
*** jlibosva has joined #openstack-infra		08:20
*** HACKING-TWITTER has joined #openstack-infra		08:20
*** HACKING-TWITTER has quit IRC		08:20
SergeyLukjanov	jeblair, fungi, clarkb, unfortunately I'll be in flight in time of infra meeting, so, most probably /me will be unavailable today	08:21
*** HACKING-TWITTER has joined #openstack-infra		08:21
*** HACKING-TWITTER has joined #openstack-infra		08:23
*** HACKING-TWITTER has joined #openstack-infra		08:24
openstackgerrit	Sergey Lukjanov proposed a change to openstack-infra/config: Remove obsolete #openstack-climate channel https://review.openstack.org/94913	08:25
*** dstanek is now known as dstanek_zzz		08:25
*** HACKING-TWITTER has joined #openstack-infra		08:26
*** HACKING-TWITTER has joined #openstack-infra		08:27
*** HACKING-TWITTER has joined #openstack-infra		08:29
*** arnaud__ has quit IRC		08:29
*** arnaud has quit IRC		08:29
*** liusheng has joined #openstack-infra		08:30
*** zz_johnthetubagu is now known as johnthetubaguy		08:30
openstackgerrit	Joshua Hesketh proposed a change to openstack-infra/zuul: Add gerrit reviews into patchset approvals https://review.openstack.org/97390	08:34
openstackgerrit	Joshua Hesketh proposed a change to openstack-infra/zuul: Allow a pipeline to specify alternative gerrit acc https://review.openstack.org/97391	08:34
openstackgerrit	Joshua Hesketh proposed a change to openstack-infra/zuul: WIP Add support for negative requirements https://review.openstack.org/102726	08:34
openstackgerrit	Joshua Hesketh proposed a change to openstack-infra/zuul: Add support for negative requirements https://review.openstack.org/102726	08:35
*** mflobo has joined #openstack-infra		08:38
*** mflobo has quit IRC		08:38
*** andreaf_ has joined #openstack-infra		08:39
*** doude has joined #openstack-infra		08:39
hashar	jhesketh: :-)	08:42
jhesketh	Hey hashar	08:42
*** ildikov has quit IRC		08:42
hashar	jhesketh: my little Zuul cloner is working more or less as expected. James told me he will eventually review it soon ™	08:42
jhesketh	excellent :-)	08:42
jhesketh	I've been excited for it	08:42
hashar	jhesketh: thank you for all your encouragements and reviews over the last few months	08:43
jhesketh	anytime!	08:43
hashar	I started using it to build an integration job that test mediawiki/core + all the extensions we have on Wikimedia website	08:43
hashar	example output https://integration.wikimedia.org/ci/job/mediawiki-core-extensions-integration/52/consoleFull :-D	08:43
hashar	that spam a lot. I believe that will help turbo-hipster / third party CI and maybe devstack will be interested in it	08:43
hashar	we will see	08:43
jhesketh	neat !	08:44
openstackgerrit	Valeriy Ponomaryov proposed a change to openstack-infra/config: Set docs jobs for Manila project as votable https://review.openstack.org/103803	08:46
openstackgerrit	Valeriy Ponomaryov proposed a change to openstack-infra/config: Set docs jobs for Manila project as votable https://review.openstack.org/103803	08:49
*** rcarrillocruz has quit IRC		08:52
*** ociuhandu has quit IRC		08:54
*** ildikov has joined #openstack-infra		08:54
*** rcarrillocruz has joined #openstack-infra		08:57
openstackgerrit	Valeriy Ponomaryov proposed a change to openstack-infra/config: Add py33 job for python-manilaclient https://review.openstack.org/103809	08:59
*** IvanBerezovskiy1 has joined #openstack-infra		09:05
eglynn-office	folks if anyone has any spare review cycles in the next hour or two	09:06
eglynn-office	... it would be great to have a look at the ceilo-specs backlog	09:06
eglynn-office	... https://review.openstack.org/#/q/status:open+project:openstack/ceilometer-specs,n,z	09:06
eglynn-office	thanking y'all in advance :)	09:06
* eglynn-office hits the road to Paris		09:07
eglynn-office	... happy travels to anyone who's heading to the mid-cycle	09:07
hashar	eglynn-office: enjoy Paris	09:07
*** eglynn-office is now known as eglynn-office-af		09:07
*** eglynn-office-af is now known as eglynn-offic-afk		09:08
*** andreaf has quit IRC		09:10
*** andreaf_ is now known as andreaf		09:11
*** andreaf_ has joined #openstack-infra		09:11
*** yamahata has quit IRC		09:12
*** flaper87 is now known as flaper87\|afk		09:13
*** ihrachyshka has quit IRC		09:13
*** jgallard has quit IRC		09:14
*** dstanek_zzz is now known as dstanek		09:16
*** jp_at_hp has joined #openstack-infra		09:17
*** jp_at_hp has quit IRC		09:17
*** jp_at_hp has joined #openstack-infra		09:17
*** ildikov has quit IRC		09:19
*** pelix has joined #openstack-infra		09:23
openstackgerrit	Valeriy Ponomaryov proposed a change to openstack-infra/config: Added devstack job for manila https://review.openstack.org/99933	09:23
openstackgerrit	Valeriy Ponomaryov proposed a change to openstack-infra/config: Added devstack job for manila https://review.openstack.org/99933	09:24
*** dstanek is now known as dstanek_zzz		09:26
*** mmaglana has joined #openstack-infra		09:26
*** oomichi has quit IRC		09:30
*** mmaglana has quit IRC		09:31
*** basha_ has joined #openstack-infra		09:38
*** basha has quit IRC		09:39
*** basha_ is now known as basha		09:39
openstackgerrit	Thierry Carrez proposed a change to openstack-infra/config: Change ACLs to apply on proposed/* branches https://review.openstack.org/102822	09:47
*** dstanek_zzz is now known as dstanek		09:48
*** Longgeek has quit IRC		09:49
*** Longgeek has joined #openstack-infra		09:50
abhishek	Hi all: can any one review this patch and help me to get this merged?	09:51
abhishek	https://review.openstack.org/#/c/102215/	09:51
abhishek	Thank you	09:51
openstackgerrit	A change was merged to openstack-infra/jeepyb: Update global requirements https://review.openstack.org/103338	09:58
*** dstanek is now known as dstanek_zzz		09:58
*** Longgeek_ has joined #openstack-infra		10:00
*** Guest9812 has quit IRC		10:01
*** roeyc has quit IRC		10:01
*** Guest9812 has joined #openstack-infra		10:02
*** Longgeek has quit IRC		10:03
*** jgallard has joined #openstack-infra		10:04
*** skolekonov has quit IRC		10:09
openstackgerrit	A change was merged to openstack-infra/reviewstats: Added skraynev to heat-core https://review.openstack.org/103152	10:14
*** ihrachyshka has joined #openstack-infra		10:16
*** ociuhandu has joined #openstack-infra		10:18
yolanda	mordred, jeblair, i was wondering about the nodepool image-update command. Actually with our approach, it will make more sense to do: nodepool image-update dibname , and then this will rebuild image in dib, and propagate to all providers that have this image in their list, what do you think?	10:22
openstackgerrit	A change was merged to openstack-infra/config: Make py33 job voting in check and gate for Rubick https://review.openstack.org/103516	10:24
*** jcoufal has quit IRC		10:26
*** salv-orlando has joined #openstack-infra		10:26
*** jcoufal has joined #openstack-infra		10:26
*** roeyc has joined #openstack-infra		10:28
*** Guest9812 has quit IRC		10:28
openstackgerrit	Sergey Skripnick proposed a change to openstack-infra/config: Enable ceilometer in rally-dsvm jobs https://review.openstack.org/103836	10:29
openstackgerrit	A change was merged to openstack-infra/config: Add openstack-security to accessbot and eavesdrop https://review.openstack.org/103650	10:31
openstackgerrit	A change was merged to openstack-infra/config: Enable py33 gating for Stackalytics https://review.openstack.org/103654	10:31
openstackgerrit	Sergey Skripnick proposed a change to openstack-infra/config: Enable ceilometer in rally-dsvm jobs https://review.openstack.org/103836	10:34
*** yaguang has quit IRC		10:35
*** talluri has quit IRC		10:36
openstackgerrit	Christian Berendt proposed a change to openstack-infra/gerritlib: Changed home-page used in setup.cfg https://review.openstack.org/101860	10:38
openstackgerrit	A change was merged to openstack-infra/config: Add functional tests for swift3 https://review.openstack.org/102293	10:40
*** e0ne has quit IRC		10:41
openstackgerrit	Boris Pavlovic proposed a change to openstack-infra/config: Enable ceilometer in rally-dsvm jobs https://review.openstack.org/103836	10:41
openstackgerrit	Boris Pavlovic proposed a change to openstack-infra/config: Enable ceilometer in rally-dsvm jobs https://review.openstack.org/103836	10:42
openstackgerrit	Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Auth error handling https://review.openstack.org/101867	10:44
bookwar	if i have a chain of >10 dependent patchsets in Gerrit, and then commit a change to the first patchset, every dependent patchset gets changed and goes through review process?	10:46
*** chandan_kumar has quit IRC		10:47
*** dstanek_zzz is now known as dstanek		10:49
sdague	bookwar: if the other patches aren't changed, their review votes stay	10:49
*** jyuso006 has joined #openstack-infra		10:50
bookwar	but gerrit event is triggered anyway?	10:50
*** openstack has joined #openstack-infra		10:51
*** talluri has joined #openstack-infra		10:56
hashar	bookwar: if only the first patch is changed and the dependent one are not rebased, that would be only one patch being triggered	10:57
hashar	bookwar: but I guess most folks rebase the whole chain, so that is new patchets that needs testing indeed	10:57
*** dstanek is now known as dstanek_zzz		10:59
*** talluri has quit IRC		11:00
*** talluri has joined #openstack-infra		11:00
*** basha has quit IRC		11:01
*** talluri has quit IRC		11:05
*** cnesa has quit IRC		11:07
*** dims_ has joined #openstack-infra		11:07
*** e0ne has joined #openstack-infra		11:08
*** Longgeek_ has quit IRC		11:08
*** Longgeek has joined #openstack-infra		11:09
bookwar	yeah, testing is still needed for those patchsets, let's see if i an rearrange or reschedule them on the CI side	11:09
bookwar	thanks	11:10
*** tkelsey has quit IRC		11:17
*** chandan_kumar has joined #openstack-infra		11:20
openstackgerrit	Andreas Jaeger proposed a change to openstack-infra/config: Adds a docs-specs repository https://review.openstack.org/103115	11:21
openstackgerrit	Nikita Konovalov proposed a change to openstack-infra/storyboard: Fix logging usage https://review.openstack.org/103848	11:26
*** mmaglana has joined #openstack-infra		11:27
*** adalbas has joined #openstack-infra		11:28
*** basha has joined #openstack-infra		11:28
*** mmaglana has quit IRC		11:32
Kiall	jeblair / clarkb - when you have a chance, could you have a look at my comment on https://review.openstack.org/#/c/99611/ ?	11:32
*** basha has quit IRC		11:33
*** jyuso006 has quit IRC		11:35
*** basha has joined #openstack-infra		11:39
*** basha has quit IRC		11:39
*** Longgeek_ has joined #openstack-infra		11:39
*** weshay has joined #openstack-infra		11:43
*** Longgeek has quit IRC		11:43
*** weshay has quit IRC		11:44
*** weshay has joined #openstack-infra		11:44
*** ArxCruz has joined #openstack-infra		11:46
*** ArxCruz_ has joined #openstack-infra		11:47
*** skolekonov has joined #openstack-infra		11:49
*** dstanek_zzz is now known as dstanek		11:50
*** ArxCruz has quit IRC		11:51
*** dkehn__ has joined #openstack-infra		11:51
*** HACKING-TWITTER has quit IRC		11:55
*** amitpp has quit IRC		11:55
*** dkehn has quit IRC		11:55
*** HACKING-TWITTER has joined #openstack-infra		11:57
*** rlandy has joined #openstack-infra		11:58
*** HACKING-TWITTER has joined #openstack-infra		11:59
*** dstanek is now known as dstanek_zzz		12:00
*** HACKING-TWITTER has joined #openstack-infra		12:00
*** talluri has joined #openstack-infra		12:01
*** HACKING-TWITTER has joined #openstack-infra		12:02
*** ArxCruz_ has quit IRC		12:02
*** ArxCruz_ has joined #openstack-infra		12:02
*** mbacchi has joined #openstack-infra		12:03
*** HACKING-TWITTER has joined #openstack-infra		12:03
*** HACKING-TWITTER has quit IRC		12:03
*** HACKING-TWITTER has joined #openstack-infra		12:04
*** talluri has quit IRC		12:05
*** HACKING-TWITTER has joined #openstack-infra		12:06
*** cnesa has joined #openstack-infra		12:07
*** kashyap_bbiab has joined #openstack-infra		12:08
*** Longgeek_ has quit IRC		12:08
*** Longgeek has joined #openstack-infra		12:08
*** kashyap_bbiab has quit IRC		12:09
*** kashyap_bbiab has joined #openstack-infra		12:09
*** kashyap` has joined #openstack-infra		12:10
*** doug-fish has joined #openstack-infra		12:10
*** kashyap has quit IRC		12:11
*** Longgeek has quit IRC		12:13
*** kashyap_bbiab has quit IRC		12:14
*** kashyap` is now known as kashyap		12:16
*** ihrachyshka has quit IRC		12:16
*** Longgeek has joined #openstack-infra		12:19
*** jcoufal has quit IRC		12:19
*** dhellmann is now known as dhellmann_		12:19
openstackgerrit	A change was merged to openstack-infra/config: Add python34 build and job template https://review.openstack.org/102847	12:20
abhishek	Hi all: can any one review this patch and help me to get this merged?	12:21
abhishek	https://review.openstack.org/#/c/102215/	12:21
abhishek	Thank you	12:21
*** rdopieralski has joined #openstack-infra		12:21
rdopieralski	I wanted to ask what we would need to do to get openstack/tuskar-ui to auto-update its requirments to match global-requirements?	12:22
*** pdmars has joined #openstack-infra		12:23
*** jhesketh has quit IRC		12:23
chmouel	how come something like that can happen ? http://paste.openstack.org/raw/85221/ in devstack review https://review.openstack.org/#/c/65113/	12:24
Kiall	rdopieralski: I've just done that for designate - https://review.openstack.org/#/c/103316/	12:24
rdopieralski	Kiall: awesome, thank you!	12:24
*** pdmars has quit IRC		12:24
*** pdmars has joined #openstack-infra		12:25
*** dprince has joined #openstack-infra		12:26
*** bcrochet\|g0ne is now known as bcrochet		12:27
*** dims_ has quit IRC		12:32
*** dims_ has joined #openstack-infra		12:32
*** Longgeek has quit IRC		12:34
*** Longgeek has joined #openstack-infra		12:34
*** ihrachyshka has joined #openstack-infra		12:36
openstackgerrit	Christian Berendt proposed a change to openstack-infra/config: Add pep8 check/gate for stackforge/doc8 https://review.openstack.org/103863	12:36
*** mrmartin has quit IRC		12:37
openstackgerrit	Kashyap Chamarthy proposed a change to openstack-infra/elastic-recheck: Add query for Neutron functional test failures bug 1336172 https://review.openstack.org/103864	12:37
uvirtbot	Launchpad bug 1336172 in neutron "neutron/agent/linux/ovs_lib.py: int(ofport) returns 'NoneType' instead of string or a number" [Undecided,New] https://launchpad.net/bugs/1336172	12:37
*** Longgeek_ has joined #openstack-infra		12:39
*** jcoufal has joined #openstack-infra		12:39
*** Longgee__ has joined #openstack-infra		12:40
*** Longgee__ has quit IRC		12:40
*** Longgee__ has joined #openstack-infra		12:41
*** Longgeek_ has quit IRC		12:41
*** Longgeek has quit IRC		12:42
fungi	rdopieralski: you would need to add the project to http://git.openstack.org/cgit/openstack/requirements/tree/projects.txt	12:43
fungi	oh, i see Kiall already answered	12:43
*** tkelsey has joined #openstack-infra		12:43
*** ArxCruz_ has quit IRC		12:43
*** Longgeek has joined #openstack-infra		12:44
*** Longgee__ has quit IRC		12:44
openstackgerrit	Peter Belanyi proposed a change to openstack/requirements: Add tuskar-ui to projects.txt https://review.openstack.org/103868	12:46
openstackgerrit	Christian Berendt proposed a change to openstack-dev/cookiecutter: Use spaces instead of tabs for indentation https://review.openstack.org/103869	12:46
*** alexpilotti has joined #openstack-infra		12:47
fungi	chmouel: was that in the neutron functional test job?	12:47
*** ArxCruz has joined #openstack-infra		12:48
fungi	yeah, looks like there was some sort of permissions problem with that job... note that things started falling apart long before that sudo error message	12:48
*** lpaduano_freenod has left #openstack-infra		12:49
*** basha has joined #openstack-infra		12:49
openstackgerrit	A change was merged to openstack-infra/release-tools: Add spec2bp.py script https://review.openstack.org/101566	12:50
openstackgerrit	A change was merged to openstack-infra/release-tools: Add basic milestone check feature to ms2version.py https://review.openstack.org/103070	12:50
*** dstanek_zzz is now known as dstanek		12:50
*** basha has quit IRC		12:51
*** Longgeek has quit IRC		12:52
*** Longgeek_ has joined #openstack-infra		12:52
*** julim has joined #openstack-infra		12:52
fungi	chmouel: yeah, it persists across runs on the current patchset, so something about that change is breaking the neutron functional tests	12:55
*** Longgeek has joined #openstack-infra		12:56
*** xianghuihui has joined #openstack-infra		12:56
*** lcheng_ has joined #openstack-infra		12:57
*** chandan_kumar has quit IRC		12:59
openstackgerrit	Thierry Carrez proposed a change to openstack-infra/config: Add proposed/* branches support to merge_tags.sh https://review.openstack.org/102812	12:59
*** Longgeek_ has quit IRC		12:59
*** xianghui has quit IRC		12:59
ttx	fungi: tried to address your concern and mordred's "should not be that complex" remark ^	13:00
*** mfer has joined #openstack-infra		13:00
*** dstanek is now known as dstanek_zzz		13:00
*** lcheng has quit IRC		13:00
*** talluri has joined #openstack-infra		13:02
fungi	ttx: what is the possibility that this will be run for tags on master branches of projects, and should those short-circuit too if so	13:03
fungi	?	13:03
*** Guest9812 has joined #openstack-infra		13:03
*** weshay has quit IRC		13:03
*** Longgeek has quit IRC		13:03
fungi	or will it just try to merge a tag from master back into master and be a no-op?	13:04
*** Longgeek has joined #openstack-infra		13:04
*** yaguang has joined #openstack-infra		13:04
ttx	fungi: it will certainly run, for example when we push the tag for intermediary swift versions onmaster	13:04
ttx	but I think it should noop	13:05
*** yaguang has quit IRC		13:05
openstackgerrit	A change was merged to openstack/requirements: Add Designate to projects.txt https://review.openstack.org/103316	13:05
*** yaguang has joined #openstack-infra		13:05
ttx	fungi: if not, I guess we could also exclude master if it --contains TAG	13:06
*** talluri has quit IRC		13:06
*** hdd_ has joined #openstack-infra		13:07
*** chandan_kumar has joined #openstack-infra		13:07
openstackgerrit	Thierry Carrez proposed a change to openstack-infra/infra-specs: Added specification for storyboard story tags https://review.openstack.org/97211	13:09
*** blamar has quit IRC		13:09
*** pblaho_ has joined #openstack-infra		13:09
*** pblaho has quit IRC		13:09
*** Longgeek_ has joined #openstack-infra		13:10
*** lcheng__ has joined #openstack-infra		13:11
*** blamar has joined #openstack-infra		13:11
fungi	ttx: yeah, it runs under errorexit, so if the git merge call exits nonzero the job is going to be marked as failed	13:12
fungi	testing now	13:12
*** lcheng_ has quit IRC		13:13
fungi	ttx: aha! git merge responds "Already up-to-date." and exits 0 in that no-op case, so wfm	13:14
*** Longgeek has quit IRC		13:14
ttx	fungi: cool, thx for testing	13:14
*** markmcclain has joined #openstack-infra		13:15
*** dkehn__ is now known as dkehnx		13:15
*** dkliban_afk is now known as dkliban		13:16
*** weshay has joined #openstack-infra		13:16
*** yaguang has quit IRC		13:19
*** yaguang has joined #openstack-infra		13:19
*** mriedem has joined #openstack-infra		13:22
*** mriedem has quit IRC		13:22
*** mriedem has joined #openstack-infra		13:22
*** changbl has quit IRC		13:23
*** devkulkarni has joined #openstack-infra		13:25
*** ominakov has quit IRC		13:27
*** mmaglana has joined #openstack-infra		13:28
*** chandan_kumar is now known as chandankumar		13:30
*** markmcclain has left #openstack-infra		13:31
*** prad has joined #openstack-infra		13:32
*** ominakov has joined #openstack-infra		13:32
*** zz_gondoi is now known as gondoi		13:32
*** msabramo has joined #openstack-infra		13:33
*** pblaho_ is now known as pblaho		13:33
*** mmaglana has quit IRC		13:33
*** pblaho_ has joined #openstack-infra		13:33
*** amalagon has joined #openstack-infra		13:36
fungi	mordred: check your cronspam... you probably want to configure bandersnatch to not log info/debug-level messages to stdout/stderr	13:36
*** jgrimm has joined #openstack-infra		13:37
*** pblaho has quit IRC		13:37
*** pcrews has joined #openstack-infra		13:39
openstackgerrit	Ian Cordasco proposed a change to openstack/requirements: Add rfc3986 for Nova URI validations https://review.openstack.org/103892	13:41
*** mdenny has joined #openstack-infra		13:44
annegentle	dhellmann_: around? sphinx theming question for ya	13:47
annegentle	mordred: I talked to Lauren and we want to indicate a project is incubating with the docs output. I commented on https://review.openstack.org/#/c/103349/	13:48
annegentle	mordred: what I want to figure out now is the best way to provide the theme	13:48
*** basha has joined #openstack-infra		13:49
*** sarob has joined #openstack-infra		13:50
*** dstanek_zzz is now known as dstanek		13:51
* annegentle drives to work		13:52
*** devkulkarni has quit IRC		13:57
fungi	annegentle: if that sort of distinction is needed, it may be good to dynamically select it based on the programs.yaml in the governance repo (at per-project-per-release granularity presumably?)	13:57
*** om has quit IRC		13:58
*** homeless has joined #openstack-infra		13:58
*** otherwiseguy has joined #openstack-infra		13:58
*** chandan_kumar has joined #openstack-infra		13:58
*** chandankumar has quit IRC		13:59
*** basha has quit IRC		13:59
*** chandan_kumar is now known as chandan\|afk		14:00
*** chandan\|afk is now known as ciypro\|afk		14:00
*** david-lyle has joined #openstack-infra		14:01
*** nelsnelson has joined #openstack-infra		14:02
*** pblaho_ is now known as pblaho		14:02
*** eharney has quit IRC		14:02
*** chandan_kumar has joined #openstack-infra		14:04
*** roeyc has quit IRC		14:05
*** duran has joined #openstack-infra		14:05
*** mestery has quit IRC		14:05
*** flaper87\|afk is now known as flaper87		14:06
*** basha has joined #openstack-infra		14:06
*** gokrokve has joined #openstack-infra		14:06
*** gokrokve_ has joined #openstack-infra		14:07
*** basha has quit IRC		14:07
*** ominakov has quit IRC		14:08
openstackgerrit	Nikita Konovalov proposed a change to openstack-infra/storyboard: Adding Search endpoints and sqlalchemy impl https://review.openstack.org/101476	14:08
*** chandan_kumar is now known as chandankumar		14:09
openstackgerrit	Nikita Konovalov proposed a change to openstack-infra/storyboard: Adding Search endpoints and sqlalchemy impl https://review.openstack.org/101476	14:09
*** gokrokve has quit IRC		14:11
*** devkulkarni has joined #openstack-infra		14:11
*** nosnos has quit IRC		14:13
*** mestery has joined #openstack-infra		14:14
*** nosnos has joined #openstack-infra		14:14
*** basha has joined #openstack-infra		14:15
openstackgerrit	yolanda.robla proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482	14:15
*** ominakov_ has joined #openstack-infra		14:15
*** camunoz has joined #openstack-infra		14:16
*** jamielennox has quit IRC		14:17
*** camunoz_ has joined #openstack-infra		14:17
*** camunoz_gone has quit IRC		14:17
*** camunoz_ has quit IRC		14:18
*** radez_g0` is now known as radez		14:18
*** ianw has quit IRC		14:18
*** camunoz_ has joined #openstack-infra		14:18
*** ianw has joined #openstack-infra		14:18
*** nosnos has quit IRC		14:18
*** camunoz_ has quit IRC		14:18
*** ianw has quit IRC		14:18
*** camunoz_ has joined #openstack-infra		14:19
*** jamielennox has joined #openstack-infra		14:19
*** ianw has joined #openstack-infra		14:19
*** msabramo has quit IRC		14:19
*** ianw has quit IRC		14:19
*** camunoz__ has joined #openstack-infra		14:19
*** ianw has joined #openstack-infra		14:19
ArxCruz	anteaya: hello	14:19
ArxCruz	We are facing some network issues in our infra, we are already working on it	14:20
*** jamielennox_ has joined #openstack-infra		14:21
*** camunoz has quit IRC		14:21
*** camunoz has joined #openstack-infra		14:22
*** jamielennox_ is now known as jamielennox\|away		14:22
*** jamielennox has quit IRC		14:23
*** annegent_ has joined #openstack-infra		14:24
*** ianw has quit IRC		14:24
*** camunoz_ has quit IRC		14:24
yolanda	mordred, jeblair, i pushed an approach of building image once and deploying for all providers, but it still needs work, as image update doesn't need the provider to be passed (for dib), and also image-update command shouldn't need the provider in this case. Can i get some feedback?	14:24
*** camunoz__ has quit IRC		14:24
*** ianw has joined #openstack-infra		14:25
*** Longgeek_ has quit IRC		14:25
*** Longgeek has joined #openstack-infra		14:26
*** dims_ has quit IRC		14:28
*** zz_dimtruck is now known as dimtruck		14:29
*** salv-orlando has quit IRC		14:29
*** reed has joined #openstack-infra		14:30
*** vhoward- has joined #openstack-infra		14:31
*** gondoi is now known as zz_gondoi		14:34
*** nati_ueno has joined #openstack-infra		14:34
*** wenlock has joined #openstack-infra		14:35
openstackgerrit	A change was merged to openstack-infra/config: Make ATC lists sort E-mails with preferred first https://review.openstack.org/85385	14:36
openstackgerrit	A change was merged to openstack-infra/config: Modernize ATCs script to Gerrit 2.8, programs.yaml https://review.openstack.org/102698	14:38
*** nati_ueno has quit IRC		14:38
*** eharney has joined #openstack-infra		14:39
openstackgerrit	A change was merged to openstack-infra/elastic-recheck: Add query for Neutron functional test failures bug 1336172 https://review.openstack.org/103864	14:39
uvirtbot	Launchpad bug 1336172 in neutron "neutron/agent/linux/ovs_lib.py: int(ofport) returns 'NoneType' instead of string or a number" [Undecided,New] https://launchpad.net/bugs/1336172	14:39
*** gokrokve_ has quit IRC		14:40
jeblair	Kiall: so the main thing is that code is used for both long-running servers, like review.o.o, etc, as well as test nodes. i don't think we want to change anything about our long-running servers, so i think the puppet needs to grow to handle both cases separately.	14:40
sdague	fungi: we've got a config revert coming in for something that's causing 100% failure in neutron functional tests	14:41
sdague	which means blocking tempest / devstack / neutron changes	14:41
*** dims_ has joined #openstack-infra		14:41
fungi	sdague: oh, exciting	14:41
jeblair	yolanda: i'll take a look in a bit, thanks	14:42
sdague	also, is there something to be done to clear out all the stuck check jobs? (like fail them)	14:42
openstackgerrit	Henry Gessau proposed a change to openstack-infra/config: Revert "Proper passing of SUDO flag for neutron functional tests" https://review.openstack.org/103909	14:43
*** dkehn__ has joined #openstack-infra		14:43
fungi	sdague: push new patchsets, or a zuul restart	14:43
yolanda	jeblair, thx, i only updated the part of build and update images, but i think the imageupdate command and checks needs some rewrite in that case	14:44
*** zz_gondoi is now known as gondoi		14:44
yolanda	but now image-update needs provider to be passed	14:44
sdague	fungi / jeblair: https://review.openstack.org/103909 would be good to move through fast	14:44
sdague	fungi: ok, I suspect most of the authors don't know their patches are stuck because they got no negative feedback	14:45
jeblair	sdague, fungi: is the cause of the stuck jobs known?	14:45
sdague	jenkins03 died yesterday	14:46
*** dkehnx has quit IRC		14:46
fungi	jeblair: yes, jenkins03's network connectivity completely dropped yesterday morning	14:46
fungi	jeblair: rebooting it brought it back to being reachable again	14:46
jeblair	oh, probably the rax network maint	14:46
jeblair	oh, maybe not then if manual intervention was required	14:47
fungi	jeblair: nothing obvious in its logs indicating why. it was responsive from the virtual console but with no backdoor passworded account to log in with, i couldn't investigate live from that end	14:47
*** msabramo has joined #openstack-infra		14:48
Kiall	jeblair: Okay, that sounds reasonable.. I'll take a stab at it	14:49
*** hdd_ has quit IRC		14:49
fungi	jeblair: investigation showed we lost contact with nothing else, just jenkins03. figured if a reboot didn't clear it or if it recurred, i'd open a ticket with rackspace about it	14:50
*** chuckC has joined #openstack-infra		14:50
jeblair	fungi, sdague: i'm assuming the problem that was trying to be solved on the neutron functional tests is that the OS_SUDO_TESTING env var wasn't available to tox and the test runner	14:50
jeblair	fungi, sdague: so probably the command wants -E, or we want to set the env variable in tox.ini	14:51
*** Longgeek has quit IRC		14:51
*** Longgeek has joined #openstack-infra		14:51
sdague	jeblair: possibly, I'd rather revert to working	14:51
sdague	and give those folks a chance to regroup with the right change later	14:52
*** gokrokve has joined #openstack-infra		14:52
jeblair	sdague: i'm assuming we'll have to deal with it again at some point. i'd prefer one change to two. but i can also just be stingy reviewing a later change.	14:54
*** xianghuihui has quit IRC		14:54
*** eharney has quit IRC		14:54
*** devkulkarni has joined #openstack-infra		14:54
*** rdopieralski has quit IRC		14:55
*** james_li has joined #openstack-infra		14:55
*** mzoeller has joined #openstack-infra		14:56
*** flaper87 is now known as flaper87\|afk		14:57
sdague	jeblair: honestly, no idea. Right now it's blocking a bunch of stuff, including the devstack change that probably helps with the ES backlog. The neutron team is in favor of a revert	14:58
HenryG	I vote for a revert to give us more time to find the real issue/solution. Right now all of neutron is up in arms.	14:58
jeblair	fungi: i'm preparing for a zuul restart	14:58
openstackgerrit	A change was merged to openstack-infra/config: Revert "Proper passing of SUDO flag for neutron functional tests" https://review.openstack.org/103909	14:58
fungi	jeblair: okay	14:58
HenryG	Thanks!	14:58
*** PLATOSCAVE has joined #openstack-infra		14:59
*** david-lyle has quit IRC		14:59
*** HACKING-TWITTER has quit IRC		15:00
*** david-lyle has joined #openstack-infra		15:00
mordred	fungi: nod (re cronspam)	15:00
*** marun_afk is now known as marun		15:01
*** adrian_otto has joined #openstack-infra		15:01
openstackgerrit	James E. Blair proposed a change to openstack-infra/config: Comment on current state of neutron functional test https://review.openstack.org/103919	15:01
jeblair	sdague, HenryG: sure, but we've gone from obviously-broken to un-obviously broken.	15:01
*** dstanek is now known as dstanek_zzz		15:02
sdague	jeblair: we've also gone from 100% blocking non neutron projects, to letting them merge code	15:02
jeblair	sdague, HenryG: i'd prefer the tree either work, or explain why it doesn't actually do what it looks like it says it does	15:02
jeblair	sdague: if you think my request is unreasonable, feel free to -1 that patch	15:03
*** flaper87\|afk is now known as flaper87		15:03
*** david-lyle has quit IRC		15:03
*** Longgeek has quit IRC		15:03
*** david-lyle has joined #openstack-infra		15:03
*** david-lyle has quit IRC		15:03
sdague	jeblair: I'm fine with your comment	15:04
*** david-lyle has joined #openstack-infra		15:04
*** moted has joined #openstack-infra		15:05
jeblair	restarting zuul	15:06
*** otherwiseguy has quit IRC		15:06
*** ominakov_ has quit IRC		15:06
*** otherwiseguy has joined #openstack-infra		15:08
*** gokrokve has quit IRC		15:08
*** HACKING-TWITTER has joined #openstack-infra		15:08
*** HACKING-TWITTER has joined #openstack-infra		15:09
*** dstanek_zzz is now known as dstanek		15:09
*** PLATOSCAVE has quit IRC		15:09
*** HACKING-TWITTER has quit IRC		15:09
*** HACKING-TWITTER has joined #openstack-infra		15:09
*** sarob has quit IRC		15:10
jeblair	clarkb, fungi, mordred: the new pipeline requirements prevent zuul enqueue from enqueing changes in gate (they don't pass the clean check req)	15:10
*** lttrl has joined #openstack-infra		15:10
*** sarob has joined #openstack-infra		15:10
*** ominakov_ has joined #openstack-infra		15:10
jeblair	i've rechecked the 3 things in the gate queue manually	15:11
*** david-lyle has quit IRC		15:12
*** talluri has joined #openstack-infra		15:12
fungi	jeblair: ohh... good to know. i wonder how we'd want that override to work (obviously there are some requirements that we want to be able to skip in that situation and others we want still enforced as a safety check?)	15:12
fungi	or should we add a zuul vote subcommand?	15:13
jeblair	fungi: oh, that's in interesting idea -- simulate a gerrit event?	15:13
fungi	seems like there might be more than one use case for that as a feature	15:14
hashar	zuul event --trigger gerrit -- --code-review +2 --workflow +1 -m "Oh yeah!"	15:14
jeblair	hashar: you forgot "--username kool-aid-man"	15:14
*** sarob has quit IRC		15:15
*** yolanda has quit IRC		15:15
*** jpich has quit IRC		15:15
fungi	bwahaha	15:15
*** jpich has joined #openstack-infra		15:16
*** yolanda has joined #openstack-infra		15:16
hashar	I should be attending your infra meeting in a few hours. Hoping to quickly introduce my evil Zuul Cloner	15:16
jeblair	hashar: neato	15:17
*** eharney has joined #openstack-infra		15:17
*** praneshp has joined #openstack-infra		15:20
*** basha has quit IRC		15:21
hashar	and I found out that 'git fetch' from a Zuul local repo takes up to 15 seconds :/	15:23
hashar	it apparently send all the local refs/zuul/ to Gerrit and do a diff which takes a while	15:23
jeblair	hashar: yeah, we need to do something to clean up old refs periodically	15:24
*** talluri has quit IRC		15:24
*** ihrachyshka has quit IRC		15:24
hashar	I thought about altering the ref format using year/month to namespace them	15:24
hashar	would make time based garbage collection trivial	15:24
jeblair	hashar: good idea	15:25
*** talluri has joined #openstack-infra		15:25
hashar	I am dreaming about Zuul workflow overnight	15:25
jeblair	yaay! :)	15:25
fungi	there might already be a git feature to find refs older than a certain date?	15:26
fungi	if so, namespacing them might not be necessary to accomplish that	15:26
fungi	git gc --prune=<date>	15:27
jeblair	i didn't think refs had that info; i thought there were merely dumb pointers	15:27
fungi	i guess --prune just handles loose objects	15:27
jeblair	fungi: yeah, and there won't be that many because most of them will have refs pointed at them	15:28
fungi	yep	15:28
fungi	though the fs timestamp on the ref file might give it away	15:29
*** mmaglana has joined #openstack-infra		15:29
*** talluri has quit IRC		15:29
*** adrian_otto has quit IRC		15:29
jeblair	fungi: except i think we pack them? so most of them will just be in one file	15:30
*** changbl has joined #openstack-infra		15:32
fungi	ahh, yeah. poking around in the trees on zm01 to see if there	15:32
*** skolekonov has quit IRC		15:33
hashar	yup git pack-refs is in a cron	15:33
hashar	I have stolen that cron on my setup :D	15:33
*** dimtruck is now known as zz_dimtruck		15:33
*** mmaglana has quit IRC		15:33
hashar	unrelated note for pelix : we have a bit more information for Gerrit hash mismatch bug https://bugzilla.wikimedia.org/show_bug.cgi?id=53895#c22 \O/	15:34
hashar	out for some fresh air, bbl	15:36
mordred	fungi, jeblair: we're on updated images everwhere except two images	15:36
YorikSar	fungi: btw, how about merging my workaround for "hash mismatch" in git-review tests?	15:36
*** Shrews_ is now known as Shrews		15:36
*** virmitio has joined #openstack-infra		15:36
*** timrc-afk is now known as timrc		15:36
*** hashar has quit IRC		15:36
fungi	YorikSar: i'm okay with it, was just hoping another core reviewer would chime in as well	15:36
mordred	I'm kicking those by hand right now	15:36
YorikSar	https://review.openstack.org/#/c/98354/	15:36
*** mmaglana has joined #openstack-infra		15:38
pelix	hashar: wow that's a nice suble bug :)	15:38
*** IvanBerezovskiy1 has left #openstack-infra		15:38
YorikSar	fungi: Hm... Who of core team might know what that's all about?	15:38
sdague	so once this merges - https://review.openstack.org/#/c/103658/ - kicking off a nodepool image build would be good. Right now we're downloading the trove image every time.	15:39
fungi	YorikSar: clarkb has also been looking at that bug from time to time	15:39
*** flaper87 is now known as flaper87\|afk		15:40
YorikSar	fungi: Oh, right. I see him in CC.	15:40
fungi	YorikSar: though hashar's link is interesting--if we get a gerrit point release with that fixed mina-sshd the problem may just go away	15:41
YorikSar	clarkb: Can you please take another look at https://review.openstack.org/#/c/98354/ and approve it if you're OK with it?	15:41
fungi	YorikSar: my only reservation about retrying ssh connections automatically in the tests is that there's a potential for it to mask other bugs	15:41
*** msabramo has quit IRC		15:42
*** flaper87\|afk is now known as flaper87		15:42
fungi	the commit message is hilarious too... "[SSHD-330] Handshake fails (wrong shared secret) 1 out of 256 times"	15:42
YorikSar	fungi: I think we should test with Gerrit release that is deployed in community. And I doubt anyone would want to upgrade it soon,	15:42
*** lttrl has left #openstack-infra		15:42
*** gangil has joined #openstack-infra		15:43
*** lttrl has joined #openstack-infra		15:43
fungi	YorikSar: well, we currently test with 2.8.5, and this would probably end up in 2.8.6 which we'd be likely to upgrade to in production fairly quickly as well	15:43
YorikSar	fungi: Retry happens only if 'hash mismatch' string is found. So it shouldn't hide other bugs, only this one.	15:43
*** gokrokve has joined #openstack-infra		15:43
fungi	YorikSar: good point	15:43
fungi	"The shared secret returned by KeyAgreement.generateSecret() is a byte array, which can (by chance, roughly 1 out of 256 times) begin with zero byte (some JCE providers might strip this, though)."	15:44
fungi	classic	15:44
YorikSar	fungi: From what it looks like we've achieved 9 out of 6 ratio in tests.	15:44
YorikSar	*out of 10	15:44
*** adrian_otto has joined #openstack-infra		15:45
*** dstanek is now known as dstanek_zzz		15:47
mordred	I think 9 out of 6 is what I aim for	15:47
YorikSar	hashar, fungi, jeblair: I think simple 'find .git/refs/zuul -not -ctime 10 -delete' would clean those refs good enough.	15:50
fungi	YorikSar: as pointed out, packed refs defeat that	15:50
fungi	YorikSar: since they all end up as lines in the .git/packed-refs file	15:50
*** markwash has joined #openstack-infra		15:50
YorikSar	fungi: But still it would send all refs to Gerrit wouldn't it?	15:51
YorikSar	fungi: Ah, you mean, this solution sucks. Right.	15:51
*** pballand has joined #openstack-infra		15:51
*** basha has joined #openstack-infra		15:52
fungi	YorikSar: yeah, looking at the data stored for packed refs, namespacing them does indeed seem to be about the only sane solution after all	15:52
*** dangers_` is now known as dangers		15:53
YorikSar	fungi: I think you can do the same find but with logs instead refs to find refs that should be deleted...	15:53
*** thedodd has joined #openstack-infra		15:54
*** markmc has quit IRC		15:55
fungi	maybe with the reflog... but does expiring unreachable refs from the reflog also remove the corresponding refs from the repo?	15:56
*** hashar has joined #openstack-infra		15:56
*** basha has quit IRC		15:57
*** praneshp has quit IRC		15:57
YorikSar	fungi: No, it won't. But we can at least find them there.	15:57
*** ominakov_ has quit IRC		15:58
*** annegent_ has quit IRC		15:58
YorikSar	fungi: And then sed them out of packed-refs	15:59
zaro	morning	15:59
*** praneshp has joined #openstack-infra		15:59
abhishek	Hi all: can any one review this patch and help me to get this merged?	16:00
abhishek	https://review.openstack.org/#/c/102215/	16:01
abhishek	Thank you	16:01
*** hashar is now known as hasharMeeting		16:01
*** msabramo has joined #openstack-infra		16:01
fungi	YorikSar: i'm a little uncomfortable directly manipulating files in .git just because it's not really a stable api for git. we'd probably still want to use actual git calls to delete the refs once we identify them	16:01
*** vhoward- has left #openstack-infra		16:01
*** chandankumar has quit IRC		16:02
*** zz_dimtruck is now known as dimtruck		16:02
jeblair	fungi: i'm not aware of a way to delete packed refs other than manipulating (or removing) the packed-refs file :/	16:02
*** Ng has quit IRC		16:02
fungi	oh, ewwww	16:03
*** gondoi is now known as zz_gondoi		16:03
*** david-lyle has joined #openstack-infra		16:03
YorikSar	jeblair: git tag -d zuul/smth should do that	16:03
*** Ng has joined #openstack-infra		16:03
openstackgerrit	Monty Taylor proposed a change to openstack-infra/devstack-gate: Stop doing tricky things with mirrors https://review.openstack.org/103363	16:03
jeblair	YorikSar: what a strange thing for the tag command to do :)	16:03
fungi	i would have expected git branch -D to be able to do it	16:04
fungi	but i guess not	16:04
*** Ng has quit IRC		16:04
pleia2	good morning	16:04
YorikSar	Oh, git tag -d refs/zuul/smth	16:04
jeblair	well, neat :)	16:05
YorikSar	fungi: branches are a subset of refs	16:05
* mordred learns something new every day		16:05
mordred	fungi, jeblair ^^ I think that devstack-gate patch is the next thing that wants to land for the bandersnatch mirror	16:06
YorikSar	Oh, wait. Verified that. Not tag. Direct manipulations with refs are handled by update-ref	16:06
*** Ng has joined #openstack-infra		16:06
openstackgerrit	Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Restrict 1336172 query on build_name https://review.openstack.org/103933	16:08
*** whoops has joined #openstack-infra		16:09
*** salv-orlando has joined #openstack-infra		16:10
*** unicell has quit IRC		16:10
*** sarob has joined #openstack-infra		16:11
*** e0ne has quit IRC		16:11
*** mzoeller has quit IRC		16:11
*** anode has joined #openstack-infra		16:11
*** sarob_ has joined #openstack-infra		16:13
YorikSar	jeblair, fungi: So smth like this: cd .git/logs; find zuul -cname 10 \| sed 's\|./\|delete \|' \| git update-ref --stdin	16:13
mordred	jeblair: you know a lot about python logging ... if this is the log module for bandersnatch: http://paste.openstack.org/show/85249/	16:14
*** mmaglana has quit IRC		16:14
mordred	jeblair: is there a way I can control that with a logging config file?	16:14
*** unicell has joined #openstack-infra		16:14
*** crosebraugh has joined #openstack-infra		16:15
*** mmaglana has joined #openstack-infra		16:15
*** crosebraugh has left #openstack-infra		16:15
*** sarob has quit IRC		16:15
*** rcarrillocruz has quit IRC		16:15
mordred	jeblair: I have to make a patch to bandersnatch to be able to do that don't I?	16:16
*** rcarrillocruz has joined #openstack-infra		16:16
*** signed8bit has joined #openstack-infra		16:16
jeblair	ianw: in 101110, why did you chose to sort the requests twice rather than change the getPriority function to incorporate a "time waiting" term?	16:17
YorikSar	mordred: Yes	16:17
* mordred cries a little bit		16:17
*** sarob_ has quit IRC		16:17
YorikSar	mordred: But it's just one call that can happen after this method is called.	16:18
fungi	YorikSar: in your example, did you mean -ctime? and .git/refs instead of .git/logs?	16:18
jeblair	mordred: yeah, that's nicely hard-coded!	16:18
*** zhiyan is now known as zhiyan_		16:19
*** zz_gondoi is now known as gondoi		16:19
*** mmaglana has quit IRC		16:19
YorikSar	fungi: "-ctime n" means "file's status has been changed more that n*24 hours ago"	16:19
fungi	YorikSar: yeah, you said -cname (not a valid find option) which is why i asked	16:20
fungi	i'm familiar with -ctime	16:20
*** gyee_ has quit IRC		16:20
*** arnaud has joined #openstack-infra		16:20
YorikSar	fungi: And cd to logs so that we can find those stale refs by their old logs.	16:20
fungi	YorikSar: yeah, those refs don't seem to have any logs	16:21
YorikSar	fungi: Oh, right, didn't see my mistake	16:21
YorikSar	fungi: Oh...	16:21
YorikSar	fungi: That's sad...	16:21
*** cindyo has joined #openstack-infra		16:21
YorikSar	fungi: How are they created then?	16:21
fungi	YorikSar: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/merger/merger.py#n133	16:22
*** mugsie_ has quit IRC		16:23
fungi	YorikSar: gitpython's repo.create() method	16:23
*** abhishek has quit IRC		16:24
fungi	er, git.reference.create	16:25
*** pballand has quit IRC		16:25
*** andreaf has quit IRC		16:26
fungi	anyway, the more i dig into this, the more sure i am that zuul is going to need to record something which can be used to identify the age of the ref, so encoding it into the ref's name seems as sane as anything	16:26
YorikSar	fungi: Well... Then I guess some Python script should remove them...	16:27
fungi	YorikSar: well, or at least (pre)identify them for us	16:27
*** Ryan_Lane has joined #openstack-infra		16:28
YorikSar	fungi: Yeah. We can't rely on any information left then.	16:28
*** gyee has joined #openstack-infra		16:28
*** talluri has joined #openstack-infra		16:28
*** dstanek_zzz is now known as dstanek		16:28
*** pballand has joined #openstack-infra		16:29
*** arnaud has quit IRC		16:30
YorikSar	fungi: Btw, are mergers heavily loaded? I wonder if it'd be better to use libgit2 bindings instead GitPython	16:32
*** amitpp has joined #openstack-infra		16:32
*** rcarrill` has joined #openstack-infra		16:32
*** arnaud has joined #openstack-infra		16:33
mordred	jeblair, YorikSar: does this look reasonable to you: http://paste.openstack.org/show/85251	16:33
jeblair	YorikSar: libgit2 (pygit2) looks like it supports creating merges, so we could probably use it.	16:34
*** bdpayne has joined #openstack-infra		16:34
jeblair	i've been starting to think that gitpython may be becomming unmaintained	16:34
mordred	jeblair: uhoh	16:34
*** rcarrillocruz has quit IRC		16:34
YorikSar	jeblair: Last release is RC in 2011...	16:35
fungi	yeah.... :/	16:35
jeblair	which, to be fair, is about when we started using it :)	16:35
mordred	jeblair: pip install pygit2 fails, btw	16:36
jeblair	yay!	16:36
mordred	jeblair: it does not seem to declare dependencies	16:36
mordred	also, libgit2 does not exist in debian	16:36
jeblair	mordred: that's because it has none! http://libgit2.github.com/	16:36
jeblair	mordred: "Zero Dependencies"	16:36
mordred	that's libgit2	16:36
jeblair	i'm attempting to make a funny	16:37
mordred	:(	16:37
YorikSar	mordred: Looks fine except wording for 'fp' and I'm not sure if extra check for existense worth it.	16:37
mordred	YorikSar: oh - well, I just copied that part from gerritbot :)	16:37
*** jgallard has quit IRC		16:38
mordred	jeblair: so our choices are start to use a new library which is very hard to install ... or adopt another abandoned library?	16:38
jeblair	mordred: where was setup_logging called before?	16:39
mordred	jeblair: it's called at the very top with no args because the author uses logging to report config errors	16:39
jeblair	mordred: or check back in on https://pypi.python.org/pypi/dulwich	16:39
jeblair	mordred: which used to be insufficient, perhaps it has changed	16:39
mordred	jeblair: https://bitbucket.org/mordred/bandersnatch/src/e3176c3426e5d5a81510c870fc486123e91d6d9a/src/bandersnatch/main.py?at=default#cl-29	16:40
YorikSar	mordred: How did pygit2 fail?	16:40
mordred	YorikSar: in two ways	16:40
mordred	YorikSar: one - there is no libgit2	16:40
mordred	oh! wait	16:40
YorikSar	mordred: It is in Ubuntu packages :)	16:40
jeblair	mordred: though dulwich is GPLv2 (not lgpl)	16:40
*** tkelsey has quit IRC		16:40
mordred	it's in ubuntu	16:40
mordred	it's not in debian	16:40
mordred	because STAB	16:41
mordred	zul: ^^ wth man?	16:41
jeblair	mordred: oO	16:41
YorikSar	mordred: Oh, you're on Debian?	16:41
mordred	YorikSar: yes. I am running debian	16:41
mordred	and I find it very angering when someone has uploaded something to ubuntu without uploading it to debian	16:41
*** dims_ has quit IRC		16:41
jeblair	it's not in precise	16:41
mordred	jeblair: it's in trusty	16:42
*** dims_ has joined #openstack-infra		16:42
mordred	jeblair: (I have a trusty node held to check on the bandersnatch stuff, so I just checked there)	16:42
mordred	YorikSar: but also, it requires cffi and pygit2 does not declare that	16:42
mordred	that's easy enough to deal with - but it is kinda super lame	16:42
fungi	seems to be in debian experimental, but not in unstable for some unspecified reason... https://packages.debian.org/libgit2	16:42
mordred	ok. at least it's been uploaded	16:43
YorikSar	mordred: Ok... Missing cffi is strange...	16:43
jeblair	they seem to accept pull requests	16:43
*** duran has quit IRC		16:43
YorikSar	mordred: And here it fails even with cffi and libgit2 :)	16:43
mordred	oh. well,	16:44
fungi	looks like it was uploaded into debian/experimental by the ubuntu maintainer of libgit2	16:44
mordred	they're not going to accept that pull request	16:44
mordred	because they use distutils and not setuptools	16:44
mordred	and they recently accepted a pull request to add to the readme that you should install cffi first	16:44
mordred	I hate python developers	16:44
fungi	request for an upload to unstable: https://bugs.debian.org/745960	16:44
*** talluri has quit IRC		16:45
fungi	wow... "	16:45
fungi	"The primary reason for not uploading this to Debian Unstable is that the ABI changes with every release."	16:45
pleia2	yowch	16:45
*** basha has joined #openstack-infra		16:45
*** talluri has joined #openstack-infra		16:45
fungi	that may be a good reason for us not to switch to it just yet... ;)	16:45
mordred	I'm going to, yeah, say - uhm, wow	16:46
YorikSar	fungi: Python bindings follow libgit2's versions.	16:46
YorikSar	So ABI might not be a problem.	16:46
*** talluri has quit IRC		16:46
*** talluri has joined #openstack-infra		16:46
*** basha has quit IRC		16:46
fungi	YorikSar: well, it's a problem if distro maintainers of the c lib aren't comfortable with its development stability	16:46
fungi	also links to https://github.com/libgit2/libgit2/issues/2398	16:46
*** mmaglana has joined #openstack-infra		16:47
mordred	that does not seem stable enough for us to use	16:47
*** nati_ueno has joined #openstack-infra		16:48
*** hasharMeeting has quit IRC		16:48
*** yaguang has quit IRC		16:48
YorikSar	mordred: I wonder how GitHub bears using it :)	16:49
fungi	one up-side to abandoned projects... their development is very stable ;)	16:49
*** bdpayne has quit IRC		16:50
fungi	YorikSar: when did github become a free software project and try to support their project running on a variety of different platforms?	16:50
reed	fungi, yes, like a clockwatch stuck at 11:10 will show the exact time twice in a day and another watch running fast will almost never	16:51
fungi	i suspect they just pick a version and use that, and upgrade their systems to a new version when they test it	16:51
*** talluri has quit IRC		16:51
*** pelix has quit IRC		16:52
fungi	which works fine if you're running as proprietary infrastructure, but not so well for a cross-platform free software project	16:52
*** ominakov has joined #openstack-infra		16:53
YorikSar	Well... In context of Python service (that wouldn't ever care about ABI) that's primarily used in our infrastructure, I don't see downsides of using such lib given it had packages in Ubuntu.	16:53
dstufft	mordred: you can use install_requires with distutils	16:54
dstufft	mordred: distutils will just ignore it, and pip/easy_install patch shit to make it work	16:54
mordred	dstufft: yup. but it doesn't do anything	16:54
mordred	dstufft: well, that does not work with pygit2	16:54
mordred	dstufft: they have it in install_requires	16:54
dstufft	output?	16:54
mordred	https://bitbucket.org/ctheune/bandersnatch/pull-request/5/add-ability-to-configure-logging-with/diff	16:54
mordred	btw	16:54
*** sarob has joined #openstack-infra		16:54
*** jcoufal has quit IRC		16:54
*** Ng has quit IRC		16:55
fungi	YorikSar: the question is whether the pygit2 module is bound to a specific libgit2 version, and how you support a user doing a pip install of zuul onto different distros which may have a variety of libgit2 (and thus pygit2) versions, since the latest pygit2 on pypi probably doesn't support every libgit2 version in use	16:55
dstufft	use build_clib and include the source of libgit2 for static linking for maximum distro hate	16:56
YorikSar	fungi: It is tightly bound to libgit2 version.	16:56
YorikSar	Overall I get it.	16:56
*** pblaho has quit IRC		16:56
mordred	dstufft: http://paste.openstack.org/show/85256/	16:56
*** ihrachyshka has joined #openstack-infra		16:56
openstackgerrit	Andreas Jaeger proposed a change to openstack-infra/config: Cleanup manual jobs https://review.openstack.org/103646	16:56
YorikSar	Of course we could do an enteprise move and bundle .so library with Zuul :)	16:56
*** anode has quit IRC		16:56
mordred	YorikSar: no	16:56
mordred	YorikSar: I know you're joking - but no	16:57
*** Ng has joined #openstack-infra		16:57
fungi	YorikSar: right, so pip install zuul wouldn't be able to automatically pull in the right pygit2 from pypi to support a particular platform's libgit2	16:57
*** arnaud has quit IRC		16:57
dstufft	mordred: oh they are importing cffi in setup.py, they need to do a little more machinations to make that work	16:57
mordred	sigh	16:57
*** SumitNaiksatam has quit IRC		16:57
mordred	well, in general, it's not worth futzing with	16:57
mordred	the underlying c lib is unstable	16:57
dstufft	https://github.com/pyca/cryptography/blob/master/setup.py	16:57
*** anode has joined #openstack-infra		16:58
YorikSar	mordred: (about logging patch) But htis way you'll have all debug logs from all this initialization.	17:00
mordred	YorikSar: I made the patch a little simpler in the pull request above	17:00
*** devkulkarni has quit IRC		17:01
*** devkulkarni has joined #openstack-infra		17:01
YorikSar	Ok, I've disturbed this channel with libgit2 enough. I guess I should take a look at dulwich proposed by jeblair...	17:01
*** derekh_ has quit IRC		17:01
openstackgerrit	Monty Taylor proposed a change to openstack-infra/config: Log bandersnatch to file https://review.openstack.org/103942	17:02
mordred	fungi: ^^ that should fix cronspam	17:02
*** zzelle_ has joined #openstack-infra		17:02
fungi	mordred: does http://logs.openstack.org/63/103363/2/check/check-tempest-dsvm-full/644b2c1/logs/devstacklog.txt.gz#_2014-07-01_16_13_15_066 mean that it went to pypi.python.org for the download instead of using the bandersnatch mirror?	17:02
*** dkehn__ is now known as dkehn		17:02
mordred	fungi: we haven't fixed things for devstack yet ... https://review.openstack.org/#/c/103363/	17:02
mordred	fungi: but looking	17:02
*** dkehn is now known as dkehnx		17:03
*** atiwari has joined #openstack-infra		17:03
fungi	mordred: i know--that's a devstack run from the 103363 change	17:03
fungi	i'm evaluating the logs for the most recent patchset (even though it hasn't reported just yet)	17:03
fungi	before approving	17:03
mordred	hrm	17:04
*** chandan_kumar has joined #openstack-infra		17:04
mordred	fungi: it DOES appear to have done that ...	17:04
*** basha has joined #openstack-infra		17:04
*** adrian_otto has left #openstack-infra		17:05
krtaylor	anteaya, re: qa/infra midcycle meetup - any 3rd party topics? none on agenda	17:05
*** jpich has quit IRC		17:05
mordred	fungi:	17:05
mordred	2014-07-01 16:08:10.756 \| + sudo install -D -m0644 -o root -g root /home/jenkins/.pip/pip.conf '~root/.pip/pip.conf'	17:05
mordred	2014-07-01 16:08:10.758 \| install: cannot stat â€˜/home/jenkins/.pip/pip.confâ€™: No such file or directory	17:05
fungi	mordred: ahh, yep, my eyes glossed right over that error	17:05
mordred	fungi: it's possible this ran on a host that was pre-dependent patch?	17:06
*** harlowja_away is now known as harlowja		17:06
* fungi looks		17:06
*** flaper87 is now known as flaper87\|afk		17:06
fungi	devstack-trusty in hpcloud-b5	17:06
mordred	fungi: any way to tell how old the node is?	17:07
fungi	mordred: well, we can guess	17:07
jeblair	mordred: what's the dependent patch?	17:07
*** melwitt has joined #openstack-infra		17:08
fungi	mordred: the image for that az completed updating right at 2 hours ago, about an hour before that job started	17:08
mordred	fungi: the devstack-trust hpcloud-b5 image is 1.96 hours old	17:08
mordred	jeblair: the one that installs the pip.conf file in ~jenkins	17:08
fungi	mordred: so if we think we would have cycled through all our devstack-trusty nodes in an hour's time, then it should have had the new script	17:09
jeblair	2014-07-01 16:00:46,505 DEBUG nodepool.NodeLauncher: Launching node id: 670096	17:09
jeblair	fungi: no need to guess ^	17:09
fungi	oh, right!	17:09
fungi	logs... duhhh	17:09
*** melwitt1 has joined #openstack-infra		17:09
fungi	so yes, it should have built from the latest image there	17:09
jeblair	mordred: any chance you have a link to the dependent change?	17:09
mordred	jeblair: one sec	17:10
fungi	i can hold a node in that az and poke into the slave scripts to see if they're correct	17:10
mordred	jeblair: https://review.openstack.org/#/c/103354/	17:10
mordred	this: https://review.openstack.org/#/c/103354/9/modules/openstack_project/manifests/slave.pp is the important bit	17:10
*** SumitNaiksatam has joined #openstack-infra		17:11
fungi	i'll confirm presence of those files	17:11
mordred	fungi: I've got them in rax-ord, fwiw	17:12
*** basha has quit IRC		17:12
*** melwitt has quit IRC		17:12
*** sarob_ has joined #openstack-infra		17:13
*** KurtMartin is now known as kmartin		17:13
jeblair	mordred: the select mirror script will write them, right?	17:13
fungi	mordred: yeah, they exist with the correct ownership/permissions and contents	17:14
jeblair	mordred: so if you check a used node, you would expect to find them there, but written by that script	17:14
jeblair	mordred: only an unused node would tell you for sure, yeah?	17:14
*** amcrn has joined #openstack-infra		17:14
fungi	oh, right	17:14
* fungi checks the image build logs instead		17:15
*** amcrn has quit IRC		17:15
clarkb	o/	17:15
*** bdpayne has joined #openstack-infra		17:16
*** basha has joined #openstack-infra		17:17
*** sarob_ has quit IRC		17:17
jeblair	mordred: i don't think that file is used by the nodepool puppet invocation	17:17
*** msabramo has quit IRC		17:19
jeblair	mordred: i think it's openstack_project::single_use_slave -> [jenkins::slave, openstack_project::slave_common]	17:19
fungi	single_use_slave	17:19
fungi	yeah	17:19
fungi	was just tracing that, and completely agree	17:19
*** Alexei_987 has joined #openstack-infra		17:20
jeblair	openstack_project::slave appears to be the entry point for long running slaves	17:21
*** whoops has quit IRC		17:21
*** james_li has quit IRC		17:21
*** rcarrillocruz has joined #openstack-infra		17:22
*** whoops has joined #openstack-infra		17:22
*** dstanek is now known as dstanek_zzz		17:22
*** rcarrill` has quit IRC		17:23
*** aaronorosen is now known as arosen		17:23
*** melwitt1 is now known as melwitt		17:24
*** Guest9812 has quit IRC		17:24
*** devkulkarni has quit IRC		17:24
*** ociuhandu has quit IRC		17:25
*** dstanek_zzz is now known as dstanek		17:25
*** mmaglana has quit IRC		17:26
mordred	gah	17:26
mordred	so - slave_common then?	17:26
*** whoops has quit IRC		17:26
fungi	mordred: probably, if the goal is to avoid duplication	17:27
openstackgerrit	Monty Taylor proposed a change to openstack-infra/config: Put the mirror config into slave_common https://review.openstack.org/103954	17:28
*** mmaglana has joined #openstack-infra		17:28
clarkb	so it looks like the mirror stuff isn't all happy and that is being worked ^ is part of that	17:29
clarkb	jeblair: fungi: I did not follow why things are not ending up in the gate	17:29
*** markwash_ has joined #openstack-infra		17:29
*** lttrl has quit IRC		17:29
clarkb	we seemed to go from restarting zuul to nothing can end up in the gate	17:29
*** dimtruck is now known as zz_dimtruck		17:30
fungi	clarkb: when jeblair did a zuul restart and wanted to re-enqueue things into the gate, the new clean check pipeline requirement prevented the zuul enqueue cli subcommand from actually working for enqueuing into the gate pipeline	17:30
*** e0ne has joined #openstack-infra		17:30
clarkb	we have a new clean check pipeline requirements?	17:30
fungi	clarkb: because zuul had already removed the previous jenkins +1 verify vote when it enqueued them itself before	17:30
*** markwash has quit IRC		17:31
*** markwash_ is now known as markwash		17:31
clarkb	I think that is what I am missing in the middle there	17:31
fungi	clarkb: they were moved from trigger filters to pipeline requirements	17:31
fungi	after the zuul restart last weekend to pick up the new patches	17:31
jeblair	made the gate config much simpler	17:31
clarkb	oh I thought we knew that wouldn't work?	17:32
clarkb	or maybe that was an old iteration	17:32
*** whoops has joined #openstack-infra		17:32
clarkb	oh weekend now I know why I am so confused :)	17:32
fungi	heh	17:32
mordred	fungi: can you +2 https://review.openstack.org/#/c/103942/1 as well - I'd like to land both so that we can start update-image-ing all of the images	17:32
fungi	mordred: yep, i was digging in to make sure that exec actually runs that under a shell so that the redirects will work as intended	17:32
mordred	fungi: ooh, good call	17:33
fungi	oh, right... this is not an actual puppet exec, it's a crontab line	17:33
fungi	+2 from me	17:34
*** melwitt has quit IRC		17:34
*** melwitt has joined #openstack-infra		17:34
*** msabramo has joined #openstack-infra		17:34
openstackgerrit	A change was merged to openstack-infra/nodepool: Make template and node hostnames configurable https://review.openstack.org/90883	17:34
clarkb	oh and check continues to use trigger requirements. maybe that was the big that wasn't quite expected to work with pipeline requires. in any case I grok now. There were zuul changes over the weekend and they affected reenqueing change today	17:35
cindyo	fungi: My first stackforge review was merged, and package on pypi!! How do we set the version number in the futuer?	17:35
cindyo	for the pypi package?	17:35
fungi	cindyo: when you push a signed tag with a version number, it should (if the right jobs and configuration are in place in your project) upload it with that version number	17:36
cindyo	fungi: I used to use git tag '1.0.1' since we have pbr	17:36
jeblair	mordred, fungi: jenkins::jenkinsuser defines file { '/home/jenkins/.pip':	17:36
jeblair	mordred, fungi: is that going to be a problem?	17:36
clarkb	jeblair: it will cause a duplicate namevar error	17:37
cindyo	fungi: COOL! so same tag process, but then it's automatically built	17:37
fungi	jeblair: oh, maybe... yes	17:37
cindyo	yes I use a signed tag	17:37
*** basha has quit IRC		17:37
fungi	cindyo: right, same process. just make sure you're using pbr for your packaging, and that 'tox -evenv python setup.py sdist' creates a properly-named tarball in dist/	17:38
fungi	cindyo: and the pypi registration for your project needs to have openstackci as a package index maintainer, and you need to have pypi jobs for your project in zuul's layout.yaml (in the openstack-infra/config repo)	17:39
*** pblaho has joined #openstack-infra		17:39
*** ominakov_ has joined #openstack-infra		17:40
*** signed8bit is now known as signed8bit_ZZZzz		17:40
*** zz_dimtruck is now known as dimtruck		17:41
*** ominakov has quit IRC		17:42
mordred	jeblair: so really we can just not do file { '/home/jenkins/.pip' and require jenkinsuser ...	17:42
fungi	jeblair: mordred: in fact... http://logs.openstack.org/54/103954/1/check/gate-config-puppet-apply-precise/e0465d8/console.html#_2014-07-01_17_34_35_590	17:42
cindyo	fungi i think we have the registration right, would the pypi package have built automatically without that?	17:42
jeblair	btw, what's it take to fix that job and make it vote?	17:43
fungi	cindyo: it would have been built but our systems would have been unable to upload successfully... it wasn't clear from your comment that it had been uploaded automatically or you had uploaded the tarball yourself	17:43
fungi	jeblair: i believe the problem right now is that it tries to apply things like our cgit module on an ubuntu precise node	17:44
fungi	which are only written to work on centos 6 at the moment	17:44
clarkb	yes it takes everything and applies it to one node type	17:44
openstackgerrit	Monty Taylor proposed a change to openstack-infra/config: Put the mirror config into slave_common https://review.openstack.org/103954	17:44
cindyo	fungi: the upload worked too, I was able to install it with pip	17:44
clarkb	the centos6 job has the inverse problem	17:44
fungi	cindyo: what's the project name, out of curiosity?	17:45
mordred	fungi, jeblair: nibalizer and jesusaurus had some ideas on using some tool to do a proper version of that job	17:45
cindyo	fungi: python-monascaclient, if you see problems please let me know. Thnks!	17:45
clarkb	mordred: beaker	17:46
*** melwitt has quit IRC		17:46
*** sarob has quit IRC		17:47
jeblair	oh, that's where that's coming from. :)	17:47
*** melwitt has joined #openstack-infra		17:48
jeblair	i never knew the motivation for using beaker	17:48
jeblair	we had that conversation about how beaker was trying to do what zuul and nodepool do. then we ran out of time and didn't finish talking about it.	17:48
*** arnaud has joined #openstack-infra		17:48
clarkb	right I think the conversation started on the wrong foot	17:48
*** arnaud__ has joined #openstack-infra		17:48
clarkb	it should have started with we have a test, it only does rudimentary puppet apply. We can use beaker to do this properly	17:49
clarkb	then figure out the overlap with existing tools	17:49
jeblair	clarkb: i didn't think it was a contentious conversation	17:49
clarkb	it wasn't	17:49
clarkb	but it never framed the problem being solved	17:49
jeblair	well, anyway, i hope we can continue it at some point	17:50
clarkb	so there was a bit of confusion around why we would even use beaker	17:50
fungi	cindyo: i don't see the openstackci account as having permission to upload to https://pypi.python.org/pypi/python-monascaclient so it looks like someone must have uploaded it manually. also i don't see any tags in your git repository: http://git.openstack.org/cgit/stackforge/python-monascaclient/	17:50
clarkb	++	17:50
jeblair	clarkb: if beaker doesn't work out, i'm sure we can find a way to filter the node types by os and fix up the apply test	17:51
cindyo	fungi: it was the first review (adding a .gitreview) and it was automatically built and uploaded I think. We were told to create the name and version ahead of time.	17:51
cindyo	on pypi	17:52
fungi	cindyo: right, got it. usually i just put in a version of 0 when i register a pypi project	17:52
clarkb	jeblair: we can, but I think beaker makes a lot more sense when framed in a "let us test puppet apply without noop on appropriate nodes automagically and do testing with the result" way rather then. There is this puppet tool that overlaps with our tools what do we do	17:53
cindyo	ok, makes sense. In the future we'll use the signed tags	17:53
*** dhellmann_ is now known as dhellmann		17:53
*** rcarrillocruz has quit IRC		17:53
clarkb	jeblair: I really like the idea of having beaker just spin up containers and on a nodepool provided node and runnin tests against that. Because it makes it easy for any dev to reproduce on their dev machines	17:53
fungi	cindyo: so anyway, make sure to add openstackci as a package index owner in the pypi registration	17:53
*** Guest9812 has joined #openstack-infra		17:54
cindyo	fungi ok will make that change	17:54
jeblair	clarkb: sure, as long as that's a reasonable test environment; however, i'm sure it's better than most of the puppet testing we have, so it probably is :)	17:54
clarkb	mordred: safe to approve 103942?	17:54
clarkb	it looks safe /me approves	17:55
cindyo	fungi: is a package index owner the same as adding a Role of Owner for openstackci?	17:56
cindyo	to our project	17:56
fungi	cindyo: i believe so, yes	17:56
cindyo	ok, tks!	17:56
*** Guest9812 is now known as Hal_		17:56
clarkb	mordred: the common slave move looks good to me as well	17:57
fungi	cindyo: otherwise, it looks like you have the correct upload jobs defined for it in zuul's configuration, and pbr appears to create a proper sdist on a signed tag, so should be good to go. just be aware that only members of https://review.openstack.org/#/admin/groups/382,members can push a signed tag to that project	17:57
nibalizer	jeblair: sure lets continue that conversation :)	17:58
openstackgerrit	A change was merged to openstack-infra/config: Log bandersnatch to file https://review.openstack.org/103942	17:59
nibalizer	my only concern with docker is 'does stuff like iptables work in docker?'	17:59
nibalizer	i could just investigate that	17:59
clarkb	nibalizer: it won't but we are already working on that for disk image builder	17:59
clarkb	nibalizer: so I think items like that may be mostly solved	17:59
clarkb	actually it may be that iptables does work because network namespaces (that would be cool)	18:00
nibalizer	mk ya i've seen monty's point	18:00
nibalizer	is there use of the other lxc deal breakers in infra/config i.e. nfs/iscsi?	18:00
sdague	hmmm... so do we have any theories about the pip install socket timed out?	18:01
sdague	http://logs.openstack.org/58/103658/1/check/check-grenade-dsvm/f2661dd/logs/grenade.sh.txt.gz#_2014-07-01_15_45_56_784	18:01
*** signed8bit_ZZZzz is now known as signed8bit		18:01
sdague	because that was on a rax node	18:01
clarkb	nibalizer: not for infra	18:01
*** rcarrillocruz has joined #openstack-infra		18:01
sdague	oh, never mind, it wasn't	18:01
cindyo	fungi: without the signed tag pushed, on a merge no package would be built and uploaded - correct?	18:01
clarkb	sdague: I think that is what 103954 is supposed to address	18:01
*** Ryan_Lane has quit IRC		18:01
*** Hal_ has quit IRC		18:01
clarkb	nibalizer: for devstack yes, so the testing of openstack can't happen that way but the testing of infra puppet should be ok	18:01
sdague	it's on an hp node that has 3ms ping to mirror.rackspace.com	18:02
zul	mordred: hmmm?	18:02
nibalizer	sure	18:02
*** timrc is now known as timrc-afk		18:02
nibalizer	i think tthe folks working on stackforge/puppet-* would probably prefer to spin up real nodes, but infra/config can probably use the dockers	18:03
*** jamielennox\|away is now known as jamielennox		18:04
fungi	cindyo: it should upload a snapshot tarball at http://tarballs.openstack.org/python-monascaclient/python-monascaclient-master.tar.gz on each new change which is merged	18:05
*** dimtruck is now known as zz_dimtruck		18:05
clarkb	dhellmann: are you around today? looking for a quick review on moving oslo* py27/doc/pep8 jobs to trusty	18:05
jeblair	nibalizer: so here's the situation: we have a tool that spins up nodes for test jobs -- nodepool	18:05
jeblair	nibalizer: it can spin up single nodes, or multiple nodes for use in a single test	18:05
openstackgerrit	A change was merged to openstack-infra/config: Allow integer-only release tags https://review.openstack.org/95257	18:06
jeblair	nibalizer: the tests we run can do anything they want on those nodes	18:06
*** zz_dimtruck is now known as dimtruck		18:06
jeblair	nibalizer: so it's conceivable that the stackforge/puppet-* folks could run a multi-node puppet openstack deploy test	18:06
nibalizer	yes	18:07
jeblair	nibalizer: i'm not sure that we want to do a 30 node infra deployment. maybe someday, but let's consider that a stretch goal :)	18:07
nibalizer	i got some push back from you on, i think, wasting a node as the controller	18:08
fungi	that might be interesting as a check experimental	18:08
nibalizer	which i think is yes wasteful, but kinda necessary	18:08
nibalizer	jeblair: oh no i never wanted 30 infra nodes	18:08
nibalizer	should be at most 4 or something, i figure you just need to check the pipeline of things working together	18:08
clarkb	nibalizer: you need gerrit -> zuul -> jenkins -> slave -> logs -> logstash -> elasticsearch	18:08
nibalizer	but yea 30 nodes, duplicate infra in ci, then test the crap out of, that would be pretty awesome	18:08
jeblair	nibalizer: yes -- i would like to not waste a node on the controller. though, i would be more likely to waste a node as a controller in a multi-node situation	18:09
clarkb	nibalizer: I think that is our biggest node graph	18:09
nibalizer	yea the multinode situation is where you need the controller	18:09
*** anode has quit IRC		18:09
jeblair	nibalizer: but if you take a step back, we don't need to waste a node telling another single node to run a job. we have the world's biggest system for doing that already. :)	18:09
nibalizer	clarkb: hrm. well thats only 8	18:09
*** harlowja is now known as harlowja_away		18:10
nibalizer	jeblair: ya, i figure we can tell beaker to use localhost and there might be some weirdness to that but we can probably overcome any of that	18:10
jeblair	nibalizer: so, pretending that i don't know anything about beaker -- at some point, beaker logs into the second node and runs something	18:10
*** thedodd has quit IRC		18:11
jeblair	nibalizer: what's the difficulty in just making a job that just does the "runs something" part?	18:11
*** anode has joined #openstack-infra		18:12
nibalizer	jeblair: i think the way to make that work is to tell it to ssh into a node called localhost :)	18:12
*** arnaud__ has quit IRC		18:13
*** arnaud has quit IRC		18:13
nibalizer	its a little gross, but for the check each server independent of the others I think its a good middle ground	18:13
jesusaurus	jeblair: i dont know very much about beaker myself, but i think that beaker conflates 'log in to remote system' with 'make server-spec easier to use'	18:13
nibalizer	ya, what we have here is a unixy system with specific roles for different tools, beaker just does everything	18:14
jesusaurus	jeblair: so we could just use server-spec directly, but we lose some of the ease-of-use that beaker gives us	18:14
jeblair	jesusaurus: server-spec is the thing beaker runs to actually run a test?	18:14
*** W00die has quit IRC		18:14
jeblair	https://github.com/puppetlabs/beaker/wiki/Overview suggests to me that beaker just runs whatever tests you tell it to	18:15
jesusaurus	jeblair: server-spec is the format the tests are written in aiui	18:15
nibalizer	http://serverspec.org/	18:15
jesusaurus	if bodepd is around, he can describe it better	18:15
nibalizer	or anode or hogepodge	18:16
jesusaurus	he and i chatted about this for a while at the summit	18:16
clarkb	AaronGr: if you are around I would like to talk about 101603 to make sure my -1 isn't completely missing the point	18:17
*** salv-orlando_ has joined #openstack-infra		18:17
mordred	jesusaurus, nibalizer: in this case, the "ease of use" may be a misnomer ... in our case, it's not actually 'easier' ... since it conflicts with the system we already have for doing many of those takss	18:18
*** salv-orlando has quit IRC		18:18
*** salv-orlando_ is now known as salv-orlando		18:18
*** johnthetubaguy is now known as zz_johnthetubagu		18:19
nibalizer	ya we could just write serverspec tests and a scrip to do puppet apply	18:19
nibalizer	and we'd be sorta golden	18:19
clarkb	mordred: so I disagree on that statement	18:19
clarkb	mordred: because we want tests to be reproduceable by devs	18:19
hogepodge	hi nibalizer	18:19
*** ociuhandu has joined #openstack-infra		18:19
mordred	clarkb: yah	18:19
mordred	I get taht	18:19
clarkb	mordred: so if this makes it easy for devs then maybe overlap is fine	18:19
nibalizer	i like the idea of integrating with proper beaker because then the stackforge/puppet-* folks and us can use the same tooling	18:19
mordred	I'm just saying "if we only use $blah" we lose ease of us	18:19
nibalizer	and us and the rest of the puppet community can use the same tooling	18:19
nibalizer	which has cool feel good advantages, but might not be really worth it	18:20
wenlock	hey guys, so have you guys not run into any puppet exceptions today that look like http://paste.openstack.org/show/85260/	18:20
mordred	so- maybe we should make a patch to beaker to allow us to tell it in infra that we don't need it to do node management	18:20
nibalizer	and i'm pretty new to all of this	18:20
jesusaurus	yeah, i really like using the same tooling as the rest of the puppet community, rather than us doing our own thing off to the side	18:20
nibalizer	hogepodge: ohai	18:20
mordred	wenlock: nope	18:20
clarkb	wenlock: yes that is basically the hpcloud is broken go drinking issue	18:20
mordred	wenlock: ignore me, listen to clarkb	18:21
clarkb	mordred: that is facter trying to talk to metadata server	18:21
clarkb	mordred: and timing out because it isn't there	18:21
mordred	ah - yes. that fun	18:21
mordred	didn't we patch that out?	18:21
clarkb	mordred: no, jeblair -1.9'd it iirc	18:21
mordred	gotcha	18:21
mordred	jesusaurus, nibalizer: yes to using same tools as rest of puppet community	18:22
fungi	if it removes a valid reason to go drinking, i'm fully -2 on it	18:22
hogepodge	The fundamental problem of testing multiple systems is not having the sub test-harness pollute your test environment. I would think that a Docker container for Beaker could mitigate that. It has the ability to target providerless nodes.	18:22
mordred	I think I just want to push back hard on us communicating that it actually makes part of our world HARDER if beaker insists on doing provisioning tasks	18:22
clarkb	wenlock: hpcloud doesn't always hook up the metadata server. when that happens facter fails	18:22
mordred	hogepodge: right, but we dont' need a docker container	18:22
mordred	I think it's a great idea for beaker to know how to do that stuff ...	18:22
clarkb	wenlock: I had luck submitting a support ticket about it once. But not much luck since then	18:23
wenlock	clarkb, yes, we knew that tool, we started a helion_public_ipv4 facter in our code as well, but this one is causing puppet to basically role over	18:23
nibalizer	mordred: so i think that the designers of beaker WANT you to use WHATEVER you want to do to provision nodes	18:23
mordred	nibalizer: awesome	18:23
nibalizer	you write a hypervisor plugin to beaker	18:23
mordred	nononono	18:23
hogepodge	wouldn’t beaker pollute the test environment with libraries?	18:23
mordred	I don't want beaker to do that	18:23
nibalizer	mordred: no thats what they want	18:23
clarkb	wenlock: so I stopped spinning up centos6 nodes in east where it failed 100% of the time	18:23
nibalizer	the problem is that, for us, that puts the cart before the horse	18:23
wenlock	clarkb wanted to hear about approaches you guys took for how to fix it... i was thinking about meta programing in a fix to work around the function thats failing	18:23
clarkb	wenlock: see above. go drinking :)	18:23
mordred	nibalizer: I know. what I want is for us to be able to tell beaker to skip the "make me a new node" step	18:24
wenlock	clarkb hahaha!!! gotcha	18:24
clarkb	wenlock: if you depend on metadata server and it isn't there you are pretty dead in the water	18:24
mordred	hogepodge: I think, given what we're testing, "polluting the test env with libraries" is less of an inssue	18:24
hogepodge	mordered that functionality exists.	18:24
nibalizer	if we gave the beaker node credentials and a bit of code, it could api call into hp cloud or rax and provision its nodes and have its fun	18:24
mordred	sweet	18:24
mordred	hogepodge: if that exists, then I think we're golden	18:24
wenlock	clarkb but thats the problem we don't want to depend on it but because well its facter , puppet knows better	18:24
nibalizer	that is more or less how puppet uses it and intends for you to extend it	18:24
jeblair	nibalizer: that's not possible	18:24
mordred	nibalizer: we dont' want it to provision nodes	18:24
clarkb	wenlock: I don't follow	18:24
nibalizer	jeblair: i know	18:24
jesusaurus	mordred: it looks like there is a --no-provision argument we can use	18:24
clarkb	wenlock: you need metadata server for things like ssh key injection and so on	18:24
mordred	AWESOME	18:24
* mordred is excited now		18:24
jeblair	nibalizer: our test nodes will never have permission to do that because they run untrusted code submitted over the internet	18:24
clarkb	wenlock: it breaks a lot more than justfacter	18:24
wenlock	clarkb even if you don't use the facter values, it still crashes puppet	18:24
nibalizer	i know its not what we want, but i feel i've not said this to all of you before so i want to get it out	18:24
clarkb	wenlock: you just happen to see it in facter	18:25
mordred	nibalizer: :)	18:25
nibalizer	jeblair: just trying to sync stuff up	18:25
clarkb	wenlock: correct, so think of this as a canary	18:25
wenlock	clarkb right it breaks the entire puppet run	18:25
clarkb	wenlock: if facter can't talk to metadata server nothing else can and you shouldn't use that node	18:25
nibalizer	mordred: so there is a patch i made which instead of provisioing, you tell beaker what ip to log into and it gets it done	18:25
clarkb	wenlock: you can probably work around it depending on your actual needs by eg deleting the ec2.rb facter file	18:26
wenlock	clarkb but it's not just a node thing	18:26
nibalizer	so we could leverage the mulitnode nodepooling jobs	18:26
clarkb	wenlock: this was mordred's proposal	18:26
clarkb	wenlock: ?	18:26
wenlock	clearkb we see it on almost all our nodes	18:26
jeblair	mordred: i believe nibalizer's suggestion is that we use the no-provision argument and tell beaker to log into localhost to run its test	18:26
clarkb	wenlock: yes I know	18:26
*** UtahDave has joined #openstack-infra		18:26
clarkb	wenlock: on my account I cannot spin up any centos6 nodes in east	18:26
jesusaurus	nibalizer: did you test using --no-provision at all? does that just use localhost?	18:26
nibalizer	https://review.openstack.org/#/c/102071/	18:26
clarkb	wenlock: they fail 100% fo the time with this error	18:26
wenlock	clarkb for me it's ubuntu on west	18:27
nibalizer	jesusaurus: i'm not passing command line args, im setting in the nodeset file what to do	18:27
jeblair	mordred, nibalizer: i believe that functionally, that will work for us, but it's an awkward solution due to the fact that beaker seems to be conflating test environment setup with test running	18:27
mordred	jeblair: correct. that's what I'm saying.	18:27
jesusaurus	jeblair: yes, that is exactly the problem we are trying to work around	18:27
hogepodge	jeblair, right, the beaker host machine just becomes another test node	18:27
wenlock	clarkb and it looks like an easy fix, just catch the Timeout exception properly in this sucker /usr/lib/ruby/vendor_ruby/facter/util/ec2.rb	18:27
clarkb	wenlock: except not	18:27
clarkb	wenlock: because you need the metadata server usually	18:28
nibalizer	hogepodge: are yall running beaker in jenkins right now?	18:28
clarkb	wenlock: to eg add the first account's ssh keys	18:28
mordred	clarkb: do they?	18:28
clarkb	wenlock: or determine what your external IP is	18:28
mordred	clarkb: that shold have already happened	18:28
mordred	because they've connected and are running puppet	18:28
wenlock	clarkb exaclty thus implement helion_ec2_public_ipv4	18:28
nibalizer	jeblair: i agree with your last statement	18:28
clarkb	wenlock: right but it won't work	18:28
wenlock	one that works for hpcloud	18:28
mordred	once you're to "run puppet" - you should be able to ignore the metadata service	18:28
hogepodge	puppet testing is using beaker internally from jenkins	18:28
clarkb	wenlock: if metadata server isn't accepting connections you lose	18:28
clarkb	mordred: wenlock explicitly talked about adding a special ec2 fact	18:29
clarkb	mordred: so I assumed they needed it	18:29
mordred	ah. gotcha	18:29
clarkb	wenlock: the fix here is to keep pushing hpcloud to fix it	18:29
wenlock	clarkb right we don't need the fact for sure, but because ec2.rb is bugged, even if your not using it, your gona fail	18:29
*** james_li has joined #openstack-infra		18:29
clarkb	wenlock: in the mean time you may get away with deleting ec2.rb from facter	18:29
wenlock	clarkb hmmm delete it	18:30
clarkb	wenlock: yes but anything that needs metadata will fail	18:30
mordred	clarkb: right. which might be fine if you don't need the metadata	18:30
jeblair	so if the consensus is the best way to use beaker to run a single node test on an already-allocated node (without wasting an extra control node) is ssh to localhost, i'm okay giving that a try	18:30
mordred	jeblair: ++	18:30
nibalizer	awesome	18:30
clarkb	jeblair: wait why not containers?	18:30
nibalizer	im happy to post up a review to demo that	18:30
mordred	clarkb: why add the complexity?	18:31
clarkb	containers seem so much better to me	18:31
jeblair	(and ftr, i'm okay with a control node for multi-node system)	18:31
mordred	why?	18:31
mordred	we have a cloud	18:31
clarkb	mordred: so that you can have a gerrit talk to a zuul	18:31
mordred	alreayd	18:31
wenlock	isn't container spin up like 5 times faster?	18:31
clarkb	mordred: and I can run it on my laptop	18:31
clarkb	wenlock: spin up isn't reall an issue	18:31
clarkb	for me it is more about how do I test this lcoally?	18:31
clarkb	containers with beaker make that really easy	18:31
mordred	I'm assuming that beaker can already do that	18:31
mrodden	clarkb: ++	18:31
mordred	right	18:31
wenlock	true	18:31
nibalizer	clarkb: i think jelair is saying lets get single node working first before we tackle multi node, in part because we can get consensus	18:31
mordred	but WE dont' have to use a container model in infra with beaker	18:32
wenlock	same reason i like vagrant	18:32
wenlock	so it uses docker for its' containers or something else?	18:32
nibalizer	you know what would be really cool is run the multinode tests on nodepool and on docker and see if anything is different	18:32
clarkb	mordred: except for the go reproduce $failure issue	18:32
hogepodge	If you get single node working, you get the proof of concept of beaker hosted on its own test node.	18:32
mrodden	i actually use test-kitchen (which i think works with puppet, but sounds like beaker is their official thing) with LXC or an OpenStack API for testing	18:32
clarkb	anyways ignore me	18:32
nibalizer	clarkb: no i think running it on your laptop is a good use case	18:33
clarkb	starting somewhere is more important	18:33
hogepodge	It will also give you an indication of if a container is required to host beaker. Despite the complexity, it think a contained beaker is the correct solution.	18:33
sdague	looks like python33 is broken across the board on oslo-incubator	18:33
sdague	http://logs.openstack.org/15/99015/2/check/gate-oslo-incubator-python33/1f4da50/console.html#_2014-07-01_15_22_19_360	18:33
*** harlowja_away is now known as harlowja		18:33
mrodden	the server_specs are the common verification layer, not sure why that wouldn't be the best option here considering its easy to run them in localhost mode	18:33
nibalizer	i think having two files in nodesets, one being nodepool.yml and one being docker.yml or vagrant.yml is important	18:33
jeblair	clarkb: i'm not opposed to containers too, i have a small concern that the test env may not repro production	18:33
mordred	I am opposed to containers	18:33
nibalizer	and local developers could use the other nodesets	18:33
mordred	strongly	18:33
clarkb	jeblair: the single node test case doesn't either	18:33
clarkb	mordred: why?	18:33
mordred	they are an optimization. if they work as an optiization, then neat	18:34
mordred	but, if this DOES NOT WORK with actual nodes if we ahve them	18:34
mordred	then something is broken	18:34
jeblair	mrodden raises a good point	18:34
clarkb	so the reason is local reproduceablility	18:34
clarkb	its the whole reason for eg dib	18:34
mordred	I'm not _Eventually_ opposed to containers	18:34
wenlock	clarkb so devstack , local reproducible, why not that?	18:35
jeblair	clarkb: ideally, if it's broken for real, it's broken in the container too. and if it isn't, then testing in containers is less testing.	18:35
wenlock	run those test locally in devstack?	18:35
mordred	I'm just opposed to them as "The Answer" - because all they are is an easy way to have access to multiple things that look like machines	18:35
*** UtahDave has left #openstack-infra		18:35
clarkb	wenlock: devstack isn't locally reproduceable if the current situation is evidence	18:35
hogepodge	I don’t understand. I don’t want beaker poking around a node that’s being provisioned. It’s a way of creating a wall between the test sub-harness and the machines you’re testing on.	18:35
clarkb	mordred: thats fine	18:35
clarkb	mordred: I don't think anyoen is saying they are the answer	18:35
*** Hal_ has joined #openstack-infra		18:36
clarkb	mordred: I am saying it is a good balance between what our test infra does, what beaker does, and what we should attempt to target as a reasonable use case	18:36
*** thedodd has joined #openstack-infra		18:36
mrodden	containers and real nodes are different, so it would have to be tested in both, but its easy to find 80% of the problems with just LXC and they are faster...	18:36
*** ominakov has joined #openstack-infra		18:37
clarkb	mordred: right now we have almost no testing of this sort	18:37
*** ihrachyshka has quit IRC		18:37
jeblair	mrodden: the node already exists before the test starts; spin up time is not a concern because it is '0'.	18:37
nibalizer	ya if you just set the container test to run first	18:37
clarkb	mordred: we do puppet apply --noop	18:37
mrodden	right so simple case now would be run locally on the node	18:37
nibalizer	then you'll hit a lot of the problems before you pay for starting all those vms	18:37
mordred	I believe we've already determined that the container is not needed to move forward	18:37
*** ociuhandu has quit IRC		18:37
nibalizer	oh wait our systme donesn't bail does it?	18:37
mordred	we dont' pay for starting all those vms	18:37
jesusaurus	clarkb: that would change if we start using beaker	18:37
clarkb	jesusaurus: right	18:38
openstackgerrit	Elizabeth K. Joseph proposed a change to openstack-infra/nodepool: Fix Configuration link in docs https://review.openstack.org/103961	18:38
mordred	they're already started	18:38
mrodden	i think i would just do a puppet-apply and run the server_specs, you'll have to write server_specs for beaker too	18:38
clarkb	jesusaurus: so I can run puppet apply --noop just about anywhere safely	18:38
clarkb	jesusaurus: if you go to beaker ssh localhost that changes	18:38
jesusaurus	oh right	18:38
clarkb	it makes testing locally harder	18:38
nibalizer	mrodden: you don't write server spec for beaker you write a slightly different syntax called beaker-rspec	18:38
mrodden	that seems weird...	18:38
*** ominakov_ has quit IRC		18:39
mrodden	can you plug in server_specs?	18:39
wenlock	why wouldn't you use puppetlabs spec helper?	18:39
nibalizer	this by the way is what the beaker-rspec tests for my puppetboard module look like: https://github.com/nibalizer/puppet-module-puppetboard/blob/master/spec/acceptance/class_spec.rb	18:39
clarkb	but I think getting something is more important at this point than arguing fine points without any data	18:39
nibalizer	for the curious	18:39
mordred	clarkb: ++	18:40
mordred	so - to summarize	18:40
clarkb	so I am fine with the beaker ssh localhost step. But I also think we shouldn't throw out alternatives that make local testing easier	18:40
mordred	we have a system that provides large numbers of available nodes that have full root access for things runing on them	18:40
clarkb	imo one of the biggest problems we have today is that our massive CI system makes it too easy to throw local testing out	18:40
mordred	any solution we work on that also solves the problem of "getting a node is costly"	18:40
mordred	is doing something wrong for us	18:40
mordred	otoh	18:41
mordred	clarkb's point about local testing is important	18:41
mordred	so if there were a way to have the tests that we run in infra also be able to be run in a local container for a dev	18:41
*** hashar has joined #openstack-infra		18:41
clarkb	I think sdague is the only person that runs devstack + tempest locally	18:41
mordred	without infra having to run them in a container	18:41
mordred	that would be teh awesome	18:41
sdague	clarkb: just about	18:41
nibalizer	so using beaker means we can have two nodeset files (the specifications of the machines that beaker will configure) and we could have one called nodepool.yml and one called docker.yml and one called vagrant.yml	18:42
nibalizer	nodepool.yml runs in our ci, docker.yml and vagrant.yml can be run at home	18:42
jeblair	nibalizer: yeah, that's kind of along the lines i was thinking	18:43
clarkb	nibalizer: so the problem with that is that the ones used not in ci will break in about 2 weeks	18:43
mordred	nibalizer: ++	18:43
nibalizer	im confident that clark is already typing 'if we dont test vagrant/docker in ci it will break'	18:43
nibalizer	oh i was write	18:43
nibalizer	er right	18:43
mordred	how about	18:43
mordred	we try nibalizer's thing	18:43
mordred	for a bit	18:43
mordred	and then	18:43
mordred	if we discover that vagrant/docker breaks	18:43
mordred	we can test it	18:43
jesusaurus	i like the multi-file approach to start with	18:43
mordred	not because we need docker for our testing - but because at that point we're testing that the dev env works	18:44
mordred	which I can get behind	18:44
jeblair	mordred: assuming we have at least one vagrant/docker user, sure. :)	18:44
mordred	jeblair: yup	18:44
*** arnaud has joined #openstack-infra		18:44
wenlock	if vagrant/docker break and it remains broken, it means they hold no value for developers anyway	18:44
openstackgerrit	Elizabeth K. Joseph proposed a change to openstack-infra/nodepool: Fix Configuration link in docs https://review.openstack.org/103961	18:44
mordred	I want to see someone caring about that case before we care about in infra	18:44
clarkb	wenlock: that is not true	18:44
nibalizer	also if the docker/vagrant stuff breaks its probably because the ubutun/centos images used are having drifftt from what we provision in nodepool, and so if we're actually buidling those in dib we could publish docker and vagrant images	18:44
clarkb	wenlock: stuff bitrots	18:44
mordred	clarkb: I agree with wenlock	18:44
nibalizer	which would be kinda cool	18:44
clarkb	wenlock: all the time. That doesn't have to do with holding value for devleopers	18:45
nibalizer	okay /me -> shower -> infra meeting -> worldcup	18:45
sdague	mordred: based on bitrot of fedora stuff in devstack, it doesn't mean it doesn't hold value	18:45
nibalizer	thansk a lot everyone for making this conversation fun and productive!	18:45
wenlock	so its more than just the local test use case then	18:45
jeblair	clarkb, mordred, wenlock: i think if they keep breaking and don't get fixed, they hold little value	18:45
sdague	it means that devs ragequit because stuff that's supposed to work doesn't	18:45
mordred	right now the set of devs we're talking about is about 15	18:46
jeblair	clarkb, mordred, wenlock: in the case of fedora/devstack, people kept fixing fedora and we said "oh, okay, so we should test that then".	18:46
wenlock	mordred much larger if we're using it	18:46
mordred	this is a different scope than devstack	18:46
mordred	right	18:46
mordred	it's still not 2000	18:46
mordred	we don't need to solve this on devstack scale out of the gate	18:46
wenlock	true... <50	18:46
mordred	we may need to solve it	18:46
mordred	just not first thing	18:46
sdague	mordred: so, unrelated to this conversation. Something is not right with wheels and py33	18:47
mordred	sdague: EXCELLENT	18:48
sdague	as far as I can tell all olso-incubator jobs are failing	18:48
*** anode has quit IRC		18:48
clarkb	oh cool	18:48
sdague	http://logs.openstack.org/60/103960/1/check/gate-oslo-incubator-python33/444dbb7/console.html#_2014-07-01_18_27_55_639	18:48
mordred	sdague: you gota link to a fail? we _Are_ having half-finished mirror issues	18:48
clarkb	maybe I can move them to trusty now >_> and then if anything breaks it was already broken	18:48
sdague	"Routes is in an unsupported or invalid wheel"	18:48
*** melwitt has quit IRC		18:48
mordred	wow	18:49
mordred	dstufft: ^^	18:49
sdague	it's across the board from what I can see	18:49
mordred	sdague: this might have something to do with the old mirror not really pulling wheels in properly	18:49
sdague	though ES is backed up again, so seeing is tough	18:49
clarkb	oh its py33, that won't be trustied	18:50
dstufft	uh what	18:50
wenlock	clarkb can you point me to the change that was -2'd for the ec2 factered thing... i want to see some history there	18:50
mordred	dstufft: on the new bandersnatch mirror, we're seeing ^^	18:50
mordred	http://logs.openstack.org/60/103960/1/check/gate-oslo-incubator-python33/444dbb7/console.html#_2014-07-01_18_27_55_639	18:50
mordred	dstufft: which is telling us something about an invalid wheel	18:50
clarkb	wenlock: if I can find it	18:50
dstufft	does it work if you install it manually outside of a slave	18:50
* mordred trying		18:50
wenlock	clarkb thanks ... crossing fingers	18:51
sdague	anyway, I need to run off to csa pickup. Figured I'd raise the issue, as it is blocking all of the incubator	18:51
clarkb	wenlock: https://review.openstack.org/#/c/87360/ that was easier than I expected	18:51
wenlock	clarkb awesome sauce ty ty !	18:51
openstackgerrit	Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Notification Priority Constant https://review.openstack.org/103966	18:52
mordred	clarkb: trusty no have python 3.3	18:53
clarkb	mordred: right	18:53
clarkb	we will switch everything tp py3.4 in a bit	18:53
*** melwitt has joined #openstack-infra		18:53
fungi	looks like http://pypi.openstack.org/simple/Routes/ has Routes-2.0-py27-none-any.whl and Routes-2.0-py33-none-any.whl matching the requested spec, as well as a variety of tarballs	18:53
wenlock	clarkb ok cool... this was the same conclusion i was reaching as well on deleting it outright... if someone runs our stuff on a cloud that doesn't have issues with ec2 , then all is good... btw, we see this on private openstack clouds too	18:54
wenlock	clarkb its not just a hpcloud issue	18:54
*** UtahDave has joined #openstack-infra		18:54
clarkb	wenlock: it may be an openstack issue then	18:54
*** jlibosva has quit IRC		18:54
clarkb	wenlock: and someone like hpcloud with reproduceable cases of it should fix it :P	18:54
wenlock	clarkb to me it looks like there service is falling over to be honest, and its a puppet bug in that they don't handle the timeout exceptions	18:54
clarkb	wenlock: yes, puppet definitely has a bug (issue is the exception thrown by the http stuff doesn't inherit from the base exception class)	18:55
wenlock	clarkb which is why i was thinking a metaprogram that offers an update to the exception handling might be ok	18:55
wenlock	ie; if they have our alternate facter, then we fix with a metaprogram there exception bug	18:55
mordred	dstufft, clarkb: routes installs in a virtualenv from our mirror on python3.4	18:55
wenlock	plus they depricate that function in newer puppet	18:55
clarkb	wenlock: well facter and puppet are decoupled ish	18:56
clarkb	wenlock: also I think I decided Icouldn't side load a fix	18:56
dstufft	mordred: has the routes thing happened more than once?	18:56
wenlock	clarkb yeah, but it cause puppet core to fail	18:56
clarkb	wenlock: because facter loads facts funny	18:56
mordred	dstufft: it's happening in all of our oslo-incubator builds apparently	18:56
clarkb	wenlock: so by the time you sideload the fix you are already dead	18:56
fungi	mordred: clarkb: dstufft: though is python 3.4 going to download the 27 or 33 whl?	18:56
* mordred gonna try something else		18:56
dstufft	fungi: 33	18:57
dstufft	IIRC	18:57
fungi	okay	18:57
dstufft	i'm lik e99% sure	18:57
mordred	fungi: there Routes-2.0-py33-none-any.whl	18:57
wenlock	clarkb leme test my idea here.... maybe your right on that part	18:57
wenlock	clarkb i was thinking facters could re-write that code	18:57
clarkb	wenlock: I didn't test it I just raed through the code with a (former) facter maintainer and we decided it wouldn't be pretty if it was possible at all	18:57
*** ianw has quit IRC		18:57
*** alexpilotti has quit IRC		18:57
fungi	mordred: yep, see my earlier comment about what we have in our mirror for that	18:58
clarkb	wenlock: iirc facter loads all bundled facts first	18:58
clarkb	wenlock: then evaluates user provided code	18:58
fungi	mordred: clarkb: dstufft: i can successfully install the 33 whl from our mirror by hand into a 3.3 venv too	18:58
*** ianw has joined #openstack-infra		18:58
clarkb	wait why are we using wheels?	18:58
mordred	I'm now trying to install the oslo-incubator requirements from our mirror	18:58
clarkb	tox 1.6.1 provided pip shouldn't wheel by default	18:58
*** derekh_ has joined #openstack-infra		18:59
fungi	clarkb: good question. presumably gate-oslo-incubator-python33 is using newer pip than provided in the venv initially?	18:59
clarkb	fungi: maybe?	18:59
dstufft	it must be	18:59
*** marcoemorais has joined #openstack-infra		18:59
mordred	yes	19:00
mordred	install_command = pip install -U {opts} {packages}	19:00
clarkb	mordred: but that should use pip from tox	19:00
mordred	the -U, when it hits the pbr requirement, will cause pip to get upgraded	19:00
clarkb	mordred: but ti won't restart pip	19:00
clarkb	mordred: so that only hits us if pip is called twice?	19:00
mordred	h - wait	19:01
mordred	nope. it didn't	19:01
mordred	it's fine	19:01
dstufft	actually i'm real confused now	19:01
dstufft	I think the error message is only in pip develop branch	19:01
*** sarob has joined #openstack-infra		19:01
*** olaph has joined #openstack-infra		19:02
fungi	i'm rerunning it outside our mirror to upstream pypi just to confirm this really is mirror-specific	19:02
clarkb	welcome to the twilight zone	19:02
jeblair	meeting time...	19:02
dstufft	no I'm wrong, stupid git	19:02
dstufft	its' 1.5.3+	19:03
*** UtahDave has quit IRC		19:04
openstackgerrit	Adrian Otto proposed a change to openstack/requirements: Add python-marconiclient to requirements https://review.openstack.org/103970	19:05
yolanda	mordred, jeblair, were you able to take a look at nodepool change? is that a good approach to continue?	19:09
jeblair	yolanda: not yet	19:09
dhellmann	clarkb: sorry, I was afk yesterday and am still catching up on the resulting backlog; would waiting until tomorrow put you far behind?	19:10
clarkb	dhellmann: no that should be fine	19:10
dhellmann	clarkb: is there a patch to review, or do you just mean to do some testing by hand?	19:11
clarkb	dhellmann: I did testing by hand already for oslo* as reported by gerrit ls-projects. did the py27/docs/pep8 tests	19:11
*** ildikov has joined #openstack-infra		19:11
dhellmann	clarkb: ok, and py26 runs on fedora, right?	19:11
clarkb	dhellmann: and https://review.openstack.org/#/c/103660/ is up for review	19:11
clarkb	dhellmann: centos6	19:11
dhellmann	I was close	19:12
*** devkulkarni has joined #openstack-infra		19:12
fungi	so running the py33 tox env myself, i do get Routes 2.0 installed from pypi (according to pip list) and seem to be running pip 1.5.6	19:12
clarkb	fungi: hax	19:12
dhellmann	clarkb: if the jobs work, it's probably ok. I haven't looked at the review backlog though -- did you need any changes to oslo code or tox configs?	19:12
clarkb	dhellmann: no changes needed	19:13
clarkb	dhellmann: I was surprised that oslosphinx has no docs :)	19:13
dhellmann	clarkb: ironic, no? :-)	19:13
openstackgerrit	James Polley proposed a change to openstack-infra/nodepool: Track last allocations to ensure forward-progress https://review.openstack.org/101110	19:14
dhellmann	clarkb: https://bugs.launchpad.net/oslo/+bug/1336491	19:15
uvirtbot	Launchpad bug 1336491 in oslo "oslosphinx needs documentation explaining how to use it" [Undecided,New]	19:15
clarkb	dhellmann: in any case I was just hoping for a sanity check from you. I am fairly confident in the switch as I tested the 3 tests for all the oslo* projects	19:16
clarkb	dhellmann: I can wait for tomorrow if that helps you	19:16
dhellmann	clarkb: did you test any of the oslo projects that don't have oslo in the name? (pbr, taskflow, stevedore, cliff, pycadf)	19:16
clarkb	dhellmann: I did not, but they won't be grabbed by my regex so should be fine for this round	19:17
dhellmann	clarkb: ok	19:17
*** MarkAtwood has joined #openstack-infra		19:18
dhellmann	clarkb: I'll +1 the patch, and you can go ahead today. There's no need for you to wait if you've already tested and don't need me to test. Thanks for asking!	19:18
clarkb	great thanks	19:19
openstackgerrit	Khai Do proposed a change to openstack-infra/config: Setup a private gerrit instance for security reviews https://review.openstack.org/47937	19:19
*** amcrn has joined #openstack-infra		19:19
mordred	fungi, dhellmann: I cannot reproduce the oslo-incubator fail locally	19:19
fungi	mordred: nor can i	19:20
*** arnaud has quit IRC		19:20
*** ildikov has quit IRC		19:20
fungi	i'll need to hold a node and dig deeper	19:20
*** UtahDave has joined #openstack-infra		19:21
*** UtahDave has left #openstack-infra		19:21
*** moted has quit IRC		19:22
*** doude has quit IRC		19:22
*** prad has quit IRC		19:23
openstackgerrit	Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Session states now use Notifications https://review.openstack.org/103979	19:24
mfisch	Is there a way to get Jenkins to run the test suite on trunk? I have a stackforge module that I'm almost certain is broken in its current state	19:24
fungi	mfisch: run which test suite?	19:25
fungi	mfisch: oh, you mean without running on a change	19:25
mfisch	fungi: yeah, like a null commit	19:25
fungi	mfisch: we do that with periodic bitrot jobs	19:25
mfisch	I susect that the underlying deps changed on puppet-openstack client as the tests appear to be broken through no fault of my own	19:26
fungi	mfisch: though if this is a one-time need i suppose you could just upload a change to something insignificant, like the readme	19:26
mfisch	fungi: simple enough	19:26
*** yolanda has quit IRC		19:27
*** rcarrill` has joined #openstack-infra		19:28
*** lcheng__ has quit IRC		19:29
*** rcarrillocruz has quit IRC		19:30
*** signed8bit has quit IRC		19:30
*** pballand has quit IRC		19:30
*** ildikov has joined #openstack-infra		19:30
*** prad has joined #openstack-infra		19:30
*** signed8bit has joined #openstack-infra		19:31
*** chandan_kumar has quit IRC		19:31
*** yolanda has joined #openstack-infra		19:31
mfisch	fungi: this project may be abandoned, I will talk to the commuity and maybe it should just be removed from gerrit. Thanks for the idea	19:34
*** pballand has joined #openstack-infra		19:38
*** jp_at_hp has quit IRC		19:38
*** amitpp has quit IRC		19:41
krotscheck	clarkb: Are there HP things happening in the office today?	19:45
clarkb	krotscheck: apparently	19:45
clarkb	krotscheck: I decided to wfm to catch infra meeting and game more easily	19:46
krotscheck	clarkb: I did the same.	19:46
*** devkulkarni has quit IRC		19:46
krotscheck	Well, more because sunshine.	19:46
krotscheck	But now I’m sligtly crispy	19:46
*** mestery has quit IRC		19:46
*** mrmartin has joined #openstack-infra		19:46
*** cnesa2 has joined #openstack-infra		19:46
openstackgerrit	Doug Hellmann proposed a change to openstack-infra/infra-specs: Add spec for adding cross-project unit test jobs https://review.openstack.org/95885	19:47
*** jaypipes is now known as jaypipes-matchti		19:47
*** jaypipes-matchti is now known as jaypipes-GOOAL		19:47
openstackgerrit	Doug Hellmann proposed a change to openstack-infra/infra-specs: Add spec for adding cross-project unit test jobs https://review.openstack.org/95885	19:47
*** mestery has joined #openstack-infra		19:48
*** sarob has quit IRC		19:48
*** sarob has joined #openstack-infra		19:48
*** sarob has quit IRC		19:53
*** devkulkarni has joined #openstack-infra		19:54
*** annegent_ has joined #openstack-infra		19:55
*** MarkAtwood has quit IRC		19:55
*** SumitNaiksatam has quit IRC		19:56
*** arnaud has joined #openstack-infra		19:58
fungi	clarkb: wfb	20:00
fungi	(work from bar)	20:00
clarkb	jeblair: fungi: I am going to settle into watching the game, but then my plan is to delete elasticsearch01 and its volume to free up some quota, then add 1TB ssd volume to es07 and pvmove to that vlume	20:00
clarkb	that sound good to you guys?	20:00
fungi	clarkb: cool beans	20:01
*** zehicle_at_dell has joined #openstack-infra		20:01
clarkb	also lunch. I have lox apparently	20:01
fungi	i'm going to hold a py3k node real quick and see what i can figure out about the oslo-inc job, then disappear for an early dinner before coming back to polish the git-review release tasks	20:01
clarkb	fungi: have fun	20:01
clarkb	fungi: also don't forget to watch the game :P	20:01
fungi	clarkb: i suppose i can find a stream (or a suitable bar). i don't have teevee at the new place	20:02
*** krtaylor has quit IRC		20:02
fungi	seemed not terribly important	20:02
*** dhellman_ has joined #openstack-infra		20:02
*** HACKING-TWITTER has quit IRC		20:03
hashar	your meetings are nice. I hope I haven't overwhelmed you with my copy / pasted sentences	20:04
fungi	hashar: not at all. your input is very important to us	20:04
*** gokrokve has quit IRC		20:05
hashar	I guess I am representing the archetype of an happy "customer" :D	20:05
hashar	love the product, love the guys behind it and use it daily.	20:06
*** annegent_ is now known as annegentle_		20:06
*** anode has joined #openstack-infra		20:06
*** HACKING-TWITTER has joined #openstack-infra		20:06
*** HACKING-TWITTER has quit IRC		20:06
*** annegentle_ is now known as Guest79795		20:06
fungi	hashar: i could say the same about wikipedia and mediawiki	20:07
fungi	you all do a great job	20:07
*** cnesa2 has quit IRC		20:07
hashar	thanks!	20:07
*** HACKING-TWITTER has joined #openstack-infra		20:08
*** HACKING-TWITTER has quit IRC		20:08
*** krtaylor has joined #openstack-infra		20:08
* krotscheck is trying to come up with a sane API endpoint for subscribing to a resource. Anyone have opinions on /v1/subscription/{resourcetype}/{resourceid}?		20:09
*** HACKING-TWITTER has joined #openstack-infra		20:09
*** HACKING-TWITTER has quit IRC		20:09
fungi	mordred: dhellmann: sdague: dstufft: so i have an update... i can reproduce the "Routes is in an unsupported or invalid wheel" error straight from pypi.python.org, so it's not our bandersnatch mirror causing it at least	20:09
fungi	working now to narrow down the reproducible case	20:10
krotscheck	Concrete example would be /v1/subscription/project/232	20:10
dstufft	fungi: what version of routes	20:10
fungi	dstufft: the only one they've uploaded a wheel for... 2.0	20:11
*** HACKING-TWITTER has joined #openstack-infra		20:11
*** moted has joined #openstack-infra		20:11
krotscheck	….but then they’re user specific. So a subresource maybe under user?	20:11
* krotscheck wanders off to throw ideas on a wall and see which one feel best.		20:12
dstufft	lol	20:12
dstufft	fungi: I bet	20:12
fungi	dstufft: http://paste.openstack.org/show/85270/	20:13
*** adalbas has quit IRC		20:13
*** Guest79795 is now known as theannegentle		20:13
dstufft	fungi: Routes-2.0-py33-none-any.whl has both a Routes-2.0.dist-info and a Routes.egg-info	20:13
dstufft	I bet pkg_resources is getting confused	20:13
fungi	dstufft: sounds like we should update our version spec to include !2.0 and file a bug with them	20:14
*** e0ne has quit IRC		20:14
*** mbacchi has quit IRC		20:14
sdague	wendar: on the MD server thing, are all the fails that you see in neutron envs?	20:14
*** dims__ has joined #openstack-infra		20:15
fungi	dstufft: the traceback from the pip log: http://paste.openstack.org/show/85271/	20:15
*** Hal_ has quit IRC		20:16
*** SumitNaiksatam has joined #openstack-infra		20:16
dstufft	fungi: yea, that means that pip couldn't find a Wheel-Version	20:16
dstufft	Wheel-Version exists inside of a dist-info, not inside of an egg-info	20:16
fungi	dstufft: pip==1.5.6, setuptools==3.6	20:17
fungi	mordred: ^	20:17
fungi	dhellmann: ^	20:17
*** dims_ has quit IRC		20:18
mordred	fungi: so it'll only show up in our tox-based things because of the pin	20:18
mordred	it woudl be great if tox would release the next version	20:19
*** HACKING-TWITTER has quit IRC		20:19
*** sarob has joined #openstack-infra		20:19
*** dims__ has quit IRC		20:20
fungi	looks like their bug tracker is https://github.com/bbangert/routes/issues	20:20
fungi	aha! https://github.com/bbangert/routes/issues/31	20:20
fungi	"Incorrect wheel for py3k"	20:20
clarkb	why are we using 1.5.6 with tox?	20:20
clarkb	should be 1.4.x iirc	20:20
fungi	clarkb: virtualenv 1.11.6	20:21
clarkb	oh did we update virtualenv? interesting	20:21
fungi	clarkb: that's what i find on py3k-precise nodes	20:21
sdague	fungi: so can we block Routes 2.0 ?	20:22
fungi	mordred: so reqs pin of Routes to exclude 2.0 and reference that upstream bug?	20:22
clarkb	ok thats not bad	20:22
sdague	is that the fix?	20:22
fungi	sdague: i suppose?	20:22
*** HACKING-TWITTER has joined #openstack-infra		20:22
fungi	if there's consensus, i'll submit a reqs change asap	20:22
mordred	++	20:22
sdague	fungi: I'm good with that	20:23
*** gokrokve has joined #openstack-infra		20:23
fungi	inbound	20:23
*** sarob has quit IRC		20:23
openstackgerrit	Jeremy Stanley proposed a change to openstack/requirements: Exclude Routes 2.0 https://review.openstack.org/103991	20:25
fungi	sdague: mordred: ^	20:25
*** e0ne has joined #openstack-infra		20:25
sdague	fungi: +2	20:25
sdague	I like the topic as well :)	20:26
*** PLATOSCAVE has joined #openstack-infra		20:26
fungi	that wheel was apparently uploaded to pypi 2013-11-17, so it was just our failure to mirror wheels ourselves that shielded us from it before now	20:27
fungi	and with that, i'm heading to dinner	20:27
*** mbacchi has joined #openstack-infra		20:27
sdague	enjoy	20:28
sdague	and thanks	20:28
*** mbacchi has quit IRC		20:28
*** HACKING-TWITTER has quit IRC		20:28
*** mbacchi has joined #openstack-infra		20:28
*** gokrokve has joined #openstack-infra		20:29
*** ArxCruz has quit IRC		20:29
*** nati_ueno has quit IRC		20:30
*** ogelbukh has quit IRC		20:30
*** julim has quit IRC		20:32
*** dims__ has joined #openstack-infra		20:33
*** devkulkarni has quit IRC		20:33
*** julim has joined #openstack-infra		20:33
*** gokrokve has quit IRC		20:33
*** julim has quit IRC		20:33
*** MarkAtwood has joined #openstack-infra		20:34
*** sarob has joined #openstack-infra		20:35
*** devkulkarni has joined #openstack-infra		20:36
*** MarkAtwood has quit IRC		20:38
*** amcrn has quit IRC		20:38
*** dims__ has quit IRC		20:41
*** saper has quit IRC		20:41
marun	jeblair: Given that we can't seem to replicate the gate failure locally for neutron-dsvm-functional, would it be reasonable to move it back to the experimental queue so it can be run with sudo enabled and not block merges?j	20:42
marun	jeblair: (non-voting of course)	20:42
*** MarkAtwood has joined #openstack-infra		20:42
*** gokrokve has joined #openstack-infra		20:43
*** e0ne has quit IRC		20:43
arosen	Hi was wondering if someone could review this one. I was hoping to start pushing code here :) https://review.openstack.org/#/c/102697/	20:43
*** otherwiseguy has quit IRC		20:43
*** andreykurilin_ has joined #openstack-infra		20:46
*** MarkAtwood has quit IRC		20:47
*** lttrl has joined #openstack-infra		20:47
*** mbacchi has quit IRC		20:50
*** signed8bit is now known as signed8bit_ZZZzz		20:51
*** pdmars has quit IRC		20:51
*** prad has quit IRC		20:52
*** pblaho has quit IRC		20:53
*** lttrl has quit IRC		20:53
*** prad has joined #openstack-infra		20:55
*** crc32 has joined #openstack-infra		20:56
*** dprince has quit IRC		20:57
*** ominakov has quit IRC		20:58
*** misterpink has joined #openstack-infra		21:01
*** rcarrillocruz has joined #openstack-infra		21:02
mordred	fungi: that's actually great - the new mirror has actually caught a real-world error that we had let through	21:02
mordred	jeblair: ^^	21:02
*** rcarrill` has quit IRC		21:04
sdague	zomg the trove image thing landed finally!	21:04
sdague	when do nodepool images normally rebuild?	21:05
clarkb	I haev added an ssd volume to es07 as /dev/xvdc. Now I need to format, pvcreate, then pvmove /dev/xvdb1 /dev/xvdc1 ?	21:05
sdague	only took 3 rechecks on 2 different bugs :)	21:05
clarkb	and as final step vgreduce main /dev/xvdb1 ?	21:06
marun	does anyone know whether the 'stack' user is supposed to be able to use password-less sudo?	21:06
clarkb	fungi: ^ you did this before	21:06
*** prad has quit IRC		21:06
marun	the errors seen in this failure log suggest otherwise: http://logs.openstack.org/90/102490/4/check/check-neutron-dsvm-functional/9db189a/console.html#_2014-07-01_03_31_05_126	21:06
marun	sudo: no tty present and no askpass program specified	21:06
*** lttrl has joined #openstack-infra		21:06
clarkb	marun: I think it should doesn't devstack use that user in that way?	21:07
clarkb	sdague: ^	21:07
marun	clarkb: I'm not sure what the sudo error would indicate other than a password being required	21:08
sdague	no, devstack sets that up, but devstack-gate removes those permissions before tests run	21:09
sdague	https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L306	21:09
marun	sdague: sigh	21:10
clarkb	marun: you can use the jenkins user maybe?	21:10
sdague	clarkb: I thought we took it's permissions away as well?	21:10
clarkb	sdague: it may, but it needs to sudo during cleanup so I would be a bit surprised if it did	21:11
sdague	clarkb: it might be more limitted	21:11
marun	so if I need to enable sudo testing to make sure the fixes for https://bugs.launchpad.net/neutron/+bug/1336172 are going to work, do I have to disable voting or is there another way?	21:12
sdague	marun: also, I thought these were things that were expected to migrate into tempest, in which case, they need to not be calling out to sudo	21:12
uvirtbot	Launchpad bug 1336172 in neutron "neutron/agent/linux/ovs_lib.py: int(ofport) returns 'NoneType' instead of string or a number" [Critical,New]	21:12
marun	sdague: no, these are not slated for tempest	21:12
marun	sdague: these are functional tests of neutron's system interaction	21:12
clarkb	which should use rootwrap right?	21:12
marun	sdague: i.e. low level, and requiring sudo to run	21:12
clarkb	which I assume is what devstack allows to happen	21:13
clarkb	otherwise how does it work	21:13
marun	clarkb: hmmm	21:13
marun	clarkb: I think the initial conception was not to require rootwrap, but there's no reason that can't be changed	21:13
jeblair	marun, sdague: jenkins should have full sudo perms, and yeah, after devstack runs, sudo perms are removed	21:13
sdague	yeh, rootwrap is the hole that you need	21:13
clarkb	well if you are testing system interaction isn't the way to do that rootwrap?	21:13
marun	clarkb: that would also serve to postitively validate that rootwrap is configured correctly	21:13
sdague	clarkb: ++	21:13
jeblair	agreed	21:13
jeblair	marun: and that's actually why we remove stack user sudo perms -- to make sure that rootwrap is tested correctly in devstack	21:14
clarkb	jeblair: your lvm foo is probably better than mine. Does my list of steps above look correct? I need to partition, pvcreate, then pvmove?	21:14
marun	ok, fair enough.	21:14
marun	so, to my previous question...	21:14
marun	how do I validate the fixes for https://bugs.launchpad.net/neutron/+bug/1336172 (fix the TypeError and replace sudo with rootwrap)?	21:15
uvirtbot	Launchpad bug 1336172 in neutron "neutron/agent/linux/ovs_lib.py: int(ofport) returns 'NoneType' instead of string or a number" [Critical,New]	21:15
marun	Do I need to disable voting or is there another way?	21:15
*** otherwiseguy has joined #openstack-infra		21:15
marun	I don't want to cause any more havoc.	21:15
sdague	marun: so if you just change to doing rootwrap calls, that should be self testing	21:16
*** mrmartin has quit IRC		21:17
jeblair	clarkb: i think so, and the commands in http://ci.openstack.org/sysadmin.html#cinder-volume-management should help for the first part of that	21:17
marun	sdague: the problem is that the tests don't run without defining OS_SUDO_ENABLED at present, so that developers can run the functional job and skip tests that require privileged commands by default	21:17
clarkb	thanks	21:17
sdague	marun: the issue is that config changes aren't self tested	21:18
sdague	marun: propose a version that defaults it on	21:18
sdague	see that it runs	21:18
sdague	WIP it so it doesn't land	21:18
sdague	then you'll know if it works	21:18
marun	sdague: how do I run the neutron functional job in response to a config repo patch?	21:18
sdague	marun: you don't	21:18
marun	sdague: :/	21:18
sdague	what I'm saying is make your fix patch	21:19
sdague	put another patch on top of it which hard codes OS_SUDO_ENABLED=True	21:19
sdague	in the neutron tests	21:19
marun	sdague: ah, gotcha	21:19
sdague	don't land the 2nd one, just use it to make sure it works	21:19
marun	sdague: my gate-foo is weak :(	21:19
sdague	marun: yeh, no worries, we do these funny tail patches quite often to force things like this	21:20
clarkb	ok pvmove started	21:27
clarkb	running in a screen owned by me	21:27
sdague	clarkb: when does nodepool respin images?	21:27
clarkb	sdague: 1400	21:28
clarkb	UTC	21:28
sdague	ok, well about an hour ago the trove image fix landed in devstack. Once we get images based on that, we'll stop downloading trove image on every devstack run, which is about 1 minute time, and a ton of log lines	21:28
sdague	and part of what I think is impacting the indexer	21:29
clarkb	ok, I can kick them off manually now	21:29
sdague	yeh, that would be great	21:29
sdague	thanks much!	21:29
*** amalagon has quit IRC		21:29
openstackgerrit	Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] Subscription https://review.openstack.org/104004	21:30
*** Ryan_Lane has joined #openstack-infra		21:30
clarkb	sdague: I am starting with rax-ord and once those images are done will do them en mass	21:31
sdague	cool	21:32
*** dims__ has joined #openstack-infra		21:36
clarkb	jeblair: fungi: I called the es07 volume ssd01... hopefully that doesn't create any confusion but it made sense to me to have the cinder side describe that bit	21:36
clarkb	jeblair: fungi: you can always `pvs` and compare that to cinder show	21:36
*** james_li has quit IRC		21:37
mordred	jeblair, fungi: https://review.openstack.org/#/c/103954/ <-- need this to get our builds using the mirror again - devstack are all going to pypi.p.o	21:37
clarkb	oh maybe I shouldn't rebuild everything so fast and wait for 103954	21:38
mordred	clarkb: yes please	21:39
*** dhellman_ has quit IRC		21:40
*** zehicle_at_dell has quit IRC		21:42
*** radez is now known as radez_g0n3		21:43
*** mfer has quit IRC		21:44
*** dkliban is now known as dkliban_afk		21:45
*** bcrochet is now known as bcrochet\|afk		21:45
*** lcheng has joined #openstack-infra		21:49
clarkb	10% on pvmove	21:50
*** r1chard_afk is now known as r1chardj0n3s		21:50
*** jhesketh has joined #openstack-infra		21:52
jhesketh	Morning	21:52
clarkb	o/	21:53
*** signed8bit_ZZZzz is now known as signed8bit		21:55
clarkb	jhesketh: any chane you can review 103954 so that we can get that in and build new images?	21:56
mordred	jhesketh: yes please	21:57
mattoliverau	Morning all	21:58
*** jgrimm has quit IRC		21:58
mordred	morning mattoliverau	21:59
*** theannegentle has quit IRC		22:00
*** mrda-away is now known as mrda		22:02
jhesketh	mordred: Is there a reason the .pip dir isn't ensured in that change?	22:03
*** andreykurilin_ has quit IRC		22:03
clarkb	jhesketh: yup the jenkins::slave manifest does it	22:04
*** hashar has quit IRC		22:06
jhesketh	ah cool	22:06
jhesketh	clarkb, mordred: approved	22:07
clarkb	ty	22:07
openstackgerrit	A change was merged to openstack-infra/config: Put the mirror config into slave_common https://review.openstack.org/103954	22:10
*** Ryan_Lane has quit IRC		22:11
mordred	woot	22:13
*** james_li has joined #openstack-infra		22:15
*** pcm__ has quit IRC		22:15
*** weshay has quit IRC		22:16
*** gondoi is now known as zz_gondoi		22:17
*** zehicle_at_dell has joined #openstack-infra		22:17
*** jp_at_hp has joined #openstack-infra		22:18
clarkb	pvmove is at 25%	22:18
*** chuckC has quit IRC		22:26
clarkb	mordred: I am rebuilding images now	22:26
mordred	clarkb: awesome	22:26
openstackgerrit	Aaron Rosen proposed a change to openstack-infra/config: Add python-congressclient as new project to stackforge https://review.openstack.org/102697	22:27
*** dstanek is now known as dstanek_zzz		22:28
*** sarob has quit IRC		22:29
*** r1chardj0n3s is now known as r1chard_afk		22:31
*** doug-fish has left #openstack-infra		22:32
derekh_	ianw: jeblair clarkb added comment to https://review.openstack.org/#/c/101110/8	22:34
*** lcheng has quit IRC		22:35
clarkb	derekh_: cool	22:35
fungi	back from eats	22:37
clarkb	fungi: I am rebuilding images now to pick up a couple of fixes	22:37
clarkb	fungi: and the es07 pvmove is in progress	22:37
clarkb	and the US lost	22:37
fungi	one of the many things to appreciate about life on the beach... shrimp boil	22:37
clarkb	that is what hapepend while you were eating	22:37
clarkb	ooh shrimp boil	22:37
fungi	clarkb: and wow, that's a fast pvmove!	22:37
ianw	derekh_: thanks. don't think being proportional with sorting the waiting queue would be too big a change; i just thought that is best as a follow-on as it's already a big enough change	22:38
*** otherwiseguy has quit IRC		22:38
*** devkulkarni has quit IRC		22:38
clarkb	++	22:38
mordred	clarkb: do you have a 'good' process to rebuild all fo the images?	22:38
*** rlandy has quit IRC		22:38
clarkb	I really do like the approach taken in ianw's change	22:38
clarkb	mordred: I fire up screen, start one build per screen window	22:38
*** thedodd has quit IRC		22:38
fungi	mordred: i open a ton of separate windows in a screen session on nodepool and yeah	22:39
mordred	clarkb: ok. so the answer is "no"	22:39
fungi	that	22:39
clarkb	mordred: makes it easy to stay on top of where things are for various builds	22:39
clarkb	mordred: its not bad	22:39
derekh_	ianw: yup, sounds good to me, once its in action we'll know if more is needed	22:39
* mordred considers a "rebuild all" command		22:39
fungi	for ornery builds which need lots of retries (old hpcloud 1.0 az2 will live in infamy) i have one-liner shell script which loops until it succeeds and then dies	22:39
clarkb	mordred: well you usually don't want that	22:40
clarkb	mordred: because you want to start small	22:40
clarkb	one or two bad images is so much better than 50	22:40
fungi	rebuild this, now rebuild that, now heck with it just rebuild all the others i'm sure they'll work	22:40
mordred	yah	22:41
mordred	well	22:41
mordred	I mean	22:41
mordred	so you just described a canary build	22:41
*** derekh_ has quit IRC		22:41
mordred	which means it still should be possible to have doing this be a thing that nodepool knows how to do	22:41
*** PLATOSCAVE has quit IRC		22:41
clarkb	mordred: its not even a canary build so much as when clouds are cranky they get cranky	22:41
mordred	sure. but I guess it depends on why you're rebuilding	22:41
*** MarkAtwood has joined #openstack-infra		22:42
fungi	not that the new guard is at all related to the old guard, but wow cool! http://www.datacenterknowledge.com/archives/2014/06/30/openstack-integration-in-the-works-for-cray-supercomputers/	22:42
*** HACKING-TWITTER has joined #openstack-infra		22:43
*** james_li has quit IRC		22:43
clarkb	jroll: the containers discussion in the solum meeting has me curious. is your docker image build working well now?	22:43
*** dims__ has quit IRC		22:44
clarkb	fungi: is that how you know we have made it?	22:44
clarkb	cray is supporting us therefore we win?	22:44
jroll	clarkb: the post job? I don't think so, but I haven't checked in on it	22:44
clarkb	jroll: ya that job	22:44
clarkb	also :(	22:44
fungi	clarkb: when wang supercomputers claws its way back out of the scrap heap and becomes a corporate sponsor, that's when we'll know we've won ;)	22:44
jroll	clarkb: oh, JayF added this: https://github.com/openstack/ironic-python-agent/blob/master/imagebuild/coreos/full_trusty_build.sh	22:44
jroll	clarkb: just need to update the job	22:45
clarkb	jroll: cool	22:45
jroll	clarkb: I can put that patch up, if you point it out to me. jay just signed off for the day, I believer	22:45
jroll	-r	22:45
jroll	point out where that patch should be*	22:46
*** lttrl has quit IRC		22:46
clarkb	jroll: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/ironic-python-agent-jobs.yaml that file	22:46
jroll	thanks	22:46
fungi	(not that wang labs ever made a supercomputer, but they totally should once they're back from the grave)	22:46
clarkb	jroll: basically update the shell script in that job to call out to your script	22:46
jroll	yep, just had know idea where it was :)	22:46
clarkb	37.9%!	22:46
clarkb	also it is warm in seattle today	22:46
clarkb	ok I am going to take the plunge and trusty oslo* and trove* now	22:49
jroll	clarkb: is my cwd at the root of the project?	22:49
*** dstanek_zzz is now known as dstanek		22:49
clarkb	jroll: yes, you end up in root of your repo	22:49
jroll	great, thanks	22:49
mordred	clarkb: ++	22:50
*** changbl has quit IRC		22:50
jesusaurus	nibalizer: im looking at https://review.openstack.org/#/c/83678/15/modules/openstack_project/manifests/puppetmaster.pp and im wondering: what is the @{u} doing in that cron command? where does that come from?	22:50
SlickNik	clarkb: ++ on trusty trove :) (And thanks!)	22:52
clarkb	jesusaurus: iirc git magic	22:52
clarkb	jesusaurus: its a special ref alias for upstream main tracking branch or some such	22:52
openstackgerrit	Jim Rollenhagen proposed a change to openstack-infra/config: Use build script in IPA repo https://review.openstack.org/104012	22:53
jroll	clarkb: ^	22:53
jroll	brb	22:53
jesusaurus	clarkb: oh, fancy. TIL	22:53
mordred	jroll: your real name is so much longer than your irc nick!	22:55
*** r1chard_afk is now known as r1chardj0n3s		22:56
*** msabramo has quit IRC		22:56
dtroyer	is there a magic URL to find a commit in gerrit given the SHA?	22:58
*** dims__ has joined #openstack-infra		22:58
clarkb	dtroyer: if you put the git sha in the search box and search for it you will see what the url looks like	22:58
dtroyer	I get the usual change url: https://review.openstack.org/#/c/103168/	22:59
clarkb	oh huh I thought it gave you a listing, one sec I will dig up what it is	22:59
clarkb	dstufft: /#q,$1,n,z where $1 is the sha1	23:00
clarkb	er dtroyer ^	23:00
fungi	you haz tabcompletefail	23:00
dtroyer	coolness, thanks	23:01
mordred	jeblair, fungi: SSL certs...	23:01
*** virmitio has quit IRC		23:01
fungi	mordred: oh, right, i saw that in scrollback this morning and then got sidetracked	23:01
mordred	fungi: pip carps at people now if it's pointed at non ssl	23:01
fungi	i have not actually ordered any x.509 certs for openstack.org subdomains before	23:02
clarkb	neither have I	23:02
mordred	fungi: so we should really probably put a cert on pypi.openstack.org	23:02
*** rcarrill` has joined #openstack-infra		23:02
clarkb	++	23:02
fungi	being the only one on the team working directly for the foundation now, i'm okay if it falls on me to start taking care of those, but pointers from jeblair on prior process would be helpful	23:03
openstackgerrit	A change was merged to openstack-infra/config: Comment on current state of neutron functional test https://review.openstack.org/103919	23:03
*** rcarrillocruz has quit IRC		23:03
*** pballand has quit IRC		23:03
fungi	mordred: however, if it's merely a cosmetic error in this case, there's some counterargument that it will act to discourage people from using our mirror in their production configurations	23:04
clarkb	mordred: do we need to rebuilds the bare-* nodes too?	23:04
clarkb	mordred: I started with devstack-*	23:04
jroll	mordred: soooo long.	23:04
clarkb	fungi: I am going to completely change the subject	23:05
clarkb	fungi: how d oyou want to handle all of the new project requests up there that use the old version of the gerrit acl files	23:05
clarkb	fungi: maybe you want to update them using your handy script?	23:05
fungi	clarkb: i think landing them as-is is probably fine, and then we can run another cleanup pass of that script in a week or so, and after that the chances of old patterns being cargo-culted are dwindling to nonexistent most likely	23:06
clarkb	fungi: ok	23:06
fungi	after that if we still see people adding new projects with outmoded configuration (for example copied from a previous review) we can politely ask them to revisit the docs	23:07
mordred	clarkb: yeah. we should really do all of them	23:07
clarkb	mordred: ok	23:07
*** oomichi has joined #openstack-infra		23:07
*** sarob has joined #openstack-infra		23:07
*** chuckC has joined #openstack-infra		23:07
clarkb	arosen: I am going to update your congressclient change really quickly with a thing then +2 it	23:07
fungi	the whole point of that script was so that we could do this in an eventually-consistent fashion with minimal disruption to review process	23:08
arosen	clarkb: cool thanks	23:08
mordred	fungi: well, I think a) in the future pip is going to change from warning to error	23:08
mordred	fungi: and b) we should ssl just because	23:08
mordred	fungi: also, dstufft wants us to	23:08
fungi	mordred: i believe those are all (three) wholly valid reasons	23:08
*** msabramo has joined #openstack-infra		23:09
*** whoops has quit IRC		23:09
fungi	so anyway, i'll try to remember to hash out the details with jeblair if he wants/needs to no longer be doing that himself	23:09
*** zzelle_ has quit IRC		23:10
clarkb	two small changes	23:10
*** ildikov has quit IRC		23:10
*** eharney has quit IRC		23:11
openstackgerrit	Clark Boylan proposed a change to openstack-infra/config: Add python-congressclient as new project to stackforge https://review.openstack.org/102697	23:11
*** nati_ueno has joined #openstack-infra		23:11
clarkb	arosen: ^ we really do like to restrict the group that can push tags as cleanup is painful	23:11
clarkb	arosen: and stackforge/python-designateclient had snuck back into patchset 4	23:12
arosen	clarkb: cool that's fine with me, we can restrict it to me for now if you want.	23:12
openstackgerrit	A change was merged to openstack-infra/config: Enable ceilometer in rally-dsvm jobs https://review.openstack.org/103836	23:13
clarkb	almost to 50% o nthe pvmove	23:13
openstackgerrit	A change was merged to openstack-infra/config: Add py33 job for python-manilaclient https://review.openstack.org/103809	23:13
fungi	clarkb: do you feel it's negatively impacting cluster performance as much as node rolls would?	23:13
*** dimtruck is now known as zz_dimtruck		23:13
arosen	clarkb: cool thanks for merging that for me :)	23:14
clarkb	fungi: no I think this is mostly sane	23:14
fungi	awesomesauce	23:14
clarkb	arosen: well its +2 not +A yet :)	23:14
clarkb	maybe fungi or mordred or jhesketh will change that	23:14
*** otherwiseguy has joined #openstack-infra		23:14
arosen	clarkb: whoops k	23:14
openstackgerrit	A change was merged to openstack-infra/config: Set docs jobs for Manila project as votable https://review.openstack.org/103803	23:14
openstackgerrit	A change was merged to openstack-infra/config: Add functions & cleanup install_puppet.sh https://review.openstack.org/103735	23:15
clarkb	fungi: it is definitely having an affect but rebalances are really bad compared to this imo	23:15
jhesketh	arosen: what do you need reviewed?	23:15
*** wenlock_ has joined #openstack-infra		23:15
mordred	done	23:15
jeblair	fungi: wfm -- they're $10 a pop so i doubt hp would care if i expensed it, but it seems more correct for the foundation to do so...	23:15
mordred	arosen: +A	23:15
jhesketh	arosen: also, I need you to take a look at comments I had for you from a while back: https://review.openstack.org/#/c/77347/	23:15
arosen	thanks mordred	23:15
dstufft	mordred: pip will likely always allow HTTP, but the other two are totally legit	23:15
mordred	jeblair: I would approve that expense - but also, yeah. jbryce should really buy it :)	23:16
mordred	dstufft: ssssh. the misinformation is our friend	23:16
dstufft	I also din't think not having security is a good incentive anyways, people are amazingly willing to do all sorts of grossly inseure things if it makes their life better in the moment	23:16
jeblair	fungi: moreover, i try to get new certs around july of every year -- just so i only have to worry about it once a year	23:16
fungi	jeblair: what a coincidence--it's july!	23:16
jeblair	fungi: so now's probably a good time to check and see what's up :)	23:16
*** wenlock has quit IRC		23:17
*** wenlock_ is now known as wenlock		23:17
fungi	jeblair: should i just generate new csrs for anything coming up for expiration, and do you happen to have a comprehensive list of which ones i should keep tabs on?	23:18
clarkb	jenkins.o.o can probably go self signed?	23:18
ianw	mordred: all oslo-incubator-python33 failing with "Routes is in an unsupported or invalid wheel"; the only thing i can find via google is https://review.openstack.org/#/c/92221/ which you wrote; any tips?	23:19
ianw	mordred: example failing log http://logs.openstack.org/60/103960/1/check/gate-oslo-incubator-python33/444dbb7/console.html	23:19
dstufft	Routes has a borken wheel	23:19
clarkb	ianw: routes 2.0 whl is broken. requirements update pushed by fungi to remove that version	23:19
fungi	ianw: fix is at the top of the gate, about to merge	23:19
jeblair	fungi: nothing expires this year :)	23:19
jeblair	fungi: http://paste.openstack.org/show/85284/	23:20
openstackgerrit	A change was merged to openstack-infra/config: Revert "Downgrade puppetlabs-apache to version 0.0.4." https://review.openstack.org/103258	23:20
fungi	jeblair: convenient--have you been getting multi-year i guess?	23:20
openstackgerrit	A change was merged to openstack-infra/config: Use build script in IPA repo https://review.openstack.org/104012	23:20
jeblair	fungi: 2 year, yeah	23:20
* dstufft is glad we have a single star cert and we just route everything through TLS termination		23:20
dstufft	we = PSF	23:20
ianw	clarkb / fungi : thanks	23:20
* jeblair is very glad that the cert on jenkins.o.o can't be used for any actually important hosts		23:20
fungi	dstufft: yeah, for us (cloud using peeps) that would add an additional point of risk for outages	23:21
fungi	if we had dedicated servers in a data center together, different story perhaps	23:21
jeblair	and the tls termination approach would require us to additionally secure the termination<->server path against an untrusted cloud provider	23:21
fungi	yup	23:22
dstufft	fungi: pft that's what a globally distributed load balancing CDN is for :D	23:22
fungi	s/untrusted cloud provider/dstufft's employer/ ;)	23:22
*** salv-orlando has quit IRC		23:22
jeblair	(which isn't to say that we don't occasionally trust our cloud providers, just that we grumble about it when we do)	23:22
fungi	we like to grumble	23:23
*** lcheng has joined #openstack-infra		23:23
clarkb	argle bargle	23:23
fungi	curmudgeons the lot of us	23:23
fungi	glop glyf!	23:23
jeblair	coremudgeons even	23:23
*** MarkAtwood has quit IRC		23:23
fungi	those as well, yes	23:23
clarkb	seriously how hard is dns that doesn't blacklist good ips	23:23
dstufft	sounds pretty hard	23:23
jeblair	fungi: so anyway, that's the list -- i have a script on ci-puppetmaster that generates csrs	23:23
dstufft	(idk)	23:23
dstufft	(im joking too)	23:23
fungi	clarkb: apparently about as hard as policing your tenants who run compromised/malicious payloads	23:24
gothicmindfood	everything is hard at HP	23:24
gothicmindfood	(everything)	23:24
mordred	gothicmindfood wins	23:24
clarkb	gothicmindfood: I think that is why we use rax	23:24
* clarkb hides		23:24
mordred	I was just telling wenlock some things about that earlier	23:24
fungi	except it's rax that's blacklisting our servers for someone else's dns queries, so whatevs	23:24
jeblair	fungi: ~root/certs/gencert.sh; will create csrs with standard info	23:24
mordred	oh right	23:24
fungi	jeblair: thanks! added to the notes	23:25
clarkb	fungi: right but hpcloud doesn't even give us consistent servers of dnssec	23:25
clarkb	fungi: they say "use this it doesn't work and I don't know where it came from or what it will be tomorrow. have fun"	23:25
fungi	jeblair: have you been using a fresh csr for each renewal, or an original csr on file with the vendor?	23:25
jeblair	fungi: and i use namecheap comodo domain validated certs cause they're $10	23:25
jeblair	fungi: fresh	23:25
wenlock	i bet that is the problem with ec2 metadata props too	23:25
jeblair	since the command makes everything anyway	23:25
fungi	jeblair: wfm. thanks again!	23:26
clarkb	wenlock: possibly	23:26
wenlock	those http request are being blocked	23:26
wenlock	from some of hp's range	23:26
clarkb	wenlock: but likely not. since two different clouds	23:26
openstackgerrit	A change was merged to openstack-infra/config: Add proposed/* branches support to merge_tags.sh https://review.openstack.org/102812	23:26
wenlock	clarkb not really	23:26
jeblair	fungi: and then i use "hostmaster@openstack.org" as the approver email; password for the account is in the usual place	23:26
wenlock	clarkb it's 15.x.x stuff	23:26
fungi	jeblair: yep. perfect	23:26
clarkb	wenlock: metadata happens over the 169 network iirc	23:26
wenlock	and it leaks out of the cloud	23:26
dstufft	fungi: maybe you should stop spinning up servers with lame IP addresses then!	23:26
clarkb	wenlock: so it shouldn't leak and should be entirely local	23:27
mordred	fungi, jeblair, clarkb: SO ... I had disabled puppet over the weekend because I was going to apply the ansible chance	23:27
clarkb	mordred: yes?	23:27
fungi	dstufft: well, we spin up so many servers, statistically many of them will get lame ip addresses. it's the way of the universe	23:27
clarkb	I think I +2'd the stack should be good to go right?	23:27
mordred	and I jst remembered - so I went and undisabled it and ran a puppet agent --test ...	23:27
mordred	and it applied a change that should have been applied back in april	23:27
clarkb	woo	23:28
dstufft	software.	23:28
fungi	i trust some of this software more than my own wetware	23:28
dstufft	the only way to really be happy with computer is to keep lowering your expectations until it matches what the computer decides to give you	23:28
jeblair	mordred: where?	23:29
openstackgerrit	A change was merged to openstack-infra/config: remove non-existing job https://review.openstack.org/103412	23:30
openstackgerrit	A change was merged to openstack/requirements: Exclude Routes 2.0 https://review.openstack.org/103991	23:30
fungi	ianw: ^	23:31
mordred	jeblair: on the puppetmaster	23:32
openstackgerrit	A change was merged to openstack-infra/config: Stop running pypi-mirror for old mirror creation https://review.openstack.org/103360	23:32
mordred	jeblair: so, I mean, it was a TINY amount of changes	23:32
*** devkulkarni has joined #openstack-infra		23:32
mordred	but still	23:33
openstackgerrit	A change was merged to openstack-infra/config: Add python-congressclient as new project to stackforge https://review.openstack.org/102697	23:34
fungi	dhellmann: https://review.openstack.org/104022	23:38
*** sarob has quit IRC		23:38
clarkb	yum is so slow	23:39
clarkb	mtreinish: we should pacman everyting	23:39
ianw	fungi: that will auto-merge, right?	23:39
fungi	ianw: huhwhat?	23:39
*** lcheng has quit IRC		23:40
fungi	ianw: requirements proposal changes like 104022 still get reviewed and approved like any normal change	23:40
fungi	ianw: though you can see if it's working by watching https://jenkins07.openstack.org/job/gate-oslo-incubator-python33/438/console	23:40
fungi	fwiw, it's been running long enough that i'm pretty sure it's not going to bail on the Routes wheel problem (it'	23:41
fungi	s past the installdeps stage now)	23:41
*** MIDENN_ has joined #openstack-infra		23:41
ianw	fungi: sorry, yeah i meant "will have a change automatically created to fix the issue" i.e. don't do anything :) https://review.openstack.org/#/c/104022/ i didn't realise it was basically instant	23:42
fungi	and it succeeded	23:42
fungi	ianw: yeah, https://jenkins.openstack.org/job/propose-requirements-updates/90/console did that	23:43
fungi	ianw: it runs as a post pipeline job on openstack/requirements commit merges	23:43
*** mdenny has quit IRC		23:44
ianw	fungi: cool. i've fulfilled my learn something about -infra quota for today and it's only 9:30 :)	23:45
clarkb	9:30 ianw where d oyou sit?	23:45
clarkb	australia?	23:45
*** nelsnelson has quit IRC		23:46
ianw	clarkb: sydney australia. i about half stand/half sit, after I read that a standing desk was like the equivalent of running 3 marathons a year or something	23:46
* fungi stands all day every day, and also walks about 10km every morning before work. totally worth it		23:48
fungi	ianw: get a good commercial grade anti-fatigue mat to stand on. it made all the difference for me. i can stand through a 12+ hour workday nonstop without problems	23:49
fungi	the kind they use in restaurant kitchens. a little expensive, but totally worth it	23:50
clarkb	the problem with workfing from home to watch world cup is I miss out on my walks	23:50
*** penguinRaider has joined #openstack-infra		23:50
*** Sukhdev has joined #openstack-infra		23:52
*** devkulkarni has quit IRC		23:53
clarkb	argle bargle	23:54
clarkb	image builds race puppet creating projects in the project list	23:55
clarkb	mordred: any chance you can look into why python-congressclient isn't a thing yet?	23:55
clarkb	images won't work until that is sorted and I am melting	23:56
jhesketh	clarkb: so.. what do we need to do to remove the envinject module from Jenkins?	23:56
clarkb	jhesketh: we need to put one in shutdown mode (07 is the usual suspect), wait for jobs to finish on it, shut it down, remove the plugin, start it again	23:57
*** unicell has quit IRC		23:57
clarkb	then use it as a guinea pig to make sure nothing terrible has happeend	23:57
clarkb	and if not we usually do the remaining master two at a time	23:57
*** sabari has joined #openstack-infra		23:58
jhesketh	clarkb: at the moment puppet only has the module removed from jenkins-dev... should we try it out there somehow?	23:58
clarkb	mordred: looking at logs Tue Jul 1 23:15:05 UTC 2014 was the last time puppet ran on review.o.o?	23:59
clarkb	jhesketh: we could get nodepool running on jenkins dev as well as zuul-dev	23:59
clarkb	jhesketh: I know zaro was working on that a while back and don';t know where it got left off	23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!