Wednesday, 2013-09-18

*** atiwari has quit IRC00:00
openstackgerritA change was merged to openstack-infra/config: Add groups-dev.openstack.org node  https://review.openstack.org/4260800:02
*** sarob has quit IRC00:03
*** sarob_ has quit IRC00:04
*** xchu has joined #openstack-infra00:06
*** senk has joined #openstack-infra00:07
fungiin a bit i'll try firing up a vm for that ^00:07
*** jergerber has quit IRC00:09
openstackgerritLucas Alvares Gomes proposed a change to openstack/requirements: Update WSME minumium to 0.5b5  https://review.openstack.org/4705900:11
*** kpavel has quit IRC00:13
*** senk has quit IRC00:14
*** nosnos has joined #openstack-infra00:18
fungijeblair: here's that fun sigpipe issue on centos6-8 analyzed a bit http://paste.openstack.org/show/47200/ (i've moved the offending workspace to workspace/gate-nova-python26.broken in case anyone else wants to play with it, and put the slave back in service now)00:21
*** amotoki_ has quit IRC00:22
*** svarnau has quit IRC00:22
fungithe offending commit was apparently from a test of https://review.openstack.org/3860300:23
*** amotoki has quit IRC00:26
*** Ryan_Lane has quit IRC00:38
*** dkehn has quit IRC00:39
*** dkehn has joined #openstack-infra00:41
*** dcramer__ has joined #openstack-infra00:42
mroddencurious:       "username => 'gerrig', # This is not a typo, well it isn't anymore."00:43
clarkbmrodden: it was a typo, but by the time anyone realized it was easier to just live with it00:45
clarkbmrodden: comment is there to avoid changes to "fix" it00:45
mroddenmust really make the OCD people shift in their seats00:45
mroddenso... jenkins-jobs update gets run whenever the config changes00:47
mroddenwhich is whenever puppet runs?00:47
clarkbmrodden: only whenever the JJB config files change00:47
mroddenyeah00:48
mroddensends a notify to run00:48
mroddenso how often does puppet check that00:48
*** jhesketh__ has quit IRC00:48
*** jhesketh__ has joined #openstack-infra00:48
pleia2every 10 minutes00:48
clarkbwith the git behind it updating every 15 minutes, so it actually varies and is a little annoying like that00:49
mroddengotcha00:49
mroddenthanks00:49
mroddenwondering if i should run my jenkins-jobs update in a chef/puppet manner or just have a manually configured jenkins job to run it periodically00:50
clarkbmrodden: I think either way should work00:53
*** amotoki has joined #openstack-infra00:54
mroddenyeah, i'd like to work on my chef/puppet skills, but at the same time its totally overkill for what i'm doing right now00:54
mroddenplus its easier to get reports on failures from Jenkins at the moment00:55
*** zzs has joined #openstack-infra01:00
*** zzs has left #openstack-infra01:04
*** nati_ueno has quit IRC01:09
*** yongli_away is now known as yongli01:15
*** weshay has joined #openstack-infra01:29
*** markmc has quit IRC01:30
*** yongli has quit IRC01:31
*** pcrews has quit IRC01:34
*** cody-somerville has quit IRC01:40
*** yongli has joined #openstack-infra01:47
*** gyee has quit IRC01:47
*** changbl has joined #openstack-infra01:47
fungiworth noting, we're working on making that more event-driven by combining puppet with salt01:50
*** mriedem has joined #openstack-infra01:52
*** dkliban has joined #openstack-infra01:54
*** marun has quit IRC01:56
fungiso that puppet configuration will be applied more or less immediately once a change merges to the config repo01:56
*** rfolco has joined #openstack-infra01:56
fungirather than on a timer01:56
*** sarob has joined #openstack-infra01:57
fungithat reminds me, i should try to get the next stage of that up for review tomorroy01:58
fungitomorrow01:58
*** adalbas has quit IRC01:59
*** mriedem has quit IRC02:02
*** mriedem has joined #openstack-infra02:03
*** CaptTofu has quit IRC02:06
*** hasharBusy is now known as hashar02:08
*** cody-somerville has joined #openstack-infra02:13
*** rfolco has quit IRC02:15
*** dims has quit IRC02:15
*** sarob has quit IRC02:15
*** sarob has joined #openstack-infra02:16
*** amotoki_ has joined #openstack-infra02:17
*** sarob has quit IRC02:21
mordredfungi: ++02:27
fungibut now, firing up the groups-dev vm for mrmartin02:29
*** jerryz has quit IRC02:38
*** yaguang has joined #openstack-infra02:41
*** sdake_ has joined #openstack-infra02:50
*** jhesketh__ has quit IRC02:50
*** jhesketh__ has joined #openstack-infra02:55
morganfainberglifeless, ping02:59
mordredAlex_Gaynor: how 'funny': https://travis-ci.org/alex/alchimia/jobs/1148049003:07
mordredAlex_Gaynor: py.test seems to be discovering tests in such a way that it looks inside of the .egg that's in the source dir because of the setup_requires03:07
mordredand then finds the tests in the pbr package03:08
mordredwhich it cannot run03:08
*** mriedem has quit IRC03:08
*** mestery has joined #openstack-infra03:10
*** mestery has quit IRC03:10
*** mestery has joined #openstack-infra03:10
*** sdake_ has quit IRC03:12
*** davidlenwell_ has joined #openstack-infra03:12
*** sld_ has joined #openstack-infra03:13
*** clayg_ has joined #openstack-infra03:13
*** mkoderer_ has joined #openstack-infra03:13
*** _sirushti has joined #openstack-infra03:13
*** jhesketh has quit IRC03:13
*** mkerrin has quit IRC03:13
*** gaelL has quit IRC03:13
*** olaph has quit IRC03:13
*** sld has quit IRC03:13
*** clayg has quit IRC03:13
*** Daviey has quit IRC03:13
*** adarazs has quit IRC03:13
*** mkoderer has quit IRC03:13
*** Reapster has quit IRC03:13
*** davidlenwell has quit IRC03:13
*** sirushti has quit IRC03:13
*** _sirushti is now known as sirushti03:13
*** adarazs_ has joined #openstack-infra03:14
*** hashar has quit IRC03:14
*** mkoderer_ is now known as mkoderer03:14
*** mkerrin has joined #openstack-infra03:15
*** zehicle_at_dell has joined #openstack-infra03:16
*** gaelL has joined #openstack-infra03:18
*** jhesketh has joined #openstack-infra03:18
*** portante has quit IRC03:18
*** w_ has joined #openstack-infra03:18
*** Daviey has joined #openstack-infra03:18
*** Reapster has joined #openstack-infra03:18
*** clayg_ is now known as clayg03:18
*** mestery has quit IRC03:19
*** motoki has joined #openstack-infra03:19
*** weshay has quit IRC03:19
*** amotoki_ has quit IRC03:19
*** portante has joined #openstack-infra03:20
*** ArxCruz has quit IRC03:21
*** ArxCruz has joined #openstack-infra03:22
*** sarob has joined #openstack-infra03:26
mordredAlex_Gaynor: ok. pr updated03:30
*** vogxn has joined #openstack-infra03:31
*** sarob has quit IRC03:32
*** pcrews has joined #openstack-infra03:39
*** dkliban has quit IRC03:45
*** dkliban has joined #openstack-infra03:46
*** dkliban has quit IRC03:54
*** vipul has quit IRC03:59
*** vipul has joined #openstack-infra04:01
*** thomasm has joined #openstack-infra04:04
*** dkliban has joined #openstack-infra04:12
*** marun has joined #openstack-infra04:12
*** ArxCruz has quit IRC04:13
*** thomasm has quit IRC04:23
*** xchu has quit IRC04:25
*** ryanpetrello has quit IRC04:27
*** dkliban has quit IRC04:28
lifelessmorganfainberg: pong04:30
*** SergeyLukjanov has joined #openstack-infra04:32
*** ryanpetrello has joined #openstack-infra04:32
*** Ryan_Lane has joined #openstack-infra04:35
*** pcrews has quit IRC04:47
*** boris-42 has joined #openstack-infra04:58
morganfainberglifeless, i forgot what i was going to ask =/   I'll ask tomorrow if i can remember04:59
lifelessmorganfainberg: kk05:04
*** jpeeler has quit IRC05:08
*** jpeeler has joined #openstack-infra05:17
*** vogxn has joined #openstack-infra05:17
*** CaptTofu has joined #openstack-infra05:18
*** nicedice has quit IRC05:26
*** zehicle has quit IRC05:32
*** CaptTofu has quit IRC05:34
*** pblaho has joined #openstack-infra05:37
*** zehicle_at_dell has quit IRC05:44
*** zehicle has joined #openstack-infra05:45
*** vogxn has quit IRC05:51
*** boris-42 has quit IRC05:59
*** CaptTofu has joined #openstack-infra06:03
*** sld_ has quit IRC06:04
*** sld_ has joined #openstack-infra06:04
*** DinaBelova has joined #openstack-infra06:17
*** vogxn has joined #openstack-infra06:26
*** DinaBelova has quit IRC06:28
*** vogxn has quit IRC06:30
*** sdake_ has joined #openstack-infra06:44
*** sdake_ has quit IRC06:44
*** sdake_ has joined #openstack-infra06:44
*** DinaBelova has joined #openstack-infra06:51
*** senk has joined #openstack-infra06:54
*** mrmartin has joined #openstack-infra07:01
*** fbo_away is now known as fbo07:03
*** senk has quit IRC07:03
*** sdake_ has quit IRC07:04
*** odyssey4me has joined #openstack-infra07:05
*** Ryan_Lane has quit IRC07:10
*** SergeyLukjanov has quit IRC07:20
*** hashar has joined #openstack-infra07:31
*** openstack has joined #openstack-infra09:17
*** SergeyLukjanov has joined #openstack-infra09:20
*** DinaBelova has joined #openstack-infra09:23
*** sbadia has joined #openstack-infra09:26
*** kpavel has joined #openstack-infra09:26
*** nati_ueno has quit IRC09:27
*** michchap has quit IRC09:32
*** thomasbiege has joined #openstack-infra09:36
*** ryanpetrello has quit IRC09:43
*** thomasbiege has quit IRC09:44
*** dizquierdo has joined #openstack-infra10:05
*** pcm_ has joined #openstack-infra10:09
*** pcm_ has quit IRC10:10
*** pcm_ has joined #openstack-infra10:11
*** jcoufal has joined #openstack-infra10:15
*** branen__ has quit IRC10:16
*** nosnos has quit IRC10:18
*** ryanpetrello has joined #openstack-infra10:37
*** oubiwann has quit IRC10:44
*** oubiwann has joined #openstack-infra10:54
*** michchap has joined #openstack-infra10:59
*** Protux has quit IRC11:17
*** adalbas has joined #openstack-infra11:19
*** Protux has joined #openstack-infra11:20
chmouelrahmu: can you going to http://quickscreenshare.com/share?t=TqkGwPGYwGTTPkwYP&n=Chmouel&s=011:22
chmouelthis is for a test of screen sharing11:22
rahmuI'm missing the java plug in for my browser11:23
*** Protux has quit IRC11:23
chmouelwhy you guys don't have the java plugin11:24
rahmubecause fuck java11:26
rahmuI installed, I'm connecting to your link11:26
rahmuI installed *it*, ...11:26
rahmuTimed out trying to connect!11:26
rahmuchmouel: ^11:26
chmouelrahmu: ah yeah i stopped11:27
chmouellet me try again11:27
chmouelrahmu: http://quickscreenshare.com/share?t=TqkGwPGYwGTT3GwkU&n=Chmouel&s=011:27
rahmuSharing Ready!11:28
rahmuWaiting for Chmouel to start sharing.11:28
chmoueloops11:28
chmouelcan you see me?11:28
chmouelnot sure if i closed11:28
chmouellet me try again http://quickscreenshare.com/share?t=TqkGwPGYwGTTkwkTU&n=Chmouel&s=011:28
chmouelrahmu: ^11:28
rahmusharing ready11:29
chmoueldoes that work?11:30
rahmuyes I see you11:31
chmouelrahmu: ^11:31
chmouelcan you see my screen properly?11:31
rahmuI'm seeing part of the screen, not all of it11:31
chmouelyeah11:31
chmouelwell it kinda work11:31
chmouelit's a bit weird to use11:31
*** BobBallAway is now known as BobBall11:32
chmoueloops sorry guys chatted to rahmu  in the wrong window ;)11:33
*** cthulhup_ has joined #openstack-infra11:33
*** cthulhup_ has quit IRC11:33
*** cthulhup has joined #openstack-infra11:34
*** flaper87|afk is now known as flaper8711:44
*** ArxCruz has joined #openstack-infra11:48
*** dcramer__ has quit IRC11:52
mordredchmouel: dammit man! now there's all this talking in here! ;)11:55
chmouelmordred: hah11:55
chmoueleurope office hours can get lonely here11:55
mordredchmouel: sleep later!11:56
mordredchmouel: but I know what you mean, when I travel to europe, it's very weird11:56
chmouelnot a bad idea at all, i can just as well get drunk by 9AM11:56
flaper87Did someone mention getting drunk?11:57
flaper87my irc client highlighted that11:57
flaper87:D11:57
mordredflaper87: you have configured a very sensible irc highlight that I should copy11:58
chmouelwell btw: if you guys knows any screen sharing app that works well with linux and others and can do >10 (i.e: no skype/hangout) that would be nice12:00
*** sandywalsh has joined #openstack-infra12:01
*** adalbas has quit IRC12:02
*** cppcabrera has joined #openstack-infra12:05
*** adalbas has joined #openstack-infra12:10
*** mriedem has joined #openstack-infra12:14
*** thomasbiege1 has joined #openstack-infra12:18
*** adalbas has quit IRC12:19
*** mestery has joined #openstack-infra12:19
*** mestery has quit IRC12:19
*** mestery has joined #openstack-infra12:20
*** dkliban has joined #openstack-infra12:25
*** rfolco has joined #openstack-infra12:26
*** thomasbiege1 has quit IRC12:30
*** michchap has quit IRC12:30
*** jhesketh__ has quit IRC12:30
*** michchap has joined #openstack-infra12:30
*** flaper87 is now known as flaper87|afk12:37
*** michchap has quit IRC12:38
*** mestery has quit IRC12:40
*** mestery has joined #openstack-infra12:40
*** mestery has quit IRC12:40
*** mestery has joined #openstack-infra12:41
*** motoki has quit IRC12:47
*** weshay has joined #openstack-infra12:47
*** thomasm has joined #openstack-infra12:48
*** mestery has quit IRC12:56
*** dkliban has quit IRC13:03
*** matty_dubs|gone is now known as matty_dubs13:03
*** adalbas has joined #openstack-infra13:03
*** DinaBelova has quit IRC13:08
*** prad has joined #openstack-infra13:08
*** prad has quit IRC13:08
openstackgerritArx Cruz proposed a change to openstack-infra/config: Adds support to specify a different jenkis_jobs_builder username  https://review.openstack.org/4715613:11
*** julim has joined #openstack-infra13:11
ArxCruzALL: is gate-ci-docs broken ?13:22
*** ArxCruz has left #openstack-infra13:22
*** ArxCruz has joined #openstack-infra13:22
*** dcramer__ has joined #openstack-infra13:22
*** ArxCruz has quit IRC13:23
*** DennyZhang has joined #openstack-infra13:23
*** ArxCruz has joined #openstack-infra13:23
ArxCruzALL: is gate-ci-docs job broken? https://review.openstack.org/#/c/47156/  is saying that fails on this job… wondering why...13:24
*** jpeeler has quit IRC13:25
*** jpeeler has joined #openstack-infra13:26
*** jpeeler has quit IRC13:26
*** jpeeler has joined #openstack-infra13:26
*** zehicle_at_dell has joined #openstack-infra13:28
*** dcramer__ has quit IRC13:29
*** mrodden has quit IRC13:31
* ttx yawns13:32
*** mestery has joined #openstack-infra13:34
*** DinaBelova has joined #openstack-infra13:36
*** mrodden has joined #openstack-infra13:44
*** dkliban has joined #openstack-infra13:45
*** tstevenson has joined #openstack-infra13:46
*** DinaBelova has quit IRC13:47
*** cthulhup has quit IRC13:49
*** cthulhup has joined #openstack-infra13:51
*** mrmartin has quit IRC13:54
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add pep8 checks to tripleo-image-elements  https://review.openstack.org/4716413:54
*** afazekas has joined #openstack-infra13:56
*** dcramer__ has joined #openstack-infra13:56
*** mrmartin has joined #openstack-infra13:58
*** dcramer__ has quit IRC14:01
*** amotoki has quit IRC14:04
*** EmilienM has quit IRC14:05
*** dims has joined #openstack-infra14:05
*** EmilienM has joined #openstack-infra14:07
*** zehicle_at_dell has quit IRC14:11
*** julim has quit IRC14:12
*** markmcclain has joined #openstack-infra14:14
*** DinaBelova has joined #openstack-infra14:19
mordredttx: any chance you remember what the issues are six versions were?14:20
mordredzul: ^^ ?14:20
*** prad has joined #openstack-infra14:21
mordredwe've got some things that want new six and some that want old six and I know that it was problematic for somebody14:21
zulmordred:  not sure14:21
ttxmordred: I may not have been your thinking sparring partner on that one14:21
mordredttx: dammit14:22
jd__mordred: wsme14:22
mordredjd__: ah - so wsme is the thing that wants <1.4?14:22
jd__mordred: wanted, until 0.5b414:23
*** flaper87|afk is now known as flaper8714:23
jd__the patch that unpin six is merging right now I think14:23
mordredjd__: oh good14:23
jd__mordred: https://review.openstack.org/#/c/46081/ 4 minutes left says Jenkins14:23
mordredjd__: woohoo! that makes another thing I'm working on better!14:23
jd__cool :-)14:24
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/config: Add Python 3.3 gate for Oslo-Incubator  https://review.openstack.org/4717014:25
Mithrandirhmm, any chance somebody could approve https://review.openstack.org/#/c/44525/ ?14:26
AJaegerand could that famous somebody also take care of https://review.openstack.org/#/c/45574/ , please?14:27
Mithrandirget in line. :-P14:28
openstackgerritMonty Taylor proposed a change to openstack-infra/reviewstats: Add openstack/requirements to the oslo program  https://review.openstack.org/4717314:29
AJaegerMithrandir, sure, that was implied by the "also" ;)14:30
*** datsun180b has joined #openstack-infra14:30
*** CaptTofu has quit IRC14:31
ArxCruzALL: is gate-ci-docs job broken? https://review.openstack.org/#/c/47156/  is saying that fails on this job… wondering why...14:31
kiallHah - I got that exact error from an unrelated project's `build_sphinx` command.. I suspected something, somewhere, got upgraded.. but I've not looked14:33
mordredsphinx did an upgrade14:34
mordredupstream14:34
mordredand it's broken14:34
ArxCruzoh, okay14:35
kiallmordred: I can see the next evolution of the openstack/requirements mirror stuff's getting more and more complex :)14:35
ArxCruzmordred: is this means that my patch will not pass ?14:35
MithrandirAJaeger :-)14:35
*** kpavel has quit IRC14:35
mordredArxCruz: I believe we're trying to land some patches to pin sphinx14:35
mordredbut fungi was wokring on that and I don't know the status14:35
ArxCruzok, cool14:36
fungimordred: https://review.openstack.org/46984 fixes it for us14:36
yportnovamordred: please look at new project adding https://review.openstack.org/#/c/46919/14:36
ArxCruzit's my first patch to the project so i'm anxious :)14:36
fungimordred: the issue we saw had to do with a broken *beta* release of sphinx on pypi14:36
fungimordred: so using new enough pip avoids pulling in beta releases14:36
openstackgerritafazekas proposed a change to openstack-infra/devstack-gate: Let devstack to decide about swift storage  https://review.openstack.org/4666314:37
mordredfungi: http://logs.openstack.org/56/47156/1/check/gate-ci-docs/cd779b8/console.html .. 2013-09-18 13:16:33.004 | Running Sphinx v1.2b214:37
fungimordred: yep14:38
kiallfungi: I'm speechless.. Agaian. pip/PyPy/Setuptools has a tendency to do that ;)14:38
dstufft;[14:38
fungimordred: with the change in 46984 we end up running sphinx 1.1.3 instead of 1.2b214:39
mordredOH - we need to land that change14:39
fungiwithout needing a pin to keep out 1.2 (which presumably won't be broken like 1.2b2, but you never know)14:39
*** zehicle_at_dell has joined #openstack-infra14:39
fungimordred: yes14:40
*** thomasbiege1 has joined #openstack-infra14:40
mordredfungi: technically, that tox.ini needs minversion = 1.614:40
mordred in the [tox] section - because install_command is new in 1.614:40
mordredfungi: maybe I should code review that14:40
fungiplease do, and i'm happy to add that minversion or whatever else you think it needs14:41
mordredothe rthan that - I think we should land that change so that config repo becomes unbroken14:41
fungii basically just played around with merging options from nova's tox.ini (since their sphinx jobs were getting the release version properly) until i found which line was making that happe14:42
fungin14:42
fungii would not be at all surprised to discover that it only works because of some side effect and is not the *correct* fix, but it was a conversation starter at least14:42
*** dcramer__ has joined #openstack-infra14:43
BobBallmordred: https://review.openstack.org/#/c/47114/ - fix for run_tests.sh :)14:44
dstufftfungi: mordred it fixes it becasue tox adds --pre by default to the install command14:45
dstufftfungi: and by providing your own install command you won't be using --pre14:45
mordreddstufft: AAAAAAAAAH14:45
jd__we need an upstream police with an intervention squad14:45
mordredthank you. that's very helpful14:45
*** pblaho has quit IRC14:45
mordredand explains why that works (the -U didn't seem right to me)14:45
*** mestery has quit IRC14:45
dstufft-U will just make it so you use the latest packages if you're running them in an already created venv.. which tox doesn't IIRC but I don't recalll for sure14:46
*** pblaho has joined #openstack-infra14:47
*** julim has joined #openstack-infra14:47
*** ruhe has joined #openstack-infra14:47
*** mrmartin has quit IRC14:48
fungiyolanda: if you're still struggling with the push to refs/meta/config for the "all-projects" project, i think you should be able to create an acl rule to allow it, but yes that's a bit of a chicken and egg problem if what you're trying to do is update the acl through that mechanism. you can circumvent by pushing to the underlying filesystem, but then you miss out on gerrit's syntax checks of that14:48
fungifile14:48
*** ruhe has quit IRC14:48
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Make sure pip upgrades itself in tox  https://review.openstack.org/4698414:50
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Add openstack/requirements to the oslo program  https://review.openstack.org/4717314:50
yolandafungi, yes, i find that i need to automate ACL but i cannot because of ACL permissions...14:50
openstackgerritA change was merged to openstack-infra/reviewstats: Add openstack/requirements to the oslo program  https://review.openstack.org/4717314:50
*** thomasbiege1 has quit IRC14:53
yolandaso there isn't any workaround of automating that, it's a manual process using UI at first time?14:53
mordredyolanda: so far - we have not figured out a more automated initial step yet14:54
yolandamordred, ok, i'll leave that as a manual step then14:55
*** fbo is now known as fbo_away14:56
*** kpavel has joined #openstack-infra14:56
*** pcrews has joined #openstack-infra14:59
*** datsun180b_ has joined #openstack-infra15:00
*** datsun180b has quit IRC15:02
*** datsun180b_ is now known as datsun180b15:02
fungiyolanda: well, technically we the last few gerrits we built, we simply copied that acl file in from another gerrit which was already set correctly. you could probably get a very simple template acl file which gerrit thinks is suitable and which gets you the access you need for further acl updates, and then embed that in your initial configuration and drop it in place when setting up a new gerrit15:02
*** zehicle_at_dell has quit IRC15:04
*** zehicle_at_dell has joined #openstack-infra15:04
yolandafungi, i didn't know it was possible15:06
fungiyolanda: sure. gerrit stores its git repositories on the filesystem. you can manipulate them directly with git rather than going through gerrit's jgit service, and if it breaks you also get to keep the pieces15:08
fungiand all-projects is "just another git repository" (at least on the 2.4 release we're still running, no idea if that changes in newer releases)15:09
yolandafungi, so the process is just similar but with local git inside gerrit machine, instead of using gerrit interface, right'15:10
yolanda?15:10
fungiyolanda: yep15:10
yolandanice, i'll do a try15:10
yolandathx15:10
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add requirements and publish jobs to marconi  https://review.openstack.org/4718315:10
*** flaper87 is now known as flaper87|afk15:12
*** branen has joined #openstack-infra15:12
NobodyCamgood morning infra15:12
*** nati_ueno has joined #openstack-infra15:12
NobodyCampaste.openstack.org down??15:12
fungiNobodyCam: i hope not!15:13
mordredNobodyCam: wow. it sure does seem to be not running15:13
fungii was about to use it15:13
fungii'll find out why15:13
* fungi sighs15:13
NobodyCam:) Ty fungi ... and good morning !!!15:13
fungiNobodyCam: good morning to you too15:14
NobodyCam:)15:14
*** w_ is now known as olaph15:15
NobodyCamfungi: are you going to HK?15:15
fungiNobodyCam: oh yeah, getting there on saturday so i can help set up15:16
NobodyCam:) awesome :)15:16
*** senk has joined #openstack-infra15:18
openstackgerritEric Harney proposed a change to openstack/requirements: Require python-novaclient>=2.15.0  https://review.openstack.org/4718515:21
*** julim has quit IRC15:23
*** thomasbiege1 has joined #openstack-infra15:24
*** thomasbiege1 has quit IRC15:25
*** markmcclain has quit IRC15:25
*** julim has joined #openstack-infra15:26
*** DennyZhang has left #openstack-infra15:26
fungiokay, found what killed paste.o.o...15:27
fungi[Wed Sep 18 13:38:36 2013] Out of memory: Kill process 5918 (python) score 527 or sacrifice child15:27
*** burt has quit IRC15:27
*** DennyZhang has joined #openstack-infra15:27
fungihttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=148&rra_id=all15:27
*** datsun180b has quit IRC15:28
*** datsun180b has joined #openstack-infra15:28
fungii think we should probably grow it to a 2gib flavor15:28
*** bnemec_ has quit IRC15:28
*** afazekas has quit IRC15:29
fungiactually no. it didn't get above 40% actually in use (aside from buffers/cache)15:29
NobodyCambut a good chunck is of the ram is cache15:29
NobodyCam:-p15:29
NobodyCamya15:30
NobodyCamanything odd in the log around 6:2015:31
NobodyCamsomehting happened right around then15:31
fungii wonder if this might have happened due to memory pressure on the compute node? http://pastebin.com/i6bjkkP115:35
dimsAnyone seen this yet? Sphinx v1.2b2 seems to be bad (http://logs.openstack.org/70/47170/1/check/gate-ci-docs/3fb4fbc/console.html) - switching to v1.1.3 seems to fix the problem15:35
fungidims: yep, https://review.openstack.org/4698415:36
*** marun has quit IRC15:37
fungininja'd it in for now, since i think i addressed mordred's concern over the initial patchset15:37
openstackgerritA change was merged to openstack-infra/config: Make sure pip upgrades itself in tox  https://review.openstack.org/4698415:38
annegentlehey does anyone here know for sure if translators are given ATC status and a registration code for summits?15:40
annegentlettx: or mordred, do you know if translators get ATC status?15:40
ttxannegentle: no, they (currently) don't15:41
ttx(not a program)15:41
annegentlettx: ok, I thought not, thanks for confirming15:41
annegentlettx: ah right.15:41
ttxannegentle: if it was a program, i would at least consider giving top language coordinators the status15:42
fungihowever, if they submit one small whitespace fix to any project, boom atc15:42
annegentlettx: sure15:42
ttxannegentle: but like fungo said, maybe not to every string contributor15:42
ttxfungi*15:42
Shrewsfungi, jeblair: seems that publishing a draft doesn't trigger zuul. This is a known issue?15:43
fungiShrews: it's a known behavior15:43
Shrewsok, just checking15:43
fungiShrews: gerrit 2.4 does not emit an event on draft publication15:43
fungii think that's addressed in 2.5 or maybe 2.615:43
fungibut zuul would also need to know to look for it15:43
*** thomasbiege1 has joined #openstack-infra15:45
fungiokay, done prodding at things on paste.o.o which i was afraid to lose over a reboot, so rebooting it now to make sure any lingering side effects from the earlier oom situation are cleared15:46
fungishould be working again now15:46
NobodyCam:)15:46
*** thomasbiege1 has quit IRC15:46
NobodyCamw00t you rock fungi !15:47
NobodyCamTY15:47
fungiapologies for the delay15:47
fungiyep, seems working... http://paste.openstack.org/show/47216/ (transfer of the oom in its dmesg)15:47
*** dcramer__ has quit IRC15:49
*** markmcclain has joined #openstack-infra15:50
*** nati_ueno has quit IRC15:51
*** danger_fo_away is now known as danger_fo15:54
jd__fungi: should I do something particular about http://logs.openstack.org/85/47185/1/check/gate-requirements-install/fcf0e77/console.html ?15:55
*** hemnafk is now known as hemna15:56
*** CaptTofu has joined #openstack-infra15:57
openstackgerritRyan Petrello proposed a change to openstack-infra/config: Provide a more generic run-tox.sh.  https://review.openstack.org/4314515:57
*** markmcclain has quit IRC15:58
fungijd__: recheck no bug. it's an unfortunate situation that to test installability of new requirements, we obviously can't use the mirror built from our current list of requirements. thus failures on pypi.python.org can cause job failures (but that only runs on requirement changes, so shouldn't be too frequent a nuisance)15:59
ryanpetrellofungi: thanks for https://review.openstack.org/#/c/46984/, seems to have resolved the sphinx issue I saw in gate-ci-dosc w/ https://review.openstack.org/#/c/43145/15:59
ryanpetrello*gate-ci-docs16:00
fungiryanpetrello: yep, i'm going back through and rechecking any other changes which hit that issue while i'm thinking about it16:00
*** thomasbiege1 has joined #openstack-infra16:01
*** thomasbiege1 has quit IRC16:02
*** svarnau has joined #openstack-infra16:02
*** matty_dubs is now known as matty_dubs|lunch16:03
dimsfungi, thx16:04
*** dprince has joined #openstack-infra16:05
fungidims: my pleasure. it was a fun issue to fix ;)16:05
kiallfungi: might be an issue with https://review.openstack.org/4698416:07
kiallSpecifcally, with the -U16:07
*** pcrews has quit IRC16:07
fungikiall: in what way?16:07
kiallhttps://review.openstack.org/#/c/47193/ is failing after that change, the difference is we're no longer getting WebTest==1.3.3 (like test-requirements.txt says), we're instead getting 2.0.something16:08
*** markmc has joined #openstack-infra16:08
fungihrm16:08
kiallI'm about to push a patchset without the -U to see what happens16:08
kiallThe "stock" tox default doens't include that16:08
fungikiall: agreed. however we also use -U on nova and lots of other projects, so i would expect it to have exhibited problems before now16:09
clarkbwow scrollback16:10
kiallhttp://paste.openstack.org/show/47220/ <-- Stock tox logic, just for reference16:10
fungii was mostly trying to keep consistent with how we'd done it elsewhere16:10
kiallAnd .. PS#4 without the -U passed. I've no idea why the -U triggers an upgrade beyond "WebTest==1.3.3".. But, it does..16:11
openstackgerritSean Dague proposed a change to openstack-infra/reviewstats: add new tempest core contributors  https://review.openstack.org/4719916:12
fungikiall: you may need some additional logic from nova's tox.ini to handle your requirements files16:12
fungikiall: they have some lines in there to specify how they get installed in tox-land16:13
clarkbkiall: fungi: is there a tl;dr on the issue?16:13
*** thomasbiege1 has joined #openstack-infra16:13
*** julim has quit IRC16:13
*** marun has joined #openstack-infra16:13
fungiclarkb: for docs builds of our config project, tox was happily using an older pip which pulled in a new and very broken beta release of sphinx from pypi16:14
clarkbnova does special things because libvirt cannot be installed from pypi so it relies on site packages being enabled, When you do that you have to be careful that you don't get additional things from site packages16:14
kiallfungi: I was getting the same sphinx issue, hence trying to port your fix and noticed this :)16:14
fungikiall: yep, understood16:14
clarkboh interesting, and old pip does that because it doesn't look at versions as being semantic16:15
fungiclarkb: anyway, according to dstufft this is because tox passes --pre by default, and overriding it with a pip install command which does not use --pre allowed tox to upgrade pip in the virtualenv before installing deps16:15
clarkb:/ seems like tox should make that configurable16:16
kiallfungi: looks like all the requirement file related stuff is already there.. I reckon 1.3.3 gets installed, and then something else upgrades it..16:17
kiallBut - I've not looked hard enough to say that with any real confidence :)16:17
fungii cargo-culted the install_command option from nova's tox.ini which solved it for us, but designate (which is seeing the same issue in its tests) has discovered that having tox use pip -U is upgrading at least some of their requirements beyond revs they capped16:17
*** odyssey4me has quit IRC16:18
fungikiall: i was mostly curious whether https://git.openstack.org/cgit/openstack/nova/tree/tox.ini#n14 helped16:19
*** zehicle_at_dell has quit IRC16:19
kiallfungi: already have things that way: https://git.openstack.org/cgit/stackforge/designate/tree/tox.ini#n1016:20
fungikiall: for some reason i don't see it reflected in https://review.openstack.org/#/c/46984/2/tox.ini16:21
fungioh, nevermind16:21
kiallthat's your change :D16:21
fungii was looking at my patch, not yours16:21
* fungi has waaay too many tabs open16:21
kiallI'm going to take a wild guess that tox is calling pip once per "deps" line .. I'm also gonna guess the order is reversed for some reason, meaing a pip install -U test-requirements.txt && pip install -U requirements.txt16:22
*** DennyZhang has quit IRC16:22
fungipossibly. i also wonder why this isn't biting nova16:22
kiallWith the cap being in test-requirements, and Flask or Pecan depending on WebTest too, then, boom, it upgrades16:23
fungiyeah, it's possible they have a latent problem there which they're just not running into because the order happens to be okay for them at the moment16:23
kiallyea.. Uploading another patchset with -U, and the order reversed.. That's proof enough for me to move on ;)16:24
kiallDamn.. With the order reversed, I still get WebTest==2.0.916:26
kiall(rather than the expected 1.3.3)16:26
*** svarnau_ has joined #openstack-infra16:27
*** svarnau has quit IRC16:28
fungii'm surveying a few other projects to see if they dep on sphinx in their tox.ini and whether they override install_command16:28
BobBalldtroyer: regarding local.conf - should we fail to stack if you have both [[local:localrc]] and localrc file, telling people to remove localrc or add an IGNORE_LOCALCONF setting in localrc?16:30
fungikiall: i think most other projects are getting away with it by putting sphinx in test-requirements.txt instead of a deps line in tox.ini16:30
SpamapShttps://git.openstack.org/cgit/openstack/tripleo-incubator/plain/devtest.m <--- shouldn't this be a 40 ?16:31
fungikiall: at least that's the only theory i have so far16:31
*** pblaho has quit IRC16:31
SpamapSmy browser shows it as a 20016:31
kiallWell, with or without the -U, the install_command gets the correct non-prerelease version of Sphinx, it's another random dep that's changing!16:31
SpamapSblank page 200.. very confusing :p16:31
fungiSpamapS: yeah, no idea whether that's a bug in cgit or something we've done16:33
fungiSpamapS: easy solution--don't make typographical or cut-and-paste errors16:33
openstackgerritA change was merged to openstack-infra/reviewstats: add new tempest core contributors  https://review.openstack.org/4719916:33
dtroyerBobBall: I'm worried about breaking scripted stuff, that's why just use localrc if it exists.  it takes specific action (rm localrc) on the user's part to change16:34
fungiso, after hunting in the apache logs, i've determined how to make lodgeit consume all your base^Wmemory16:35
fungireloaded one particular paste three times and now its memory consumption looks like... http://paste.openstack.org/show/47223/16:35
*** DinaBelova has quit IRC16:35
fungione more reload of the offending page will easily tip it over into oomsville16:36
fungigonna restart lodgeit on there now before the kernel tells it to bug out16:36
BobBalldtroyer: fair enough - I'd go with your latest suggestion then!16:37
*** jpich has quit IRC16:38
*** SergeyLukjanov has quit IRC16:39
*** thomasbiege1 has quit IRC16:40
fungifun, looks like jjb is also trying to build its docs with the sphinx prerelease16:41
*** julim has joined #openstack-infra16:45
*** markmcclain has joined #openstack-infra16:45
*** changbl has quit IRC16:48
*** zehicle_at_dell has joined #openstack-infra16:51
*** thomasbiege1 has joined #openstack-infra16:52
fungiokay, so the broken paste is a ~61mib entry consisting of >800k lines of ubuntu package build log data16:54
fungii am going to delete that from the table in the db and see if there are some limits we maybe need to set in lodgeit's config for our own sanity16:54
*** matty_dubs|lunch is now known as matty_dubs16:55
*** Ryan_Lane1 has joined #openstack-infra16:56
*** svarnau has joined #openstack-infra16:56
*** svarnau_ has quit IRC16:56
*** BobBall is now known as BobBallAway16:57
*** uvirtbot has quit IRC17:00
*** pcrews has joined #openstack-infra17:04
*** uvirtbot has joined #openstack-infra17:04
*** tstevenson_ has joined #openstack-infra17:07
*** tstevenson has quit IRC17:10
*** kpavel has quit IRC17:12
*** cppcabrera is now known as cppcabrera_afk17:13
*** julim has quit IRC17:15
*** julim has joined #openstack-infra17:17
*** fbo_away is now known as fbo17:18
*** yidclare has joined #openstack-infra17:23
*** dizquierdo has left #openstack-infra17:23
*** zehicle_at_dell has quit IRC17:23
*** jerryz has joined #openstack-infra17:24
*** zehicle_at_dell has joined #openstack-infra17:24
*** svarnau_ has joined #openstack-infra17:29
*** nati_ueno has joined #openstack-infra17:29
*** markmcclain has quit IRC17:32
*** svarnau has quit IRC17:32
*** danger_fo is now known as danger_fo_away17:34
ArxCruzfungi: any chance you approve https://review.openstack.org/#/c/47156/ ? ;D17:36
*** ryanpetrello has quit IRC17:36
*** ryanpetrello has joined #openstack-infra17:37
pleia2fungi: I think it's something with our configuration (redirects?), my cgit server 404s properly, ie: http://15.185.127.146/cgit/openstack-infra/config/plain/run_puppet.s17:39
fungiArxCruz: it looks safe, but probably needs some debate. the stuff inside the openstack_project module is fairly opinionated about variable definitions and is meant more as an example that something to be directly reused, so if there are features of that which haven't been generalized and would benefit from it, moving them out of the openstack_project module to a lower layer might make more sense17:40
fungipleia2: wouldn't surprise me. i'll have a look in our rewrite log17:41
pleia2fungi: thanks17:41
ArxCruzfungi: :(17:44
fungiArxCruz: +2'd but needs input from others i think17:45
*** EmilienM has quit IRC17:46
ArxCruzfungi: thanks, we here are trying to create an environment as similar as possible to openstack-infra, that's why we are using those modules17:47
ArxCruzwe are trying to avoid workarounds here, and I believe this will help others who's trying to do the same17:48
fungiArxCruz: well, "as similar as possible" means you get our long-standing typographical errors too ;)17:48
ArxCruzfungi: hehehe i keep the comment :)17:48
fungibut yes, i see your point, and i agree that the change is probably warranted17:49
ArxCruzwas too funny in the boring world of puppet17:49
fungiif i didn't think there was a slim chance of disagreement over it, i'd have approved because it is a very trivial change17:49
*** EmilienM has joined #openstack-infra17:49
ArxCruzand help me a lot here :) one less thing in my puppet list of errors :)17:51
fungimostly worried that it could be a slippery slope to hoisting all our openstack-infra-specific implementation details to where they can be overridden in the global site manifest17:51
fungiwhen really the goal is to make anything which should be reused into stand-alone modules, and provide the openstack_project module merely as an example of how to combine them17:52
ArxCruzfungi: but it's not :) the global site manifest remains intact, and continues to work in the same way, I've been testing in my internal environment for days :)17:53
fungirather than having the openstack_project module itself be a reusable portal to those modules and then make it so the implementation details can all overridden externally, which adds complexity if it becomes more prevalent17:53
ArxCruzfungi: agree17:54
*** cody-somerville has quit IRC17:56
fungioh, i know your patch still defaults to what we're doing and won't break us, but if we do that for every possible bit of configuration data we've currently hard-coded into the openstack_project module it does get more complex in ways that openstack itself doesn't exercise, and wouldn't notice if it broke in those ways. and also it would just generally make the manifests in openstack_project longer17:56
fungiand expose more surface where accidental errors could get introduced17:56
fungiworst case scenario obviously, but it's something i think we need to be mindful of17:57
*** Ryan_Lane1 has quit IRC17:58
*** Ryan_Lane has joined #openstack-infra17:58
*** cppcabrera_afk is now known as cppcabrera18:00
ArxCruzfungi: I got it, actually, I'm having several problems with hard-coded configurations :) so, from one side, keeps openstack-infra project easy, for the other side, make my work harder :)18:00
fungii totally understand18:01
*** sarob has joined #openstack-infra18:13
*** cody-somerville has joined #openstack-infra18:15
*** zehicle_at_dell has quit IRC18:17
*** zehicle_at_dell has joined #openstack-infra18:17
fungipleia2: i think it might be haproxy...18:18
fungi192.237.223.224 - - [18/Sep/2013:16:28:57 +0000] "GET /cgit/openstack/tripleo-incubator/plain/devtest.m HTTP/1.1" 404 - "https://review.openstack.org/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0"18:18
pleia2fungi: ah, interesting18:19
fungithat was from the access log on git0118:19
clarkbdangit haproxy18:19
*** cppcabrera has quit IRC18:19
clarkbwell haproxy is just doing tcp forwarding18:20
clarkbit isn't doing anything with http18:20
fungiwell, i don't know for sure that it's haproxy, but apache's responding with a 404 so something it changing that on the way back18:20
*** mrmartin has joined #openstack-infra18:20
fungijust need to determine what18:20
mrmartinre18:20
*** cthulhup has quit IRC18:21
fungimrmartin: i brought up the groups-dev server last night before i fell asleep, but there are errors around the drush calls. i meant to iron them out and get a fix up for review, but have been a little busy today18:21
fungihope to get to it in a bit18:22
mrmartinfungi: yeah, I saw that, something went wrong18:22
fungiif you know specifically what's wrong, i'll leave it i your capable hands18:22
mrmartincan I get a temporary root access for that instance?18:22
*** changbl has joined #openstack-infra18:22
mrmartinI just saw, that final setup was not gone through18:23
mrmartincould you send me this hiera variable to me: groups_dev_site_admin_password ?18:23
fungimrmartin: sure, just a sec18:23
mrmartinfungi: the major steps are ok, because we have a running drupal anyway18:24
mrmartinI want to move some parts to the gating processes18:24
mrmartinbut let's see something working first18:24
*** mitul has joined #openstack-infra18:24
*** melwitt has joined #openstack-infra18:27
mrmartinfungi: it seems to be that drush had some problems, can I see the logs of this server somewhere?18:27
fungimrmartin: according to puppet, 'drupal-site-deploy.sh -in /etc/drupal_site/groups-dev.openstack.org.config' returned a nonzero exit code... http://paste.openstack.org/show/47229/18:28
mrmartinthnx18:28
fungimrmartin: i can run that by hand and see what it reports on stdout/stderr if that will help18:28
mrmartinplease do it, it could help a lot18:29
*** EmilienM has quit IRC18:29
mrmartinI'm thinking on a method, how could we restart a redeploy process if something fails.18:29
fungimrmartin: http://paste.openstack.org/show/47231/18:31
mrmartineh18:31
fungimysql client missing?18:31
mrmartinit have problems with the mysql service18:31
*** EmilienM has joined #openstack-infra18:32
mrmartinthe drush install process tries to recreate the database somehow. I need to check it how this happens,18:32
*** cppcabrera has joined #openstack-infra18:32
fungiokay18:33
mrmartinmaybe it tries to drop the entire database or just drop existing tables.18:33
mrmartinso it could be a permission issue.18:33
*** thomasbiege1 has quit IRC18:33
fungii can check the mysql server logs for any audit info18:33
mrmartinok, give me a few minutes, I'll check the drush source code.18:33
fungioh, i guess this isn't actually running a mysql server anyway18:35
clarkb18:35
clarkbwhoops18:35
fungipresumably some other sort of database backend (sqlite?)18:36
mrmartinfungi: as I know it is a first project to use the mysql service18:36
mrmartinfungi: hiera('groups_dev_site_mysql_host') is the exact host18:36
fungiahh, right!18:36
fungiokay, i totally spaced on that. sorry. yes this is using the trove database as a service at rackspace18:37
*** rahmu has left #openstack-infra18:38
fungiokay, so from the error output on that deploy script, it looks like drush is trying to use the command-line mysql client and doesn't find it ("sh: 1: mysql: not found")18:39
fungidoes it have some sort of fallback to an internal mysql client library or something instead?18:39
*** hashar has joined #openstack-infra18:40
mordredTOO MUCH SCROLLBACK18:40
fungimordred: ENODRUNK18:40
mrmartinfungi: I see the same18:40
clarkbmordred: that is how I feel18:41
clarkbmordred: are you in town yet?18:41
mrmartincould you check whether you see an install mysql-client package?18:41
mrmartinas I know php5-mysql not depends on mysql-client18:41
*** mitul has left #openstack-infra18:42
fungimrmartin: i had already checked earlier, which is why i was asking. it's not installed. on a whim i've installed it now to see if that helps, and if so i'll add a review to install it on the groups servers from now on18:42
mrmartinfungi: I guess drush tries to create a groups_dev database, because it is not exists.18:42
*** thomasbiege1 has joined #openstack-infra18:43
mrmartinthe difference is, that I always had a pre-created database on the mysql server. It seems to be it is missing, and drush tried to create a new one.18:43
mordredclarkb: getting closer18:43
fungimakes sense18:43
fungimordred: flap your arms harder--you'll get there sooner18:44
mordredfungi: but they're so tired already!18:44
mrmartinfungi: I never tried that before, and I don't have too much experience with Trove, so anything could happen.18:44
fungimrmartin: yeah, i'm waiting for it to give me some feedback--should know in a moment18:45
mrmartinthe life could be boring without strange bugs :)18:45
*** thomasbiege1 has quit IRC18:46
fungiyes, or maybe i'd have enough time to take up flower arranging18:46
mordredjog0: we really need to get hacking.rst publishing18:49
jog0++18:50
*** mestery has joined #openstack-infra18:50
fungimrmartin: it's been running for a while... according to the process list it's on '/usr/bin/php /usr/share/php/drush/drush.php --php=/usr/bin/php si -y groups --db-url=mysql://...'18:50
jog0mordred: look in openstack-qa18:51
fungimrmartin: been going for about 8 minutes so far18:51
fungidefinitely farther than it got when there was no mysql client installed18:51
mrmartinfungi: yeap, another issue I found with mysql settings, it import translation files and it means a hell lot of insert and select18:51
fungii assume that's just going to happen once though (per site anyway)18:52
*** senk has quit IRC18:52
mrmartinfungi: mysql innodb requires this setting: innodb_flush_log_at_trx_commit=218:52
*** markmcclain has joined #openstack-infra18:52
fungimrmartin: on the server or on the client?18:52
fungithat seems like a server option to me18:52
mrmartinon the server, if we can do it on trove18:52
mrmartinbecause the default settings is 1, that means a commit after every insert, which is safe18:53
fungihub_cap: ^ is that something we have control over from the trove client end?18:53
mrmartinbut hell slow in this case18:53
mrmartinit is a weak point of drupal, we can do several workarounds, like importing only one translation for dev site, or hack it with some direct import instead drupal inserts.18:54
*** Protux has joined #openstack-infra18:54
openstackgerritMonty Taylor proposed a change to openstack-dev/hacking: Move the hacking guidelines to sphinx docs  https://review.openstack.org/4722618:55
mrmartinon my current notebook, this process took 5 minutes, and without innodb setting it took more then 30 minutes18:55
*** annegentle has quit IRC18:56
mordredjog0: ^^18:56
Mithrandircan anybody explain what's going on in http://logs.openstack.org/25/44525/5/gate/gate-jenkins-job-builder-docs/91d10f9/console.html ?18:57
jog0mordred: nice, will we have a job to build  the docs per patch18:58
mordredjog0: already there18:58
jog0one step ahead, awesome18:58
*** davidlenwell_ is now known as davidlenwell18:58
Mithrandirfungi: thanks for the approval!18:58
fungiMithrandir: yeah, but we have a sphinx problem in jjb, so i need to toss up a patch for that18:59
mordredjog0: we apparently haven't landed any hacking patches since we added them18:59
fungiMithrandir: once it's worked around, we'll be able to get yours merged18:59
jog0heh18:59
Mithrandirfungi: ah, ok, so that's the problem, nothing wrong with my fix, right?18:59
mordredjog0: hrm18:59
mordredactually18:59
fungiMithrandir: nope, just old pip likes to download prerelease versions of things from pypi, and sphinx has a broken 1.2b2 release up at the moment19:00
jog0I think its what fungi just said19:00
Mithrandirfungi: ugh.19:01
fungimrmartin: i had to start it over. i let puppet run it originally, but the timeout just killed it so i've restarted it directly from the command line this time19:01
mrmartinfungi: ok, I'm sure it is a db backend issue, because it was finished with the provided timeout settings19:01
mrmartinif i add some extra logging to drush, could we redirect it to a central log service?19:02
fungimrmartin: we don't really have a central log service at the moment. unless you count logstash but it's not operating as a traditional syslog sink19:03
clarkbfungi: and it probably never will be19:03
*** mestery has quit IRC19:03
clarkbfungi: the existing logstash/elasticsearch setup is 100% public and we don't want our server logs to be that public19:03
openstackgerritJerry Zhao proposed a change to openstack-infra/zuul: add forge author identity privilege to support pushing refs to gerrit  https://review.openstack.org/4723319:04
clarkbcould use a second instance that is more restrictive though19:04
mrmartinok, it could be great to see somehow the deploy logs19:04
fungiclarkb: yep, agreed19:04
mrmartinanyway, I can send-in a patch to groups and temporarily disable to translation file import.19:04
fungimrmartin: well, for this first try i can at least let the current init run to completion so we can see if there's any other issues lurking in its shadow19:05
*** Ryan_Lane has quit IRC19:06
*** Ryan_Lane has joined #openstack-infra19:06
mrmartinwe can import translations from admin ui, so it will speed up the deploy process, until we found a solution to mysql performance19:07
*** UtahDave has joined #openstack-infra19:07
fungiokay19:08
*** cppcabrera has quit IRC19:09
*** Yportnova_ has joined #openstack-infra19:10
mrmartinfungi: the changes are landed in git repo, may I ask you to retry the drush deployment?19:13
fungimrmartin: sure, just a moment and i'll restart it19:13
fungimrmartin: the previous run ended with "sitedeploy completed in 884 seconds."19:13
fungijust seconds ago19:14
fungiretrying it under puppet now19:14
mrmartingreat, the site is looking better19:14
fungifinished too quickly that time. do i need to clear the deploy flag file?19:14
fungiso that it will deploy again?19:15
mrmartinyes, please19:15
mrmartinif the git head commit is changing, it automatically removes the deploy flag file.19:15
mrmartinI want to move the first phase, the drush make into jenkins jobs, and move the entire translation importing process there.19:16
*** cppcabrera has joined #openstack-infra19:16
fungiokay19:19
fungisounds like a great idea19:19
*** zehicle_at_dell has quit IRC19:19
mrmartinand I want to post some deployment status logs somewhere, a not-so open logstash is a good idea, but I guess it takes more time to implement19:20
*** zehicle_at_dell has joined #openstack-infra19:20
*** sdake_ has joined #openstack-infra19:20
fungiwell, mainly just because we need to set up some authentication. we already have a great model for how to deploy public logstash at a variety of scales thanks to clarkb's tireless work19:21
fungiprobably wouldn't be a huge stretch from there to rolling out a separate instance for log aggregation, but we already have people who can log into systems and look in /var/log as well19:22
mrmartinhow serious could be getting an ssh access to this server?19:23
*** senk has joined #openstack-infra19:23
fungimrmartin: fairly easy. it's just a code review to add your username and ssh public key to the right puppet manifest and also reading and agreeing to our ssh access policy19:24
mrmartinfungi: I know the process, I want to ask whether the infra team supports getting an access to this server?19:24
fungimrmartin: i have no objection to you having ssh and sudo access on the groups-dev server, just need to confirm with mordred, jeblair and clarkb for completeness19:25
fungibut it makes sense for you to be able to troubleshoot things there more rapidly than having us relay error messages and performance data to you19:26
mrmartinas I remember, initially this patch included an ssh access for Stefano, but it was removed later19:26
*** Ryan_Lane has quit IRC19:26
mrmartinfungi: yeap, it is a strong argue, we need to find a good balance between responsiveness and security.19:27
*** mrodden1 has joined #openstack-infra19:27
*** dkliban has quit IRC19:27
*** mrodden has quit IRC19:28
*** senk has quit IRC19:28
fungimrmartin: so cleared everything from /tmp and reapplied via puppet, but this time get a message saying the drupal-site-deploy.sh step returned an exit code of 137. i'll try running it again by hand to see what's going on19:28
mrmartinok, thanks19:29
fungimrmartin: also, long-term, putting those fag files somewhere under /var/lib or in /srv probably makes more sense, so that they won't get blown away by tmpreaper19:29
*** sandywalsh has quit IRC19:30
*** mestery_ has joined #openstack-infra19:31
mrmartinIt is a good question, whether this redeploy process must be run automatically...19:31
*** vipul is now known as vipul-away19:32
*** vipul-away is now known as vipul19:32
fungier, flag files19:32
*** Yportnova_ has quit IRC19:34
*** mestery_ has quit IRC19:37
*** sdake_ has quit IRC19:41
*** jcoufal has quit IRC19:42
*** DennyZhang has joined #openstack-infra19:43
openstackgerritJeremy Stanley proposed a change to openstack-infra/jenkins-job-builder: Make sure tox does not call pip install with --pre  https://review.openstack.org/4723919:43
fungiMithrandir: ^ that's the fix for our sphinx build failures on jjb19:43
NobodyCamgate-requirements-install getting SSLError: The read operation timed out19:43
*** Ryan_Lane has joined #openstack-infra19:43
Mithrandirfungi: nod, I don't really know tox, so I won't +1 it, even though I assume you've tested it and it's good.19:44
NobodyCamthat a known issue? didn't see anything on recheck page19:44
*** Protux has quit IRC19:44
NobodyCambut  /me does need glasses19:44
fungiMithrandir: yep, no worries. i was mainly just pointing it out as a matter of completeness19:44
*** Protux has joined #openstack-infra19:45
*** zzs has joined #openstack-infra19:45
fungiNobodyCam: that's generally the slave getting an intermittent connection failure to pypi.python.org (or more likely to the fastly cdn front-ending that)19:45
fungiNobodyCam: our requirements jobs can't use our mirror which is built from the requirements list, since they may be testing additions to that list19:46
NobodyCam:) lol very true19:47
*** vipul is now known as vipul-away19:47
NobodyCamso i'll try another recheck in a bit19:47
Mithrandirfungi: appreciated. :-)19:47
NobodyCamty fungi :)19:47
fungiNobodyCam: the assertion is that those jobs are very infrequently run, and so people don't need to recheck to often because of them19:47
*** sandywalsh has joined #openstack-infra19:47
*** vipul-away is now known as vipul19:47
*** sdake_ has joined #openstack-infra19:48
fungibecause we theoretically don't make changes to our dependencies lists very often19:48
NobodyCamlol19:48
NobodyCamsure you dont19:48
fungihowever, you're, i think, the fourth person who has asked me that question in the past 24 hours19:48
NobodyCamprob on this patch https://review.openstack.org/46757 ???19:48
NobodyCamlol19:49
NobodyCamgah wrong review19:49
NobodyCamhttps://review.openstack.org/#/c/46081/19:50
NobodyCamthat one! :-p19:50
*** anteaya has joined #openstack-infra19:50
NobodyCammorning anteaya hope your feeling better19:50
anteayathanks NobodyCam, I can sit up now19:54
anteayasorry I didn't get to spend more time with you :(19:54
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Add mkiss user to groups-dev instance  https://review.openstack.org/4724119:54
*** mestery has joined #openstack-infra19:55
NobodyCamcompletely understandable19:55
anteayanext time19:55
NobodyCamgetting sick sucks19:55
anteayaat the summit then19:55
*** markmcclain has quit IRC19:55
NobodyCamhong kong19:55
anteayanot at all how I planned to spend my time in Seattle, no19:55
anteayayes, hk19:56
*** mestery has quit IRC19:56
NobodyCam:)19:56
*** adalbas has quit IRC19:57
*** mestery has joined #openstack-infra19:57
*** sandywalsh has quit IRC19:57
*** mestery has quit IRC19:58
*** markmcclain has joined #openstack-infra19:59
*** julim has quit IRC20:02
*** cppcabrera is now known as cppcabrera_afk20:03
fungimrmartin: the earlier puppet process i had started to rebuild/redeploy ate the virtual machine. it was so bogged down (not sure whether i/o starvation or what) that it was essentially unusable, so i'm rebooting it20:03
mrmartinfungi: was that before the translation import patch?20:04
fungimrmartin: after i think, but it was in such a state i couldn't confirm, so i'll check it once it's back up20:05
mrmartinok, thanks. I added this patch: https://review.openstack.org/#/c/47241/20:05
fungiyeah, git show says 871130f Disable language import20:06
fungimrmartin: on https://review.openstack.org/47241 can you leave a review comment stating that you've read and agree to follow the practices described in http://ci.openstack.org/sysadmin.html#ssh-access ?20:06
mrmartinof course20:07
fungithat way we have a good record of it20:07
mrmartindone.20:08
*** Yportnova_ has joined #openstack-infra20:10
*** nicedice has joined #openstack-infra20:10
fungithanks!20:12
*** sandywalsh has joined #openstack-infra20:13
*** jcoufal has joined #openstack-infra20:13
*** mestery has joined #openstack-infra20:13
fungistarted the puppet apply again once the vm was reachable and once it got past the build-dist-post step load has started climbing (5-minute load average is already up to 10). seems to be swapping heavily and rapidly running out of memory20:15
*** yolanda has quit IRC20:15
fungiroughly 90% of the cpu utilization is waiting for i/o20:15
fungimrmartin: what size vm should this need? i picked a 1gb flavor but can resize to something larger20:17
mrmartin:)20:17
mrmartinyeap, 1GB is small for Drupal 7, could we start with 2GB at least?20:17
fungilooks like it's eating 1gb ram and 1gb swap at the moment20:17
fungii think we probably ought to do 4gb, honestly20:17
*** zehicle_at_dell has quit IRC20:17
fungii'l resize it now20:17
mrmartinlet's try it20:17
fungibecause it's chewed through 2gb of total virtual memory already during the build/deploy with no signs of stopping20:18
mrmartinI have a HP cloud instance with 2GB memory, and it is running well20:19
fungiokay, i'll do 2gb initially then20:19
*** Yportnova_ has quit IRC20:19
*** bnemec has joined #openstack-infra20:19
fungiit's resizing to a 2gb flavor, but will probably take a few minutes20:20
fungii also need to leave shortly, but will come back later and check in on it to repeat the deployment tests. i know it's getting late for you, but i can catch you up tomorrow20:21
*** sdake_ has quit IRC20:22
mrmartinok, let's talk about the results tomorrow20:22
mrmartinanyway, thanks for the help20:23
*** senk has joined #openstack-infra20:24
fungimy pleasure20:28
*** Yportnova_ has joined #openstack-infra20:28
*** senk has quit IRC20:29
*** annegentle has joined #openstack-infra20:29
mrmartinbye20:29
*** mrmartin has quit IRC20:29
*** zehicle_at_dell has joined #openstack-infra20:29
*** markmcclain has quit IRC20:30
* fungi is stepping out for a few hours and will be back later20:30
*** SergeyLukjanov has joined #openstack-infra20:31
*** dkliban has joined #openstack-infra20:37
*** Yportnova_ has quit IRC20:37
*** jpeeler has quit IRC20:43
*** pblaho has joined #openstack-infra20:43
*** boris-42 has quit IRC20:51
*** cppcabrera_afk is now known as cppcabrera20:51
*** branen has quit IRC20:51
*** branen has joined #openstack-infra20:52
*** grizzled has joined #openstack-infra20:54
*** mestery has quit IRC20:54
*** jpeeler has joined #openstack-infra20:59
*** senk has joined #openstack-infra21:01
*** mrodden1 has quit IRC21:01
*** svarnau_ has quit IRC21:03
*** senk has quit IRC21:03
*** ArxCruz has quit IRC21:04
*** DinaBelova has joined #openstack-infra21:04
*** matty_dubs is now known as matty_dubs|gone21:05
*** rfolco has quit IRC21:07
*** DinaBelova has quit IRC21:09
*** senk has joined #openstack-infra21:11
*** mrodden has joined #openstack-infra21:12
*** ryanpetrello_ has joined #openstack-infra21:12
*** sdake_ has joined #openstack-infra21:12
*** sdake_ has quit IRC21:12
*** sdake_ has joined #openstack-infra21:12
*** ryanpetrello has quit IRC21:14
*** ryanpetrello_ is now known as ryanpetrello21:14
*** grizzled has quit IRC21:14
*** shardy is now known as shardy_afk21:14
*** DennyZhang has quit IRC21:15
*** pcm_ has quit IRC21:20
*** CaptTofu has quit IRC21:20
*** sdake_ has quit IRC21:20
*** sdake_ has joined #openstack-infra21:20
*** sdake_ has quit IRC21:20
*** sdake_ has joined #openstack-infra21:20
*** CaptTofu has joined #openstack-infra21:20
*** cppcabrera has quit IRC21:22
*** weshay has quit IRC21:23
*** davidhadas_ has joined #openstack-infra21:24
*** vipul is now known as vipul-away21:25
*** davidhadas has quit IRC21:26
*** thomasm has quit IRC21:28
*** sarob has quit IRC21:28
*** vipul-away is now known as vipul21:29
*** datsun180b has quit IRC21:29
*** sarob has joined #openstack-infra21:29
*** sarob has quit IRC21:34
pleia2clarkb: updated diagram based on our chat this morning: https://wiki.openstack.org/wiki/File:Os-ci-workflow-09182013.png21:34
*** mestery has joined #openstack-infra21:38
*** mestery has quit IRC21:40
*** zehicle_at_dell has quit IRC21:48
*** sdake_ has quit IRC21:48
*** Ryan_Lane has quit IRC21:52
*** Ryan_Lane has joined #openstack-infra21:52
*** jcoufal has quit IRC21:52
jerryzanybody can help me with a zuul-doc test problem?21:53
*** pblaho has quit IRC21:55
*** CaptTofu has quit IRC21:56
*** CaptTofu has joined #openstack-infra21:56
*** senk has quit IRC22:04
*** sarob has joined #openstack-infra22:07
*** julim has joined #openstack-infra22:07
*** sbadia has left #openstack-infra22:07
*** sarob has quit IRC22:07
*** sbadia_ has joined #openstack-infra22:07
*** sarob has joined #openstack-infra22:08
*** tstevenson_ has quit IRC22:08
*** dims has quit IRC22:09
*** zzs has left #openstack-infra22:13
*** nicedice has quit IRC22:15
*** portante is now known as portante|afk22:15
*** julim has quit IRC22:15
*** nicedice has joined #openstack-infra22:16
*** mriedem has quit IRC22:17
*** dims has joined #openstack-infra22:23
*** CaptTofu has quit IRC22:24
*** pcrews has quit IRC22:24
*** CaptTofu has joined #openstack-infra22:25
*** CaptTofu has quit IRC22:25
*** CaptTofu has joined #openstack-infra22:25
*** julim has joined #openstack-infra22:26
*** CaptTofu has quit IRC22:27
anteayajerryz: what problem are you having?22:27
*** CaptTofu has joined #openstack-infra22:28
jerryzanteaya: http://logs.openstack.org/33/47233/1/check/gate-zuul-docs/7b794af/console.html22:28
* anteaya looks22:28
jerryzanteaya: i also tested the master branch without any change on my local machine, same problem22:29
anteayahmmm22:29
anteayait might be a dependency availability issue22:30
*** CaptTofu has quit IRC22:30
*** CaptTofu has joined #openstack-infra22:31
anteayaTypeError: must be unicode, not str - has there been a change to Sphinx lately I wonder22:32
*** dkliban has quit IRC22:32
*** mrodden has quit IRC22:32
anteayawell Sphinx was updated yesterday22:33
anteayaso there is a strong possibility your error jerryz is related to the new version of Sphinx22:33
anteayadhellmann Alex_Gaynor ^ any insight?22:34
*** changbl has quit IRC22:35
*** sbadia_ is now known as sbadia22:36
jerryzprobably22:37
jerryzi take a look at the log which was successful22:37
anteayathis page says Sphinx-1.2b2 was uploaded yesterday: https://pypi.python.org/pypi/Sphinx22:37
jerryzthe build that was successful was based on 1.2b122:38
anteayathis page says the current version is 1.1.3: http://sphinx-doc.org/22:38
anteayaokay22:38
anteayajerryz: so you are saying 1.2b1 is successful for you22:39
jerryzfor someone else22:39
anteayacan you see what the version of sphinx is for the unsuccessful build?22:39
anteayajust so we are sure22:39
jerryz1.2b222:39
anteayathere we are22:39
anteayaokay22:39
anteayaso your tests won't pass until we address this22:40
anteayaand I have no knowledge how to do so22:40
jerryzspecify the requirement version number?22:40
anteayawe can certainly try22:40
anteayawould you like to offer a patch?22:40
jerryzin test-requirements.txt , the version number is not specified22:40
anteayapining requirements is exciting business22:41
anteaya*pinning22:41
anteayago ahead and offer a patch pinning it22:41
anteayaand we shall see what comes of it, hopefully it is as easy as having it merged22:41
*** nicedice_ has joined #openstack-infra22:42
jerryzok22:42
anteayathanks jerryz22:42
*** nicedice has quit IRC22:45
funginot really back yet, but jerryz have a look at https://review.openstack.org/47239 (similar change should address it elsewhere i think)22:46
*** michchap has joined #openstack-infra22:46
fungithe problem is, beta releases should be allowed to be broken, and we simply shouldn't use them22:46
* fungi disappears again for a bit22:47
*** julim has quit IRC22:47
*** jhesketh_ has joined #openstack-infra22:48
jerryzwill test it out22:50
*** prad has quit IRC22:51
*** cthulhup has joined #openstack-infra22:52
*** senk has joined #openstack-infra22:52
*** portante|afk is now known as portante22:54
*** cthulhup has quit IRC22:55
*** sbadia has quit IRC22:56
*** sbadia has joined #openstack-infra22:57
*** julim has joined #openstack-infra22:59
openstackgerritJerry Zhao proposed a change to openstack-infra/zuul: remove --pre option from tox's pip install command  https://review.openstack.org/4727723:15
*** julim has quit IRC23:16
openstackgerritDevananda van der Veen proposed a change to openstack-infra/config: Disable python33 voting for python-ironicclient  https://review.openstack.org/4727823:17
*** dstufft has quit IRC23:19
*** dstufft_ has joined #openstack-infra23:19
*** michchap has quit IRC23:26
*** michchap has joined #openstack-infra23:29
*** sbadia has quit IRC23:31
*** vipul is now known as vipul-away23:31
*** pcrews has joined #openstack-infra23:32
*** sbadia has joined #openstack-infra23:32
*** vipul-away is now known as vipul23:34
jerryzfungi: thanks, it works , i have proposed it to zuul https://review.openstack.org/4727723:35
*** mriedem has joined #openstack-infra23:35
fungijerryz: i see that, awesome23:39
*** fbo is now known as fbo_away23:40
*** hemna is now known as hemnafk23:42
openstackgerritJerry Zhao proposed a change to openstack-infra/zuul: remove --pre option from tox's pip install command  https://review.openstack.org/4727723:45
clarkbfungi: mordred: https://review.openstack.org/#/c/47278/23:45
fungiwhen did that happen? was it always broken or did they regress?23:46
clarkbfungi: I think keystoneclient regressed23:47
fungiahh, looks like mordred and i double-tapped one again23:47
openstackgerritA change was merged to openstack-infra/config: Disable python33 voting for python-ironicclient  https://review.openstack.org/4727823:48
fungikeystoneclient py3k regression is most disappoint23:49
clarkbyeah :(23:50
clarkbfungi: we should get better at making those voting as they start to apss23:50
clarkbfungi: to prevent regressions23:50
fungisomeone shoulda... yeah23:50
clarkbAlex_Gaynor: ^23:50
fungiwhat you said23:50
* fungi is clearly someone. maybe i shoulda23:51
fungii hate to say someone else should do stuff. i am as much to blame as anyone23:51
*** changbl has joined #openstack-infra23:52
*** jhesketh__ has joined #openstack-infra23:52
*** dcramer_ has joined #openstack-infra23:53
*** jcoufal has joined #openstack-infra23:54
clarkbfungi: I think it is fair, as much as I would like to have working py3k everywhere, the projects them selves are responsible...23:54
fungithis is true23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!