Thursday, 2014-01-02

*** dstanek has joined #openstack-infra00:01
*** mgagne has quit IRC00:06
*** jcooley_ has joined #openstack-infra00:13
*** dstanek has quit IRC00:14
*** senk has quit IRC00:17
*** jcooley_ has quit IRC00:19
*** senk has joined #openstack-infra00:21
*** rfolco has quit IRC00:23
*** rakhmerov has joined #openstack-infra00:23
*** adam_g has quit IRC00:24
*** adam_g has joined #openstack-infra00:24
*** adam_g has quit IRC00:24
*** adam_g has joined #openstack-infra00:24
*** rakhmerov has quit IRC00:27
*** zehicle_at_dell has quit IRC00:28
*** zehicle_at_dell has joined #openstack-infra00:29
*** zehicle has quit IRC00:29
*** zehicle has joined #openstack-infra00:30
*** sdake has quit IRC00:30
*** adam_g has quit IRC00:35
*** adam_g has joined #openstack-infra00:35
*** adam_g has quit IRC00:35
*** adam_g has joined #openstack-infra00:35
*** rongze has joined #openstack-infra00:42
*** rongze has quit IRC00:46
*** yaguang has joined #openstack-infra00:52
*** senk has quit IRC00:53
*** senk has joined #openstack-infra00:54
*** jcooley_ has joined #openstack-infra01:00
*** boris-42 has quit IRC01:05
*** yongli has joined #openstack-infra01:05
*** jcooley_ has quit IRC01:06
*** jcooley_ has joined #openstack-infra01:07
*** banix has joined #openstack-infra01:10
*** rongze has joined #openstack-infra01:15
*** senk has quit IRC01:16
*** banix has quit IRC01:18
*** banix has joined #openstack-infra01:20
*** senk has joined #openstack-infra01:22
*** praneshp has joined #openstack-infra01:22
*** senk has quit IRC01:23
*** rakhmerov has joined #openstack-infra01:24
*** rakhmerov has quit IRC01:28
*** jcooley_ has quit IRC01:31
*** rongze has quit IRC01:34
*** banix has quit IRC01:36
*** zehicle_at_dell has quit IRC01:37
*** zehicle has quit IRC01:37
*** rongze has joined #openstack-infra01:49
*** fifieldt has joined #openstack-infra01:54
*** dcramer_ has joined #openstack-infra02:01
*** senk has joined #openstack-infra02:11
*** fifieldt has quit IRC02:15
*** rakhmerov has joined #openstack-infra02:24
*** rakhmerov has quit IRC02:29
*** larrycai_ has joined #openstack-infra02:38
*** senk has quit IRC02:40
*** adam_g has quit IRC02:48
*** adam_g has joined #openstack-infra02:48
*** adam_g has quit IRC02:48
*** adam_g has joined #openstack-infra02:48
*** dguitarbite has joined #openstack-infra03:22
*** rakhmerov has joined #openstack-infra03:25
*** rakhmerov has quit IRC03:30
*** dguitarbite has quit IRC03:34
*** banix has joined #openstack-infra03:35
*** nati_ueno has joined #openstack-infra04:04
*** changbl has quit IRC04:05
*** changbl has joined #openstack-infra04:10
*** nati_ueno has quit IRC04:16
*** rakhmerov has joined #openstack-infra04:26
*** rakhmerov has quit IRC04:30
*** loq_mac has joined #openstack-infra04:39
*** julim has joined #openstack-infra04:46
*** rongze has quit IRC04:47
*** rongze has joined #openstack-infra04:48
*** rongze has quit IRC04:52
*** banix has quit IRC04:57
*** ArxCruz has quit IRC05:11
*** rongze has joined #openstack-infra05:18
*** rongze has quit IRC05:20
*** rongze_ has joined #openstack-infra05:20
*** adam_g has quit IRC05:21
*** adam_g has joined #openstack-infra05:21
*** adam_g has quit IRC05:21
*** adam_g has joined #openstack-infra05:21
*** adam_g has quit IRC05:24
*** adam_g has joined #openstack-infra05:24
*** adam_g has quit IRC05:24
*** adam_g has joined #openstack-infra05:24
*** fifieldt has joined #openstack-infra05:24
*** rongze_ has quit IRC05:25
*** rakhmerov has joined #openstack-infra05:27
*** rakhmerov has quit IRC05:29
*** rakhmerov has joined #openstack-infra05:29
*** chandankumar has joined #openstack-infra05:48
*** loq_mac has quit IRC05:55
clarkbwhat if we deny push access to all users on refs/draft/*06:18
*** rongze has joined #openstack-infra06:18
*** rakhmerov has quit IRC06:22
*** AaronGr_Zzz is now known as AaronGr06:22
*** rongze has quit IRC06:23
*** rongze has joined #openstack-infra06:28
*** rakhmerov has joined #openstack-infra06:35
*** jhesketh_ has joined #openstack-infra06:47
*** rakhmerov has quit IRC06:48
*** fifieldt has quit IRC06:49
*** coolsvap has joined #openstack-infra06:50
*** fifieldt has joined #openstack-infra06:50
*** zigo has joined #openstack-infra06:56
*** fifieldt has quit IRC06:58
*** garyk has joined #openstack-infra07:03
*** yfried has joined #openstack-infra07:04
*** yfried has left #openstack-infra07:04
*** fifieldt has joined #openstack-infra07:14
*** rcarrillocruz has joined #openstack-infra07:32
*** coolsvap has quit IRC07:37
*** coolsvap has joined #openstack-infra07:39
*** rongze has quit IRC07:45
*** rongze has joined #openstack-infra07:51
*** adam_g has quit IRC08:07
*** adam_g has joined #openstack-infra08:08
*** adam_g has quit IRC08:08
*** adam_g has joined #openstack-infra08:08
*** fbo_away is now known as fbo08:12
*** dguitarbite_ has joined #openstack-infra08:12
*** hashar has joined #openstack-infra08:13
*** praneshp_ has joined #openstack-infra08:16
*** praneshp has quit IRC08:19
*** praneshp_ is now known as praneshp08:19
*** zhiyan has joined #openstack-infra08:24
*** fifieldt has quit IRC08:29
*** rakhmerov has joined #openstack-infra08:29
*** coolsvap has quit IRC08:44
*** coolsvap has joined #openstack-infra08:53
*** zhiyan has quit IRC08:55
*** jpich has joined #openstack-infra09:00
*** adam_g has quit IRC09:02
*** adam_g has joined #openstack-infra09:03
*** adam_g has quit IRC09:03
*** adam_g has joined #openstack-infra09:03
*** rakhmerov has quit IRC09:03
*** talluri has joined #openstack-infra09:06
*** derekh has joined #openstack-infra09:10
*** larrycai_ has quit IRC09:30
*** lcestari has joined #openstack-infra09:42
*** praneshp has quit IRC09:45
*** coolsvap has quit IRC09:47
*** zhiyan has joined #openstack-infra09:58
*** rakhmerov has joined #openstack-infra10:00
*** coolsvap has joined #openstack-infra10:03
*** rakhmerov has quit IRC10:04
*** zhiyan has quit IRC10:09
*** jooools has joined #openstack-infra10:15
*** yaguang has quit IRC10:23
*** fifieldt has joined #openstack-infra10:27
*** dguitarbite_ has quit IRC10:43
*** fifieldt has quit IRC10:43
*** rongze_ has joined #openstack-infra10:43
*** rongze has quit IRC10:46
*** rongze_ has quit IRC10:48
*** boris-42 has joined #openstack-infra10:50
*** ArxCruz has joined #openstack-infra10:58
*** rakhmerov has joined #openstack-infra11:01
*** rakhmerov has quit IRC11:05
*** rongze has joined #openstack-infra11:14
*** boris-42 has quit IRC11:16
*** boris-42_ has joined #openstack-infra11:16
*** rongze_ has joined #openstack-infra11:18
*** rongze has quit IRC11:19
*** rongze_ has quit IRC11:22
*** dstanek has joined #openstack-infra11:28
*** jroovers has joined #openstack-infra11:30
*** boris-42_ is now known as boris-4211:36
*** jorisroovers has joined #openstack-infra11:39
*** jroovers has quit IRC11:41
*** hashar has quit IRC11:44
*** rakhmerov has joined #openstack-infra12:02
*** rongze has joined #openstack-infra12:02
*** rakhmerov has quit IRC12:05
*** talluri has quit IRC12:17
*** CaptTofu has quit IRC12:31
*** CaptTofu has joined #openstack-infra12:31
*** rongze has quit IRC12:36
*** rongze has joined #openstack-infra12:37
*** rongze has quit IRC12:42
*** gfa has left #openstack-infra12:48
*** jhesketh_ has quit IRC12:48
*** jorisroovers has quit IRC12:49
*** fifieldt has joined #openstack-infra12:58
*** b3nt_pin` is now known as beagles12:59
*** beagles is now known as Guest797513:00
*** weshay has joined #openstack-infra13:01
sdaguemorning folks, happy new year13:01
*** CaptTofu has quit IRC13:02
*** coolsvap has quit IRC13:02
*** CaptTofu has joined #openstack-infra13:02
*** rakhmerov has joined #openstack-infra13:03
*** rongze has joined #openstack-infra13:03
*** smarcet has joined #openstack-infra13:04
*** rakhmerov has quit IRC13:07
*** Guest7975 has quit IRC13:08
fungisdague: what's so happy about it? means i have to remember to type "2014" everywhere now13:09
fungimajor inconvenience13:09
sdaguewho types years?13:11
*** b3nt_pin has joined #openstack-infra13:12
*** rfolco has joined #openstack-infra13:13
sdagueanyone know when jog0 is flying?13:17
sdagueI was going through his grenade patches last night, and wanted to get together with him on them13:18
*** xchu has joined #openstack-infra13:25
*** dcramer_ has quit IRC13:26
*** johnthetubaguy has joined #openstack-infra13:27
fungii don't recall whether he said when he was departing13:30
*** fifieldt has quit IRC13:37
*** b3nt_pin is now known as beagles13:44
*** mriedem has joined #openstack-infra13:44
*** MIDENN_ has quit IRC13:46
*** hashar has joined #openstack-infra13:46
*** mdenny has joined #openstack-infra13:47
openstackgerritThierry Carrez proposed a change to openstack-infra/devstack-gate: Add oslo.rootwrap to the devstack gate  https://review.openstack.org/6467513:48
* fungi will be back shortly...13:56
*** CaptTofu has quit IRC14:00
*** CaptTofu has joined #openstack-infra14:00
openstackgerritRussell Bryant proposed a change to openstack-infra/config: zuul status: add a visual hint for change headers  https://review.openstack.org/6380914:01
ArxCruzALL: Does anyone working on PPC cirros image ?14:01
openstackgerritThierry Carrez proposed a change to openstack-infra/config: Add oslo.rootwrap to the integrated gate  https://review.openstack.org/6467714:01
*** rakhmerov has joined #openstack-infra14:04
sdagueArxCruz: I don't think so14:05
*** rakhmerov has quit IRC14:08
ttxsdague: I pushed the various changes that you mentioned for oslo.rootwrap devstack-gate integration at: https://review.openstack.org/#/q/status:open+topic:rootwrap-gate,n,z14:10
ttxlet me know if I missed one14:10
*** dripton has joined #openstack-infra14:10
*** mfer has joined #openstack-infra14:11
*** tma996 has joined #openstack-infra14:12
sdaguecool, will check it out in a few14:13
*** dprince has joined #openstack-infra14:16
*** ttx has quit IRC14:24
*** ttx has joined #openstack-infra14:25
*** ttx has quit IRC14:25
*** ttx has joined #openstack-infra14:25
*** xchu has quit IRC14:32
fungiArxCruz: you might be able to make a very stripped-down debian/powerpc image which wouldn't be much bigger than cirros14:32
ArxCruzfungi: the problem is when I ran tempest it ask for cirros image, and some tests fails because the image is x86 :/14:33
openstackgerritThierry Carrez proposed a change to openstack-infra/config: Add logging to #openstack-relmgr-office  https://review.openstack.org/6468314:37
*** eharney has joined #openstack-infra14:40
*** ruhe has joined #openstack-infra14:41
*** ryanpetrello has joined #openstack-infra14:43
*** rakhmerov has joined #openstack-infra14:43
maurosrArxCruz: you can specify which image you want to use in devstack14:44
ArxCruzmaurosr: yeah, that's what I'm working right now14:44
ArxCruz:)14:44
maurosrArxCruz: then you just need to set the url of the debian-ppc image that fungi mentioned14:44
maurosrok14:44
*** markmcclain has quit IRC14:54
fungiArxCruz: right, i suspect for a given processor architecture, you will either need an emulator hypervisor (qemu can emulate x86 on powerpc, right?) or an image of the correct arch14:55
*** ruhe has quit IRC14:55
ArxCruzfungi: yeah, qemu can't emulate x86 on ppc14:56
fungiArxCruz: oh, too bad. i wonder if bochs would work14:59
ArxCruzbochs?15:00
*** dkliban has joined #openstack-infra15:01
fungiit's an x86 emulator i used to use "back in the day" (before qemu and friends existed) and still seems to be around15:01
*** dcramer_ has joined #openstack-infra15:04
sdagueArxCruz: or build your own cirros15:04
ArxCruzsdague: i know a few guys at IBM is trying to build one, but no success so far :/15:06
fungithe benefit to bochs is that it's a full system emulator, not just a paravirt implementation15:07
fungiso no need to support syscall passthrough or anything like that, which means it has more of a chance of being usable on entirely separate platforms15:07
*** Ajaeger has joined #openstack-infra15:08
fungithough i've not tried to use it for network-accessible virtual machines (that was a very new concept at the time i was using it), so no idea whether it's usable for that15:08
sdaguefungi: yeh, but then it would be *super* slow15:09
sdague2nd level kvm guests executing full emulation15:10
sdaguenot so good for running tempest tests15:10
ArxCruzand I'm not sure if it has ppc64 support15:11
ArxCruzi mean, if there's binaries for ppc64 :)15:12
annegentle_fungi: have a minute to assist in real-time with the signed tagging?15:12
*** rakhmerov has quit IRC15:15
*** jpeeler has joined #openstack-infra15:16
*** jpeeler has quit IRC15:16
*** jpeeler has joined #openstack-infra15:16
*** senk has joined #openstack-infra15:16
*** dkehn_ is now known as dkehn15:17
*** croberts has joined #openstack-infra15:17
mferfolks, my review for the config repo is failing jenkins now. the recommendation is to rebase. Do you rebase something that's already out there or should it actually be a merge? Once I share code I never rebase. This is why I ask.15:18
Ajaegermfer: which review is it?15:19
mferhttps://review.openstack.org/#/c/62069/15:19
mferthis originally passed but infa isn't making new projects right so it's on hold15:19
mferthat's why it was code reviewed with X15:20
Ajaegermfer: Download it and rebase it manually. The repository has been moved forward and your patch does not apply anymore - so, before it can go in, it might need some tweaking...15:21
mferrebase or merge or doesn't matter?15:21
Ajaegermfer: git checkout master;git pull;git review -d 62069; git rebase origin/master;git review - unless you get a conflict, then you need to resolve the conflicts15:21
Ajaegermfer: git rebase, you want your patch forward ported to the tip of the tree15:22
mferAjaeger thanks. i don't usually rebase once the code is shared publically. that's why i ask.15:22
*** coolsvap has joined #openstack-infra15:22
anteayamfer: good asking15:23
anteayawe don't merge, only rebase15:23
anteayagerrit is the only system that merges15:23
Ajaegermfer: There seems to have been changes that make it impossible for the patch to apply cleanly, therefore you need to rebase.15:23
mfergood to know. this was my first request so I'm just learning15:23
anteayamfer: welcome aboard15:23
anteayaand thanks for asking, it saves time in the long run15:24
Ajaegermfer: Welcome - and if there's more you need to know, just ask.15:24
mferoh I will. Thanks :)15:24
*** senk has quit IRC15:24
Ajaegeranteaya: why does it say "rebase or merge" at https://wiki.openstack.org/wiki/GerritJenkinsGit#Gerrit_Merge_Problems ?15:24
AjaegerI guess that should be only rebase...15:25
*** markmcclain has joined #openstack-infra15:26
*** senk has joined #openstack-infra15:27
anteayagood question, Ajaeger15:27
anteayaI think it should be only rebase15:27
AjaegerOk, I'll change it...15:28
anteayalet's hear from fungi and jeblair before we change15:28
* Ajaeger waits ;)15:28
anteayaah okay, that works too15:28
anteayaif there is objection we can change it back15:28
anteayaI say this and I am going to be awol in about an hour15:28
anteayaso I'll leave changes to you, but I do think it should be only rebase15:29
anteayafair enough?15:29
AjaegerOk, blame me for beiing to quick - reverts are cheap in the wiki ;)15:29
anteayaAjaeger: my kind of contributor15:30
anteaya:D15:30
fungiannegentle_: sorry, back. i think my irc client has stopped raising a bell in my terminal multiplexer, so i keep responding late to people. should i check my e-mail, or are you just wanting to work through it in here?15:31
Ajaegeranteaya: Fixed and reworked the paragraph slightly. mfer could you check the description at https://wiki.openstack.org/wiki/GerritJenkinsGit#Gerrit_Merge_Problems  to see whether it answers your problem?15:32
*** rakhmerov has joined #openstack-infra15:33
*** rongze has quit IRC15:34
fungimfer: rebasing published branches in a normal git workflow is indeed often frowned upon, but for code review it works out better (though you still should avoid gratuitously rebasing when there are no merge conflicts)15:34
mferAjaeger it does. what would be useful is an example of how to go about the rebase here. I say that for new folks who aren't familiar with rebasing or all of git review. I went a year working with git before I rebased the first time.15:34
mferfungi makes sense. thanks15:34
anteayamfer: http://anteaya.info/blog/2013/03/25/please-rebase-your-change-and-upload-a-new-patchset/15:34
anteayafeel free to use anything helpful15:35
Ajaegermfer, search for rebase at https://wiki.openstack.org/wiki/GerritWorkflow15:35
Ajaegermfer and if you could enhance the wiki with an example, it would be great. I guess anteaya and myself can review it as needed ;)15:35
mferAjaeger thanks. I was more thinking for future folks who are not familiar than for myself.15:36
mferkk15:36
fungiAjaeger: anteaya: the article is correct in stating that gerrit will ff *or* merge (it isn't suggesting that the contributor should do that manually, just stating what will potentially happen to the change when it comes time for it to get added officially to the target branch after gating)15:36
anteayafungi: yes15:36
anteayaAjaeger just removed the part in the wiki that looked to me like it was recomending contributors merge locally15:37
anteayawhich I think is contra-indicated with our workflow15:37
fungiahh, yes, i was still catching up on scrollback. i'll take a look at the dif15:37
fungif15:37
anteayak15:37
*** prad has joined #openstack-infra15:37
fungiAjaeger: your new wording helps, thanks!15:38
Ajaegerfungi: thanks for the review!15:38
fungithough we really should mostly just point people to https://wiki.openstack.org/wiki/Gerrit_Workflow#Long-lived_Topic_Branches15:39
fungithat article is generally more visible in the onboarding documents, so that new contributors will hopefully see it first15:40
fungiso that's where we try to keep the development workflow recommendations (and the GerritJenkinsGit article is intended to be more behind-the-scenes explanations of stuff)15:40
Ajaegerfungi: The page I changed is mentioned in the gerrit message in case of conflicts. But let's add a link to the Workflow page.15:40
fungiahh, good point. i had forgotten we linked to GerritJenkinsGit from the merge conflict message15:41
fungiwe might ought to put up a change to gerrit's config to adjust that15:42
AjaegerFor now I put the link into the page.15:43
fungiawesome--thanks again15:43
*** prad has quit IRC15:44
*** banix has joined #openstack-infra15:44
*** rongze has joined #openstack-infra15:44
*** yidclare has joined #openstack-infra15:45
jeblairclarkb, fungi: let's merge https://review.openstack.org/#/c/63809/3 to help with the discoverability of the new zuul status page15:45
fungioh, i see, the link in the review comment is just to the test failures section, not to the Gerrit_Merge_Problems section15:45
jeblairsdague: happy new year to you! :)15:45
sdaguehey, so I kicked the hornet nest again on the gerrit upgrade on infra list15:46
sdaguewhen you get a chance15:46
jeblairsdague: people were working on it through the holidays15:47
sdaguecool15:47
jeblairsdague: please don't rush it, or if it's really important, please help out.  i think it's very important that the upgrade be well tested and configured.  it's a huge change, and with >1k users, we need to be careful about how we make changes like that.15:48
fungisdague: yeah, at the moment we've run into some substantial changes in the underlying cla feature implementation which will require some special handling on upgrade15:49
jeblairfungi: perhaps we should stop dealing with the cla in gerrit.15:49
Ajaegerfungi: yeah, it points to the wrong place indeed ;(15:49
openstackgerritA change was merged to openstack-infra/config: zuul status: add a visual hint for change headers  https://review.openstack.org/6380915:49
jeblairsdague: https://blueprints.launchpad.net/openstack-ci/+spec/gerrit-2.8-upgrade15:50
jeblairsdague: that links to an etherpad where we've been working through just some of the issues.15:50
fungijeblair: well, we'd still need some means of denying contributions from non-cla-signers for certain projects, or agreement that we'll stop bothering with automated enforcement15:50
annegentle_fungi: oh sorry now I'm back :) yes, I'd like help on irc if you can15:51
fungiannegentle_: sure, how far did you get?15:51
jeblairfungi: can you summarize the cla situation in the etherpad?15:51
annegentle_fungi: I copied the .gnupg directory onto the cloud server15:51
sdaguejeblair: ok, cool. so it's a little hard for me to see from that etherpad what's decided and done, and what's stilll a blocker15:51
*** sdake has joined #openstack-infra15:52
*** sdake has quit IRC15:52
*** sdake has joined #openstack-infra15:52
fungijeblair: yeah, i'll add the details there in a moment15:52
jeblairsdague: almost nothing is done; so far the work has focused on actually getting it running and discovering what the problems are (eg, cla)15:52
jeblairs/done/complete/15:53
fungiannegentle_: does gpg2 --list-keys return anything?15:53
annegentle_fungi: yep, including what I think is the private key15:53
annegentle_fungi: but then the git tag command gives back this: https://gist.github.com/anonymous/502126886b5f473c8cd015:54
annegentle_fungi: did I need to do anything to tell git about the key?15:55
fungiannegentle_: does 'gpg2 --list-keys anne@openstack.org' return anything?15:57
fungiit might be that you didn't have a uid for that address on your key15:57
*** prad has joined #openstack-infra15:58
sdaguejeblair: ok, fair. Which is why I threw the thread out there.15:58
annegentle_fungi: yep, it's listing the one I expect it to16:02
jeblairsdague: responding16:03
*** CaptTofu_ has joined #openstack-infra16:06
*** CaptTofu has quit IRC16:07
*** UtahDave has joined #openstack-infra16:08
*** thedodd has joined #openstack-infra16:08
fungiannegentle_: interesting... well, you can tell git to use a specific key by running 'git config --global --add user.signingkey <your_key_id>'16:09
fungiwhere <your_key_id> is the last 16 or so digits of the fingerprint displayed by 'gpg2 --fingerprint anne@openstack.org' (omitting spaces)16:10
*** jgrimm has joined #openstack-infra16:12
annegentle_fungi: ok sounds like a good thing to try16:12
*** lcheng has joined #openstack-infra16:13
bknudsonI've got a change that failed pep8, but can't recreate it here...16:14
bknudsonhttp://logs.openstack.org/74/64474/1/gate/gate-reviewday-pep8/db995bd/console.html#_2014-01-02_15_42_20_74616:14
bknudsonNo distributions at all found for lazr.authentication (from lazr.restfulclient>=0.9.11->launchpadlib->reviewday==0.2.0)16:14
NobodyCamGood morning infra.. and Happy New Year16:15
annegentle_fungi: ok that did it!16:16
*** thedodd has quit IRC16:16
NobodyCami just got a change that failed to build python-mysql?16:16
*** ^d has joined #openstack-infra16:17
*** ^d has joined #openstack-infra16:17
fungiannegentle_: great!16:17
annegentle_fungi: ok so one more issue, I can't seem to push to gerrit16:18
annegentle_fungi: here's what I see https://gist.github.com/anonymous/365dfe0c41465a65fbde16:18
Ajaegerfungi: Is the acl setup correctly?16:19
annegentle_NobodyCam: happy new year!16:19
*** thedodd has joined #openstack-infra16:21
*** markmcclain has quit IRC16:21
fungibknudson: looks like maybe that run had trouble reading the links from https://pypi.python.org/simple/lazr.authentication/ (seems it's not hosted on pypi and instead links to lp for tarball downloads)16:21
fungiannegentle_: i suspect we haven't added you to that group yet (so Ajaeger was close on the cause). fixing16:22
sdagueso 3 out of 4 changes in the gate are failing docs jobs16:23
annegentle_fungi: thanks16:23
*** blamar has joined #openstack-infra16:23
annegentle_sdague: let's ditch sphinx ;)16:23
sdaguethough the pattern seems hard to figure out. 2 failed on distribute16:23
sdagueand 1 on pbr16:23
bknudsonfungi: strangely, the pep8 job had worked just a couple of days ago.16:23
*** jergerber has joined #openstack-infra16:23
fungiannegentle_: looks like that was it. try again and see if it's any better. also, you can add/remove additional members of openstack-docs-ptl at https://review.openstack.org/#/admin/groups/260,members now as needed16:24
sdagueannegentle_: well, they are also failing pep8 and unit tests16:24
annegentle_ok whew fungi thanks! I was mostly just glad I still remember the passphrase ha ha16:24
annegentle_sdague: yeah let's ditch pep8 next16:24
fungiannegentle_: ;)16:24
sdaguedocs is just the job I assume is least likely to be random fails16:24
annegentle_sdague: yeah it's usually solid, and everyone switched their requirements, right?16:25
*** coolsvap has quit IRC16:25
*** prad_ has joined #openstack-infra16:26
*** prad has quit IRC16:26
*** prad_ is now known as prad16:26
*** prad has quit IRC16:26
sdagueannegentle_: as far as I know16:27
annegentle_fungi: Ajaeger: ok, tag 0.1 for openstack-doc-tools16:27
fungibknudson: aha, i should read logs more carefully... "Some externally hosted files were ignored (use --allow-external lazr.authentication to allow)."16:28
annegentle_sdague: I have a search now on Gerrit for all rst changes for all repos, and it seems to be working, and only one has even been proposed over break, and hasn't merged. So yeah random.16:28
annegentle_(over break I mean last 2 weeks or so)16:28
*** andreaf has joined #openstack-infra16:28
Ajaegercongrats annegentle_ for the tag!16:29
annegentle_Ajaeger: whoo hoo16:29
Ajaegerfungi: I don't see it yet at https://pypi.python.org/pypi/openstack-doc-tools/ - how long will it take?16:29
* Ajaeger sees the tag in git16:29
sdaguewell, I think this is something fundamental going on16:29
AjaegerOh, there it is! GREAT!16:29
sdagueabout distribute16:29
fungibknudson: i'm thinking reviewday may need a tox.ini patch to add --allow-external on the pip command16:30
Ajaegerwhoo hoo, annegentle_ !16:30
bknudsonfungi: when I ran pep8 locally, it worked... pip freeze shows "lazr.authentication==0.1.2"16:30
annegentle_Ajaeger: wow that's fast, fungi and all y'all that's amazing16:30
fungibknudson: your version of pip is probably old16:30
annegentle_Some kinda software factory you're running here.16:30
fungiannegentle_: it's great when it finally works!16:30
Ajaegerannegentle_: if you now approve https://review.openstack.org/#/c/64561/, infra can approve https://review.openstack.org/#/c/64567/16:31
Ajaeger(review and approve I mean)16:31
AjaegerI'm removing the WIP...16:31
fungibknudson: latest pip no longer follows external links by default (requirements listed in pypi but not hosted on pypi)16:31
bknudsonfungi: maybe it works locally because of a local pip cache and previous tox runs with different config?16:32
fungiNobodyCam: i want to say there was a new release and we ninja-pinned it in openstack/requirements last week16:32
bknudsonfungi: did pip change just in the last couple of days?16:32
fungibknudson: quite possibly16:32
* fungi checks16:33
*** senk has quit IRC16:33
sdaguefungi: I think we have some bad unit test nodes16:33
fungisdague: have a log?16:33
*** senk has joined #openstack-infra16:34
annegentle_Ajaeger: testing 64561 now16:34
sdaguehttps://jenkins01.openstack.org/job/gate-nova-python27/14588/console16:34
sdaguefungi: go look at http://status.openstack.org/zuul/16:34
sdaguethere is a really odd pattern there16:34
fungiNobodyCam: nevermind--i thought we pinned something, but looking back at global-requirements.txt i guess not16:34
sdaguewhich is something very systematic16:34
Ajaegertox did not install the package properly, any ideas? http://paste.openstack.org/show/59642/16:35
NobodyCamfungi: https://review.openstack.org/#/q/status:open+project:openstack/ironic,n,z note the first three reviews16:35
*** hashar has quit IRC16:35
sdaguemaybe not bad nodes16:35
sdaguenova and cinder aren't passing any of docs, pep8, or unit tests16:36
fungisdague: yeah, looks like something's trying to globally install pbr as part of the job (rather than into a virtualenv) seeing if i can narrow it down16:36
devanandaironic's failing all tests16:36
devanandalooks like MySQL-python?16:37
devanandafungi: http://paste.openstack.org/show/59643/16:38
sdaguewell there was a new MySQL-python release today16:38
fungiheh... /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_poimts'16:38
sdagueso that's a good chance that it's the break16:38
Ajaegerfungi: any idea what could be wrong with the package http://paste.openstack.org/show/59642/ - or is there a better place to ask?16:39
jeblairfungi: where do you see poimts?16:39
HenryGgit.openstack.org is not responding to my requests ?16:40
fungijeblair: jsonpatch seems to have a broken setup.py, but i think it's unrelated to the issue16:41
*** pcrews_ has joined #openstack-infra16:41
anteayaHenryG: I can open it in a browser, what is the request you are trying?16:41
jeblairsdague, fungi: i don't see a new mysql-python on the mirror...16:42
jeblairsdague: also, 1.2.4 looks like it was released last yer16:42
jeblairyear16:42
HenryGgit remote update (on neutron). But it finally came through after about 3 or 4 minutes16:42
fungijeblair: pip 1.5 was released today. i think it may not dtrt for us under tox?16:43
sdaguejeblair: oh, right, wrong year :)16:43
jeblairheh16:43
jeblairHenryG: the git farm doesn't seem unusually loaded16:43
fungidstufft: mordred: seems we're quite possibly suffering some major breakage with today's pip 1.5 release under tox (seems to be trying to globally install things rather than using a virtualenv?)16:44
sdaguenice...16:44
sdagueso why is it only breaking on nova derived projects?16:44
fungisdague: perhaps system packages16:45
jeblairthe errors i'm seeing are due to mysql-python16:45
* anteaya tries git remote update on neutron16:45
fungisdague: er, sitepackages16:45
jeblairs/due to/triggered by/16:45
*** gyee_ has joined #openstack-infra16:45
anteayaHenryG: there were some pauses, but finished by a count of 1016:45
jeblairwhich is used by nova and ironic16:45
anteayaHenryG: might it have something to do with your proxies?16:46
fungijeblair: the log sdague linked me to was pip trying to write to /usr/local/lib from the requirements installation phase of a unit test job16:46
jeblairfungi: which one is that?16:46
fungijeblair: https://jenkins01.openstack.org/job/gate-nova-python27/14588/consoleFull16:46
jeblairfungi: https://jenkins01.openstack.org/job/gate-nova-pep8/17283/consoleFull is typical of the widespread failures seen currently on zuul16:46
HenryGanteaya: it seems back to normal now16:46
* fungi looks16:46
anteayaHenryG: very good16:46
jeblairfungi: i think the different jobs may be failing in different ways16:47
jeblairfungi: i think my link is typical of a pep8 failure, and sdague's is typical of a unit test failure16:47
jeblairpossibly just due to installation order?16:48
*** rongze has quit IRC16:48
sdaguejeblair: probably16:48
sdaguethat would definitely be a nasty change of default interaction between pip and tox though16:49
anteayahere's my cab16:49
*** rongze has joined #openstack-infra16:49
sdagueanteaya: enjoy the flight16:49
fungijeblair: the recursion there makes me wonder if pip 1.5 is tripping over mysql-python's direct declaration of distribute as a requirement in ways which 1.4.x did not16:49
anteayasdague: thanks16:49
jeblairfungi: yeah; maybe we have multiple problems with pip 1.516:50
fungijeblair: well, another issue with 1.5 was also just pointed out by bknudson (no longer following external links without adding an option to enable that behavior)16:50
dstufftfungi: 1.5 no longer hits external urls16:50
funginot so much an issue as something we need to take into account for non-mirror-enforced projects16:51
jeblair*nod*16:51
fungidstufft: the other issues we're seeing so far are pbr under tox is trying to write to /usr/local/lib when sitepackages=True, and hitting max recursion depth on mysql-python's attempt to install distribute as a requirement16:51
dstufftfungi: got a log by any chance?16:52
fungidstufft: in fact i do16:52
fungidstufft: https://jenkins01.openstack.org/job/gate-nova-pep8/17283/consoleFull is the mysql-python/distribute recursion16:53
fungidstufft: https://jenkins01.openstack.org/job/gate-nova-python27/14588/consoleFull is the pbr trying to write to system directories behavior16:53
*** tma996 has quit IRC16:55
fungijeblair: sdague: my concern is that our modus operandi for pinning misbehaving dependencies while we work through the issue won't be possible for pip, since we install is separately from normal dependencies16:55
fungier, install it separately16:55
sdaguefungi: agreed16:56
sdaguethis one we're going to need to leave broken until we have a real fix16:56
bknudsonHow do I get tox to use the new pip? I did "sudo pip install pip --upgrade" but "tox -r -e pep8" still shows "pip 1.4.1 from /home/bknudson/dev/reviewday/.tox/pep8/lib/python2.7/site-packages (python 2.7)"16:56
*** zzelle has joined #openstack-infra16:56
fungibknudson: you need to upgrade it with './.tox/pep8/bin/pip install pip --upgrade' instead16:57
russellbfungi: may want to broadcast a status message with openstackstatus16:57
fungirussellb: thanks for the reminder--will do16:57
fungi#status alert gating is severely hampered while we attempt to sort out the impact of the pip 1.5 release... no ETA for solution yet16:58
openstackstatusNOTICE: gating is severely hampered while we attempt to sort out the impact of the pip 1.5 release... no ETA for solution yet16:58
*** ChanServ changes topic to "gating is severely hampered while we attempt to sort out the impact of the pip 1.5 release... no ETA for solution yet"16:58
russellbgreat thanks :)16:58
fungibknudson: and then try running tox without -r (since that may cause it to recreate it with older pip)16:59
bknudsonfungi: right, but then it doesn't try to reinstall the deps which is where it looks like the problem is.16:59
dstufftfungi: did you by any chance happen to upgrade virtualenv?17:00
*** branen has joined #openstack-infra17:02
fungidstufft: i'll check17:02
fungibknudson: oh, good point17:02
jeblairfungi: pip installs virtualenv via puppet, right?17:02
fungijeblair: yes, in modules/openstack_project/manifests/base.pp17:04
zaromorning17:04
fungidstufft: yes, pip freeze on our slaves reflect the same version as the latest virtualenv on pypi.python.org17:05
dstufftfungi: I'm pretty sure that the MySQL-python problem is a setuptools problem17:05
fungidstufft: jeblair: also released toay17:05
fungier, today17:05
dstufftand you're getting the latest setuptools because we updated setuptools inside of virtualenv17:06
*** garyk has quit IRC17:06
dstufftor rather it's a problem with MySQL-python trying to force distribute17:06
dstufftand the latest setuptools changed something that is breaking that17:06
fungidstufft: makes sense. i think we'd already determined that mysql-python is broken for declaring a distribute requirement?17:06
fungiyeah, that17:06
fungiso maybe this will force their hand in ways which our pestering did not17:07
fungi#status alert gating is severely hampered while we attempt to sort out the impact of today's pip 1.5/virtualenv 1.11 releases... no ETA for solution yet17:07
openstackstatusNOTICE: gating is severely hampered while we attempt to sort out the impact of today's pip 1.5/virtualenv 1.11 releases... no ETA for solution yet17:07
*** ChanServ changes topic to "gating is severely hampered while we attempt to sort out the impact of today's pip 1.5/virtualenv 1.11 releases... no ETA for solution yet"17:07
dstufftfungi: the pbr error happens during the execution of pbr's setup.py17:11
dstufftfungi: so that's either a setuptools error, or due to the nature of pbr, it's an error within pbr itself17:11
dstufftpossibly an error the way pbr calls pip17:11
jeblairdstufft: which error is this?  (can you give a log link?)17:12
dstuffthttps://jenkins01.openstack.org/job/gate-nova-python27/14588/consoleFull17:12
fungidstufft: i didn't mean to imply that pip is broken, just that the behavior changes will need to be sorted out17:12
dstufftfungi: oh I know :)17:12
dstufftJust trying to narrow things down17:12
fungidstufft: appreciated17:13
mordredcan someone point me at the pbr error?17:13
*** ruhe has joined #openstack-infra17:13
jeblairhttps://jenkins01.openstack.org/job/gate-nova-python27/14588/consoleFull17:13
fungimordred: ^17:13
jeblairso i was chatting with mordred on the phone; he thinks it's unlikely that we will convince the python-mysqldb maintainer to make the distribute change (he has tried)17:14
fungimordred: so far i've only noticed that when pbr is being installed under tox with sitepackages=True, if that helps narrow it down17:14
mordredfungi: stellar17:14
jeblairand making a small fork of that to fix setup.py might not be a crazy idea17:14
fungijeblair: mordred: well, at this point it very well may be that 'pip install mysql-python' has ceased working entirely with pip 1.5, so maybe they'll reconsider their position17:15
*** lcheng has quit IRC17:15
mordredthat is correct - it flat does not work - just tested17:16
mordred  534  virtualenv testmysql17:17
mordred  535  source testmysql/bin/activate17:17
mordred  536  pip install mysql-python17:17
mordredfails with latest pip and latest virtualenv17:17
*** rongze has quit IRC17:17
fungithey are now down to either revisiting their insistence on that choice, or deciding to no longer support installation via latest pip17:17
*** rongze has joined #openstack-infra17:18
*** rongze has quit IRC17:18
*** jaypipes has joined #openstack-infra17:18
dstufftmordred: fungi locally it doesn't work with pip 1.4.1 && setuptools 2.0.2 either17:19
dstufftnor does it work with easy_install17:19
*** jaypipes has quit IRC17:19
*** rongze has joined #openstack-infra17:19
zzellehello, i am not sure but it seems pbr use --index-url or --extra-index-url ?17:19
fungiwow, so basically they've simply alowed mysql-python to completely bitrot into uninstallability17:20
dstufftnor does it work with ``python setup.py install``17:20
zzellewhich is no more supported by pip in version 1.517:20
dstufftzzelle: hrm?17:20
dstufft--index-url and --extra-index-url are still supported in 1.517:21
mordredzzelle: and pbr only uses those in its integration tests17:21
mordrednothing in execution17:21
zzelleoups, need to sleep17:21
zzellei read half words17:21
fungizzelle: i think you read the release notes backwards... the entry which mentions those is saying --use-mirrors, -M, and --mirrors are no longer supported17:21
mordreddstufft, fungi: pbr installs into a clean venv with no problem with 1.5 - I have pbr installed globally, so I'm having a hard time testing with system-site-packages enabled :)17:22
mordreddstufft: "/usr/bin/python -c "import setuptools;__file__='/home/jenkins/workspace/gate-nova-python27/build/pbr/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/tmp.FeFChvCkt3/pip-v6F8TN-record/install-record.txt"17:23
mordreddstufft: why is it using /usr/bin/python there?17:23
mordredthis should be in a virtualenv17:23
dstufftmordred: oh hrm17:23
dstufftI missed that17:23
*** krtaylor has quit IRC17:24
dstufftmordred: it just uses sys.executable17:24
*** markmcclain has joined #openstack-infra17:24
dstuffttrying to think why that'd be /usr/bin/python17:24
mordreddstufft: shouldn't that be $venv/bin/python within a virtualenv?17:24
mordredotherwise, it's going to be doing entirely the wrong thing17:24
dstufftmordred: yes17:25
dstufftand it is in my test venv17:25
mordredsame here17:25
mordredfwiw, https://github.com/emonty/MySQLdb1 has two changes I made to the repo to 'fix' the distribute issues17:27
*** rwsu has joined #openstack-infra17:27
clarkbmaybe tox and new virtualenv dont do the right thing re sys.executable above17:28
*** ruhe has quit IRC17:28
dstufftfungi: mordred are you able to repro the pbr issues locally at all?17:28
mordreddstufft: I have not reproed the pbr issues - however, my patches on MyQLdb1 do allow it to be installed with new pip17:28
mordredwhich I have just verified17:28
sdaguefungi: if you have a moment - https://review.openstack.org/#/c/63889/ would be nice to land, to help the neutron team move forward on getting isolated tests working17:29
fungidstufft: well, i can reproduce it on one of our test systems, so should be able to duplicate the same scenario on a similar virtual machine17:29
fungidstufft: i'll see what i can work out17:30
clarkbsdague: fungi: I have approved17:31
sdagueclarkb: great17:31
sdaguethanks17:31
clarkb(don't know if the puppet stuff is sufficiently decoupled from tox to pass)17:31
fungisdague: 63889 needs a rebase17:31
fungisdague: layout.yaml was refactored/reorganized since that change was drafted17:32
sdagueok, let me do that17:32
*** senk has quit IRC17:32
*** rakhmerov has quit IRC17:33
openstackgerritSean Dague proposed a change to openstack-infra/config: Adds neutron-isolated to Tempest gate as experimental  https://review.openstack.org/6388917:33
*** jpich has quit IRC17:33
mordredhttps://github.com/farcepest/MySQLdb1/pull/4417:34
mordredthere is a pull request for it17:34
mordredhowever, if we get no traction with that, I still contend that, since it's an almost unmoving project, we could do a shallow fork, release it as python-mysqldb and change our stuff to consume that instead if we can't get andy to merge that and cut a new release17:35
dstufftlol17:35
dstufftMysql-python and python-mysql17:35
mordredright?17:35
mordredwell, python-mysqldb, since that's what the module is actually called17:36
fungii guess the two would be runtime-compatible so distros could substitute the not-fork when running our software17:36
mordredyup17:37
*** lcheng has joined #openstack-infra17:37
fungiassuming all which gets altered is setup.*17:37
mordredI have NO INTEREST in touching the code17:37
fungimordred: though SpamapS would apparently love some help maintaining the debian packages for mysql if you get time ;)17:38
openstackgerritA change was merged to openstack-infra/config: Adds neutron-isolated to Tempest gate as experimental  https://review.openstack.org/6388917:38
*** krotscheck has joined #openstack-infra17:39
*** DennyZhang has joined #openstack-infra17:39
SpamapSyes please :)17:39
SpamapSA few people have stepped up of late.17:39
SpamapSThough I think we have a new set of secret oracle CVE's for 5.5 coming soon. :-/17:40
*** jaypipes has joined #openstack-infra17:40
fungiaren't all of oracle's cves effectively secret (or at least so opaque as to be useless)?17:41
mordredhttps://github.com/emonty/MySQLdb1/tree/python-MySQLdb17:42
mordredthere is a branch that we can cut a fork from, should we decide to go that route17:42
*** rongze has quit IRC17:45
dstufftmordred: hrm17:46
mordredI'd rather get upstream fixed though17:47
dstufftmordred:17:47
dstufft2014-01-02 15:58:13.628 | py27 installdeps: -r/home/jenkins/workspace/gate-nova-python27/requirements.txt, -r/home/jenkins/workspace/gate-nova-python27/test-requirements.txt17:47
dstufft2014-01-02 15:58:13.629 | WARNING:test command found but not installed in testenv17:47
dstufft2014-01-02 15:58:13.629 |   cmd: /usr/bin/pip17:47
dstufft2014-01-02 15:58:13.629 |   env: /home/jenkins/workspace/gate-nova-python27/.tox/py2717:47
dstufftfrom the pbr log17:47
dstufftit looks like tox was executing the global pip17:47
dstufftbecause it didn't find a pip installed into the venv?17:47
mordredoh. because of --system-site-packages17:47
zarofungi: would you be able to add me to gerrit admin group on review-dev?17:47
*** rongze has joined #openstack-infra17:47
dstufftmordred: this should be outside of that, this is pure $PATH I think17:48
fungizaro: sure thing17:48
mordreddstufft: nah - I just had that problem locally - because of system-site-packages, and because my global env has latest pip17:48
mordredI can't install pip into the virtualenv17:48
mordredlike, it used to automatically install one - but as best I can tell, it's letting system-site-packages allow it to not do that now17:49
dstufftmordred: ugh17:49
chmouelwhen I send a commit for review is the commit get pushed to github before it gets approved?17:49
openstackgerritMatt Farina proposed a change to openstack-infra/config: New project request: PHP-Client  https://review.openstack.org/6206917:49
mordredwhich honestly breaks everything17:49
mordredso I think we can consider taht a but in virtualenv perhaps17:49
dstufftmordred: this was probably a regression because we switched virtualenv from sdist to wheel17:49
fungizaro: i've added you just now17:49
mordredchmouel: the commit can be found on github by its sha17:49
mordreddstufft: yah. that makes sense17:49
*** harlowja has joined #openstack-infra17:49
chmouelmordred: ah yeah i just figured that, how's that work?17:50
mordredthe wheel code paths are a little different17:50
zarofungi: thanks.17:50
chmouelmordred: if it's not approved it will be orphas there?17:50
chmouelmordred: orphans17:50
mordredchmouel: yes.17:50
mordredchmouel: https://github.com/openstack/swift/commit/$sha17:50
chmouelmordred: yep figured from my example to https://github.com/openstack-dev/devstack/commit/00b434182e3c04976e03b94490359fa26e71ef6917:51
mordredshould show you any commit that's been pushed up to gerrit for swift, even if it's not merged17:51
fungichmouel: or https://git.openstack.org/cgit/openstack/swift/commit/?id=$sha17:51
chmoueli just wasn't expecting that17:51
mordredfungi: I believe with 2.8 we can actually change the gitweb urls to cgit perhaps17:51
mferfolks, i have a half dev half infra question... in Go (and i'm looking at golang-client on stackforge) you can have remote projects. When doing that you reference the full uri for the import path and Go can fetch and manage it for you. For example, github.com/stackforge/golang-client. Is it ok to point the full import path (which will need to be in the codebase itself) to github?17:52
chmouelfungi: nice, it makes nicer for code review on large patch i think17:52
clarkbmordred: I think we may need to set the slash escaping settings on apache for cgit too (maybe)17:52
mordredmfer: if it's something that you ever expect to use inside of infra systems17:52
dstufftmordred: can I bother you to raise an issue for virtualenv?17:52
mordredmfer: you should reference git.o.o instead, because github goes down17:52
mordreddstufft: I will do that right now17:52
mordredmfer: however, it's a stackforge project, so honestly you can do github if you want to - just know that you'll hit intermittent testing issues because of network17:53
*** derekh has quit IRC17:53
mordreddstufft: https://github.com/pypa/virtualenv/issues/52117:54
mfermordred can't really use openstack testing infra right now it seems. I've already investigated that :/17:55
chmoueljeblair: you are mentioning here https://review.openstack.org/#/c/62377/ that bug #124256 is a blocker for new projects on stackforge but this seems to go to a gnome bug17:55
uvirtbotLaunchpad bug 124256 in gnome-app-install "gnome-app-install crashed with TypeError in show_description() (dup-of: 122914)" [Undecided,New] https://launchpad.net/bugs/12425617:55
uvirtbotLaunchpad bug 122914 in gnome-app-install "gnome-app-install crashed with TypeError in show_description()" [Undecided,Fix released] https://launchpad.net/bugs/12291417:55
mordredjeblair: based on the above ^^ your suggestion on the phone of attempting to pin virtualenv at an earlier version until that gets fixed may be the right short-term choice17:55
*** garyk has joined #openstack-infra17:57
*** DennyZha` has joined #openstack-infra17:57
jeblairchmouel: sorry, https://bugs.launchpad.net/openstack-ci/+bug/124256917:57
uvirtbotLaunchpad bug 1242569 in openstack-ci "manage-projects error on new project creation" [Critical,In progress]17:57
fungimordred: yeah, so i can confirm that trying to tox -epy27 in a fresh nova clone is ending up with no pip installed at all in the virtualenv17:57
clarkbfungi: jeblair: if we can add a project or two we should be able to confirm 1242569 is actually fixed now17:58
chmoueljeblair: ok thanks, I updated the review17:58
clarkbfungi: mordred: the other option (not really easy) is to use cloud archive libvirt and libvirt-python17:59
*** DennyZhang has quit IRC17:59
clarkband stop using system packages completely17:59
mfermordred what i really needed was a holiday break. i just had a realization around my problem with it on the git infra! yayz17:59
mordredclarkb: well, hrm. that is an option, and it is a place we want to be eventually17:59
mordredI think all three need to happen18:00
fungimordred: also, as for --allow-external, adding that to the install_command in tox.ini doesn't seem to be causing it to get picked up, which i think we're going to need to get working if we expect people to be able to run tests under tox on their own systems without our mirror18:00
mordredfungi: wow, really?18:00
mordredinstall_command should be used verbatim18:01
fungii'm trying again with tox -r now just to make sure18:01
*** DennyZha` has quit IRC18:01
zulhmmm...cloud archive?18:01
fungimordred: still seems to not work18:01
fungiinstall_command = pip install --allow-external -U {opts} {packages}18:01
mordreddstufft: that's the right incantation right ^^ ?18:01
fungi[...]18:02
fungiSome externally hosted files were ignored (use --allow-external to allow).18:02
dstufft--allow-extenral PACKAGENAME18:02
mordredoh wow18:02
dstufftthere's also an --allow-all-external18:02
dstufftI think it's called18:02
mordredah - we shoudl start with that for now, unless we know all of the external packages we need to whitelist18:02
fungiaha!18:02
dstufftmordred: you'll probably need --allow-unverified too18:03
fungiyeah, trying18:03
dstufftwhich doesn't have an -all- variant18:03
dstufftbecause it's a security sensitive switch18:03
mordredI am in support of these switches in general - it looks like we might need to just list each package both ways explicitly rather than with -all for ease of tracking18:04
mordreddo we get a list of which external packages were skipped?18:04
fungimordred: not without iteration. it bails on the first failure18:04
fungii'll work up a list for nova18:04
mordredfungi: sweet. thanks. I believe there should only be four, based on last time we looked in to this18:05
mordreddstufft: with 1.5, wheels are on by default, right?18:05
Alex_GaynorWe added a new one in at some point recently18:05
Alex_Gaynormordred: yes18:05
clarkbwe shouldn't need any external right? the mirror protects us18:06
clarkbat least for projects that only use the mirror18:06
clarkboh but then everyone would have to use the mirror...18:06
mordredright18:06
mordredwe don't need the list for the gate, but we'll need it for devs18:07
mordredOH18:07
mordredwe'll need to update the mirror script with a whitelist18:07
mordredso that it doesn't bomb out trying to fetch externally hosted deps18:07
fungiyep18:07
*** svarnau has joined #openstack-infra18:07
zarofungi: how is contrib agreement setup in gerrit 2.4?18:08
mordredhttps://etherpad.openstack.org/p/pip1.5Upgrade18:08
mordredI'm starting a list18:08
zarofungi: just noticed that it's now in the project.config in ver 2.818:08
dstufftmordred: sorry for that :)18:09
*** rongze has quit IRC18:09
fungizaro: see the etherpad--i've already updated it with the issues i identified for the cla a couple days ago18:09
mordreddstufft: no worries! I'm just glad the pbr error wasn't a pbr error :)18:09
mordredjeblair, fungi: I think I've captured each of the things we've talked about there, yeah? have I missed one?18:12
mordredzul: we were discussing cloud archive as a way to potentially accelarate getting a new enough libvirt that we can start using the new python-libvirt from pypi and stop doing system-site-packages in our tox.ini configs18:13
jeblairmordred: i missed something; what did you determine the cause of the "pbr error that wasn't a pbr error" to be?18:13
fungimordred: what did the workaround for "error: could not create '/usr/local/lib/python2.7/dist-packages/pbr': Permission denied" end up being?18:13
mordredjeblair: bug in virtualenv18:13
mordredjeblair: (bullet 1)18:14
zarofungi: ok. that looks good. i think one that additional thing that needs to be done is to add the a 'contributor agreement' section to project.config because the upgrade does not do that for you.18:14
zarowill add my notes as well.18:14
jeblairmordred: got it, thx; fungi i think that's the answer to your question too, right?18:14
fungizaro: the upgrade did in fact add that section based on the db contents. i checked18:14
zulmordred:  ah...well trusty has 1.2 and will have libvirt-python packaged (just maing its way to the archive)18:15
zulthat reminds me devstack will be needed to be updated as well18:15
fungijeblair: possibly. though downgrading virtualenv may not allow me to effectively iterate through the whitelist i need to build18:15
mordredjeblair: re: pinning virtualenv - I think that'll require manual effort on every slave18:15
jeblairmordred, fungi: virtualenv is installed by puppet via pip; how do we downgrade/pin that?18:16
fungiso i may resort to hand-patching the system installed virtualenv18:16
fungi(on my local test system18:16
fungi)18:16
zarofungi: in all-projects ?  i don't see it.18:16
jeblairmordred: does the pip provider in puppet accept a version argument?18:16
fungizaro: git clone /home/gerrit2/review_site/git/All-Projects.git && cd All-Projects && git fetch origin refs/meta/config && git checkout FETCH_HEAD18:16
mordredjeblair: I'm not sure there is a way to tell puppet to downgrade an installed pip package. I believe that we can express it in the puppet as "installed" rather than "latest" to keep puppet from borking us18:16
mordredjeblair: I _think_ it does - but I'm not sure I've ever tested whether it is smart enough to downgrade18:17
AaronGrmordred: yes, put a version in place of "latest"18:17
zarofungi: ohh i'm just looking at the gui.  so i guess it doesn't appear on the gui.18:17
AaronGrmordred: (provided you're using the system's package manager to install it)18:17
mordredAaronGr: we're not - we're using pip18:18
*** zzelle has left #openstack-infra18:18
jeblairmordred: okay, so manually downgrade on >50 machines, and either (a) set to installed, or (b) set to version [preferred if it works because it allows us to spin up new slaves with the correct config]18:18
jeblairfungi: what's the state of salt; can we execute the downgrade command on all the slaves using it?18:19
mordredjeblair: ++18:19
fungijeblair: i believe it should work, yes18:19
* fungi checks to see whether it's remained sane since last look18:19
jeblairmordred: what's the version we need?18:20
clarkbugh run_tests.sh18:20
*** johnthetubaguy has quit IRC18:20
clarkbmordred: jeblair AaronGr putting a version in with the pip provider should work18:21
mordredjeblair: < 1.1118:21
mordredjeblair: 1.10.1 I believe is what we were using before this upgrade18:21
*** markmcclain has quit IRC18:21
clarkbmordred: why wouldn't it?18:21
clarkbthis is actually a fun side effect that pip --upgrade will also downgrade18:21
mordredoh - will it?18:22
clarkbpretty sure it will18:22
jeblairpatching puppet now18:22
mordredyes. I verify that -U downgrades18:22
clarkbif pip --upgrade virtualenv==1.10.1 did not do the right thing it wouldn't work but it does do the correct thing18:22
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Pin virtalenv to 1.10.1  https://review.openstack.org/6470918:22
*** hogepodge has joined #openstack-infra18:23
jeblairclarkb, mordred, AaronGr: ^18:23
mordredok. that gets us past that problem18:23
mordredwhat do we want to do about mysql?18:23
* mordred needs to get on the road, will have internet while in car - but would love to do higher bandwidth on this particular point18:24
clarkbmordred: we could just switch to supporting only postgres18:24
clarkb(sorry couldn't resist)18:24
mordredheh18:24
mordredfungi is working on 3 - 4 is longer term18:25
mordreddhellmann: any chance you know/have pull with Andy?18:25
fungithis seems to work... sudo salt precise10.slave.openstack.org cmd.run 'pip install -U virtualenv\<1.11'18:26
mordredfungi: great. I _think_ the puppet change will just handle it - but good to know that's a good backup if it doesn't18:26
fungivirtualenv==1.10.118:26
fungimordred: all the slaves seem to be responding as functional salt minions still, so s/precise10/*/ there ought to blast it everywhere18:27
mordredcool18:27
dhellmannmordred: sorry, I don't think so18:28
*** mdenny has quit IRC18:28
*** david-lyle has quit IRC18:28
openstackgerritA change was merged to openstack-infra/config: Pin virtalenv to 1.10.1  https://review.openstack.org/6470918:28
dhellmannmordred: oh, wait, I do think I know him18:29
*** mdenny has joined #openstack-infra18:29
*** branen_ has joined #openstack-infra18:29
*** david-lyle has joined #openstack-infra18:29
dhellmannmordred: I'll email him18:29
mordredjeblair: ok. I've done the python setup.py register for python-MySQLdb and have given openstackci full access to the account. I have uploaded the 'fixed' release. I went ahead and did that in case you guys decide to move forward with that plan and I'm unreachable18:31
mordredmy preference is that we delete that and fix things - but I'm about to be driving through the desert18:32
fungimordred: yeah, so trying to work around the virtualenv issue, because pip 1.4.1 doesn't grok --allow-unverified i can't effectively work out a whitelist purely by running tests under tox (it seems to refuse to use the virtualenv's pip even if i install one into it)18:32
clarkbmordred: when do you get on a plane for lca?18:32
*** branen has quit IRC18:32
mordredclarkb: not until Monday - I had a thing come up18:32
jeblairmordred: yeah, let's not bother importing it into gerrit; this shouldn't be maintained in the long run.18:33
mordredfungi: how about make a virtualenv without system-site-packages, and then just do pip install -U -r requirements.txt by hand into it?18:33
mordredjeblair: kk18:33
*** rakhmerov has joined #openstack-infra18:33
mordredjeblair: I did also tag the commit I used as 1.2.418:33
jaypipesclarkb: all of that makes sense, only the failed import of pyudev is also present when using tox on a fresh checkout of Neutron....18:33
jeblairmordred: so now we propose a change to requirements to switch to it, yeah?18:33
clarkbjaypipes: really? it isn't for me and that is the only difference other than possibly a distro thing18:33
fungimordred: well, trying to iterate on nova which will need sitepackages=True i think, but maybe i don't care since i just want to see deps install18:34
jaypipesclarkb: so I don't think the setup_hook is working as you state... I'm not sure what is different b/w the gate and my machine :(18:34
clarkbjaypipes: that is a possibility18:34
jaypipesclarkb: I will try and do some more digging.18:34
jeblairmordred: do you want to go aheand and do that, or are you signing off now?18:34
clarkbit does work on my test machine and in the gate. both sets of machines are ubuntu 12.0418:34
fungimordred: also, i've confirmed that virtualenv 1.10.1 actually *downgrades* the virtualenv's pip to 1.4.1 even if i install 1.5 in it by hand18:35
mordredjeblair: yeah. which is going to be tricky, because we're going to need to fix the mirror script, probably, due to the verified things18:35
jaypipesclarkb: yes, I'm on 13.04 as you know... not sure why/if that has anything to do with it..18:35
mordredfungi: yah. you'llk need to do $venv/bin/pip install -U pip18:35
*** praneshp has joined #openstack-infra18:35
*** sarob has joined #openstack-infra18:35
fungimordred: i did. running tox after that re-downgrades it18:35
mordredfungi: oh, right. I'm saying skip tox18:35
jeblairmordred, fungi, clarkb, dstufft, Alex_Gaynor: one of us should probably summarize the python-mysqldb situation to the mailing list; do any of you want to do that, or shall i?18:36
fungiahh, yeah, i mainly wanted to confirm i had the tox.ini correct, but maybe i just wing that part18:36
mordredjeblair: I can take a stab at it real quick18:36
jeblairmordred: ack, thx.18:36
*** markmcclain has joined #openstack-infra18:37
*** rakhmerov has quit IRC18:38
fungii'm being hounded for the past two hours to take a break to go out to lunch, so i'm going to need to pick the --allow-unverified testing back up in an hour or so. i think it's unlikely to matter for real until we have a fixed virtualenv anyway18:38
*** ryanpetrello has quit IRC18:38
*** chandankumar has quit IRC18:38
fungi(a fixed new virtualenv with pip 1.5 embedded i mean)18:38
*** nati_ueno has joined #openstack-infra18:39
clarkbsdague: https://review.openstack.org/#/c/64321/ for when you get back from lunch18:40
*** rongze has joined #openstack-infra18:40
*** hogepodge has quit IRC18:40
* fungi heads out for a bit18:40
*** jroovers has joined #openstack-infra18:41
*** rongze has quit IRC18:45
*** hogepodge_ has joined #openstack-infra18:45
mordredjeblair: https://etherpad.openstack.org/p/WMxbH5GK6Z18:46
mordredjeblair: check me on that - also, we should probably send out an email about the virtualenv issue18:46
mordredjust for informational purposes18:46
openstackgerritA change was merged to openstack-infra/zuul: Add zuul version to status.json  https://review.openstack.org/6420918:47
*** nati_ueno has quit IRC18:47
jeblairmordred: lgtm18:47
mordredjeblair: sent18:48
*** jroovers has quit IRC18:48
jaypipesguh, clarkb and now I can't reproduce the issue locally after uninstalling python-pyudev and rm -rf'ing my .venv and .tox dirs... :(18:49
mordredok. I need to run - I'll get back online as soon as we're moving in the car18:49
*** nati_ueno has joined #openstack-infra18:49
*** crank has joined #openstack-infra18:49
*** jerryz has joined #openstack-infra18:50
*** nati_ueno has quit IRC18:54
*** praneshp has quit IRC18:55
*** vipul is now known as vipul-away18:55
*** vipul-away is now known as vipul18:55
*** reed has joined #openstack-infra18:56
clarkbjaypipes: so it works now? weird18:57
*** thedodd has quit IRC18:58
*** hogepodge_ has quit IRC18:59
*** sileht has quit IRC18:59
*** praneshp has joined #openstack-infra19:00
jaypipesclarkb: well, tests are still running, which is a good sign. :)19:02
jaypipesclarkb: I just have no idea why it works now and didn't earlier.19:02
jaypipesclarkb: I'll run a non-tox test after this and see if it's isolated to the venv tester.19:02
*** rongze has joined #openstack-infra19:03
*** markmcclain has quit IRC19:04
clarkbjaypipes: ok let me know how that goes19:04
jaypipeswill do!19:04
*** jooools has quit IRC19:04
*** hogepodge has joined #openstack-infra19:04
*** markmcclain has joined #openstack-infra19:05
*** sileht has joined #openstack-infra19:05
sdagueclarkb: so that seems fine to me, though have we actually used the crm114 data to create any finger prints yet?19:06
sdagueI'd like to know it was useful before overselling the idea19:06
clarkbsdague: I am not sure we have. It does pick up on stuff though. search error_pr:[-1000.0 TO -10.0]19:07
vishyIs this a known issue? The required version of distribute (>=0.6.28) is not available,19:08
vishytrying to install mySQL-python19:08
clarkbvishy: yup, mordred sent mail to the list about it19:08
clarkband see topic19:08
vishyok cool thx19:08
vishylooks like it got merged19:10
jeblairvishy: neat.  now we just need a release.  :)19:11
vishy:)19:11
*** rongze has quit IRC19:13
sdagueclarkb: yeh, so a browse of what's coming back looks like we're going to need some sort of tool to sift those results before they become useful.19:14
sdagueso I would couch that a bit more19:14
clarkbsdague: really? those results are pretty reliable from a this is different perspective19:15
clarkbsdague: the fact the logs are noisy is a different problem :)19:15
*** markmcclain has quit IRC19:15
jeblairclarkb: i like your idea of doing something like 's/[0-9a-f]+/TOKEN/', btw; did you say you planned on writing that?19:15
clarkbjeblair: I didn't but I am happy to take a stab at it19:16
jeblairclarkb: i think it will make it more 'meaningful'; and after all, that's pretty much how we read the logs anyway.19:17
*** markmcclain has joined #openstack-infra19:17
clarkbyup19:17
*** CaptTofu_ has quit IRC19:18
*** krtaylor has joined #openstack-infra19:18
clarkbsdague: I can see how it doesn't give you the magical fingerprint but it does point you in the right direction imo. For example the current situation, log lines for isntalls pop up with low error_pr indicating a problem with package installs19:18
*** CaptTofu has joined #openstack-infra19:18
sdagueclarkb: so I'm not convinced I've seen a win yet in that list in the first 4 pages of looking19:18
clarkbsdague: the first hit I have is a win19:19
clarkbsdague: gate-nova-python26 Installing collected packages blah blah blah blah19:19
sdagueok, I thought you said we didn't have any queries based on this yet?19:19
clarkbsdague: we don't but ^ would point you straight to the current problems19:19
jeblairit's still early days with basically no tuning done to it; i think the question isn't "does throwing this thing at the problem right out of the box magically work?" versus, "is this a potentially useful with some more research and tuning".19:20
sdagueright, which is why I just suggested downplaying it19:21
clarkbsdague: I can do that, but isn't that one of the goals of the thing (I think it is fine to point it out as a goal, I can make it clear we haven't actually done that particular thing yet)19:27
clarkbbefore I forget, did anyone catch my idea of preventing people from pushing to refs/drafts/*19:27
clarkbwe can't disable drafts but we can prevent people from creating them19:27
jeblairclarkb: i did not, but i think it is an idea whose time has come.19:27
clarkbit was late last night19:27
clarkbhttps://review.openstack.org/#/c/63567/ was updated but without a comment answering jog0 and my questions19:28
clarkbwhich led me to thinking about drafts and that was my idea, prevent them from ebing created19:28
sdagueclarkb: the problem with the installing collected packages line, is then you take the general part of that string and stick it back in as a fingerprint and it's 99% success. We'll probably want some sort of reverse transform here to figure out if there is a generic relevancy here.19:28
clarkbsdague: I don't think you will be able to just copy pasta19:29
*** thedodd has joined #openstack-infra19:29
clarkbsdague: but that line claerly indicates there is a package problem of some sort19:29
clarkbbecause successful jobs don't have anything like it19:29
clarkbit would be even better if it picked up on the error pbr couldn't install line19:31
*** CaptTofu has quit IRC19:33
*** sarob has quit IRC19:33
sdagueclarkb: success jobs don't have that exact package list, but they have "Installing collected packages:" lists19:33
*** sarob has joined #openstack-infra19:33
*** CaptTofu has joined #openstack-infra19:33
sdagueanyway, it's something we'll explore over time here. I just want us to be careful on expectations.19:34
clarkbsure19:34
*** rakhmerov has joined #openstack-infra19:34
clarkbsdague: my expectation isn't that you can copy paste those lines from elasticsearch into elasticrecheck19:34
clarkbinstead they are very good hints19:34
clarkbjeblair: re the draft thing, do you think we need to take a temperature on that or just do it for the greater good19:37
sdagueclarkb: re: draft, greater good :)19:37
sdaguedrafts are terrible19:37
jeblairclarkb: i think we should (a) test it so we don't eat crow if it doesn't work, then (b) announce it to the ml with an implementation date...19:38
*** rakhmerov has quit IRC19:38
*** sarob has quit IRC19:38
jeblairclarkb: i agree with sdague that i strongly think we should do it; announcing it in advance will give people notice and an opportunity for objection19:39
jeblair(some people may actually be using it successfully; if such a person exists, they'll need to change their workflow)19:40
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Zuul status: don't toggle on link click  https://review.openstack.org/6471619:41
jeblairAlex_Gaynor: ^19:41
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Zuul status: don't toggle on link click  https://review.openstack.org/6471619:42
jeblairnow without tabs19:42
clarkbjeblair: sounds good19:42
clarkbjeblair: forgive my js fail, but line 314 is event in scope?19:43
*** senk has joined #openstack-infra19:46
*** sarob has joined #openstack-infra19:47
*** CaptTofu has quit IRC19:49
*** markmcclain has quit IRC19:49
*** CaptTofu has joined #openstack-infra19:49
krotscheckclarkb: You mentioned at one point that you managed to get xvfb running on some of the builds - I'm digging through the configs and can't find a reference anywhere. Is that a config thing? Where should I be lookign?19:50
*** burt has joined #openstack-infra19:50
clarkbkrotscheck: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/run-selenium.sh19:51
krotscheckAwesome, thanks19:51
*** markmcclain has joined #openstack-infra19:53
openstackgerritDevananda van der Veen proposed a change to openstack-infra/config: Add ironic log files to logstash indexing  https://review.openstack.org/6471719:55
jeblairmikal: may i suggest that you could make your job names smaller by not including 'gate-' in them?  the prefix is a convention we use to indicate whether a job is intended to run in the check+gate queues, or only the check queue19:56
jeblairmikal: therefore it's not really applicable to turbo-hipster19:56
*** michchap has quit IRC19:58
clarkbdevananda: to double check, ironic uses oslo logging?19:58
Ajaegeranybody here that can help with a python packaging issue - or where should I ask? Looks pbr related19:59
Ajaegerrun tox I get during the install phase "UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 6938: ordinal not in range(128)"20:00
Ajaegerdetails at http://paste.openstack.org/show/59642/20:00
*** jgrimm has quit IRC20:00
*** lcestari has quit IRC20:05
clarkbAjaeger: it is creating a manifest file and somewhere in that list is an ndash20:06
clarkbAjaeger: pbr can probably fix this by encoding the results of find_sources or you can rename the file with the ndash20:07
jeblairAjaeger: what are the contents of /home/aj/Software/vcs/OpenStack/openstack-manuals/test-requirements.txt ?20:07
jeblair(i don't understand why pbr is running in git context instead of downloading the release)20:07
Ajaegeropenstack-doc-tools20:08
Ajaegeris the single line20:08
jeblairAjaeger: i can't reproduce that behavior locally20:08
AjaegerI can run openstack-doc-tools' setup.py just fine.20:08
Ajaegerjeblair: both annegentle_ and myself could ;(20:08
*** markmcclain has quit IRC20:09
devanandaclarkb: yes -- http://git.openstack.org/cgit/openstack/ironic/tree/openstack-common.conf#n1720:10
Ajaegerjeblair: do you have any idea how to debug this?20:10
*** markmcclain has joined #openstack-infra20:11
devanandaclarkb: also, those screenlogs won't exist outside of "check experimental" right now. pending work on tempest, etc...20:11
clarkbdevananda: thats fine, if the files don't exist they are ignored20:12
AjaegerThere are files with an ndash in openstack-manuals - could that be a problem?20:15
clarkbAjaeger: possibly, I wonder if it is running in git context because of the openstack-manuals git repo?20:15
*** michchap has joined #openstack-infra20:15
clarkbalso boo for unicode file names20:15
clarkband unfortunately mordred is afk right now so can't chime in about the git context vs package20:16
jeblairAjaeger: do you have a commit you're working on?  can you push it up for review and mark it wip so i can fetch it?20:17
Ajaegerjeblair, clarkb: I think I got it thanks to your help.20:18
Ajaegerto reproduce: checkout openstack-manuals and run "tox"20:18
AjaegerI just deleted the files with an ndash in it and everything works fine...20:19
AjaegerSo, that's what I will do now - rename these files...20:19
clarkbAjaeger: well I think that may fix a symptom of a bigger problem20:20
Ajaegerclarkb: what do you mean?20:21
clarkbAjaeger: pbr shouldn't be adding the openstack-manuals files to the openstack-doc-tools manifest20:21
clarkbit shouldn't be running in a git context at all20:21
Alex_Gaynorjeblair: thanks20:22
Ajaegerclarkb: no, it shouldn't - so, what shall I do? file a bug report against pbr?20:22
clarkbAjaeger: I think that is a good start20:22
clarkbinclude the paste that you had earlier showing the exception20:23
jeblairAjaeger: ah, yes, i can reproduce that20:23
fungiAjaeger: (but also, rename s/–/-/g on the filenames might be appreciated)20:23
fungisticking to strict ascii for filenames is probably still a good idea given the way macos hfs+ uses different utf-8 normalization than typical linux/bsd filesystems20:25
Ajaegerfungi: Yeah, I rename now, won't wait for pbr to get fixed...20:25
*** vipul is now known as vipul-away20:26
fungiheck, a hacking check for non-ascii filenames might even be a good idea (not that i expect docs repos to necessarily use hacking jobs, but just in general)20:26
*** vipul-away is now known as vipul20:26
Ajaegerjeblair, clarkb, fungi : Thanks for your help! I filled this bug now: https://bugs.launchpad.net/pbr/+bug/126562220:28
uvirtbotLaunchpad bug 1265622 in pbr "pbr running in git context when it shouldn't" [Undecided,New]20:28
AjaegerCould you add some more comments to it as appropriate, please?20:29
*** rakhmerov has joined #openstack-infra20:34
*** ruhe has joined #openstack-infra20:38
Ajaegerwhat an irony: None of these four files was used at all ;(20:38
*** ryanpetrello has joined #openstack-infra20:39
*** rakhmerov has quit IRC20:39
*** mattf has joined #openstack-infra20:43
mattffolks, i'm seeing "OSError: [Errno 13] Permission denied: '/usr/lib/python2.6/site-packages/Babel-0.9.4-py2.6.egg-info/top_level.txt'" from the jenkins gate at http://logs.openstack.org/40/64540/2/gate/gate-python-savannaclient-python26/2e617d0/console.html#_2014-01-02_20_38_30_44020:45
mattfanyone seen something similar / have a suggestion for me?20:45
*** milki has quit IRC20:45
*** milki has joined #openstack-infra20:46
*** fbo is now known as fbo_away20:47
*** CaptTofu has quit IRC20:47
AjaegerNow https://review.openstack.org/#/c/64567/ can be reviewed ;)20:47
*** CaptTofu has joined #openstack-infra20:47
dhellmanndid someone already notice that https://pypi.python.org/pypi/MySQL-python/1.2.5 was released with mordred's fix?20:50
jeblairdhellmann: nope that's news, thanks!20:51
*** Ryan_Lane has joined #openstack-infra20:51
dhellmannjeblair: I don't know if it fixes the problem, but it should help :-)20:51
*** loq_mac has joined #openstack-infra20:52
jeblairmattf: https://review.openstack.org/#/c/64709/ should have fixed that, but doesn't appear to have taken effect on the centos slaves20:55
*** vipul is now known as vipul-away20:55
jeblairfungi: ^ puppet is running but pip doesn't seem to be downgrading virtualenv20:55
*** johnthetubaguy has joined #openstack-infra20:56
mattfjeblair, you recommend i just try again tomorrow?20:56
jeblairmattf: yes, or in a few hours20:57
jeblairfungi: http://paste.openstack.org/show/59692/20:57
mattfwill do20:57
*** loq_mac has quit IRC20:57
jeblair  Requested virtualenv==1.10.1, but installing version 1.1120:58
jeblaira reason would be a really cool thing to include on that line.  :(20:58
*** loq_mac has joined #openstack-infra20:58
*** Ryan_Lane has quit IRC20:59
dstufftjeblair: what version of pip is it using to do the install20:59
jeblair[root@centos6-5 ~]# pip --version20:59
jeblairpip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)20:59
dstufftIt's hitting the existing build dir bug20:59
ruhejeblair, fungi: btw, is this how nodes get scripts updated http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodepool.py#n614 ?20:59
*** croberts has quit IRC20:59
dstufftthere's a 1.11 build dir laying around and it is installing that instead of deleting it and downloading 1.10.1 and building/installing it20:59
dstufftthat was fixed in 1.4 or 1.5 I think20:59
* dstufft heads out21:00
jeblairruhe: it's how cached images get updated, jenkins slave scripts included.  if you're interested in a particular script+job combo, i can help confirm that's the right place.21:01
jeblairdstufft: thanks21:01
*** smarcet has quit IRC21:01
*** UtahDave has left #openstack-infra21:02
*** ArxCruz has quit IRC21:02
ruhejeblair: i just want to understand the flow. once a change in the slave script is merged, how do active slave nodes get this update?21:02
jeblairruhe: which slaves, and which slave scripts?21:03
*** UtahDave has joined #openstack-infra21:03
*** thedodd has quit IRC21:05
jeblairsudo salt centos*.slave.openstack.org cmd.run 'rm -fr /tmp/pip-build-root/'21:05
jeblairsudo salt centos*.slave.openstack.org cmd.run 'pip install -U virtualenv==1.10.1'21:05
jeblairfungi, clarkb: ^ i ran that21:06
ruhejeblair: for instance any file under jenkins/files/slave_scripts get modified; i see that new images get these scripts installed with slave.pp; but i can't see how existing instances get this scripts updated21:07
*** Ryan_Lane has joined #openstack-infra21:07
jeblairruhe: those files are updated by puppet run via cron on long-running nodes, and on single-use nodes, new versions will go into effect when an image update runs and popuplates a new image with the current version21:09
jeblairruhe: so yes, for nodepool based nodes, that's how they get updated21:09
jeblair(once per day, currently)21:10
ruhejeblair: thank you :) that's what i wanted to confirm21:10
*** loq_mac has quit IRC21:11
*** croberts has joined #openstack-infra21:11
*** rongze has joined #openstack-infra21:12
fungijeblair: cool21:12
*** freyes_ has quit IRC21:12
fungijeblair: salt calls seemed to have worked so far?21:12
jeblairfungi: yeah, looks like they all reported back with sane output21:13
jeblairfungi: http://paste.openstack.org/show/59701/21:13
*** freyes_ has joined #openstack-infra21:14
*** CaptTofu has quit IRC21:15
fungijeblair: awesome. also glad to know that a stale /tmp/pip-build-root will cause it to reinstall old packages (though mildly worrisome that centos6 slaves had pip 1.3.1)21:16
*** rongze has quit IRC21:17
fungithat looks very similar to the caching issues we saw with the -b option21:17
AjaegerYeah, openstack-doc-tools is now checking for annegentle_ as well ;) Thanks again, fungi, clarkb, jeblair!21:17
annegentle_Thanks y'all!21:17
fungiannegentle_: Ajaeger: of course, glad we could help!21:17
jeblairfungi: yeah, i didn't want to open any new cans which may or may not have worms.21:18
*** vipul-away is now known as vipul21:18
*** johnthetubaguy has quit IRC21:18
*** syerrapragada1 has quit IRC21:19
*** johnthetubaguy has joined #openstack-infra21:19
fungiall our cans are full of worms21:19
*** loq_mac has joined #openstack-infra21:23
openstackgerritPeter Balland proposed a change to openstack-infra/config: Remove gates from Congress  https://review.openstack.org/6473021:27
*** croberts has quit IRC21:27
*** johnthetubaguy has quit IRC21:29
*** banix has quit IRC21:31
*** dprince has quit IRC21:33
*** ryanpetrello has quit IRC21:35
*** rakhmerov has joined #openstack-infra21:35
*** rakhmerov has quit IRC21:40
*** ruhe has quit IRC21:40
*** Ajaeger has quit IRC21:41
krotscheckCan I get a sanity check on a diff from anyone interested in the JS toolchain? http://paste.openstack.org/show/59703/21:42
krotscheckIt's a followup to "Hey we want to use node for builds but don't want to install it on our boxes."21:42
jeblairkrotscheck: sure; first thing is you could just push a change up for review; gerrit has a better diff interface :).  you can mark it 'work in progress' and ping people individually if you want to make it clear you aren't ready for it to be merged....21:45
*** croberts has joined #openstack-infra21:45
jeblairkrotscheck: i think the mechanics are good, but since this is a single-project solution, it might make more sense to live as a script in the storyboard repo21:46
krotscheckjeblair: Those were my thoughts at first, but when presented with the option I usually default to generic solutions.21:47
krotscheckI'll get it into gerritt21:47
krotscheck(What with rumors of horizon wanting to use this toolchain etc etc)21:47
*** vipul has quit IRC21:48
jeblairkrotscheck: interesting rumours; i'm not sure that fits with the project support policy, and so i would not expect this to be accepted in the openstack context.21:48
*** ryanpetrello has joined #openstack-infra21:48
*** ryanpetrello has quit IRC21:48
krotscheckjeblair: Project support policy?21:48
*** vipul has joined #openstack-infra21:49
jeblairkrotscheck: yes, we don't do things that break the current stable releases of ubuntu or centos21:49
*** ryanpetrello has joined #openstack-infra21:49
david-lylekrotscheck: is this change just for testing?21:49
david-lyleif it compiles anything, that likely won't fly21:49
*** ryanpetrello has quit IRC21:50
david-lylesorry haven't gone through the diff yet21:50
*** beagles has quit IRC21:50
jeblairkrotscheck: in practice, that means we do testing on plain installations of those operating systems (or with os-vendor-provided repos like ubuntu cloud archive)21:50
krotscheckdavid-lyle: It adds the javascript toolchain (npm/grunt/bower) to WORKSPACE/.local21:50
jeblairkrotscheck: so basically, the reason why we asked you to do it this way is _because_ we can't do it for openstack (eg, horizon)21:51
krotscheckdavid-lyle: It's intended for build tools & testing only, not as an actual server.21:51
*** ryanpetrello has joined #openstack-infra21:52
krotscheckjeblair: Wait a sec, if storyboard isn't actually openstack, then why are we using infra?21:52
jeblairkrotscheck: so that's why it's more appropriate to just be in storyboard's repo.21:52
jeblairkrotscheck: storyboard isn't part of openstack, it's part of openstack-infra21:53
krotscheckArgh. Semantics. Right21:53
jeblairkrotscheck: openstack-infra exists to support the development of openstack21:53
david-lyledifferent rules apply21:53
jeblairkrotscheck: not really; openstack is a cloud computing system that includes things like compute and storage management systems21:54
jeblairkrotscheck: at the moment, no one has suggested that it also include a bug tracker21:54
jeblairhowever, that's something that the project needs for its development efforts21:54
jeblairkrotscheck: so _openstack_, as a piece (or collection) of software has some requirements around what systems it targets and supports21:55
jeblairthat doesn't necessarily affect openstack-infra projects (some of which are even in java, if you can imagine such a thing), but we try to stay as close as possible21:56
krotscheckRight, I get all that (though I might not remember most of the details).21:56
jeblairkrotscheck: for instance, we only run servers on the latest stable releases of centos and ubuntu.  so running storyboard in production will eventually bring up similar issues...21:56
jeblairkrotscheck: it's easy if it depends on packages that are in the os repo.  if not, then we have to figure out how to install them on servers, and make sure they are automatically maintained and receive security updates.21:57
*** Ryan_Lane has quit IRC21:57
krotscheckSo, is there a distinction between "here's the tools needed to build the software" and "here's the operating system that the software needs to run on"?21:58
krotscheckOr do they all need to be the same for the sake of openstack (as opposed to openstack-infra)21:59
krotscheck(Just clarifying because the package output of storyboard is static HTML, JS, CSS, and images)21:59
krotscheck(Which only need to be hosted by insert-web-server-of-choice-here)22:00
jeblairkrotscheck: building and installing static files like that sounds fairly easy; builds and installs can definitely be distinct.22:01
*** mfer has quit IRC22:01
*** CaptTofu has joined #openstack-infra22:02
krotscheckjeblair: Right - I see a lot of post-build SCP/FTP in the config files.22:02
jeblairkrotscheck: yep, building and uploading the static files to tarballs.o.o and installing releases from there sounds like a good solution.22:04
*** vipul is now known as vipul-away22:04
*** vipul-away is now known as vipul22:04
krotscheckjeblair: So given that the intent of my changes is that it should _never_ run on openstack, even though it outputs assets that will... should I still pull that script back into the storyboard repo?22:04
jeblairkrotscheck: it outputs assets that will run on a server, not "assets that will run on openstack" (we run our servers on a provider that's running openstack, but that's beside the point)22:06
jeblairkrotscheck: but yes, i think it should be in storyboard22:06
krotscheckjeblair: Ok, so horizon's web interface runs on a server?22:07
krotscheck(It's all the same to me, I just feel that other projects might want to make the toolchain)22:08
krotscheck*make use of22:08
jeblairkrotscheck: i'm not sure what you're getting at.  my point was that storyboard is not a component of openstack, and it would be good to not be confused on that point.22:08
jeblairkrotscheck: openstack projects _may not_ make use of this toolchain22:08
krotscheckSo the horizon guys are dead in the water?22:08
jeblairkrotscheck: they aren't dead in the water, they have a working product.22:08
krotscheckHuhn. Interesting22:08
clarkbjeblair: worth pointing out they have active blueprints to work on this stuff iirc22:08
jeblairclarkb: neat.  they will have to address the support issue22:09
clarkbjeblair: I agree, just pointing out that this is a thing for horizon. https://blueprints.launchpad.net/horizon/+spec/jasmine-integration22:09
clarkbI am assuming tests that don't require a DOM want to run in nodejs or similar22:10
*** thedodd has joined #openstack-infra22:10
krotscheckWell, in this context nodejs is just the task runner (ivy equivalent). The tests themselves are run in [insert configurable browser here]22:12
*** rongze has joined #openstack-infra22:12
vipulso with pip 1.5, should we be removing d2to1 from the setup.py? seeing some wierd errors with our Trove packages22:13
clarkbvipul: d2to1 should've been removed months ago22:13
vipulclarkb: you're right, maybe it still lives in our internal packaging22:14
vipulthanks22:14
krotscheckjeblair, clarkb: General apologies for being obtuse about some of these topics btw. Y'all have a lot of institutional knowledge that I'm still absorbing.22:15
*** milki has quit IRC22:17
*** rongze has quit IRC22:17
jeblairkrotscheck: no apology necessary; it takes most people several months to come up to speed.22:17
*** milki has joined #openstack-infra22:17
*** Alex_Gaynor has quit IRC22:20
openstackgerritZane Bitter proposed a change to openstack-infra/config: Separate handling of ChangeIds and Git SHAs in Gerrit  https://review.openstack.org/6473822:20
*** Alex_Gaynor has joined #openstack-infra22:25
*** hogepodge has quit IRC22:26
*** UtahDave has quit IRC22:26
*** UtahDave has joined #openstack-infra22:27
*** croberts has quit IRC22:27
*** zehicle_at_dell has joined #openstack-infra22:29
*** zehicle has joined #openstack-infra22:29
devanandaclarkb: for this morning's everything-fail, what bug # should I be reverifying against?22:31
*** mrda has joined #openstack-infra22:32
*** markmcclain has quit IRC22:32
*** rwsu has quit IRC22:33
*** mkoderer_ has quit IRC22:33
*** Ryan_Lane has joined #openstack-infra22:34
clarkbdevananda: I am not sure that there is one. I arrived late to the party22:34
*** Ryan_Lane has quit IRC22:34
devanandaclarkb: i couldn't find one looking through status.oo/rechecks22:35
devanandaclarkb: but apparently reverify doesn't do anything without a bug, any more :(22:35
*** ryanpetrello has quit IRC22:35
clarkbdevananda: but you have approval powers just reapprove22:35
clarkb(admittedly a work around but an intentional one)22:36
devanandaah. wasn't sure that would kick the job off. thanks22:36
*** mkoderer_ has joined #openstack-infra22:36
*** Alex_Gaynor has quit IRC22:36
*** rakhmerov has joined #openstack-infra22:36
mriedemdevananda: it's bug 126564122:37
uvirtbotLaunchpad bug 1265641 in nova "Can not create pbr directory due to permission denied" [Undecided,New] https://launchpad.net/bugs/126564122:37
mattfjeblair, jenkins looks back to normal for me, thanks!22:37
devanandamriedem: hm. i think we saw the python-MySQL one in our pipe, for what ever reason.22:38
devanandabut thanks :)22:39
*** mattf has left #openstack-infra22:39
*** rwsu has joined #openstack-infra22:40
*** Alex_Gaynor has joined #openstack-infra22:41
*** rakhmerov has quit IRC22:41
openstackgerritSean Dague proposed a change to openstack/requirements: wip: totally speculative exploration of sqla 0.9  https://review.openstack.org/6474222:41
clarkbsdague: fingers are crossed here22:44
*** mkoderer_ has quit IRC22:48
*** weshay has quit IRC22:49
*** mkoderer_ has joined #openstack-infra22:52
Alex_Gaynorjeblair: new collapsed UI is awesome, thaknks22:55
jeblairrussellb: ^22:55
zaroclarkb, fungi: i fixed the contributor agreement problem on review-dev.o.o22:57
fungidevananda: mriedem: there are several problems all surfacing at once in the face of new pip and virtualenv releases22:58
fungizaro: did you simply have to create that group and populate it?22:58
*** rwsu has quit IRC22:59
zaroclarkb: i'm not seeing the contact info error you reported.22:59
clarkbzaro: let me try again23:00
*** mkoderer_ has quit IRC23:00
clarkboh it says I am verified now23:01
clarkbzaro: what was the fix?23:01
sdagueclarkb: yeh, don't cross your fingers on that one. I blows up really early with keystone db setup23:01
zarofungi: no, we already have a openstack-cla group.  i just changed the ACLs to allow that group access.23:01
zaroclarkb: ^23:01
clarkbzaro: ideally we would use whatever group gerrit wants to use23:01
clarkbzaro: I can submit contact info now too23:01
*** mkoderer_ has joined #openstack-infra23:03
*** sarob has quit IRC23:03
zaroI might be wrong, but i assumed that openstack-cla is the group that contains members who accepted the cla.23:03
fungizaro: openstack-cla is a different group, actually. that's a cruft hold-over from when we used to sync groups from lp and approve cla signatures there23:04
*** sarob has joined #openstack-infra23:04
fungizaro: it looks like gerrit wanted to create a different group (added a uuid for it in the groups file but hadn't successfully inserted it into the db)23:04
zarofungi: ok.  i'll revert and try the db route.23:04
jeblairyeah, if we need to provide our own group we can use openstack-cla (it's unused), but if gerrit wants to default to a different group name, let's use that23:05
*** eharney has quit IRC23:06
zaroclarkb: FWIW, i never had problems submitting contacts even before my latest changes to ACLs.23:06
clarkbzaro: did you do it through the CLA signing page?23:07
clarkbthat is where I had the error23:07
zaroclarkb: yes.23:07
fungizaro: jeblair: the project.confg and groups files refer to a "CLA Accepted - ICLA" which is simply missing from the account_groups and related db tables23:07
zaroclarkb: and the plugins are not installed on site creation.  need to manually add those.23:08
*** sarob has quit IRC23:08
jeblairfungi: got it; creating that group and using it makes sense23:08
fungii think we simply need to create it and add all accounts to to which were "V" status for cla id #2 in the account_agreements table23:09
fungier, to it23:09
clarkbzaro: is there any way to automate that? eg can puppet put some wars or jars in special dirs?23:09
clarkbzaro: manual is bad23:09
zaroclarkb: ohh, yeah. it's just a simple copy to plugins folder then restart gerrit.  should be easily puppitable.23:10
fungii'm going to guess for plugins we just need to duplicate the gerrit.war copying pattern but for the plugin files23:10
zaroclarkb: i'm just saying that gerrit does not do it for us.23:10
*** thedodd has quit IRC23:11
*** fifieldt has joined #openstack-infra23:11
*** rwsu has joined #openstack-infra23:12
*** rongze has joined #openstack-infra23:13
*** rongze has quit IRC23:18
sdagueI'm poking a review-dev a little and putting a couple of things over in the etherpad that are hopefully helpful in discovery23:18
jeblairsdague: cool, thanks23:19
clarkbwas the etherpad linked in here recently? my grepping doesn't show it23:20
jeblairclarkb: i made a blueprint that links to the etherpad so we don't lose it, but it's here: https://etherpad.openstack.org/p/gerrit-2.8-upgrade23:21
jeblairclarkb: (if you need to find it again in the future, you can start with the blueprint)23:21
pleia2clarkb: https://etherpad.openstack.org/p/gerrit-2.8-upgrade23:21
pleia2ah, jeblair beat me to it :)23:21
clarkbty]23:21
*** dcramer_ has quit IRC23:22
jeblairsdague: note https://review.openstack.org/#/c/60080/123:23
sdaguejeblair: yep, found it as you were typing about it23:26
mordredo/23:35
mordredjeblair: it seems that we have a new mysql-python in the wild23:36
mordreddhellmann: ^^ thank you!23:36
jeblairmordred: yeah!  i think the virtualenv pinning is masking the problem with 1.2.4 at the moment23:36
mordredjeblair: awesome23:36
mordredjeblair: I'm going to delete my python-mysqldb from pypi23:37
*** rakhmerov has joined #openstack-infra23:37
jeblairmordred: so it looks like the periodic mirror update job for 2.7 times out because it takes > 1 hour23:39
jeblairmordred: https://jenkins.openstack.org/job/periodic-mirror-python27/23:39
mordredjeblair: yes. I believe this is known - I thought we'd expanded the timeout for it23:39
mordredclarkb: ^^ didn't we extend the timeout?23:39
jeblairapparently not23:39
mordredjeblair: part of the issue is that we're now double-building everything due to also building wheels23:39
mordredjeblair: so the script has to do twice as much work23:40
openstackgerritA change was merged to openstack-infra/zuul: Update the url for the zuul git repo  https://review.openstack.org/6267623:40
jeblairmordred: py26 is 90m, py27 is 60m23:40
jeblairpy26 _takes_ 80m23:40
*** ArxCruz has joined #openstack-infra23:41
*** ryanpetrello has joined #openstack-infra23:41
*** rakhmerov has quit IRC23:41
clarkbjeblair: re new change screen I believe that is a non issue, it is user configurable whether you want old or new23:42
clarkbjeblair: under your user settings preferences23:43
openstackgerritSean Dague proposed a change to openstack-infra/config: Gerrit-2.8: Add secondary index support  https://review.openstack.org/6008023:43
sdagueso, that's a stab23:44
clarkbnow I see 3.0 will remove the old screen...23:44
jeblairclarkb: what's a non-issue?  radically changing the ui for the keystone of the developer workflow?  i care.23:44
clarkbjeblair: well I didn't realize we had to change it23:45
clarkbI thought both screens would live together happily ever after23:45
clarkb(it is currently toggleable, you have a choice today, but that choice will be removed apparently)23:45
jeblairclarkb: ah, yeah.  it'd be nice if we had more time, but we don't.23:45
*** sarob has joined #openstack-infra23:46
jeblairclarkb: so given that, i'd like to try to start seeing what we can do to accomodate it, but i'm starting to lean toward old-as-default for now so we have a smoother transition23:46
clarkb++23:47
*** sarob has quit IRC23:47
*** sarob_ has joined #openstack-infra23:47
sdaguecould we light the new screen on review-dev to see the difference?23:47
clarkbsdague: its lit, you can select it under your preferences23:47
sdagueok, cool23:47
mordredwhat's the change screen difference? ah! I will go look23:47
clarkbgerrit needs someone like krotscheck to fix the UI for them >_>23:49
jeblairclarkb: _we_ fixed the ui for them.23:49
sdaguewell, there are parts of this I like better23:49
clarkbjeblair: ya23:49
jeblairthen they broke it again23:49
*** UtahDave has quit IRC23:49
mordredyeah - I don't hate the new screen - but I haven't used it in anger much, so I don't have good opinions really23:49
mordredit is radically different23:50
clarkbsdague: you need to load up a change with a bunch of dependencies and revisions then try looking at it on a non widescreen monitor23:50
clarkbmordred: the biggest problem is it grows horizontally23:50
sdaguewould it be possible to import something like the tempest change history in? I feel like some of the UI only will show up with bigger data23:50
mordredclarkb: in what way?23:50
sdagueclarkb: you have an example?23:50
sdagueclarkb: also, define "non widescreen"23:50
clarkbsdague: https://gerrit-review.googlesource.com/#/c/53392/23:51
clarkbsdague: basically changes for gerrit itself23:51
clarkbsdague: I have a chromium instance fullscreened on a 16:10 monitor and if I get anywhere near 4:3 I end up with a horizontal scroll bar23:51
jeblairsdague: anything less than 1440 pixels23:52
mordredoh. that does horizontally scroll23:52
mordredgross23:53
mordredgod. there's also SO MUCH INFO on that screen that I don't care about23:53
jeblairsdague: current gerrit requires 1116 pixels (and that's only in side-by-side diff view23:53
jeblairmordred: yeah, that's one of the things i was getting at with "crowded"23:54
mordred++23:54
mordredwell... now that it's got a REST api and the rest api is getting comprehensive23:54
sdagueso how stable are those gwt table tags?23:54
mordredwe could just roll with old screen by default for 2.8 and by 2.9 have our own web ui :)23:54
clarkbmordred: because then upgrading to 3.0 is potentially a giant process again...23:54
krotscheckclarkb: Oh man, I have so many amazing ideas for gerrit's UI...23:55
mordredclarkb: yeah. I know. I'm offering bad suggestions. grasping at straws23:55
jeblairsdague: my understanding is most of the gwt stuff is not stable.23:55
mordredclarkb: I say that- are we expecting the REST API to diverge _wildly_ for 3.0?23:55
mordredI mean, other stuff would break pretty terribly if the api just up and broke23:56
jeblairmordred: i plan on writing a text-mode interface.  i can use that to implement the inbox style workflow i've always wanted.23:56
clarkbmordred: they don't have any real indication of stability for anything as far as I can tell23:56
mordredjeblair: ++23:56
sdagueclarkb: honestly, I'd say get to 2.8 / 2.9 as fast as possible stay on the old UI, and then we can at least speak a close language as them23:56
jeblairmordred: (possible bonus for you: i think it could work offline too; you could review while on a plane)23:57
sdagueand I agree, the UX on that page is pretty bad23:57
jeblairmordred: (a plane over the ocean)23:57
*** herndon_ has joined #openstack-infra23:57
mordredsdague: we can also explore alternate UIs, like jeblair's text mode, or krotscheck doing a thing or something23:57
sdagueyep, sure23:57
mordredbecuase trying to tie those together is a pretty big bite23:57
sdagueso yeh, I'd default old UI. It's sort of a shame we weren't closer to them already, because we have a more active gerrit then they do, and could provide them some interesting user feedback23:58
jeblairok, i changed my vote on https://review.openstack.org/#/c/60094/23:58
jeblairsdague: yeah, you'd think they'd care about our install.23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!