Friday, 2012-03-30

*** jcooley has quit IRC00:01
*** devananda has quit IRC00:03
*** maplebed has quit IRC00:07
*** Radium has quit IRC00:10
*** Lance604 has quit IRC00:15
*** crobinso has quit IRC00:16
*** berendt has quit IRC00:16
*** vricci has joined #openstack-dev00:16
*** samkottler has quit IRC00:18
*** jim_cooley has quit IRC00:19
*** jcooley has joined #openstack-dev00:19
*** novas0x2a|laptop has quit IRC00:22
*** samkottler has joined #openstack-dev00:24
*** novas0x2a|laptop has joined #openstack-dev00:26
*** roge has quit IRC00:29
*** belliott has joined #openstack-dev00:30
*** mnewby has joined #openstack-dev00:38
openstackgerritVerification of a change to openstack-dev/devstack failed: Fix quantum deps  https://review.openstack.org/592800:39
*** NaZZaX has joined #openstack-dev00:40
*** sandywalsh has quit IRC00:41
*** NaZZaX is now known as nicoatcisco00:45
*** nicoatcisco is now known as NaZZaX00:45
*** danwent has quit IRC00:51
*** sandywalsh has joined #openstack-dev00:54
*** troytoman-away is now known as troytoman00:56
*** danwent has joined #openstack-dev00:56
*** infernix has joined #openstack-dev00:57
*** danwent has quit IRC01:00
*** mnewby has quit IRC01:00
*** mnewby has joined #openstack-dev01:03
*** jcooley has quit IRC01:05
*** lts has quit IRC01:09
*** winston-d has joined #openstack-dev01:14
*** alrs has quit IRC01:17
*** benthomasson has joined #openstack-dev01:18
*** gabrielhurley has joined #openstack-dev01:19
*** paulstallworth has joined #openstack-dev01:20
*** benthomasson has quit IRC01:20
*** novas0x2a|laptop has quit IRC01:23
*** spiffxp has quit IRC01:27
dolphmgabrielhurley: i might have just been looking for you in the wrong channel :P01:34
gabrielhurleydolphm: when I'm on, I try to make sure I'm signed into both this channel and #openstack-horizon01:35
dolphmgabrielhurley: i probably looked in #openstack01:35
gabrielhurleyah, yeah, I'm not always in that one01:35
gabrielhurleyI saw you marked the "admin"-ness bug as confirmed for keystone. thanks01:35
dolphmgabrielhurley: yeah, i confirmed the behavior, which i'm pretty confident differs than pre-redux01:36
gabrielhurleyI'm like 99% sure01:36
gabrielhurleysolely because I spent so long trying to figure out the global vs scoped roles when I first wrote the keystone client01:36
dolphmgabrielhurley: but termie wrote the policy stuff in redux, so i'd like to poke him about it01:37
gabrielhurleyyeah, I'd like to get his input as well01:37
gabrielhurleythe more the merrier on this topic though, since it matters to the whole stack01:37
*** winston-d has quit IRC02:05
*** zhiteng has joined #openstack-dev02:07
*** jdg has joined #openstack-dev02:10
*** roge has joined #openstack-dev02:10
*** mikeyp has left #openstack-dev02:13
*** roge has quit IRC02:15
jdgbcwaldon: ping02:15
*** mnewby has quit IRC02:16
*** roge has joined #openstack-dev02:19
*** samkottler has quit IRC02:33
*** NaZZaX has left #openstack-dev02:35
*** spiffxp has joined #openstack-dev02:36
*** danwent has joined #openstack-dev02:40
*** roge has quit IRC02:51
*** dtroyer is now known as dtroyer_zzz02:58
*** hugokuo has left #openstack-dev03:02
*** adjohn has joined #openstack-dev03:05
*** samkottler has joined #openstack-dev03:09
*** pixelbeat has quit IRC03:09
*** samkottler has quit IRC03:09
*** samkottler has joined #openstack-dev03:09
*** dtroyer_zzz is now known as dtroyer03:12
*** Gordonz has joined #openstack-dev03:17
*** jdg has quit IRC03:19
*** mjfork has quit IRC03:21
*** Gordonz has quit IRC03:23
*** adjohn has quit IRC03:42
*** adjohn has joined #openstack-dev03:44
*** blue999 has joined #openstack-dev03:44
*** zhiteng has quit IRC03:46
*** Ryan_Lane has joined #openstack-dev03:48
*** Ryan_Lane has quit IRC03:49
*** winston-d has joined #openstack-dev03:50
*** negronjl has joined #openstack-dev04:02
*** mnewby has joined #openstack-dev04:03
*** BalleS_ has joined #openstack-dev04:21
*** dolphm has quit IRC04:25
*** Susanne-Balle has quit IRC04:25
*** adjohn has quit IRC04:36
*** paulstallworth has quit IRC04:38
*** vricci has quit IRC04:38
*** gyee has quit IRC04:57
*** sdake has quit IRC04:58
*** samkottler has quit IRC05:02
*** hattwick has quit IRC05:02
*** blue999 has quit IRC05:04
*** gabrielhurley has quit IRC05:07
*** mnewby_ has joined #openstack-dev05:18
*** mnewby has quit IRC05:21
*** mnewby_ is now known as mnewby05:21
*** Drakiz has quit IRC05:22
*** dubsquared has quit IRC05:32
*** mnewby has quit IRC05:39
*** Drakiz has joined #openstack-dev05:39
*** danwent has quit IRC05:57
*** danwent has joined #openstack-dev05:57
*** Ryan_Lane has joined #openstack-dev06:10
*** rgoodwin_ has joined #openstack-dev06:22
*** reed has quit IRC06:23
*** bourke has quit IRC06:23
*** btorch has quit IRC06:23
*** rgoodwin has quit IRC06:25
*** rgoodwin_ is now known as rgoodwin06:25
*** Kyril has quit IRC06:26
*** spiffxp_ has joined #openstack-dev06:27
*** spiffxp has quit IRC06:28
*** spiffxp_ is now known as spiffxp06:28
*** Ryan_Lane has quit IRC06:28
*** adjohn has joined #openstack-dev06:28
*** bepernoot has joined #openstack-dev06:38
*** bepernoot has quit IRC06:42
*** hugokuo has joined #openstack-dev06:42
*** bepernoot has joined #openstack-dev06:44
*** bepernoot has quit IRC06:51
*** glenc has quit IRC06:51
*** glenc_ has joined #openstack-dev06:52
*** troytoman is now known as troytoman-away06:57
*** bepernoot has joined #openstack-dev06:59
*** Lance604 has joined #openstack-dev07:01
*** dhellmann has quit IRC07:04
*** dhellmann has joined #openstack-dev07:04
*** reidrac has joined #openstack-dev07:07
*** Mandell_ has quit IRC07:10
*** bepernoot has quit IRC07:11
*** Lance604 has quit IRC07:12
*** berendt has joined #openstack-dev07:13
*** dhellmann_ has joined #openstack-dev07:19
*** dhellmann has quit IRC07:21
*** dhellmann_ is now known as dhellmann07:21
*** vricci has joined #openstack-dev07:22
*** sandywalsh has quit IRC07:22
*** LarsErikP has joined #openstack-dev07:34
*** sandywalsh has joined #openstack-dev07:35
*** bepernoot has joined #openstack-dev07:36
*** bepernoot has quit IRC07:39
*** dtroyer is now known as dtroyer_zzz07:42
*** dtroyer_zzz is now known as dtroyer07:46
*** eglynn_ has joined #openstack-dev07:52
*** dachary has joined #openstack-dev07:57
*** Mandell has joined #openstack-dev07:57
*** reed has joined #openstack-dev07:57
*** bourke has joined #openstack-dev07:57
*** btorch has joined #openstack-dev07:57
*** eglynn_ has quit IRC07:58
*** hattwick has joined #openstack-dev07:59
*** darraghb has joined #openstack-dev08:05
*** littleidea has quit IRC08:08
*** markmcclain has joined #openstack-dev08:15
*** derekh has joined #openstack-dev08:16
*** hashar has joined #openstack-dev08:18
*** hashar has joined #openstack-dev08:18
*** spiffxp has quit IRC08:21
*** eglynn_ has joined #openstack-dev08:24
*** derekh has quit IRC08:24
*** hashar has quit IRC08:25
*** markmcclain has quit IRC08:25
*** hashar has joined #openstack-dev08:26
*** hashar has joined #openstack-dev08:26
*** oneiroi has joined #openstack-dev08:30
*** winston-d has quit IRC08:32
*** littleidea has joined #openstack-dev08:32
*** derekh has joined #openstack-dev08:33
*** Drakiz has quit IRC08:38
*** pixelbeat has joined #openstack-dev08:38
*** bepernoot has joined #openstack-dev08:43
*** corXi has joined #openstack-dev08:43
berendttry to connect to novncproxy with chromium and firefox failed with: "WebSock error: Illegal basic compression received, ctl: 80".. any hints?08:43
*** bepernoot has quit IRC08:43
berendtsorry.. development channel.. reposting it in #openstack08:44
*** fc__ has quit IRC08:50
*** hashar has quit IRC08:51
*** fc__ has joined #openstack-dev09:06
*** hashar has joined #openstack-dev09:19
*** hashar has joined #openstack-dev09:19
*** Mandell has quit IRC09:32
*** vricci has quit IRC09:42
*** littleidea has quit IRC09:43
*** adjohn has quit IRC09:47
*** tryggvil_ has joined #openstack-dev10:03
*** dneary has quit IRC10:32
*** rods has joined #openstack-dev10:38
*** LinuxJedi has quit IRC10:44
*** LinuxJedi has joined #openstack-dev10:45
*** hugokuo has quit IRC10:45
*** LinuxJedi has quit IRC10:48
*** LinuxJedi has joined #openstack-dev10:49
*** Radium has joined #openstack-dev10:52
*** adjohn has joined #openstack-dev11:04
*** ghe_ has quit IRC11:07
*** asalkeld has joined #openstack-dev11:41
*** bsza has joined #openstack-dev11:47
*** Lance604 has joined #openstack-dev11:48
*** hashar has joined #openstack-dev11:52
*** hashar has joined #openstack-dev11:52
*** reed has quit IRC11:53
*** dneary has joined #openstack-dev11:54
*** mjfork has joined #openstack-dev12:01
*** reed has joined #openstack-dev12:06
*** alaski has joined #openstack-dev12:06
*** markvoelker has joined #openstack-dev12:09
*** crobinso has joined #openstack-dev12:18
*** markmc has joined #openstack-dev12:20
*** alaski has quit IRC12:26
*** alaski has joined #openstack-dev12:26
*** markvoelker has quit IRC12:29
*** Radium has quit IRC12:31
*** Radium has joined #openstack-dev12:35
*** davidkranz has quit IRC12:37
*** davidkranz has joined #openstack-dev12:37
*** Radium has quit IRC12:45
*** kbringard has joined #openstack-dev12:46
*** Shrews has joined #openstack-dev12:48
*** markvoelker has joined #openstack-dev12:52
*** thinrhino has joined #openstack-dev12:55
*** ayoung has quit IRC12:56
*** roge has joined #openstack-dev13:00
*** Shrews has quit IRC13:01
*** Shrews has joined #openstack-dev13:03
*** belliott has quit IRC13:04
*** dneary has quit IRC13:04
*** devananda has joined #openstack-dev13:07
*** devananda has quit IRC13:08
*** dprince has joined #openstack-dev13:09
*** littleidea has joined #openstack-dev13:10
*** Radium has joined #openstack-dev13:15
*** sandywalsh has quit IRC13:26
*** littleidea has quit IRC13:30
*** ayoung has joined #openstack-dev13:34
*** Asim has joined #openstack-dev13:35
*** Asim has left #openstack-dev13:36
*** sandywalsh has joined #openstack-dev13:39
*** belliott has joined #openstack-dev13:48
*** thinrhino has quit IRC13:54
*** kbringard has quit IRC13:56
*** kbringard has joined #openstack-dev13:56
*** markmcclain has joined #openstack-dev13:56
*** tryggvil_ has quit IRC13:59
*** utlemming has joined #openstack-dev14:00
*** mariocar has quit IRC14:01
*** blamar has joined #openstack-dev14:02
*** littleidea has joined #openstack-dev14:07
*** zaid_h has joined #openstack-dev14:10
*** kbringard has quit IRC14:15
*** kbringard has joined #openstack-dev14:18
*** markmc has quit IRC14:18
*** sdake has joined #openstack-dev14:20
*** devananda has joined #openstack-dev14:23
*** AlanClark has joined #openstack-dev14:28
*** PotHix has joined #openstack-dev14:31
*** GheRivero has joined #openstack-dev14:32
*** sdake has quit IRC14:52
*** zzed has joined #openstack-dev14:53
*** zzed is now known as _zzed14:54
*** sdake has joined #openstack-dev14:55
*** roge has quit IRC14:56
*** roge has joined #openstack-dev14:58
corXichmouel: ping15:01
*** flaviamissi has joined #openstack-dev15:02
*** reidrac has left #openstack-dev15:02
*** BruceZu has joined #openstack-dev15:06
*** danwent has joined #openstack-dev15:08
*** dtroyer is now known as dtroyer_zzz15:09
pvomtaylor: ping15:15
*** dtroyer_zzz is now known as dtroyer15:16
*** dtroyer is now known as dtroyer_zzz15:18
*** rnirmal has joined #openstack-dev15:18
*** sleepsonthefloor is now known as sleepsonzzz15:19
*** RobertLaptop has joined #openstack-dev15:19
*** RobertLaptop has left #openstack-dev15:20
*** derekh has quit IRC15:29
*** roge has quit IRC15:30
*** vizsla has joined #openstack-dev15:31
*** corXi has quit IRC15:31
*** roge has joined #openstack-dev15:32
*** dtroyer_zzz is now known as dtroyer15:32
pvomtaylor: can you send me the tenant id of the limits you need me to adjust?15:33
*** Radium has left #openstack-dev15:38
*** Mandell has joined #openstack-dev15:39
*** littleidea has quit IRC15:39
*** hashar has quit IRC15:43
*** clayg has quit IRC15:44
*** 65MAADRBZ is now known as clayg15:44
*** byeager has quit IRC15:50
*** hashar has joined #openstack-dev15:53
*** sdake_ has joined #openstack-dev15:56
*** utlemming has quit IRC15:57
*** sdake has quit IRC15:58
*** utlemming has joined #openstack-dev16:00
*** eglynn_ has quit IRC16:01
*** littleidea has joined #openstack-dev16:02
chmouelcorXi: pong?16:04
*** sleepsonzzz is now known as sleepsonthefloor16:05
*** dolphm has joined #openstack-dev16:06
vishyadam_g: ping16:07
*** zaitcev has joined #openstack-dev16:09
mtaylorpvo: yes. is that different than my username?16:09
vishypvo, comstud, tr3buchet,  blamar, Vek, core members in general: reviews needed on https://review.openstack.org/#change,5969 and https://review.openstack.org/#change,598316:09
tr3buchetvishy: you bet16:10
*** maplebed has joined #openstack-dev16:11
pvomtaylor: its your tenant_id16:12
pvoit shoudl be different than your username16:13
pvoits it openstackjenkins?16:13
*** derekh has joined #openstack-dev16:13
*** belliott has quit IRC16:14
vishyreview needed here as well: https://review.openstack.org/#change,593916:14
*** belliott has joined #openstack-dev16:18
vishytr3buchet: this is networking related.  Might be a little outside of the stuff you focus on, but let me know what you think: https://review.openstack.org/#change,596116:19
*** belliott has quit IRC16:20
justinsb_redbo: ping16:21
*** darraghb has quit IRC16:22
pvomtaylor: bumped the available gee bees16:22
tr3buchetvishy: looks good to me16:23
tr3buchetapproved16:23
vishytr3buchet: hmm just realized i should change the commit message as it doesn't match16:24
vishylet me block it and reapprove16:24
tr3buchethurry16:24
ttxvishy: yo16:25
vishyttx: hi16:25
vishygoing through bugs now16:26
ttxvishy: great16:26
vishyttx: i have a bunch of backporting to do and I have to check up on a couple of bugs that I targeted16:26
ttxok. We'll probably want to sync the translations before RC2 too16:26
ttxand get to the bottom of this scopting tenant/admin thing16:27
ttxscoping16:27
*** oneiroi is now known as Oneiroi^gone16:27
ttxvishy: would really be good to have RC2 out on Monday, even if that means pushing a few bugs to the known issues section of the release notes16:28
vishyi agree16:28
ttxWe need to give at the very minimum 2-3 days of baking for that RC to catch showstoppers16:29
ttxbut I don't think we can make it happen today16:29
tr3buchetvishy: i re +216:29
vishytr3buchet: cool thx16:29
ttxbcwaldon: did you get jaypipes's ack for RC2 ?16:29
vishyttx: I will make sure everything is backported16:29
vishyttx: i just can't do the cut myself, or else I could do it tomorrow?16:30
ttxvishy: the process is a bit troublesome16:30
ttxvishy: i can do it over the weekend if you greenlight me16:30
ttxit takes just 15min16:31
*** eglynn_ has joined #openstack-dev16:31
vishyttx: ok i will email you if we are good to go!16:31
ttxsounds good.16:31
bcwaldonttx: yep16:32
ttxbcwaldon: ok then. I can make it happen now ?16:32
*** spiffxp has joined #openstack-dev16:32
bcwaldongo for it16:32
ttxyeepee16:32
ttxvishy: note that for two RC2-targeted bugs I mentioned on the bug they might not be that great in the RC at this point16:34
*** belliott has joined #openstack-dev16:35
*** eglynn_ has quit IRC16:35
*** eglynn_ has joined #openstack-dev16:35
ttxvishy: The Scheduler default; https://bugs.launchpad.net/nova/+bug/96573216:35
uvirtbotLaunchpad bug 965732 in nova "Scheduler defaults to fill-first.  Spread-first is more intuitive and should be the default" [Medium,Fix committed]16:35
tr3buchetbelliott: o/16:36
ttxand the probably-topo-complex-to-fix https://bugs.launchpad.net/nova/+bug/79777016:36
uvirtbotLaunchpad bug 797770 in nova "'Socket closed' during API stress test" [High,In progress]16:36
ttxvishy: up to you if they should stay RC2-targeted. They are both above my paranoid risk threshold.16:37
*** hashar has quit IRC16:38
ttxbcwaldon: actually, will roll Glance RC2 in ~3hours16:40
*** sniperd has joined #openstack-dev16:40
*** eglynn_ has quit IRC16:48
*** heckj has joined #openstack-dev16:50
*** heckj has quit IRC16:50
*** heckj has joined #openstack-dev16:50
heckjchmouel - you up and about?16:51
*** AlanClark has quit IRC16:51
heckjayoung - you around?16:53
*** dachary has quit IRC16:53
ayoungheckj, yes16:53
heckjam doing a run through the bugs - read through this: https://bugs.launchpad.net/keystone/+bug/92844116:53
uvirtbotLaunchpad bug 928441 in keystone "document base model types for key elements within Keystone API" [High,Confirmed]16:53
heckjayoung: I think you rather took care of that with your LDAP branch for identity. Do you agree, or do you think there's more needed there?16:53
*** dachary has joined #openstack-dev16:54
ayoungwell, not sure16:54
ayoungit seems like the Model  stuff fits the letter of the law16:54
ayoungdocstring documentation  might not quite be up to it16:55
*** dolphm has quit IRC16:55
ayoungand nothin enforces it in the SQL Identity provider...16:55
heckjSo still worth nailing down some abstract base class or something that explicits identifies the things that need to get returned with objects and their relations? Or a doc to that effect in the ReST external to docstrings?16:55
ayoungSo I would close that one as fixed,  but open another for enforcing the contract with SQL16:56
ayoungthat was pretty much what termie suggested in the meeting back then16:56
ayounglet me look at the docstrings, though16:57
vishyttx: the scheduler was changed from chance which was approximately fill-first about a month ago16:57
heckjsounds good - I'll close it out then. Would you mind opening one on enforcing the contract w/ SQL? I think you've got a better handle on it than I do for describing what needs to get solved.16:57
vishyttx: so not changing it scares me a lot more16:57
adam_gvishy: pong16:57
vishyttx: we will probably be forced to leave out https://bugs.launchpad.net/nova/+bug/797770 although that one annoys me16:57
uvirtbotLaunchpad bug 797770 in nova "'Socket closed' during API stress test" [High,In progress]16:58
vishyadam_g: just made a couple reqs on your mysql prop16:58
vishyadam_g: I was going to ask you to update it if you were around so I don't have to do it when I merge16:58
*** dtroyer is now known as dtroyer_zzz16:58
adam_gvishy: for sure. gimme a few, just got online16:59
*** derekh has quit IRC16:59
*** jdg has joined #openstack-dev16:59
heckjttx: what's the impact of https://bugs.launchpad.net/keystone/+bug/963478?16:59
uvirtbotLaunchpad bug 963478 in keystone "Missing files in python-keystoneclient RC1 tarball ?" [High,Confirmed]16:59
vishywow so unittests are broken on mac17:00
vishy:(17:00
vishythere is no O_DIRECT17:00
heckjand is that something we can fix in the client, or is this something that needs to be tweaked in the tarball creation17:00
ayoungheckj, so identity/model.py only covers User, Tenant and Role.   The other Token Catalog endpoint Action and Capability are not covered17:01
ayoungI'd assmue the token stuff would be in keystone/token/modle.py or comparable location17:01
*** dachary has quit IRC17:03
heckjayoung: yeah, doesn't appear to exist there17:03
ayoungSo...defer to Folsom.  I think what we have now is sufficient for an operational Essex.17:04
ayoungI'll update the ticket17:04
*** GheRivero has quit IRC17:05
heckjthanks17:05
*** Lance604 has quit IRC17:07
*** derekh has joined #openstack-dev17:11
*** gyee has joined #openstack-dev17:13
vishycomstud: i guess we might have to degrade the socket closed errors down to known issue17:15
vishycomstud: i'm guessing that you can work around it with sufficient load balancing17:15
*** jaypipes has joined #openstack-dev17:18
*** derekh has quit IRC17:19
*** fc__ has quit IRC17:22
comstudvishy: i don't think that'll solve it.. if it's happening in compute talking to things17:23
comstudvishy: But i think degrading it is fine, since it's been around forever17:23
*** sdake_ is now known as sdake17:24
vishycomstud: looks related: https://bugs.launchpad.net/python-oops-amqp/+bug/88234617:27
vishymight have to patch amqp?17:28
uvirtbotLaunchpad bug 882346 in python-oops-amqp "IOError exceptions are not caught" [Critical,Fix released]17:28
comstudvishy: Yeah, that's really a more appropriate fix17:28
*** Oneiroi^gone is now known as oneiroi17:29
comstudhowever, since we're doing our own reconnecting in nova...17:29
comstudcatching IOError ourselves would also work17:29
vishycomstud: lets try it as a workaround and report it upstream17:29
vishyunless it is fixed already in a newer version?17:29
comstuddunno17:30
comstudit's easier to just submit my impl_kombu fix ;)17:30
comstudsubmit/approve17:30
comstudbut I'm waiting more feedback on the bug17:30
vishycomstud: right he hasn't given a new traceback17:31
*** mdrnstm has joined #openstack-dev17:32
comstudvishy: Ya17:33
comstudvishy: Ultimately either of these aren't great fixes to the real problem...17:33
comstudvishy: which seems to be that greenthreads are not getting scheduled quickly enough17:33
comstudvishy: ie, another greenthread is running for too long... or we have too many greenthreads or something.  dunno what, though.  just not enough pseudo-parallelism in some way, though, i assume.17:34
*** dneary has joined #openstack-dev17:34
*** alrs has joined #openstack-dev17:35
comstud(mysql blocking doesn't help)17:36
*** dtroyer_zzz is now known as dtroyer17:38
*** sdake has quit IRC17:38
*** sdake has joined #openstack-dev17:39
*** Ryan_Lane has joined #openstack-dev17:39
vishycomstud: hmm my reading of transport is that recv is getting zero bytes17:40
justinsb_comstud: We block on DB calls?17:40
vishycomstud: when it expects more bytes, so you are saying that the greenthread that has the socket open on the other end isn't responding and it is timing out?17:40
vishycomstud: I wonder if the flag: MSG_WAITALL    wait for full request or error --would have an effect17:42
vishyjustinsb_: correct mysql is a c library that is not monkeypatched17:42
comstudvishy: 0 byte recv == EOF.. remote end is closing the socket.17:43
comstudvishy: When I looked at this, rabbit was closing the connection on the client because protocol negotiation was taking too long17:43
comstudvishy: Which means that the client side is not getting to recv() quickly enough.17:43
comstudvishy: When the greenthread is finally scheduled, it reads the EOF17:43
comstud(or well, 0 bytes)17:43
*** sdake has quit IRC17:43
openstackgerritVerification of a change to openstack/keystone failed: don't duplicate the extra dict in extra  https://review.openstack.org/597817:44
*** sdake has joined #openstack-dev17:44
vishycomstud: interesting, the only place i see IOError raised with that message is in tcp when it gets less bytes than it is expecting17:44
comstudvishy: if the connection was still open, you'd not read 0 bytes... you'd get a -1 response and EAGAIN17:44
vishyah i see17:44
comstudi suspect if those having this issue go and look at their rabbit logs...17:45
comstudthey'll find the timeout in there17:45
justinsb_vishy: so if someone selects 1.7 million rows... :-)17:45
comstudand rabbit closing the connection17:45
vishycomstud: so increasing the timeout in rabbit might be a workaround17:45
comstudvishy: yes, if that's possible17:45
vishyjustinsb_: yes the worker will block17:45
comstudi'm not sure what the default is17:45
comstudbut it must be something relatively small17:45
*** markmcclain has quit IRC17:46
justinsb_vishy: But, more problematically, it can throw other code off.  Code that might be perfectly correct if run on its own thread.17:46
vishyjustinsb_: we tried to make db non-blocking in diablo17:46
vishyfailed17:46
vishyjustinsb_: and no one has bothered to fix it properly17:47
comstudWe should try it again17:47
*** armaan has joined #openstack-dev17:47
comstudI think it was done correctly in the first place..17:47
comstudbut something else weird didn't like it17:47
comstudi was able to get the same sqlalchemy traceback without db pool17:47
justinsb_I guess it's the same thing with libvirt, or any process call though17:47
comstudby doing something with locks17:47
justinsb_Any call that takes too long could cause protocol negotiation to time out17:47
comstudand I can't reproduce that issue now17:47
comstudso I wonder if dbpool is also now fixed.17:48
comstud'fixed'17:48
vishycomstud: something definitely wasn' tworking right17:48
vishycomstud: it would start way more threads than you requested17:48
vishyeven with threadpool of one i was seeing 4 threads iirc17:49
comstudoh, okay17:49
comstudyeah, well, that's not quite right17:49
vishybut it may be fixed now17:49
comstudI was more referring to the weird KeyError exception from sqlalchemy17:49
comstud:)17:49
justinsb_Eventlet = the Windows 3.0 model of multitasking :-)17:49
vishycomstud: who wants to try it ?17:49
comstudjustinb: pretty much17:49
vishyjustinsb_: lol17:49
comstud+s17:49
comstudI've been thinking of a really bastardized way of doing non-blocking socket stuff for non-python modules17:50
comstudessentially eventlet-like thing that you LD_PRELOAD17:51
comstudreplaces socket(), etc.17:51
comstud(libc's socket())17:51
comstudand the rest of the socket calls.17:51
comstudfun hack-day project17:51
sdaguejustinsb_: are you working any further on database optimizations at the moment?17:52
justinsb_comstud: That sounds fun, but really hard!17:52
justinsb_sdague: I don't think they are going to make Essex, so I'm focusing on my real work!17:52
comstudjustinsb: i've done some crazy stuff17:52
sdaguejustinsb_: fair enough :)17:53
comstuda lot of background in high perf socket i/o multithreaded servers in C17:53
justinsb_comstud, vishy: I think we need a summit session on this blocking topic generally17:53
comstudi just need time!17:53
comstudy'all have some time you can donate?17:53
comstudvishy: I can see if we can work out trying dbpool here in our environments.17:54
comstud(again)17:55
justinsb_comstud: If you know high performance C, there are much better ways to apply that knowledge than monkey patching some monkey's driver for a monkey DB ;-)17:55
comstudhahaha17:55
sdagueI mostly wanted to make sure I wasn't going to be duplicating work you were doing. I'm actually a little surprised how many non indexed where clauses we've got, though I'd love to understand how big these tables are expected to get to know how bad it really is.17:55
justinsb_sdague: Yes, I tried prioritizing them based on my guesses.  But that's why I really wanted to get logging in to Essex.17:56
justinsb_sdague: The discussion on that patch wore me out!17:56
sdaguejustinsb_: fair enough17:56
justinsb_sdague: As long as everything is in memory, even a huge table shouldn't be too bad17:57
vishycomstud: someone should definitely try it17:57
justinsb_sdague: And I think it should all be in memory.  Certainly not unreasonable to ask for a few Gig of RAM on a DB if you're running something the size of EC217:57
comstudvishy: I'll see what we can do.. i know we're rather busy atm.. but we also have a big interest in it17:58
vishycomstud: maybe i can get someone on my team to give it a shot17:58
vishycomstud: i will ask17:58
comstudvishy: okie17:58
justinsb_sdague: The use of MyISAM does concern me though17:58
justinsb_sdague: Terrible locking model, terrible persistence model :-)17:58
*** eglynn_ has joined #openstack-dev17:59
vishydavidkranz: are you around to give us a new traceback on that bug?18:00
*** clayg_ has joined #openstack-dev18:00
davidkranzYes. I pulled the change yesterday but it was the same. Is there something new?18:01
vishyjustinsb_: solution: set the db to innodb by default18:01
vishydavidkranz: we are unconvinced that you are getting the same traceback18:01
vishydavidkranz: you restarted everything in the system with the new code?18:01
*** jcooley has joined #openstack-dev18:01
vishydavidkranz: because that should at least give you a different error message if it doesn't fix the problem18:01
davidkranzAh. I didn't look closely at the trace back when I saw the same socket closed error. I did restart from scratch.18:02
davidkranzI'll do it again now.18:02
vishydavidkranz: great thanks18:02
*** Ryan_Lane has quit IRC18:02
vishyjustinsb_: that will stop the myiasm table issue :)18:02
vishy*isam18:02
*** flaviamissi has quit IRC18:03
justinsb_vishy: I know.  But when someone doesn't do that, it won't be themselves they blame.18:03
sdaguewhat if we just force that in a migration?18:04
justinsb_vishy: And good luck when MyISAM blocks for someone, throws up an timeout somewhere unrelated, but we can't reproduce because all the repro systems are running InnoDB!18:04
*** zigo has joined #openstack-dev18:05
justinsb_sdague: Yes, we should.  But it probably won't make it into Essex.18:05
justinsb_sdague: I think we can create a 'tuning' SQL script that we can recommend18:05
justinsb_sdague: And the distros may package that, but OpenStack itself won't18:05
sdagueit looks like that was done once with 022_set_engine_mysql_innodb.py, but then tables since then cropped up18:05
*** armaan has left #openstack-dev18:06
justinsb_sdague: Yes.  Would have been nice to have set a systemwide default in the same patch...18:06
sdagueI dislike the idea of optional bits there, because it means lots of bugs because the default isn't that way.18:07
justinsb_sdague: I agree.  Even if the patch is an improvement, forking/fragmentation is bad.18:07
*** dneary has quit IRC18:09
*** johnpostlethwait has joined #openstack-dev18:10
vishysdague: we figured setting default table type was not really something that nova should do18:11
vishysdague: but yeah it would have been nice to make all future tables innodb :(18:12
vishyif we are setting some of them18:12
vishya new migration to do that?18:12
justinsb_vishy: Any chance of it getting in to Essex?18:13
sdaguevishy: I'd be happy to make the migration, I've also got one out there just to add one of the missing keys justinsb_ identified - https://review.openstack.org/#change,597018:14
sdaguevishy: I can understand not wanting to change it for the entire environment.18:15
sdaguejustinsb_: any idea if you can set default engine per database? not really seeing that18:15
justinsb_sdague: I was thinking sqlalchemy might have a setting.  Checking now18:16
justinsb_Oh goodie.  I bet we're not setting mysql_charset either18:17
*** flaviamissi has joined #openstack-dev18:19
*** novas0x2a|laptop has joined #openstack-dev18:21
jaypipescomstud: heyo. I've done a few rounds of iterative reviews for the Nova multi-process-blueprint submission: https://review.openstack.org/#change,5762. It's looking pretty good now, but I was hoping you could check it out since it does differ a bit from the similar code in Glance and Swift and wanted to make sure you didn't see any further catches to what I brought up in earlier reviews...18:22
*** pixelbeat has quit IRC18:22
*** novas0x2a|laptop has quit IRC18:23
*** novas0x2a|laptop has joined #openstack-dev18:23
comstudjaypipes: ok.  i'll try to hit it here.. crazy busy atm.18:23
justinsb_sdague: Looks like the best would could do would be to subclass the Table class e.g. OpenstackTable18:23
comstudjaypipes: thnx for heads up, tho.  maybe i can get to it in an hour or so18:23
comstudif it's still there :)18:23
jaypipescomstud: yup, no worries mate. I appreciate it.18:24
comstudnp18:24
justinsb_sdague: i.e. not simple enough to bundle into a migration18:24
*** dtroyer is now known as dtroyer_zzz18:26
*** belliott has quit IRC18:29
*** pixelbeat has joined #openstack-dev18:31
*** johnpostlethwait has quit IRC18:31
*** johnpostlethwait has joined #openstack-dev18:32
*** belliott has joined #openstack-dev18:32
*** mnewby has joined #openstack-dev18:35
*** oneiroi is now known as Oneiroi^gone18:41
sdaguejustinsb_: well, at least that would be a folsom strategy18:43
tr3buchetvishy: haha, i love they way you taking over these things :D18:44
*** hashar has joined #openstack-dev18:47
*** hashar has joined #openstack-dev18:47
*** zigo has quit IRC18:53
justinsb_vishy: Can you backport sql_logging into Essex pls https://review.openstack.org/#change,581618:55
justinsb_vishy: If we can't fix things, at least we won't be flying blind...18:55
*** zigo has joined #openstack-dev18:56
*** rbasak has quit IRC18:58
*** davlaps has joined #openstack-dev18:58
*** johnpostlethwait has quit IRC19:00
*** johnpostlethwait has joined #openstack-dev19:01
*** rkukura has joined #openstack-dev19:01
*** pixelbeat has quit IRC19:02
*** sannes has joined #openstack-dev19:04
*** mdrnstm has quit IRC19:06
*** Aaton has joined #openstack-dev19:09
*** davidkranz has quit IRC19:12
*** Oneiroi^gone is now known as oneiroi19:17
*** oneiroi has left #openstack-dev19:17
*** maplebed is now known as maplebedAFK19:30
*** mdrnstm has joined #openstack-dev19:30
*** mdrnstm has left #openstack-dev19:31
*** mdrnstm has joined #openstack-dev19:31
*** dachary has joined #openstack-dev19:34
*** belliott has quit IRC19:37
*** Radium has joined #openstack-dev19:41
*** hashar has quit IRC19:45
*** davidkranz has joined #openstack-dev19:46
davidkranzvishy, comstud: No good news. See comments in https://bugs.launchpad.net/nova/+bug/79777019:46
uvirtbotLaunchpad bug 797770 in nova "'Socket closed' during API stress test" [High,In progress]19:46
*** gabrielhurley has joined #openstack-dev19:55
*** lorin1 has joined #openstack-dev19:58
*** davlaps has quit IRC20:00
*** dprince has quit IRC20:03
*** danwent has quit IRC20:06
*** jaypipes has quit IRC20:13
ttxheckj: needs to be tweaked in MANIFEST.in20:16
ttxheckj: impact is minimal20:16
ttxvishy: ack for scheduler20:17
heckjttx: kk20:17
heckjthanks20:17
ttxheckj: but it's an easy fix if you do RC220:17
*** danwent has joined #openstack-dev20:17
ttxheckj: I can propose a fix if nobody else does. Monday20:18
* ttx published Glance RC220:18
ttxpublishes*20:18
russellbooh20:22
* russellb goes to update Fedora with glance rc220:22
*** littleidea has quit IRC20:22
*** derekh has joined #openstack-dev20:25
*** Aaton has quit IRC20:25
*** littleidea has joined #openstack-dev20:25
sdaguedavidkranz: for kicks, do you have mysql slow query log turned on? would be curious if any ringers came out during the stress tests20:27
*** Aaton has joined #openstack-dev20:28
*** mattstep has quit IRC20:32
*** mattstep has joined #openstack-dev20:33
ttxGlance RC2 is in: https://launchpad.net/glance/essex/essex-rc220:33
*** alaski has quit IRC20:36
*** belliott has joined #openstack-dev20:36
*** markvoelker has quit IRC20:36
*** Radium has quit IRC20:37
*** alaski has joined #openstack-dev20:38
*** glenc has joined #openstack-dev20:38
*** glenc_ has quit IRC20:40
ttxvishy: about 797770, I'm ok to leave it out. It's been there all cycle, this week is not the right one to realize it needs to be fixed20:41
*** alaski has quit IRC20:44
*** alaski has joined #openstack-dev20:44
*** maplebedAFK is now known as maplebed20:46
*** gyee has quit IRC20:49
*** alaski has quit IRC20:56
*** zigo has quit IRC20:57
*** milner has quit IRC21:01
*** sandywalsh has quit IRC21:02
*** crobinso has quit IRC21:16
*** crobinso has joined #openstack-dev21:16
*** sandywalsh has joined #openstack-dev21:17
*** vizsla has quit IRC21:19
*** alrs has quit IRC21:20
*** dtroyer_zzz is now known as dtroyer21:22
*** joshuamckenty has joined #openstack-dev21:23
*** dtroyer is now known as dtroyer_zzz21:26
*** sniperd has quit IRC21:31
*** jdg has quit IRC21:32
*** mdomsch_ has quit IRC21:34
*** mdomsch has quit IRC21:34
*** mdomsch_ has joined #openstack-dev21:35
*** lorin1 has quit IRC21:36
*** PotHix has quit IRC21:36
*** mdomsch_ has quit IRC21:36
*** Radium has joined #openstack-dev21:37
*** rnirmal has quit IRC21:38
*** _zzed has quit IRC21:41
*** heckj has quit IRC21:43
*** sniperd has joined #openstack-dev21:45
*** crobinso has quit IRC21:51
*** danwent has quit IRC21:54
*** markmcclain has joined #openstack-dev21:57
*** jk0 has quit IRC21:59
*** jk0 has joined #openstack-dev22:01
*** ChanServ sets mode: +v jk022:01
mikalAre there any nova core reviewers who aren't vishy around who can review https://review.openstack.org/#change,5799 please?22:01
kbringardoooo vishy slam out of nowhere22:02
kbringard:-p22:02
kbringardhave a good weekend friends22:03
*** kbringard has quit IRC22:03
*** sleepsonthefloor is now known as sleepsonzzz22:03
*** bsza has quit IRC22:07
*** dubsquared has joined #openstack-dev22:10
*** joshuamckenty has quit IRC22:19
*** Radium has quit IRC22:22
*** johnpostlethwait has quit IRC22:22
*** johnpostlethwait has joined #openstack-dev22:23
*** belliott has quit IRC22:30
*** danwent has joined #openstack-dev22:33
*** sniperd_ has joined #openstack-dev22:41
*** sniperd has quit IRC22:42
*** dhellmann has quit IRC22:45
*** markmcclain has quit IRC22:50
vishykbringard: :)22:58
vishyjustinsb_: we aren't, devstack does though!22:59
*** belliott has joined #openstack-dev22:59
vishycomstud: i'm thinking we just merge your fix.  It may not completely fix the problem, but at least it gives us a chance to recover from that error23:00
*** roge has quit IRC23:02
*** derekh has quit IRC23:03
*** sleepsonzzz is now known as sleepsonthefloor23:06
*** novas0x2a|laptop has quit IRC23:07
jeblairvishy: https://github.com/openstack-dev/sandbox23:09
jeblairvishy: there's a sandbox project for dev tooling workshops23:09
jeblairit's hooked up to gerrit and jenkins just like the rest.  and anyone in openstack-cla can approve changes23:10
*** berendt has quit IRC23:17
vishyjeblair: woot thx23:20
*** paulstallworth has joined #openstack-dev23:25
*** sleepsonthefloor is now known as sleepsonzzz23:25
*** gabrielhurley has quit IRC23:27
*** Lumiere is now known as Lumiere|DCUnited23:36
*** fattarsi is now known as Guest4759223:37
*** gyee has joined #openstack-dev23:38
*** Guest47592 has quit IRC23:38
*** jcooley has quit IRC23:42
*** Aaton_ has joined #openstack-dev23:44
*** Aaton has quit IRC23:45
*** westmaas has quit IRC23:45
*** ironcame12 has quit IRC23:45
*** pknouff has quit IRC23:45
*** nikhil_ has quit IRC23:45
*** ameade has quit IRC23:45
*** sniperd_ has quit IRC23:46
openstackgerritVerification of a change to openstack/nova failed: Allow snapshots in error state to be deleted.  https://review.openstack.org/598523:49
*** ayoung has quit IRC23:49
*** flaviamissi has joined #openstack-dev23:50
*** johnpostlethwait has quit IRC23:52
*** Aaton_ has quit IRC23:52
*** fattarsi has joined #openstack-dev23:53
*** soren has quit IRC23:53
*** flaviamissi has quit IRC23:55
*** asalkeld has left #openstack-dev23:55
*** flaviamissi has joined #openstack-dev23:56
*** gabrielhurley has joined #openstack-dev23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!