Thursday, 2013-05-09

« Wednesday, 2013-05-08 Index Friday, 2013-05-10 »
*** spzala has joined #openstack-dev00:02
FatDarrelgetting an error Unable to communicate with identity service: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Not Authorized"}}. (HTTP 401)00:03
*** vipul is now known as vipul|away00:05
*** dontalton has quit IRC00:05
*** Yada has quit IRC00:06
*** vipul|away is now known as vipul00:07
*** pixelbeat_ has quit IRC00:08
*** adjohn_ has joined #openstack-dev00:10
*** lloydde_ has quit IRC00:13
*** adjohn has quit IRC00:14
*** aeperezt has quit IRC00:16
jamielennoxbknudson: that error you're seeing on https://review.openstack.org/#/c/27878/ is because mysql doesn't save microseconds in the db, so comparing what went in to what came out is wrong.00:16
bknudsonjamielennox: you've seen the same thing?00:17
jamielennoxbknudson: do you think it is reasonable to trim all token expiry to the nearest second to handle this?00:17
openstackgerritA change was merged to openstack/keystone: Remove unused CONF.pam.url  https://review.openstack.org/2771400:17
jamielennoxyea, i was so focused on postgres that i forgot to run mysql00:17
openstackgerritA change was merged to openstack/python-keystoneclient: Remove duplicate test definitions  https://review.openstack.org/2726800:17
bknudsonin the testcase, I wouldn't have a problem with that.00:17
jamielennoxin general?00:17
jamielennoxi can't imagine what difference microseconds will make in token expiry at all, but it's a much further reaching change00:18
*** lloydde has joined #openstack-dev00:18
bknudsonjamielennox: could just the testcase be changed to trim the timestamp?00:19
jamielennoxyea, that's fine, i'll just keep it there then00:19
bknudsonI think it was just one testcase failing.00:20
bknudsonjamielennox: it seemed to be complaining about unicode strings rather than regular strings.00:20
bknudson- {'a': 'b',00:21
bknudson+ {u'a': u'b',00:21
bknudson?  +     +00:21
jamielennoxthat's just the way the diff is printing00:21
bknudsonok.00:21
jamielennoxu'a' == 'a'00:21
jamielennoxso it will match00:21
*** terryh has quit IRC00:22
bknudsonjamielennox: looks like the interesting part is [truncated]... "datetime(2013, 5, 9, 22, [truncated]"00:22
jamielennoxyea00:22
jamielennoxi just ran it on mysql00:22
jamielennoxold {'a': 'b', 'user_id': 'testuserid', 'expires': datetime.datetime(2013, 5, 10, 0, 22, 23, 295432), 'user': {'id': 'testuserid'}, 'id_hash': '2669ab52da734dab8dede537fecda854', 'id': '2669ab52da734dab8dede537fecda854', 'trust_id': None}00:22
jamielennoxnew {u'a': u'b', 'user_id': u'testuserid', 'expires': datetime.datetime(2013, 5, 10, 0, 22, 23), u'user': {u'id': u'testuserid'}, u'id_hash': u'2669ab52da734dab8dede537fecda854', 'id': u'2669ab52da734dab8dede537fecda854', 'trust_id': None}00:22
jamielennoxso the unicode prints in the diff but it's the microseconds in expires making it fail00:23
jamielennoxhow this is the only test that actually fails on this is the surprising part for me00:24
bknudsonis it even testing timestamps?00:24
*** Zah_ has joined #openstack-dev00:24
*** jvrbanac has quit IRC00:25
jamielennoxit's doing assertEqual on the two dictionaries, so it'll test each element i guess00:26
jamielennoxmost of the other tests relating to expires are purposefully setting a time00:26
*** jergerber has quit IRC00:26
jamielennoxthat or only testing that the ids are equal00:27
*** salv-orlando has quit IRC00:27
bknudsonif it's not explicitly a timestamp test, then I don't think anyone would have a problem with the test change.00:27
*** spzala has quit IRC00:28
*** jergerber has joined #openstack-dev00:28
FatDarrelgot an error here : ERROR : Error during puppet run : err: /Stage[main]/Nova::Keystone::Auth/Keystone_service[nova]: Could not evaluate: Execution of '/usr/bin/keystone --token 1931eb879a1c42928fe97a7eed7aa87a --endpoint http://127.0.0.1:35357/v2.0/ service-list' returned 1: Unable to communicate with identity service: {"error": {"message": "An unexpected error prevented the server from fulfilling your request. (ProgrammingErro00:29
FatDarrel(1146, \"Table 'keystone.service' doesn't exist\") 'SELECT service.id AS service_id, service.type AS service_type, service.extra AS service_extra \\nFROM service' ()", "code": 500, "title": "Internal Server Error"}}. (HTTP 500)00:29
FatDarrelwhy is keystone failing such?00:29
jamielennoxyea it's just testing that want went into the identity_api with create should be the same as what came out with a list, so dropping microseconds shouldn't matter00:30
*** zzs has left #openstack-dev00:32
jamielennoxFatDarrel: I can't speak for what puppet is doing, but it looks like it didn't set up the database00:32
FatDarrelthat is weird it should've00:32
jamielennoxhave a look in the DB keystone is connecting to and just make sure the tables are there00:33
FatDarrelaargh00:33
*** lloydde has quit IRC00:35
*** ayoung-afk is now known as ayoung00:35
ayoungjamielennox, can't trim to the second00:35
*** lloydde has joined #openstack-dev00:36
ayoungjamielennox, I am getting more annoyed at MySQL all the time.  Is there a different column type we should be using to get the smaller precision?00:36
jamielennoxfriends don't let friends use mysql00:37
ayoungjamielennox, the microseconds are kindof artificial.  If someone is creating a token in a tight loop, the microsecond precions keeps them distinct00:37
ayoungwe had that problem with unit tests00:37
ayoungjamielennox, I'm starting to think the DB world is split in to two Postgresql, and all the rest00:38
ayoungjamielennox, we also have a check in keystone/test.py  .... I'll link00:38
FatDarreljamielennox: what should I be looking for?00:38
FatDarrelalso why does thing have so many pieces00:39
jamielennoxhttps://dev.mysql.com/doc/refman/5.1/en/datetime.html00:39
ayounghttps://github.com/openstack/keystone/blob/master/keystone/test.py#L33500:39
jamielennoxFatDarrel: at the moment i would just confirm that there is a couple of tables in there, particularly service00:39
jamielennoxayoung: lol, yea i have seen that test00:40
*** lloydde has quit IRC00:40
FatDarreli deleted the tables00:41
FatDarreldoesn't db sync do that00:41
jamielennoxi don't know if doing it that way is any easier than just removing microseconds, though it may be a bit more explicity00:41
jamielennoxFatDarrel: db sync should bring the Db up to the latest version00:42
*** bdpayne has quit IRC00:42
*** ladquin_brb has quit IRC00:43
ayoungjamielennox, so I take it token tests are failing due to chopped micrseconds then?00:46
jamielennoxayoung: I re uped the patch00:46
jamielennoxcheck the bottom of https://review.openstack.org/#/c/27878/6/tests/test_backend.py00:46
ayoungjamielennox, ugh....00:47
ayoungFatDarrel, can you connect to the mysql database from the command line?00:48
FatDarrelayoung: ok00:48
jamielennoxayoung: i think that is just about as clean as popping both expiries from the dict then comparing them with the close enough test then testing the dict00:48
FatDarrelran db sync all is ok now horizon doesn't work00:48
FatDarreli set packstack to use 127.0.0.100:48
ayoungjamielennox, yeah, but it removes the reason we were doing microseconds in the first place.00:48
*** cyeoh_ is now known as cyeoh00:49
jamielennoxwhy? what difference does it make?00:49
ayoungjamielennox, distinguisches between two tokens made quickly...00:49
jamielennoxyou are doing a get_token by id, i guess it might make a difference in a list00:49
ayoungjamielennox, I suspect that with that change, we might actually see failures in sqlite00:50
*** zul has quit IRC00:50
ayoungI suspect the reason it works for that is because mysql is so slow00:50
FatDarrelso it looks like packstack is working to install but setting the installation to 127.0.0.100:50
ayoungjamielennox, actually, it fails on storing the token00:50
FatDarrelcauses horizon to break00:50
jamielennoxworks for me in sqlite00:50
ayoungjamielennox, it is an intermittent failure00:51
jamielennoxthat doesn't make sense00:51
ayoungit will fail on certain tests when the tokens are created too closely together, less than a second apart00:51
ayoungyou are only changing it for one test00:51
ayoungbut it will fail on mysql randomly00:52
jamielennoxyou're not doing 2 timestamps though, you're saving the token once, getting the data back about what was saved, doing the get and making sure what was saved is the same00:52
jamielennoxi can't see how you could have that time change in any way00:52
ayoungand, if we changed everything to truncat, sqlite will start failing00:52
ayoungjamielennox, nah, what you did is dealing with a symptom00:52
ayoungother tests will fail, I suspect00:52
jamielennoxok, so other tests00:53
ayoungnot all the time, but randomly, which makes gating hard00:53
ayoungyeah00:53
jamielennoxi haven't run the full suite against mysql for a while00:53
jamielennoxtakes to bloody long00:53
ayoungaye00:53
ayoungbknudson, mielennox, so if you don;t make that change, does the mysql test fail consistantly, or just periodically?00:54
FatDarreljust trying to make an openstack vm that I can move around and have it installed so that people can use it locally00:55
FatDarrelok off to class talk to you guys later00:55
bknudsonayoung: It's failed every time I run it. Maybe it would work once every million times when the microsecond is 0? I didn't try it.00:55
ayoungbknudson, ok, we'll commit with that fix.  Is that the only one that fails?00:56
bknudsonayoung: yes, that was the only one.00:56
jamielennoxyea, it's going to fail every time00:56
jamielennoxand that seems to be the only one00:57
ayoungbknudson, OK, I think we might take a new approach in the future. I'll talk with dolphm about it, but if mysql doesn't persist the microseconds, then we shouldn't count on them.  We can change our appaorach to say "no more than one token per second" which is, if you think about it, too small a value anyway.  We shoud rate limit it further....00:57
jamielennoxdo you know which tests will fail with mysql and the expiry time? I've got the token tests running against mysql in a loop and nothing yet00:58
bknudsonone token per user per second?00:58
ayoungbknudson, yeah00:58
ayoungI think we can just let that fail.00:58
bknudsonayoung: maybe a separate column for microseconds.00:58
ayoungbknudson, nah, we should just rate limit.  It should be a multiple of seconds at a minimum00:59
*** sean_s has joined #openstack-dev00:59
ayoungbknudson, there were other requests for rate limiting, to fend of DOS type attacks anyway00:59
bknudsonayoung: clients seem to request a lot of tokens for some reason. they don't reuse them and then complain that keystone is slow.01:00
ayoungbknudson, yeahp01:01
bknudsonunfortunately, they complain to me. It's hard to tell them to stop doing it that way.01:01
bknudsonayoung: how many tokens would you expect are needed in a running system?01:02
ayoungjamielennox, https://bugs.launchpad.net/keystone/+bug/1178041  include a commit line Bug 117804101:02
uvirtbotLaunchpad bug 1178041 in keystone "unit tests fail on mysql due to timestamp dropping microseconds" [Undecided,New] https://launchpad.net/bugs/117804101:02
uvirtbotLaunchpad bug 1178041 in keystone "unit tests fail on mysql due to timestamp dropping microseconds" [Undecided,New]01:02
*** FatDarrel has quit IRC01:03
*** maoy has quit IRC01:04
*** reed has quit IRC01:05
ayoungbknudson, if someone is making more than one identical token per second, they are attacking01:05
ayoungjamielennox, I cannot tell you how happy this change makes me, though.01:05
lifelessayoung: really ?01:05
jamielennoxayoung: done, but you'd never encounter that bug without the fixes that this change brings anyway01:06
ayounglifeless, really.  Happy as a little schoolgirl on Shprockets!01:06
lifelessayoung: what makes a token identical, out of interest ?01:06
lifelessayoung: [the really was to attacking at 1/sec]01:06
ayounglifeless, time, project, user01:06
lifelessayoung: so identical per second is (project, user) pairs ?01:06
ayounglifeless, yeah01:06
ayounglifeless, it generates a JSON document that then gets hashed01:06
lifelessok01:06
ayoungwe didn't want to have to salt it01:07
lifelessso the reason I jumped in01:07
ayoungwhat is it with the self deprecating screen nicks tonight anyway?01:07
lifelessis because bringing up large clusters will create thousands of unique credentials, and I worried that delegated tokens might trigger your attack heuristic01:07
lifelessayoung: well, I got this when I was a teen ager, uhm 199301:07
lifelessayoung: so its not so much self deprecating as burnt-in01:08
ayounglifeless, if it makes you feel better, I haven't been a teenager since Bush Sr was president01:08
lifelessayoung: I have no idea when that was :)01:09
ayoungyou weren;t known as lifeless....01:09
lifeless:)01:09
*** yidclare has quit IRC01:09
lifelessayoung: ok. so - why not salt?01:09
ayounglifeless,  so, I think it is better to let token creation fail.  > 1 / second for the same user, for the same project is suspect01:10
ayoungand salting has its own problems, not guaranteed to work distributed.01:10
ayounglifeless, ideally, you would run multiple keystone servers, just all talking to the same database.  Salting then is not going to really get you uniqueness01:11
lifelessayoung: so if we're bringing up a 5000 node cluste, we have to manually ratelimit the tokens we hand out to nodes?01:11
*** maoy has joined #openstack-dev01:11
lifelessor create separate users rather than delegating access ?01:11
lifelessor do I have the wrong end of the stick entirely?01:12
ayounglifeless, probably not.  I mean, it has to be identical token requests to make this a problem.01:12
ayoungnodes tend to use admin/static tokens, not uid/password pairs01:12
*** maoy has quit IRC01:12
ayounglifeless, it is worth thinkin about.01:12
*** zb has joined #openstack-dev01:13
lifelessayoung: so we're bringing up nodes with heat; we're going to be making tokens dynamically01:13
ayounglifeless, postgres01:13
lifelessayoung: because having multiple machines running the same tokens makes dealing with compromise harder01:13
*** alop has quit IRC01:13
ayounglifeless, well, I tend to agree with you;  the microsecond thing was the way we were dealing with it until now.01:14
lifelessayoung: microsecond I'd be less concerned about, its hard for even a distributed system to reliably collide at microsecond scale01:14
lifelessand a busy loop with random backoff for a few seconds on any particular request can cope well.01:15
bknudsonayoung: would we get a conflict in the DB? Maybe it's not a unique constraint.01:15
lifelessayoung: I'm curious about the issues with salts; is it that another keystone can't verify a just-handed-out-salt ? or that you might collide the salt with another machine?01:16
*** zaneb has quit IRC01:16
*** bknudson has quit IRC01:16
ayoungI like " busy loop with random backoff "  maybe built in to keystone itself01:18
ayounglifeless, I think we'll still have collisions with salts, unless we do a draconian sync across machines, and I think that is not called for.01:19
lifelessayoung: how many bit salts were you thinking of? Something like snowflake is collision immune01:20
ayounglifeless, feel free to provide input on the bug I filed above01:20
ayounghttps://bugs.launchpad.net/keystone/+bug/117804101:20
uvirtbotLaunchpad bug 1178041 in keystone "unit tests fail on mysql due to timestamp dropping microseconds" [Undecided,New]01:20
*** jjmb has joined #openstack-dev01:23
*** terry7 has quit IRC01:28
jamielennoxayoung: is there a need to have id in the data when you are doing a token create with (id, data)?01:28
ayoungjamielennox, only if you want to be able to find it again.01:29
jamielennoxshouldn't it just come from the id01:29
*** jaypipes has quit IRC01:29
ayoungjamielennox, id is the hash of the token.  One of the ways we allow token validation, even for pki tokens, is to fall back to querying based on the hash01:29
ayoung we could embed the Id inside of it, but then they would need to still do the popen to read the data,01:30
jamielennoxso current review, looking at https://review.openstack.org/#/c/27878/7/keystone/token/backends/kvs.py01:30
*** comay has quit IRC01:30
jamielennoxi have to do that as the id that is provided is not necessarily the same as what will be saved to the DB01:31
jamielennoxthe unique_id thing01:31
*** acfleury has quit IRC01:32
jamielennoxbknudson had a comment https://review.openstack.org/#/c/27878/5/tests/test_backend.py01:32
*** jhesketh__ has quit IRC01:32
jamielennoxabout me changing the nature of the test, he's right i hadn't understood how the extra worked01:33
*** jhesketh__ has joined #openstack-dev01:33
*** ladquin has joined #openstack-dev01:33
jamielennoxsame file L1983, if i simply remove id from data won't it simply use the token_id being passed? this pattern is repeated all over the place, and i'm not sure why the replication01:35
lifelessayoung: are you familiar with snowflakes design ?01:35
lifelessayoung: it's a distributed unique number generator [guessable but unique]01:35
ayoungjamielennox, yes, the logic should be pushed up to the manager...we weren;t really pushing that pattern in the past, so the replication was done in the individual backends.  Needs to be refactored01:36
*** torandu has quit IRC01:37
ayounglifeless, until now we had no need for it.  We could implement something comparable if required. But  we weren't doing anything that broke just distributing keystone (at least, not knowingly)01:37
lifelessayoung: commented on the bug for you01:37
ayounglifeless, thanks01:37
*** torandu has joined #openstack-dev01:38
*** ladquin is now known as ladquin_away01:39
ayoungjamielennox, we might not want to use that bug number in the fix, or we might have to split it, if we need to maintain a finer granularity.  I suspect, though, that we should be OK with 1 second...01:39
ayoungjamielennox, once you get the sql one done, you'll need to rebase https://review.openstack.org/#/c/28387/01:42
ayoungthere is a button for it right there on gerrit, but only the submitter has it.01:42
ayoungso I can;'t push the button for you.01:42
ayoungnote that the "depends on" shows it to be outdated.01:42
jamielennoxok, i was considering abandoning that and breaking it down01:44
jamielennoxat the moment no one seems game to even comment on it01:44
jamielennoxwhich is understandable, it's too big to follow01:44
jamielennoxayoung: drop the bug id then?01:45
*** erkules has quit IRC01:45
*** portante|ltp has joined #openstack-dev01:45
ayoungyeah, I think so/.  bug id is going to be used for the real fix01:45
*** rykario has joined #openstack-dev01:45
ayoungjamielennox, I think you need to break down the big commit....it is too hard to follow, and it doesn't really seem to buy much.01:46
ayoungNot a bad exercise, though, you probably understand Keystone better than most now for having done that.  And maybe we'll use that as a target for how things should look in the future01:47
ayoungI was going to bring it up at tuesdays meeting, but tuesday got bogged down in LDAP domain issues01:47
ayoungjamielennox, checking out for the night01:47
jamielennoxyea, the problem is that particularly the rearrange stuff might be hard to push through if it doesn't bring anything01:47
jamielennoxayoung: quick questoin on presenting it01:48
ayounggo for it01:48
jamielennoxif i go about a big rearrange in tests what sort of size should i be looking for with the patches01:48
jamielennox1 big rearrange patch01:48
jamielennoxor like extract backend tests01:48
jamielennoxextract legacy tests01:48
jamielennoxextract v2 tests01:48
lifelessjamielennox: anything over ~300 lines is really quite hard for reviewers.01:48
ayoungwell, smaller is always easier to review, but maybe write up a blueprint of the overall end state01:49
lifelessits a nonlinear function and some reviewers have a different threshold01:49
ayoungexplain what and why, and then people have context prior to looking at the review01:49
jamielennoxit's more that it will be quite a large depends on stack and a lot of moving things without real addition01:50
jamielennoxthose seem to be difficult to pass01:50
jamielennoxeg termie's no on tests -> keystone.tests01:50
jamielennoxbut the bp makes sense, provides a way to tie all the patches together01:51
*** soody has joined #openstack-dev01:51
ayoungjamielennox, yeah, and it will let people put in some input as well...write it up, and present the case for both the reorg and the extraction of eventlet/replace with webtest01:51
* ayoung now has to figure out what broke with the migrations on the credentials patch01:52
jamielennoxi'm good to keep those 2 concepts seperate, the rearrange should happen first but it's useful event if the eventlet thing doesn't happen01:52
jamielennoxanyway, will do, talk to you later01:53
ayoungsounds good.  jamielennox btw, I think I want to hand off the "tie token to X509/ Kerberos" to you after you iron out these issues.  I think I have enough other stuff on my plate that I can't commit to getting that done, but I think it is strategic01:54
ayoungwe can discuss in detail later, I just wanted to give you an early warning01:54
jamielennoxyea, that's fine i was thinking i'd probably be doing at least the X509 part of that01:55
ayoungjamielennox, glad to hear it.01:55
ayoungjamielennox, btw, one of my proudest hacks is ./run_tests.sh -801:56
ayoungjamielennox, it is the short form of the pep8 check, one entry per line01:56
ayounga nice to have hack for big patches01:56
jamielennoxwhich is not listed in the --help01:56
ayoungthought I had added it, guess not.01:57
ayoung I'll add it.01:57
jamielennoxvim tells me if there are pep8 problems, unfortunately in some places there are so many like unused import and unused variable that they get lost01:58
*** Mandell has quit IRC01:58
ayoungjamielennox I wonder if we can add that check optionally to our pep801:58
*** freedomhui has joined #openstack-dev01:59
ayoungjamielennox, I wonder if something in my patch broke under 2.602:00
ayounghttp://logs.openstack.org/28372/3/check/gate-keystone-python26-centos6/88/console.html.gz02:00
jamielennoxi might have a look what the difference between my version / run_test is and see. I've got a little script i use for doing database tests that i use instead of run_test02:00
*** sthaha has joined #openstack-dev02:00
ayoungreally starting to wish that devstack used puppet instead of doing it all by hand.  It would be so much more maintainable02:02
jamielennoxthat's the centos gate, i don't think that passes in general02:03
*** jasondotstar has joined #openstack-dev02:03
clarkbhowever it should be made to pass02:03
clarkbbecause oneiric will hopefully go away tomorrow. I thought fungi said all master branches for the core proejcts were passing on centos02:03
clarkbfungi: ^02:03
clarkb(oneiric is eol tomorrow so we don't want to keep those slaves around)02:04
*** gongysh has joined #openstack-dev02:04
*** bing_bu has joined #openstack-dev02:04
fungimmm02:04
fungithey... were02:05
ayoung jamielennox i think it is my patch.  It was passing before.02:05
fungidid keystone break there again?02:05
fungiahh02:05
*** danwent has quit IRC02:05
ayoungjamielennox, on https://review.openstack.org/#/c/27878/6..8/tests/test_backend.py  why areyou pooping user_id02:05
ayoungpopping02:05
*** erkules has joined #openstack-dev02:05
fungiayoung: yeah, as of tomorrow it'll start breaking again. ubuntu is abandoning oneiric as of tomorrow, and with it python 2.6. centos is our way out02:05
*** rykario has quit IRC02:06
jamielennoxhttps://review.openstack.org/#/c/27878/5..8/tests/test_backend.py02:06
jamielennoxayoung: wasn't me02:06
jamielennoxdamn it, this one: https://review.openstack.org/#/c/27878/3..8/tests/test_backend.py02:07
*** hartsocks has left #openstack-dev02:08
jamielennoxpoint is if you look at the full diff the pop user_id was there before me02:08
jamielennoxI'm not sure why it's done, but it was working so i left it02:09
ayoungjamielennox, good enough02:09
ayoungjamielennox, I tend to just look at the diff between two patches if they come in close together.  Doesn't always work02:09
jamielennoxi seemed to have a bit of a day of it yesterday that everything i put up for review had an issue i found after02:10
ayoungfungi, so the breakage I saw on my test was due to centos, not my patch>02:10
ayoung?02:10
jamielennoxthat patch is up to 8 and it's only had 1 other person comment on it02:10
ayoungjamielennox, never fear.  we'll get others to beat on it tomorrow02:10
*** beagles has quit IRC02:11
fungii'll take a look but i think dprince had gotten enough fixes into keystone master that unit tests were working for it on centos last week02:11
*** aeperezt has joined #openstack-dev02:11
ayoungjamielennox, I think this one is the record https://review.openstack.org/#/c/20289/02:12
fungithis is the danger with non-voting gate jobs, but we unfortunately have to start somewhere. tough to stay on top of without per-test latching02:12
ayoungfungi, I'm sure the red hatters in the audience will stay on top of it02:12
jamielennoxayoung: lol, that's a lot. Substantially bigger change but02:13
ayoungfungi, we kindof have a vested interest in it02:13
*** sean_s has quit IRC02:13
ayoungjamielennox, and there was a follow on patch that had all of the tests for it, with almost as many reviews02:13
*** anniec has quit IRC02:14
*** freedomhui has quit IRC02:14
fungierg, yeah gate-keystone-python26-centos6 has been failing on master all day at least, so it broke again at some point02:15
jamielennoxayoung: thanks :) https://bugs.launchpad.net/bugs/117777702:16
uvirtbotLaunchpad bug 1177777 in keystone "Only MII tokens get properly hashed" [Undecided,New]02:16
fungitrying to keep them all working across a couple dozen projects when people are allowed to merge changes that break them is not exactly easy, and i don't relish that we'll instabreak all projects which get themselves into that state when we have to cut over to testing on that platform tomorrow02:16
ayoungfungi, Do you need some help getting the Centos thing hunted down?02:19
*** nunosantos has quit IRC02:20
jamielennoxayoung: regarding guang-yee's comments on the token flush patch, do you think we worry about the grace_period thing for now or just do the docs and resubmit02:20
fungiayoung: possibly. i'm trying to find where it was last succeeding on master02:20
ayoungjamielennox, grace period is a follow on feature request02:21
fungiin hopes of narrowing down which change broke it02:21
ayoungfungi, do we have a good error message for what was failing?02:21
ayoungI don;t see it in the logs...still looking02:21
fungii can get one... just a sec02:21
*** rcleere has joined #openstack-dev02:21
ayoung AuthorizationFailure: Authorization Failed: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP Unable to establish connection to http://localhost:39... isn't a lot to go on02:22
clarkbfungi: use the logstash luke02:22
fungieep, looking at http://logs.openstack.org/28387/4/check/gate-keystone-python26-centos6/78/console.html.gz for example there's a whole slew of failures (that same patchset succeeded py26 unit tests on ubuntu oneiric). i could have sworn keystone master was working on centos/rhel last week, but maybe i was mistaken02:24
*** bmclaughlin has quit IRC02:26
*** gyee has quit IRC02:26
*** rowleyaj has joined #openstack-dev02:27
ayoungfungi, they are all from the same root source02:27
ayoungKeystone not starting02:27
ayoungguessing we are doing something python2.7 ish02:27
ayoungbut the stack trace showing what failed is missing02:27
ayoungfungi, clarkb need the keystone log from a devstack startup02:28
*** freedomhui has joined #openstack-dev02:28
*** sthaha has quit IRC02:28
clarkbayoung: its there02:28
ayounglink?02:28
clarkbayoung: can you link to a failure of some sort?02:28
fungiayoung: devstack startup? the centos failures are on unit tests instead02:28
ayoungfungi, yeah, but the keystone server side is what is failing, and we are not getting that in the log02:29
ayoungit is the client calling into the server that fails there02:29
clarkbbut devstack is not involved in unittests02:29
ayoungclarkb, see the other test failures, though, and there should be devstack/keystone issues02:29
ayoungfor example...02:29
clarkbayoung: link please...02:29
*** markmcclain has joined #openstack-dev02:29
ayounghttp://logs.openstack.org/28372/3/check/gate-tempest-devstack-vm-full/15119/02:29
ayoungthat was from my patch, and I assumed it was something I did, (i'm changing a migration) but I can't find the keystone log02:30
clarkbayoung: http://logs.openstack.org/28372/3/check/gate-tempest-devstack-vm-full/15119/logs/screen-key.txt.gz02:30
ayoungclarkb, thanks02:30
ayoung(root): 2013-05-08 21:43:48,962 CRITICAL logging wrapper Unregistered dependency: credentials_api02:30
ayoungyeah, that is me02:30
ayoungshould be specific to my patch, then, and not your root problem02:31
ayoungclarkb, OK, lets look at one that failed the gate on centos not due to my changes....02:31
ayoungdo you have one?02:31
clarkbayoung: fwiw the logs/ dir on those tempest results pages have all kinds of logs captured for you in them02:31
ayoungclarkb, I knew that, just couldn't find them under duress.02:32
* ayoung can't hack the pressure02:32
*** freedomhui has left #openstack-dev02:32
clarkbayoung: 27878, 28372, 28197, 27364 all fail the centos unittests02:33
ayoungclarkb, OK, keystone seems to be serving on http://logs.openstack.org/27364/11/check/gate-tempest-devstack-vm-full/15123/logs/screen-key.txt.gz02:34
ayounglets see...02:34
ayounghttp://logs.openstack.org/27364/11/check/gate-keystone-python26-centos6/89/console.html.gz  OK that shows the failure02:35
ayoungclarkb, not enough eggs to make an omlette02:36
ayoungI'll need a centos/rhel box to debug02:36
ayounglet me see if I have one02:37
ayoungclarkb, OK, I have a rhel 6.4 VM firing up.  I'll try to run the unit tests on there in a few02:38
clarkbawesome, thanks02:38
*** yaguang has joined #openstack-dev02:39
*** markmcclain has quit IRC02:39
clarkbayoung: you might just try running the tests  with keystone master and not on any particular change02:41
*** freedomhui has joined #openstack-dev02:43
fungiyeah, i get the impression master should be exhibiting the same failures (though i haven't disengaged one of our centos 6.3 jenkins slaves to confirm that for certain)02:43
*** sthaha has joined #openstack-dev02:47
ayoungclarkb, just dealing with getting a yum setup,  will do a git checkout in a moment02:47
*** cmark_ has joined #openstack-dev02:48
*** noslzzp has quit IRC02:50
*** cmark has quit IRC02:51
*** torandu has quit IRC02:52
*** torandu has joined #openstack-dev02:52
*** noslzzp has joined #openstack-dev02:52
*** koolhead17 has joined #openstack-dev02:54
*** soody has quit IRC02:56
*** ctracey has quit IRC02:56
*** winston-d has joined #openstack-dev02:59
*** ctracey has joined #openstack-dev03:01
*** SergeyLukjanov has joined #openstack-dev03:01
*** novas0x2a|laptop has quit IRC03:04
*** ashwini has quit IRC03:04
*** winston-d has quit IRC03:06
ayoungclarkb, fungi I have to crash for tonight.  Give me a shout if things are still borked up tomorrow and I'll keep looking03:06
fungiayoung: okay, thanks for taking a look03:07
ayoungfungi, my keystone is failing running the tests with ImportError: No module named argparse03:08
ayoung  when building the venv, so It must be something pretty basic.03:08
*** reed has joined #openstack-dev03:08
clarkbayoung: are you using tox?03:08
clarkbtox is what jenkins would use. `tox -epy26`03:08
fungimmm, yeah that implies that we're missing a requires entry for argparse (not part of stdlib until 2.7)03:09
*** soody has joined #openstack-dev03:10
*** mikal has quit IRC03:10
*** pleia2 has quit IRC03:11
ayoungfungi, nah, just that my vm is limited and I am pointing at the wrong yum repo03:11
*** mikal has joined #openstack-dev03:11
*** uvirtbot has quit IRC03:11
*** ianw has quit IRC03:11
*** pleia2 has joined #openstack-dev03:11
clarkbthe install venv script wouldn't affect jenkins (that is where argparse is imported)03:11
clarkbI would use tox03:11
fungioh, well as clarkb suggests if you use tox it should install all listed requirements into a venv for you automagically, yes03:12
ayoungclarkb, I user ./run_test.sh which builds the venv, but that needs argparse to even work right.  My vm needs something else03:12
matiudoes anyone know if keystone has a 'reauthenticate' sort of thing, where I give it an expired token, and it renews it for me (but without me having the password)03:12
ayoungmatiu, it most certainly does not03:12
matiuyeah, so if I have an expired token on the server, I'm can't continue to process the request ?03:13
ayoungmatiu, correct03:13
matiuI'm having trouble where the request comes in, we start a nova build03:13
matiuthen when glance gets its turn the token is e03:13
matiuexpired ..03:14
ayoungmatiu, how long are your tokens good for, and how long does the build take?03:14
matiuas it was like 24 hours after creation03:14
matiubuild can take 15 mins03:14
matiuI'm thinking to add an option in the keystoneclient middleware03:14
*** ianw has joined #openstack-dev03:14
ayoungmatiu, nothing will help you there03:14
matiuso that if the token is 15 minutes from expiry, it'll just wipe it out03:14
*** freedomhui has left #openstack-dev03:15
ayoungyou can;t get a new token with a longer lifespan than the origianal token03:15
ayoungunless you have password03:15
matiuit already has code in there, for wiping out tokens that are 30 seconds from expiry03:15
matiuyeah, so my plan is to fail the request at the start03:15
ayoungOK,  so sounds like you have what you need03:15
matiuand force the client to re-auth03:15
ayoungmatiu, is this grizzly?03:15
matiuthanks for the sanity check ayoung03:15
matiuit's trunk03:15
*** adjohn_ has quit IRC03:15
ayoungmatiu, use a trust instead03:15
ayoungif you can03:16
matiuit's like from last November03:16
matiuwhat's trust ?03:16
ayoungits a constrained delegation03:16
matiudo you have any links for that ?03:16
ayounginstead of using the origianl token, you create a trust and use that to fetch a new one.03:16
ayoungyeah...one sec03:16
matiuthanks so much ayoung03:17
ayoungmatiu, https://github.com/openstack/identity-api/blob/master/openstack-identity-api/src/markdown/identity-api-v3-os-trust-ext.md03:17
matiuah, so that's not released yet eh ?03:17
ayoungthis is what that feature was built for, matiu but it has not been consumed much yet, so you might be breaking new ground03:17
ayoungits in grizzly, so it is released03:17
*** rowleyaj has left #openstack-dev03:17
matiuah sweet03:17
ayoungyou are welcome.  It ate up a good two months of my life03:18
matiuThanks ayoung . You're awesome03:18
ayoungno,  barely above average03:18
ayoungmatiu, I am going to be signing off here.  fire me an email if you need more info.  ayoung@redhat.com03:19
matiuhave a great night ayoung03:19
*** CaptTofu has quit IRC03:21
*** ijw has quit IRC03:25
*** ayoung has quit IRC03:29
*** stevemar has joined #openstack-dev03:31
*** soody has quit IRC03:32
*** anniec has joined #openstack-dev03:33
openstackgerritA change was merged to openstack/keystone: Replace password to "***" in the debug message  https://review.openstack.org/2648703:36
*** uvirtbot has joined #openstack-dev03:36
*** soody has joined #openstack-dev03:36
*** anniec_ has joined #openstack-dev03:39
*** anniec has quit IRC03:40
*** anniec_ is now known as anniec03:40
*** Mandell has joined #openstack-dev03:46
*** soody has quit IRC03:52
*** melwitt has quit IRC03:52
*** danwent has joined #openstack-dev03:54
*** soody has joined #openstack-dev03:54
*** jergerber has quit IRC03:54
*** soody has quit IRC04:04
*** soody has joined #openstack-dev04:05
*** jimfehlig has quit IRC04:05
*** SergeyLukjanov has quit IRC04:07
*** jasondotstar has quit IRC04:10
*** winston-d has joined #openstack-dev04:13
*** abhishekkr has quit IRC04:14
*** anniec has quit IRC04:18
*** bdpayne has joined #openstack-dev04:21
*** soody has quit IRC04:24
*** rcleere has quit IRC04:26
*** nati_ueno has joined #openstack-dev04:31
*** soody has joined #openstack-dev04:32
*** koolhead17 has quit IRC04:33
*** swaT30 has quit IRC04:34
*** aeperezt has quit IRC04:41
*** stevemar has quit IRC04:44
*** abhishekkr has joined #openstack-dev04:44
*** koolhead17 has joined #openstack-dev04:46
*** mynamiesdeleted is now known as mynameisdeleted204:51
*** glikson has joined #openstack-dev04:52
*** ijw has joined #openstack-dev04:55
*** topol has joined #openstack-dev04:55
*** sandywalsh has quit IRC04:58
*** zaitcev has quit IRC04:59
*** prekarat has joined #openstack-dev05:04
*** bdpayne has quit IRC05:07
*** sacharya has quit IRC05:11
*** bdpayne has joined #openstack-dev05:12
*** soody has quit IRC05:14
*** nati_ueno has quit IRC05:17
*** Gman2 has joined #openstack-dev05:20
*** lloydde has joined #openstack-dev05:20
*** freedomhui has joined #openstack-dev05:24
*** abhishekkr_ has joined #openstack-dev05:27
*** abhishekkr has quit IRC05:30
*** crandquist has joined #openstack-dev05:31
openstackgerritA change was merged to openstack/cinder: Copy the RHEL6 eventlet workaround from Oslo  https://review.openstack.org/2802205:34
*** Zah_ has quit IRC05:35
*** vipul is now known as vipul|away05:35
*** Mandell has quit IRC05:35
*** sandywalsh has joined #openstack-dev05:36
*** LiLin has joined #openstack-dev05:36
*** LiLin has left #openstack-dev05:36
*** freedomhui has quit IRC05:38
*** Aarti has joined #openstack-dev05:38
*** navid_ has joined #openstack-dev05:38
*** freedomhui has joined #openstack-dev05:40
*** vipul|away is now known as vipul05:41
*** freedomhui has quit IRC05:41
*** freedomhui has joined #openstack-dev05:42
*** vartom11110 has joined #openstack-dev05:43
*** ijw has quit IRC05:44
*** dguitarbite has joined #openstack-dev05:52
*** vartom11110 has quit IRC05:52
*** souvik has joined #openstack-dev05:56
*** brunnhilde has quit IRC05:57
*** yaguang is now known as yaguang_afk05:57
*** Zah_ has joined #openstack-dev05:58
*** prekarat has quit IRC06:07
*** garyk has joined #openstack-dev06:09
*** jhesketh__ has quit IRC06:19
*** adjohn has joined #openstack-dev06:20
*** prekarat has joined #openstack-dev06:20
*** jprovazn has joined #openstack-dev06:21
*** juice_otg has joined #openstack-dev06:23
*** adjohn has quit IRC06:23
*** juice_ot_ has joined #openstack-dev06:24
*** danwent has quit IRC06:26
*** freedomhui has joined #openstack-dev06:27
*** juice_otg has quit IRC06:28
*** kenperkins has quit IRC06:29
*** topol has quit IRC06:32
*** cmark_ has quit IRC06:33
*** cmark has joined #openstack-dev06:33
*** koolhead17 has quit IRC06:36
*** giroro_ has joined #openstack-dev06:36
*** juice_ot_ has quit IRC06:37
*** Ruetobas has quit IRC06:39
*** maploin has joined #openstack-dev06:41
*** maploin has quit IRC06:41
*** maploin has joined #openstack-dev06:41
*** michchap has quit IRC06:43
*** michchap has joined #openstack-dev06:43
*** yuanz has joined #openstack-dev06:45
*** vipul is now known as vipul|away06:48
*** vipul|away is now known as vipul06:48
*** koolhead17 has joined #openstack-dev06:48
*** mkollaro has joined #openstack-dev06:48
*** mkollaro has quit IRC06:48
*** nati_ueno has joined #openstack-dev06:49
*** glikson has quit IRC06:51
*** dguitarbite has quit IRC06:53
*** jcoufal has joined #openstack-dev06:54
*** jclift has joined #openstack-dev06:55
*** reidrac has joined #openstack-dev06:56
*** PooBee has joined #openstack-dev06:56
*** jtomasek has joined #openstack-dev06:56
*** PooBee has left #openstack-dev06:57
*** yongli has joined #openstack-dev06:58
*** bdpayne has quit IRC06:59
*** mindpixel has joined #openstack-dev06:59
*** lloydde has quit IRC07:01
*** sarob has quit IRC07:04
*** dguitarbite has joined #openstack-dev07:04
*** sarob has joined #openstack-dev07:04
*** freedomhui has quit IRC07:06
*** bing_bu has quit IRC07:06
*** afazekas has joined #openstack-dev07:06
*** utlemming has joined #openstack-dev07:07
*** baba has quit IRC07:08
*** megha has joined #openstack-dev07:08
*** megha is now known as baba07:09
*** giulivo has joined #openstack-dev07:13
*** mmagr has joined #openstack-dev07:14
*** glikson has joined #openstack-dev07:16
*** eglynn has joined #openstack-dev07:17
*** bing_bu has joined #openstack-dev07:18
*** FnordDownUnder has quit IRC07:19
*** shardy_afk is now known as shardy07:20
*** m4xmr has joined #openstack-dev07:23
*** m4xmr has left #openstack-dev07:24
*** eglynn has quit IRC07:25
*** johnthetubaguy has joined #openstack-dev07:28
*** mkollaro has joined #openstack-dev07:29
*** salv-orlando has joined #openstack-dev07:29
*** juice_otg has joined #openstack-dev07:31
*** lloydde has joined #openstack-dev07:31
*** adjohn has joined #openstack-dev07:34
*** aloga has joined #openstack-dev07:36
*** ndipanov has joined #openstack-dev07:38
*** adjohn has quit IRC07:38
*** eglynn has joined #openstack-dev07:39
*** lloydde has quit IRC07:41
*** Mandell has joined #openstack-dev07:41
*** souvik has quit IRC07:42
*** prekarat1 has joined #openstack-dev07:43
*** juice_otg has quit IRC07:44
*** mkollaro has quit IRC07:47
*** prekarat has quit IRC07:47
*** utlemming has quit IRC07:50
*** psedlak has joined #openstack-dev07:52
*** boris-42 has joined #openstack-dev07:52
*** jaume has joined #openstack-dev07:52
*** jaume is now known as dev_sa07:53
*** afazekas has quit IRC07:54
*** michchap has quit IRC07:54
*** afazekas has joined #openstack-dev07:54
*** michchap has joined #openstack-dev07:54
*** prekarat has joined #openstack-dev07:57
*** prekarat1 has quit IRC07:58
*** boris-42 has quit IRC08:01
*** jpich has joined #openstack-dev08:02
*** ndipanov has quit IRC08:03
*** jcoufal has quit IRC08:05
*** jcoufal has joined #openstack-dev08:06
*** kpavel has joined #openstack-dev08:06
*** bing_bu has quit IRC08:06
*** afazekas has quit IRC08:10
*** prekarat has quit IRC08:15
*** freedomhui has joined #openstack-dev08:17
*** athomas has joined #openstack-dev08:19
*** ndipanov has joined #openstack-dev08:19
*** mynameisdeleted2 has quit IRC08:21
*** mynameisdeleted2 has joined #openstack-dev08:21
*** prekarat has joined #openstack-dev08:22
*** afazekas has joined #openstack-dev08:22
*** lucasagomes has joined #openstack-dev08:22
*** prekarat has quit IRC08:22
*** prekarat has joined #openstack-dev08:22
*** derekh has joined #openstack-dev08:26
*** nati_ueno has quit IRC08:26
*** nati_ueno has joined #openstack-dev08:27
*** nati_ueno has quit IRC08:31
*** lloydde has joined #openstack-dev08:32
*** gnorth has quit IRC08:32
*** bing_bu has joined #openstack-dev08:32
*** alpha_ori has quit IRC08:32
*** alpha_ori has joined #openstack-dev08:32
*** tkammer has joined #openstack-dev08:35
*** AnilV4 has quit IRC08:36
*** lloydde has quit IRC08:36
*** markmc has joined #openstack-dev08:39
*** Mandell has quit IRC08:41
*** AnilV4 has joined #openstack-dev08:41
*** jruzicka has joined #openstack-dev08:46
*** sarob has quit IRC08:49
*** sarob has joined #openstack-dev08:49
*** danpb has joined #openstack-dev08:51
*** tkammer has quit IRC08:52
*** roz has quit IRC08:54
*** sarob has quit IRC08:54
*** winston-d has quit IRC08:54
openstackgerritA change was merged to openstack/quantum: Don't run metadata proxy when it is not needed  https://review.openstack.org/2643108:55
*** belmoreira has joined #openstack-dev08:57
*** pixelbeat_ has joined #openstack-dev08:59
*** afazekas has quit IRC09:01
*** souvik has joined #openstack-dev09:05
*** tkammer has joined #openstack-dev09:05
*** prekarat has quit IRC09:07
*** ndipanov has quit IRC09:08
*** boris-42 has joined #openstack-dev09:12
*** ndipanov has joined #openstack-dev09:22
*** roz has joined #openstack-dev09:24
*** bing_bu has quit IRC09:29
*** lloydde has joined #openstack-dev09:33
*** lloydde has quit IRC09:37
rozI submitted a change and jenkis jobs ran (more than 12 hours ago)  but I can't see SmokeStack, do I need to take any actions?09:42
rozand second question some jenkins jobs are reporting a FAILURE but they are non-voting jobs. What that means?09:43
*** jhesketh_ has joined #openstack-dev09:45
johnthetubaguydepends on the job I think, could be a pylint error09:46
johnthetubaguyit should link to the error log09:46
rozone of them are errors in tempest tests,  I'd like to understand which errors are considered "safe"09:47
*** maploin has quit IRC09:49
johnthetubaguyI think it depends why it failed, have to read the logs really09:50
johnthetubaguybut I think that tempest one is just not proving reliable yet, hence its optional09:50
johnthetubaguyits probably fine09:50
johnthetubaguybut there are experts on this, and that is not me!09:51
rozok thanks for your response.09:51
*** johnthetubaguy1 has joined #openstack-dev09:51
*** prekarat has joined #openstack-dev09:52
*** afazekas has joined #openstack-dev09:55
*** johnthetubaguy has quit IRC09:55
*** yaguang_afk has quit IRC10:07
*** athomas has quit IRC10:08
*** athomas has joined #openstack-dev10:11
*** baba has quit IRC10:13
*** mkollaro has joined #openstack-dev10:14
*** koolhead17 has quit IRC10:16
*** koolhead17 has joined #openstack-dev10:16
*** alexpilotti has joined #openstack-dev10:20
*** SergeyLukjanov has joined #openstack-dev10:21
*** Aarti has quit IRC10:23
*** johnthetubaguy1 is now known as johnthetubaguy10:26
*** baba has joined #openstack-dev10:31
*** lloydde has joined #openstack-dev10:33
*** dprince has joined #openstack-dev10:34
*** lloydde has quit IRC10:38
*** mindpixel has joined #openstack-dev10:41
*** koolhead17 has quit IRC10:45
*** koolhead17 has joined #openstack-dev10:46
*** jcoufal has quit IRC10:46
*** souvik has quit IRC10:47
*** jhesketh_ has quit IRC10:49
*** jcoufal has joined #openstack-dev10:49
*** jhesketh_ has joined #openstack-dev10:50
*** dprince has quit IRC10:50
*** rkukura has quit IRC10:55
openstackgerritA change was merged to openstack/nova: Removes unnecessary check for admin context in evacuate.  https://review.openstack.org/2855110:55
*** deuterium has joined #openstack-dev10:56
*** beagles has joined #openstack-dev10:59
*** topol has joined #openstack-dev11:03
*** freedomhui has quit IRC11:04
*** SergeyLukjanov has quit IRC11:04
*** jhesketh_ has quit IRC11:07
*** prekarat has quit IRC11:09
*** giulivo has quit IRC11:10
*** sride has joined #openstack-dev11:13
*** souvik has joined #openstack-dev11:14
*** gongysh has quit IRC11:16
*** Yada has joined #openstack-dev11:16
*** drewlander has joined #openstack-dev11:25
*** SergeyLukjanov has joined #openstack-dev11:29
*** freedomhui has joined #openstack-dev11:30
*** pcm__ has joined #openstack-dev11:31
*** dprince has joined #openstack-dev11:32
*** ijw has joined #openstack-dev11:33
*** lloydde has joined #openstack-dev11:33
*** dguitarbite has quit IRC11:38
*** lloydde has quit IRC11:38
*** matiu has quit IRC11:38
*** mikal has quit IRC11:39
*** CaptTofu has joined #openstack-dev11:40
*** mikal has joined #openstack-dev11:41
*** tkammer has quit IRC11:52
*** wiliam_ has joined #openstack-dev11:57
*** sandywalsh has quit IRC11:57
*** SergeyLukjanov has quit IRC11:58
*** SergeyLukjanov has joined #openstack-dev11:59
*** networkstatic has quit IRC12:00
*** wiliam_ is now known as wiliam12:01
*** READ10 has joined #openstack-dev12:02
*** ijw has quit IRC12:05
*** wiliam has quit IRC12:08
*** wiliam has joined #openstack-dev12:09
*** dev_sa has quit IRC12:09
*** wiliam is now known as waa12:10
*** sandywalsh has joined #openstack-dev12:10
*** sandywalsh has quit IRC12:10
*** sandywalsh has joined #openstack-dev12:10
*** flaper87 has joined #openstack-dev12:12
*** m4xmr has joined #openstack-dev12:13
*** m4xmr has left #openstack-dev12:13
*** digitalsanctum has joined #openstack-dev12:14
*** lpmulligan has joined #openstack-dev12:16
*** tzumainn has joined #openstack-dev12:19
*** CaptTofu has quit IRC12:25
*** primeministerp has quit IRC12:25
*** mjfork has joined #openstack-dev12:27
*** SergeyLukjanov has quit IRC12:28
*** anniec has joined #openstack-dev12:30
*** anniec_ has joined #openstack-dev12:33
*** lloydde has joined #openstack-dev12:34
*** anniec has quit IRC12:35
*** anniec_ is now known as anniec12:35
*** AlanClark has quit IRC12:35
*** bknudson has joined #openstack-dev12:35
*** jjmb has quit IRC12:36
*** SergeyLukjanov has joined #openstack-dev12:37
*** adalbas has joined #openstack-dev12:38
*** acfleury has joined #openstack-dev12:38
*** ndipanov has quit IRC12:38
*** lloydde has quit IRC12:40
*** primeministerp has joined #openstack-dev12:44
*** navid_ has quit IRC12:45
*** CaptTofu has joined #openstack-dev12:49
*** freedomhui has quit IRC12:50
*** sacharya has joined #openstack-dev12:50
*** digitalsanctum has quit IRC12:50
*** ndipanov has joined #openstack-dev12:51
*** digitalsanctum has joined #openstack-dev12:52
*** martine has joined #openstack-dev12:57
*** mindpixel has quit IRC12:57
*** kbringard has joined #openstack-dev12:58
*** adalbas has quit IRC12:58
*** acfleury has quit IRC12:58
*** eharney has quit IRC13:03
*** lauria has joined #openstack-dev13:03
*** tkammer has joined #openstack-dev13:04
*** pfreund has joined #openstack-dev13:09
*** lpmulligan has quit IRC13:10
*** jayg|g0n3 is now known as jayg13:11
*** acfleury has joined #openstack-dev13:12
*** adalbas has joined #openstack-dev13:12
*** souvik has quit IRC13:12
*** radez_g0n3 is now known as radez13:13
*** michchap_ has joined #openstack-dev13:15
*** michchap has quit IRC13:15
*** salv-orlando has quit IRC13:16
*** alunduil has quit IRC13:21
*** morazi has joined #openstack-dev13:22
*** Gman2 has quit IRC13:22
*** lpmulligan has joined #openstack-dev13:22
*** bmclaughlin has joined #openstack-dev13:23
*** rnirmal has joined #openstack-dev13:24
*** sride has quit IRC13:25
*** ladquin_away is now known as ladquin13:28
*** mkollaro1 has joined #openstack-dev13:28
*** mkollaro has quit IRC13:28
*** portante|ltp has quit IRC13:29
*** afazekas_ has joined #openstack-dev13:29
*** afazekas_ has quit IRC13:29
*** afazekas_ has joined #openstack-dev13:30
*** giulivo has joined #openstack-dev13:30
*** afazekas has quit IRC13:31
*** giulivo has quit IRC13:33
*** jecarey has joined #openstack-dev13:33
*** bswartz has quit IRC13:34
*** lloydde has joined #openstack-dev13:34
*** giulivo has joined #openstack-dev13:35
*** deuterium has quit IRC13:35
*** navid_ has joined #openstack-dev13:38
*** lloydde has quit IRC13:39
*** ayoung has joined #openstack-dev13:40
*** eharney has joined #openstack-dev13:40
*** sthaha has quit IRC13:43
*** sthaha has joined #openstack-dev13:43
*** sthaha has joined #openstack-dev13:43
*** rkukura has joined #openstack-dev13:44
*** morazi has quit IRC13:45
*** morazi has joined #openstack-dev13:45
*** deuterium has joined #openstack-dev13:48
*** aelkikhia has joined #openstack-dev13:48
*** deuterium has joined #openstack-dev13:49
*** sthaha has quit IRC13:50
*** boris-42 has quit IRC13:50
*** markmcclain has joined #openstack-dev13:50
*** dguitarbite has joined #openstack-dev13:51
*** stevemar has joined #openstack-dev13:51
*** sacharya has quit IRC13:52
*** aelkikhia has left #openstack-dev13:52
*** AlanClark has joined #openstack-dev13:52
*** sthaha has joined #openstack-dev13:53
eharneyhttp://docs.openstack.org/developer/cinder/ has a broken link to "Cinder 2011.2 docs"13:53
*** jimfehlig has joined #openstack-dev13:53
*** jjmb has joined #openstack-dev13:54
*** maoy has joined #openstack-dev13:55
*** markmc has quit IRC13:56
*** cloudchimp has joined #openstack-dev13:58
*** jasondotstar has joined #openstack-dev13:59
*** HenryG has joined #openstack-dev13:59
*** utlemming has joined #openstack-dev14:01
*** jjmb has quit IRC14:02
*** jjmb has joined #openstack-dev14:03
*** topol has quit IRC14:03
*** bswartz has joined #openstack-dev14:03
*** ttx has quit IRC14:03
*** rcleere has joined #openstack-dev14:04
ladquineharney, thanks for the heads up!14:05
*** redbeard2 has joined #openstack-dev14:05
*** ttx has joined #openstack-dev14:06
*** ttx has quit IRC14:06
*** ttx has joined #openstack-dev14:06
*** mrodden has joined #openstack-dev14:06
*** alunduil has joined #openstack-dev14:06
*** jjmb has quit IRC14:06
*** swaT30 has joined #openstack-dev14:07
*** kenperkins has joined #openstack-dev14:08
*** acfleury has quit IRC14:09
*** adalbas has quit IRC14:10
*** gordc has joined #openstack-dev14:10
*** jvrbanac has joined #openstack-dev14:12
*** ayoung has quit IRC14:12
*** galstrom_zzz is now known as galstrom14:14
*** networkstatic has joined #openstack-dev14:17
*** salv-orlando has joined #openstack-dev14:19
*** ttx has quit IRC14:20
*** ttx has joined #openstack-dev14:21
*** ttx has quit IRC14:21
*** ttx has joined #openstack-dev14:21
*** adalbas has joined #openstack-dev14:23
*** acfleury has joined #openstack-dev14:24
*** aelkikhia1 has joined #openstack-dev14:25
*** utlemming has quit IRC14:25
*** galstrom is now known as galstrom_zzz14:26
*** ayoung has joined #openstack-dev14:27
*** jaypipes has joined #openstack-dev14:28
*** acfleury has quit IRC14:28
*** bmclaughlin has quit IRC14:29
*** Aarti has joined #openstack-dev14:31
*** wwallnrr__ has quit IRC14:32
*** hartsocks has joined #openstack-dev14:32
*** garyk has quit IRC14:33
*** spzala has joined #openstack-dev14:33
*** datsun180b has joined #openstack-dev14:35
*** ndipanov has quit IRC14:36
*** topol has joined #openstack-dev14:36
*** zyluo has joined #openstack-dev14:39
*** terryh has joined #openstack-dev14:40
*** diogogmt has joined #openstack-dev14:41
*** FunnyLookinHat has joined #openstack-dev14:42
*** dolphm has joined #openstack-dev14:42
*** bmclaughlin has joined #openstack-dev14:42
*** cdsvrd has joined #openstack-dev14:43
*** aelkikhia has joined #openstack-dev14:45
*** souvik has joined #openstack-dev14:45
*** diogogmt has quit IRC14:47
*** ndipanov has joined #openstack-dev14:48
*** aelkikhia1 has quit IRC14:48
*** diogogmt has joined #openstack-dev14:49
*** boris-42 has joined #openstack-dev14:50
*** SergeyLukjanov has quit IRC14:50
*** salv-orlando_ has joined #openstack-dev14:50
*** bmclaughlin has quit IRC14:50
*** andrewbogott_afk is now known as andrewbogott14:50
*** mrodden has quit IRC14:50
*** FunnyLookinHat has quit IRC14:52
*** salv-orlando has quit IRC14:53
*** bmclaughlin has joined #openstack-dev14:53
*** salv-orlando_ is now known as salv-orlando14:54
*** FunnyLookinHat has joined #openstack-dev14:54
*** zaitcev has joined #openstack-dev14:55
*** derekh has quit IRC14:56
*** otherwiseguy has joined #openstack-dev14:57
*** garyk has joined #openstack-dev14:58
*** woodspa has quit IRC14:58
*** yidclare has joined #openstack-dev14:58
spzaladolphm: good morning, Dolph! there?15:00
*** danwent has joined #openstack-dev15:00
*** souvik has quit IRC15:00
dolphmspzala: o15:00
dolphmo/15:00
spzaladolphm: I sent an email about my scheduled vacation next week (5/13 to 5/17), not sure if you had chance too look at it..so wanted to let you know. :)15:01
*** woodspa has joined #openstack-dev15:01
spzaladolphm: also I updated the patch https://review.openstack.org/#/c/27364/15:01
spzaladolphm: not sure if we are going with the approach but if you can take a look at it :)15:01
spzaladolphm: I guess if we don't reach to a conclusion by this week, I should handover it to you or someone else considering my vacation next week?15:02
dolphmspzala: we can patch your patch if necessary, no worries15:02
dolphmspzala: did you see my comments about the domain crud operations?15:03
dolphmspzala: it doesn't make sense to only revise list_domains()15:03
spzaladolphm: OK, that sounds great..thanks!15:03
*** mmagr has quit IRC15:04
spzaladolphm: hmmm.. I modified crud operation in the latest patch based on your comments15:04
*** mkollaro1 has quit IRC15:04
*** katylava has joined #openstack-dev15:05
spzaladolphm: please let me know if I missed anything on your comments15:06
*** navid_ has quit IRC15:07
*** salv-orlando has quit IRC15:07
*** mrodden has joined #openstack-dev15:07
*** aelkikhia has left #openstack-dev15:07
*** reidrac has quit IRC15:08
dolphmspzala: i'm not seeing any changes to create_domain, get_domain, update_domain or delete_domain15:09
dolphmoh, you're making those changes in DomainApi...15:09
*** alop has joined #openstack-dev15:09
spzaladolphm: yes15:09
dolphmspzala: why do list_domains in the driver and everything else in DomainApi?15:10
*** markmcclain has quit IRC15:10
spzaladolphm: I think that's similar to user, projects, roles..  list_domains (or list_*) uses a common/core method15:11
HenryGI submitted a review and put "Fixes bug #B" in the Commit Message, but elsewhere in the Commit Message I also mentioned another bug #A. It seems Gerrit picked the wrong bug (bug #A) for the topic. What's the best way to fix this?15:11
spzaladolphm: so there is no specific code in DomainApi for list_domains15:12
*** jcoufal has quit IRC15:12
spzaladolphm: hope it makes sense, unless I misunderstood your question.15:12
*** jamespage_ has joined #openstack-dev15:13
*** aloga has quit IRC15:13
*** souvik has joined #openstack-dev15:13
*** drewlander has quit IRC15:14
dolphmspzala: so, the [Object]Api classes are very specific to LDAP, whereas everything in the Identity driver class isn't; the additional checks you are adding would make more sense in the higher layer15:15
*** flaper87 has quit IRC15:15
*** brunnhilde has joined #openstack-dev15:15
spzaladolphm: OK, agree with that. I wold move that in the DomainApi.15:16
*** mkollaro has joined #openstack-dev15:17
*** utlemming has joined #openstack-dev15:17
*** jamespage_ has quit IRC15:17
dolphmspzala: in the DomainApi == out of the DomainApi?15:18
*** sthaha has quit IRC15:19
*** SergeyLukjanov has joined #openstack-dev15:19
spzaladolphm: hmmm.. sorry didn't get it. you mean move the logic for "== DEFAULT_DOMAIN" to Identity?15:19
dolphmyes15:20
*** lloydde has joined #openstack-dev15:20
dolphmup with list_domains (the code there feels correct)15:20
openstackgerritA change was merged to openstack/nova: Fix variable referenced before assginment in vmwareapi code.  https://review.openstack.org/2851215:20
openstackgerritA change was merged to openstack/quantum: Imported Translations from Transifex  https://review.openstack.org/2866615:20
*** giulivo has quit IRC15:21
*** zul has joined #openstack-dev15:21
*** sacharya has joined #openstack-dev15:22
*** pabelanger_ has joined #openstack-dev15:23
*** pabelanger_ has quit IRC15:23
*** pabelanger_ has joined #openstack-dev15:23
*** pabelanger has quit IRC15:23
*** pabelanger_ is now known as pabelanger15:23
spzaladolphm: OK, sounds good.. makes sense.15:24
*** pabelanger_ has joined #openstack-dev15:24
spzaladolphm: so don't change anything in list_domains ?15:24
spzaladolphm: I like the idea of moving == in Identity...that way we won't be calling domainapi unnecessary15:27
*** Mandell has joined #openstack-dev15:27
*** zyluo has quit IRC15:27
*** adjohn has joined #openstack-dev15:30
*** utlemming has quit IRC15:31
topoldolphm, +115:31
*** utlemming has joined #openstack-dev15:33
*** ayoung has quit IRC15:33
*** jprovazn has quit IRC15:33
*** terryh has quit IRC15:33
*** epende has joined #openstack-dev15:36
*** abhishekkr_ has quit IRC15:37
*** mlavalle has joined #openstack-dev15:38
*** freedomhui has joined #openstack-dev15:38
*** crandquist has joined #openstack-dev15:39
*** hemna has joined #openstack-dev15:39
ependeAnyone know where to change the default db driver from mongodb to sqlalchemy in ceilometer?15:39
ependefor the reporting api15:40
*** lloydde has quit IRC15:40
*** lloydde has joined #openstack-dev15:43
dolphmspzala: regarding list_domains, i don't think so15:44
spzaladolphm: OK, cool. I will leave it as it is.15:45
*** epim has joined #openstack-dev15:46
*** ociuhandu has quit IRC15:46
*** bdpayne has joined #openstack-dev15:49
*** Mandell has quit IRC15:50
*** ayoung has joined #openstack-dev15:50
*** galstrom_zzz is now known as galstrom15:53
*** devoid has joined #openstack-dev15:53
*** salv-orlando has joined #openstack-dev15:53
openstackgerritA change was merged to openstack/keystone: Revoke tokens on user delete (bug 1166670)  https://review.openstack.org/2867715:54
uvirtbotLaunchpad bug 1166670 in keystone/grizzly "Deleted user can still create instances" [High,In progress] https://launchpad.net/bugs/116667015:54
*** armax has joined #openstack-dev15:54
*** devoid has left #openstack-dev15:56
*** abhishekkr_ has joined #openstack-dev15:58
*** gordc has quit IRC15:59
*** sarob has joined #openstack-dev15:59
*** ayoung has quit IRC16:00
*** ayoung has joined #openstack-dev16:01
*** sarob_ has joined #openstack-dev16:01
*** jcoufal has joined #openstack-dev16:01
*** gyee has joined #openstack-dev16:02
*** souvik has quit IRC16:03
*** sarob has quit IRC16:04
*** drewlander has joined #openstack-dev16:06
*** salv-orlando has quit IRC16:06
*** souvik has joined #openstack-dev16:07
*** mdenny has joined #openstack-dev16:07
*** dguitarbite has quit IRC16:08
ayoungbknudson, malini_ can you confirm that you are OK with the changes to https://review.openstack.org/#/c/27364/1116:10
*** souvik_ has joined #openstack-dev16:10
ayoungI'll gate my approval on yours.16:10
*** kpavel has quit IRC16:10
*** souvik has quit IRC16:11
*** souvik_ is now known as souvik16:11
*** danwent has quit IRC16:11
*** tkammer has quit IRC16:13
*** dolphm has quit IRC16:13
bknudsonayoung: I'll take a look today, but getting pulled to look at another problem.16:13
*** apevec has joined #openstack-dev16:14
*** danwent has joined #openstack-dev16:16
*** Samos123_ has quit IRC16:16
*** gordc has joined #openstack-dev16:16
apevecayoung, ping - I didn't see reply on http://lists.openstack.org/pipermail/openstack-stable-maint/2013-May/000674.html16:17
apevecayoung, you're ok delaying backport to 2013.1.2 ?16:18
*** dprince has quit IRC16:19
*** salv-orlando has joined #openstack-dev16:20
*** pfreund has quit IRC16:20
*** Samos123 has joined #openstack-dev16:20
*** yidclare has quit IRC16:21
ayoungapevec, "ok?"  no, I'm not.  LDAP is broken.  Failing to backport will mean that we delay people moving to grizzly16:22
*** dolphm has joined #openstack-dev16:22
ayoungapevec, however, I might be "resigned"16:22
spzalaayoung: thanks! Dolphm recommended to move some logic from DomainApi to Identity .. I have made those changes and running a quick test. Will be updating the patch in few16:22
*** drewlander has quit IRC16:22
*** zul has quit IRC16:22
spzalaayoung: no change in the logic itself though16:22
*** lloydde has quit IRC16:22
*** waa has quit IRC16:22
dolphmspzala: thanks!16:23
dolphmayoung: resigned from what?16:23
apevecayoung, well, master patch has not merged yet16:23
spzaladolphm: np. thank you for the looking at the changes quickly and feedback.16:23
*** networkstatic has quit IRC16:23
apevecayoung, and 2013.1.2 is due June 616:24
*** abhishekkr_ has quit IRC16:24
*** danpb has quit IRC16:24
apevecit's significant patch, better give it some soak time on master16:24
*** utlemming has quit IRC16:26
*** jamespage_ has joined #openstack-dev16:27
*** zul has joined #openstack-dev16:28
*** jamespage_ has quit IRC16:30
*** mrodden has quit IRC16:31
cdsvrdHi16:31
rozfor one of my changes jenkins ran correctly (24-ish hours ago) but smokeStack didn't. I think i need that, to trigger it again I need to submit a new patch set?16:31
cdsvrd(Sorry for english) I would like to test horizon on gentoo16:32
ayoungdolphm, "resigned to"  not "resigned from"16:32
cdsvrdand i have an "Address family not supported by protocol"16:33
cdsvrdafter authentication16:33
*** cmark_ has joined #openstack-dev16:33
dolphmayoung: i joined right before that message, so i'm totally lacking context16:33
ayoungapevec, I am not "ok" with it but I am "resigned to" it.16:34
ayoungdolphm, ha16:34
ayoungLDAP being brokn and not fixed in stable this go round16:34
*** mrodden has joined #openstack-dev16:34
dolphmah16:34
*** SergeyLukjanov has quit IRC16:34
dolphmyeah, we missed the boat on 2013.1.116:34
ayoungdolphm, however, I have been hacking apart the identity API, and I am getting more and more happy with that concept.  I think it is going to work, and work well.16:34
*** lucasagomes has quit IRC16:35
ayoungIt will be a fairly big number of lines of code patch, but they should mostly be moving from one file to another16:35
cdsvrdI think it is novaclient exception16:35
cdsvrdFile "/usr/lib64/python2.7/site-packages/novaclient/client.py" in request16:35
ayoungthe credential split is a good first hack.  shows what some of the issues will be16:35
cdsvrd2013.116:36
*** glikson has quit IRC16:36
*** adjohn has quit IRC16:36
*** jruzicka has quit IRC16:37
*** cmark has quit IRC16:37
cdsvrdhttp://pastebin.com/U9FfmzUz16:37
cdsvrdpython socket compatibity ?16:38
cdsvrdipv6 ?16:38
cdsvrdAny idea ?16:38
*** digitalsanctum has quit IRC16:38
spzalaayoung: dolphm: just updated the patch.16:39
ayoungspzala, looking16:39
spzalaayoung: thanks16:39
apevecayoung, dolphm - ok, I've removed 2013.1.1 from bug 116872616:39
ayoungmalini_, bknudson ^^16:40
uvirtbotLaunchpad bug 1168726 in keystone/grizzly "default_domain_id breaks the ability to map keystone  to ldap" [Critical,In progress] https://launchpad.net/bugs/116872616:40
spzalaayoung: stepping out for a quick lunch with team but will be back soon.16:40
*** salv-orlando has quit IRC16:40
*** digitalsanctum has joined #openstack-dev16:40
apevecdolphm, ayoung - I'll put it as "known issue" in 2013.1.1 relnotes, do you have any text suggestion?16:40
dolphmapevec: i have a related line in the grizzly release notes https://wiki.openstack.org/wiki/ReleaseNotes/Grizzly#OpenStack_Identity_.28Keystone.2916:41
dolphmapevec: "Read-only LDAP deployments..."16:41
cdsvrdhttp://dpaste.com/1129884/16:41
apevecdolphm, thanks, I'll link to that16:41
*** salv-orlando has joined #openstack-dev16:42
apevecdolphm, and thanks for adding "fixed in" there16:42
apevecgood idea!16:42
openstackgerritA change was merged to openstack-dev/devstack: Update the package name of qpid in the fedora platform  https://review.openstack.org/2852416:42
openstackgerritA change was merged to openstack/nova: Fixes typo in server-evacuate-req.xml  https://review.openstack.org/2867416:43
openstackgerritA change was merged to openstack-dev/devstack: Set MYSQL_HOST parameter to better support multi-node mode.  https://review.openstack.org/2852716:43
cdsvrdIs it right place ?16:43
*** garyk has quit IRC16:43
ayoungspzala, in https://review.openstack.org/#/c/27364/13/tests/test_backend_ldap.py  I see two functions at the bottom that look like they were copied from elsewhwer.  What is up with them?16:44
ayoungtest_authenticate_no_metadata and test_user_crud16:44
openstackgerritA change was merged to openstack-dev/devstack: Add Debian OS support in DevStack  https://review.openstack.org/2821516:45
openstackgerritA change was merged to openstack/quantum: in dhcp_agent, always use quantum.conf root_helper  https://review.openstack.org/2863016:45
*** ndipanov has quit IRC16:45
sdaguedolphm: I stuck it in the reviews but can we do the master changes for tempest first, then cherry pick, like backports in other projects16:47
sdaguealso, a bug would be awesome16:47
*** salv-orlando has quit IRC16:47
*** anniec has quit IRC16:47
*** ndipanov has joined #openstack-dev16:47
jpichcdsvrd: #openstack or http://ask.openstack.org might be a better place to ask. I'm not familiar with that particular error myself, good luck!16:48
dolphmsdague: yes, i started with folsom only because that's where the test failure was; opened as WIP due to no bug / haven't actually verified this is the cause of my failure (it just stuck out to me)16:49
sdagueok, cool16:49
sdaguedolphm: regardless, they are good fixes16:49
sdaguethe upstream code is wrong, and your fixes are right, just want to do them in an order to make sure we don't fix a bug in stable and leave it in master16:50
*** johnthetubaguy has quit IRC16:50
dolphmsure16:50
*** salv-orlando has joined #openstack-dev16:51
*** jpich has quit IRC16:51
*** markmcclain has joined #openstack-dev16:53
*** terry7 has joined #openstack-dev16:54
*** sarob_ has quit IRC16:54
*** eglynn has quit IRC16:56
*** garyk has joined #openstack-dev16:58
*** yidclare has joined #openstack-dev17:01
*** ashwini has joined #openstack-dev17:01
*** afazekas_ is now known as afazekas17:02
*** aloga has joined #openstack-dev17:03
*** adjohn has joined #openstack-dev17:05
*** mkollaro has quit IRC17:05
*** nati_ueno has joined #openstack-dev17:05
*** zodiak has joined #openstack-dev17:08
*** jamespage_ has joined #openstack-dev17:08
*** salv-orlando has quit IRC17:11
openstackgerritA change was merged to openstack/horizon: Fixing Floating IP capitalization bug  https://review.openstack.org/2860417:11
*** dprince has joined #openstack-dev17:11
*** adjohn has quit IRC17:12
*** FatDarrel has joined #openstack-dev17:12
*** danwent has quit IRC17:17
*** salv-orlando has joined #openstack-dev17:17
*** athomas has quit IRC17:17
*** drewlander has joined #openstack-dev17:18
*** networkstatic has joined #openstack-dev17:23
*** aeperezt has joined #openstack-dev17:23
*** networkstatic has quit IRC17:24
*** Mandell has joined #openstack-dev17:24
*** danwent has joined #openstack-dev17:25
*** athomas has joined #openstack-dev17:26
*** yidclare has quit IRC17:26
*** abhishekkr_ has joined #openstack-dev17:27
*** souvik has quit IRC17:27
*** jbresnah has quit IRC17:27
*** cmark_ has quit IRC17:27
*** aloga has quit IRC17:27
*** cmark has joined #openstack-dev17:28
*** yidclare has joined #openstack-dev17:28
*** SergeyLukjanov has joined #openstack-dev17:30
*** HenryG has quit IRC17:32
openstackgerritA change was merged to openstack-dev/grenade: Remove errant echo_summary calls  https://review.openstack.org/2799417:36
*** zul has quit IRC17:37
*** psedlak has quit IRC17:38
*** jamespage_ has quit IRC17:40
*** digitalsanctum has joined #openstack-dev17:43
*** afazekas has quit IRC17:43
*** dims has quit IRC17:45
*** Ryan_Lane has joined #openstack-dev17:45
*** alop has quit IRC17:47
spzalaayoung: yes, i copied them from test_backend. After the changes for ignore domain_id it was breaking the test for those two functions17:47
ayoungspzala, are they straight copies,  or copied then modified?17:48
spzalaayoung: no , I need to modify them17:48
spzalaayoung: what I modified is17:48
dolphmayoung: https://bugs.launchpad.net/keystone/+bug/117713617:48
uvirtbotLaunchpad bug 1177136 in keystone "Unit tests fail when trying to install python-memcache 1.50" [Undecided,In progress]17:48
spzalaspzala: added "enabled" based on the emulation setting17:49
ayoungspzala, ok, cool.  I can bless17:49
*** dims has joined #openstack-dev17:49
spzalaayoung: :)17:49
spzalaayoung: Thank you!17:49
*** martine has quit IRC17:49
*** glikson has joined #openstack-dev17:50
*** jamespage_ has joined #openstack-dev17:50
*** Mandell has quit IRC17:53
*** zzs has joined #openstack-dev17:54
*** Mandell has joined #openstack-dev17:54
*** rnirmal has quit IRC17:55
*** jamespage_ has quit IRC17:57
*** mlavalle has quit IRC17:58
*** terryh has joined #openstack-dev18:01
*** markmcclain has quit IRC18:01
*** HenryG has joined #openstack-dev18:02
*** markmcclain has joined #openstack-dev18:07
*** mdomsch has joined #openstack-dev18:12
*** Mandell has quit IRC18:14
*** ladquin is now known as ladquin_brb18:14
*** tzumainn has quit IRC18:14
*** salv-orlando has quit IRC18:15
bknudsonspzala: I see that dolphm had a bunch of comments in previous patch sets that weren't responded to.18:16
*** eglynn has joined #openstack-dev18:16
bknudsone.g. "i'd rather not reduce test coverage with any of these ContainsSubset changes; can the test be made more specific?"18:16
bknudsonseems pretty important to not reduce test coverage.18:16
apevecttx, ok, 2013.1.1 is ready for tagging - just one question about https://wiki.openstack.org/wiki/StableBranchRelease#Versioning18:17
apevecttx, that's no more in Grizzly right?18:17
*** reed has quit IRC18:18
apevecmordred, ^^^ FINAL = True is gone since your change to use oslo version, right?18:19
*** vartom11110 has joined #openstack-dev18:19
apevecmordred, so I just tag ?18:19
*** jtomasek has quit IRC18:19
spzalabknudson: hmmm...18:19
mordredapevec: yes. EXCEPT - I believe you'll also need to make a commit that has a change to setup.py that changes 2013.1.1 to 2013.1.2 after the tag is made18:20
mordredbut18:20
*** mlavalle has joined #openstack-dev18:20
mordredI hestitate to say I'm 100% correct with ttx18:20
apevecmorazi, yes, bump version is the next commit after version tag18:20
*** lloydde has joined #openstack-dev18:21
morazimordred, ^^ I'm suspicious I was the victim of nick collision there18:21
apevecoops sorry morazi18:22
apeveceverybody should have unique first 3 letters :)18:22
mordredhehe18:23
*** Mandell has joined #openstack-dev18:24
morazihehe, indeed18:24
moraziapevec, no worries18:24
spzalabknudson: I did reply to those comments on tests. I had chat with DolphM this morning and he didn't bring it up so thought he is okay with it.18:25
spzalabknudson: can we have that as "todo"? just a thought18:25
bknudsonspzala: sure. Since I'm interested maybe I'll take a look at it.18:27
*** comay has joined #openstack-dev18:27
spzalabknudson: thanks. I am up for working on it,so you can pass the work to me once you take a look if you want.18:28
bknudsonI thought you were on vacation?18:28
*** melwitt has joined #openstack-dev18:29
*** yidclare has quit IRC18:30
*** jaypipes has quit IRC18:31
*** baba has quit IRC18:32
*** megha has joined #openstack-dev18:32
morganfainbergayoung: ping18:34
*** zul has joined #openstack-dev18:34
spzalabknudson: :) no..I will be on vacation next week18:35
ayoungmorganfainberg, wazzup?18:36
*** jaypipes has joined #openstack-dev18:36
morganfainbergayoung: was trying to parse your comment on the token purging code.  I am doing an append (previous behavior) on at time of revocation. the only times I parse the list is when we need to flush.18:37
ayoungmorganfainberg, so, while termie is correct, it doesn't change the fact that we still need to list the tokens by user etc18:37
morganfainbergayoung: in revocation?18:38
termieayoung: you build a list, brah18:38
termieayoung: at write time18:38
ayoungtermie, and I said that you are correct18:38
termieayoung: not at read time18:38
ayoungtermie, but to build the list, we need to enumerate the tokens for the user18:38
ayoungso "when" it happens needs to change.18:39
*** vartom11111 has joined #openstack-dev18:39
termieayoung: technically, it would be cheapest to just send the service the new clock18:40
termieayoung: instead of a revocation list18:40
termieayoung: so they can validate the same way we do18:40
termieclock is what you were referring to as "score"18:40
ayoungtermie, so the service would cached the "clock" on a per user basis?18:41
morganfainbergoh ayoung sorry, mis-read the name(s) thought you commented vs termie :P18:41
morganfainbergshould have ping'd him first ;)18:41
ayoungOr, one a second would fetch the entire clock list for all users?18:41
*** pixelbeat_ has quit IRC18:42
*** martine has joined #openstack-dev18:42
ayoung"one a second"  where did that come from, not even sure what that was supposed to mean.  sorry18:42
*** vartom11110 has quit IRC18:42
arosennati_ueno: ping18:42
nati_uenoarosen: pong18:43
*** danwent has quit IRC18:43
arosennati_ueno:  quick question, why do we need: class OVSHybridIptablesFirewallDriver(IptablesFirewallDriver):18:43
termieayoung: what are your thoughts on how long this timeout is18:43
arosennati_ueno: none of the plugins use that for the devstack configurations18:43
*** eharney has quit IRC18:43
nati_uenohmm i supposed it is used in the devstack. OVSHybridIptablesFirewallDriver is for OVS18:44
nati_uenolet me check18:44
arosennati_ueno: doh your right..18:44
*** vartom11111 has quit IRC18:44
arosenlib/quantum_plugins/ovs_base:        iniset /$Q_PLUGIN_CONF_FILE SECURITYGROUP firewall_driver quantum.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver18:44
termieayoung: if it is one second, for example, it is not going to save a heck of a lot of time18:44
*** amerine has quit IRC18:44
ayoungtermie, there are a couple times outs.  On the remote services side, they fetch the revocation list, I would guees about once a minute.  The tokens in the memcached backend woud live, I guess, the TTL of the token18:44
ayoungtermie, even one a second might make a difference under load, but I was thinking once a minute.18:44
nati_uenoarosen: ah yes. bridge configuration is different, so we need specific driver for OVS18:45
termieayoung: and under your current ideas what is being fetched every minute?18:45
*** Yada has quit IRC18:45
ayoungtermie, the updated list of revoked tokens.18:45
ayoungtermie, and, yes, you are right, that list should not be reproduced everytime18:46
ayoungpainfully obvious now18:46
*** novas0x2a|laptop has joined #openstack-dev18:46
ayoungbut should instead be updated when the tokens are revoked18:46
*** mkollaro has joined #openstack-dev18:46
*** eharney has joined #openstack-dev18:46
*** eharney has quit IRC18:46
*** eharney has joined #openstack-dev18:46
termieayoung: some other questions to ask, how frequently do you htink tokens will be revoked?18:46
ayoungtermie, very infrequently18:47
*** amerine has joined #openstack-dev18:47
termieayoung: so let's say, hypothetically, our "revocation" list is a list of users with their clocks18:47
morganfainbergayoung / termie: if the tokens are being revoked quickly, there is probably another problem such as too long lived tokens.  I'd hope they are infrequently revoked18:47
termieayoung: most users will be at 0 and not included in this clock18:47
termies/clock/list/18:47
arosennati_ueno: hrm that's unfortunate. The only different between the two is _get_device_name() I wonder if there if there is anyway to just built that logic into the IpTablesFirewallDriver() so we don't need to have to closes.18:48
ayoungtermie, OK...and once all of a users revoked tokens would have timed out, we would remove those users from the list as well...to keep the list from always  growing18:48
ayoungtermie, I like it...18:48
arosennati_ueno:  and how _port_chain_name() is handled.18:49
termiek, i'm going back to code reviews, talk amongst yourselves18:49
arosennati_ueno: thx for the info though.18:49
ayoungtermie, I'll certainly do that for Havana.  I need to figure out how to get that to work for Grizzly.18:50
ayoungg stable18:50
openstackgerritA change was merged to openstack/tempest: Do super()'s tearDowns last (bug 1178337)  https://review.openstack.org/2868318:50
uvirtbotLaunchpad bug 1178337 in tempest "super's tearDown called too early" [Undecided,In progress] https://launchpad.net/bugs/117833718:50
*** danwent has joined #openstack-dev18:50
nati_uenoarosen: yeah you are right. may be we need to refactor how to handle interface name18:50
nati_uenoarosen: I'm hoping ml2 will have standard way to handle device name (nic name)18:51
morganfainbergayoung: well let me know if you need any help with that, i'm feeling like i'm missing part of the convo, so i can circle back once a direction is a bit more clear (or i've had time to digest the convo a bit more)18:51
*** lloydde has quit IRC18:51
termiemorganfainberg: what are you missing?18:51
ayoungmorganfainberg here's the deal.  For a given user, we keep a 'clock' which starts at 018:51
ayoungwhen a token is revoked, we up the score18:51
ayoungeach token also has a clock value on it18:51
*** redondos has joined #openstack-dev18:52
termieayoung: when the user is changed we up the score18:52
ayoungif the clock value on the token is less than the clock value on the user, the token is invalid18:52
termienot when a token is revoked18:52
*** eharney has quit IRC18:52
morganfainbergtermie: i think this is what i was missing, pulling out the meaning between both sides.  i am getting it now.18:52
ayoungtermie, right...upping the score is what revokes the token, although I think we would up the score also on a "revoke token" call, no?18:53
ayoung"advance the clock"18:53
termieayoung: you will revoke all tokens18:53
termieayoung: for a given user18:53
ayoungtermie, yes, I understand that.  That may well be acceptable.  If it isn't, I don't think we could use the clock approach.18:54
*** eharney has joined #openstack-dev18:54
morganfainbergtermie: that seems counter-intuitive for revoking a specific token vs. all.  does it mean we no longer support the individual token .delete() function and make it tied to the clock exclusively?18:55
termiemorganfainberg: i think revoking individual tokens is probably a pointless exercise18:55
termiey'all are welcome to come up with a compelling use case, but i'm voting yagni18:55
*** yidclare has joined #openstack-dev18:56
morganfainbergtermie: that makes your proposal significantly easier and clearer.18:56
*** yidclare has quit IRC18:56
morganfainbergi don't see anything that really makes use of delete at this point across the code base, so i dno't think thats an unreasonable change18:56
morganfainberg(besides the "revoke all tokens" code)18:57
morganfainbergwhich is covered.18:57
spzaladolphm: I replied your comments on tests for patch 10 but I realized I never 'published' the my reply,..so they seating there as draft.18:58
dolphmspzala: have you published them now?18:58
spzaladolphm: just did it18:59
ayoungso we can drop the memcached record that has all of the users tokens in it, as we don't need to list tokens for a user except to revoke them.  I'm not sure that we can do all of this in the context of Grizzly stable, though.  It fundamentally changes the format of the revocation list, and that means auth_token middleware is going to have to change.  We might need morganfainberg 's patch as a stop gap measure, and then do the clock appr18:59
ayoungoach for Havana18:59
spzaladolphm: sorry about that. I was under impression that you have looked at them and since you didn't bring that topic this morning, I thought you were okay with it.18:59
termiedolphm: am i not able to -2 things you write? or is it a branch thing? https://review.openstack.org/#/c/28423/ (not that i wanted to -2 it but i noticed i couldn't)18:59
*** sgordon has quit IRC18:59
ayoungmorganfainberg, unless you can wait until Havana18:59
ayoungtermie, we can't +2 -2 stable branch19:00
termieayoung: i am mostly unconcerned for grizzly, i'll let dolphm sweat over that stuff19:00
morganfainbergayoung: i will be using this code in my version of grizzly, but i figure if i've hit this problem someone else has.  I totally agree havana is the right place to make the clock change.19:00
spzaladolphm: bknudson mentioned about it too. I suggested if we can take that as 'ToDo' but please let me know what you think once you at my reply.19:00
termieayoung: why is that? re: -219:00
termiespzala: TODO is the proper terminology :)19:01
morganfainbergI'm totally good with scrapping the master proposal and working on havana clock approach with you guys.19:01
*** alop has joined #openstack-dev19:01
ayoungtermie, it is just a different group for that branch.  It has to go into master before then, and then the stable maintainers can say "backport or no"19:01
ayoungI've never bugged for it.19:01
spzalatermie: Hello! my first time talking to you.19:01
spzalatermie: LOL19:01
termiespzala: don't make it a habit ;)19:02
spzalatermie: no not at all19:02
ayoungmorganfainberg, OK, I'm going to write up the "clock" approach.  Are you interested in helping with the implementation?19:02
morganfainbergayoung: absolutly19:03
termiemorganfainberg: the vodka version of agreeing19:03
*** freedomhui has quit IRC19:04
ayoungtermie, BTW, I totally get it on the Cassandra backend.  I assume you would want the clock approach implemented as a common code as a prereq?19:04
morganfainbergtermie: Absolut Python? Absolut Agreement?19:05
openstackgerritA change was merged to openstack/nova: Change type of cells.deleted from boolean to integer.  https://review.openstack.org/2856719:05
*** zul has quit IRC19:05
termieayoung: not sure what cassandra has to do with it, but yes it can and probably should be common code19:05
spzalatermie: this is needed quick patch so that's why thought of 'todo' :) but won't make it a habit19:06
spzaladolphm: I need to leave to be at school bus stop to pick up my son. I will be back online soon.19:07
morganfainbergayoung: if it wasn't going to be common code, i'd advocate that it should be.  it doesn't make sense not to be.19:08
morganfainbergayoung: bad grammer, it makes sense it should, and would be an incorrect approach if it wasn't common.19:08
openstackgerritA change was merged to openstack/nova: Add pointer to compute driver matrix wiki page.  https://review.openstack.org/2868419:08
openstackgerritA change was merged to openstack/quantum: Improve checking of return values for functions in linuxbridge agent plugin  https://review.openstack.org/2784719:08
openstackgerritA change was merged to openstack/quantum: Fetch routers ports for metadata access from DB  https://review.openstack.org/2831719:08
*** vipul is now known as vipul|away19:09
*** vipul|away is now known as vipul19:09
*** jcoufal has quit IRC19:09
termiespzala: i just meant that "ToDo" is ugly19:09
termiespzala: and would hate to see it accidentally show up ever again19:09
ayoungmorganfainberg, https://blueprints.launchpad.net/keystone/+spec/clock-for-revoke-tokens19:10
*** dolphm has quit IRC19:12
*** spzala has quit IRC19:12
ayoungtermie, I meant that Cassandra as a persistent KVS is a valuable backend to have.19:13
*** Ryan_Lane has quit IRC19:13
*** nati_ueno has quit IRC19:13
termieayoung: i agree19:13
morganfainbergayoung: that looks good. so is the clock part of the KVS store or is the clock an attribute on the actual user (as in the clock never "resets" to 0)19:13
*** Ryan_Lane has joined #openstack-dev19:13
*** sgordon has joined #openstack-dev19:13
termiemorganfainberg: user19:13
ayoungmorganfainberg, I don't want it on the user object, as we might have read only users from LDAP19:13
ayoungso I'll punt on a complete answer yet19:13
termieayoung: ldap is magic and needs a supplementary datastore anyway, we don't know when ldap users change19:14
apevecrussellb, vishy - found few stable/grizzly commits w/o bug# e.g. https://review.openstack.org/2680319:14
apevecrussellb, you said there "While it seems like an optimization, this was a part of a patch series that addressed a significant performance related regression in grizzly."19:14
*** electrichead has joined #openstack-dev19:14
apevecrussellb, was there LP# for that?19:14
*** epende has quit IRC19:15
russellbapevec: dansmith ^  ?19:15
apevecso it can show up in list of fixed issues in 2013.1.119:15
russellb^^ high message compression there19:15
uvirtbotrussellb: Error: "^" is not a valid command.19:15
ayoungtermie, yeah, LDAP doesn't do notifications, but I think that is something we just have to live with.  Not sure there is a good solution there.19:15
russellbdansmith: was there a bug on the system_metadata stuff?19:15
dansmithyeah19:15
dansmithis it not in the commit?19:15
enikanorov_markmcclain: Hi. could you please take a look at https://review.openstack.org/#/c/28257/ again? The link to the wiki has been posted there.19:15
apevecit might have been series19:15
apevecso this particular patch doesn't have it19:16
dansmithjust a sec19:16
apevecdansmith, https://review.openstack.org/2680319:16
markmcclainenikanorov_: I'm meeting with some folks now, but I'll take a look in bit19:16
enikanorov_thanks!19:16
apevecdansmith, ah nm it's topic name :)19:16
dansmithapevec: https://bugs.launchpad.net/nova/+bug/116473719:17
dansmithheh19:17
uvirtbotLaunchpad bug 1164737 in nova/grizzly "Instance joins with system_metadata are critically slow" [Critical,Fix committed]19:17
termieayoung: document the things ldap won't do for you and punt it to the curb19:17
apevecdansmith, thanks19:17
dansmithyep19:17
*** lloydde has joined #openstack-dev19:18
topolayoung, I thought we said that folks would have custom code for handling LDAP notifications and we just would eventually provide some notification methods for them to use19:18
ayoungtopol, minor, minority there.  I don;t think we can count on that19:18
ayoungso, no, I didn't say that, I recall saying some LDAP server have notifications, but you can't count on it19:19
topol(topol so close to eating incredibly good italian food soon....)19:19
ayoungand if we need that behaviour, we have to poll19:19
topolayoung agreed19:19
fungiayoung: dprince: going back over all the centos jobs we have configured, if you're still trying to narrow down the keystone breakage i have some timeframe details which could help...19:19
ayoungtopol, you need to change your nick to topolino.\19:19
ayoungfungi, do you have a known good commit?19:19
ayoung I can do a git bisect if you do19:20
topolyes, translates to little mouse in italian19:20
*** lloydde has quit IRC19:20
fungiayoung: dprince: seems the issue is affecting stable/grizzly as well, and began after May 3, 2013 6:06:08 AM19:20
topolbeen called that many times19:20
ayoungtopol, more so since Cars 2, I assume19:20
*** lloydde has joined #openstack-dev19:20
topolindeed19:20
termietopol: i think you found your casual nick friday name19:20
fungiayoung: dprince: so that's narrowed to a stable branch patch in a 24-hour period. can't be too many to sift through there19:21
ayoungfungi, stable branch?  I thought this was breaking on master?19:21
fungiayoung: apparently it's both master and grizzly19:21
fungiayoung: we initially noticed it on master last night19:22
fungiayoung: but it had presumably been going on for about a week in that state19:22
ayoungfungi, do you have a good commit hash?19:22
fungiayoung: i'll get you one, just a sec19:22
fungiayoung: Checking out Revision 3aa0f45466c17d8e375595c273a27b10fea5c91c (origin/stable/grizzly)19:23
morganfainbergtermie /  ayoung: one last question, so when we modify roles for the user / grants / groups / etc, we will increment clock and revoke all tokens?  does this seem a bit far reaching (invalidate all tokens and force a reauthenticate).  it would be problematic to accomplish via something like Horizon if tyou're changing your own user (e.g. in an "administrative" context).19:23
fungiayoung: that's the nightly which succeeded for 2013-05-0319:23
*** garyk has quit IRC19:23
*** vipul is now known as vipul|away19:24
ayoungmorganfainberg, I think it is fair, but I will sleep on it19:24
morganfainbergayoung: ok19:24
morganfainbergsounds good.19:24
morganfainbergthanks! :)19:24
ayoungfungi, how about for master?19:25
fungiayoung: master's a little tougher to nail down because all i have are check jobs, no periodics19:26
termiemorganfainberg: if you change your user it is not a lot to ask you to log back in19:26
ayoungfungi, OK, I'll try it off stable19:26
termiemorganfainberg: it is more of a problem if horizon tries to do some operation immediately after changing the user19:26
termiemorganfainberg: but that is something horizon will need to fix19:26
ayoungfungi, just to confirm 3aa0f45466c17d8e375595c273a27b10fea5c91c was good?19:28
fungiayoung: dprince: ooh! actually hold off with the bisect. there were no changes between that hash and the one which failed the next night19:28
fungiayoung: it was good on that sha, but then the next night it failed on the same exact sha. i smell an external python library upgrading somewhere19:29
fungigonna diff the pip freeze output from both logs here in a moment19:29
termieayoung: background info on "credentials19:29
termieayoung: " please19:29
ayoungfungi yeah, that commit is failing, as are a bunch older than it19:30
ayoungI'm testing using ./run_tests.sh -x test_keystoneclient:KcMasterTestCase.test_admin_requires_adminness19:30
fungiayoung: right, almost certainly a moving dependency somewhere. i'll know in a bit19:30
ayoungcan't get a decent debug log out of it, though, as only the client logging is shown.19:30
ayoungtermie, so the V3 API is attempting to cover what was previously done in the EC2 extension, and we now have a "credentials" list in the identity backend.  It doesn't really belong there.  It means that, among other things, we will never have an LDAP implementation.19:32
termieayoung: i guess what i would like to see is where the discussion of implementing the credentials api happened19:33
openstackgerritA change was merged to openstack/nova: xenapi: cleanup vdi when disk too big exception raised  https://review.openstack.org/2866419:33
termieayoung: like the BP or somesuch19:33
dprincefungi: so you think this is an environmental change?19:34
*** sandywalsh has quit IRC19:34
fungiayoung: looks like the failures coincide with introduction of dnspython==1.10.0 as a dependency (there was no dnspython in the pip freeze from the working run)19:34
ayoungtermie you'll have to beat up dolphm about that.  I think the discussion happened about a year ago or more.  I don't think there was a separate BP for Crednetilas from the rest of the V3 API.19:34
dprincefungi: back in a bit...19:34
ayoungfungi, ACHA!19:34
termieayoung: k19:34
ayoungfungi, so...let me see what happens if I remove dns python from my venv...19:35
ayoungfungi, ... and that makes sense19:35
fungiayoung: it may get pulled back in again. i'm looking now to see what started depending on it19:35
ayoungdnspython pulls in IPv6 support, which is b0rk3d beyond belief in Eventlet19:35
fungiyes, as mikal will happily tell you19:35
fungii think he lost a couple weeks of his life to that recently19:36
openstackgerritA change was merged to openstack/quantum: Create a common function for method _parse_network_vlan_ranges used by plugins.  https://review.openstack.org/2865619:38
ayoungfungi, what pull;s that in?19:39
fungii can't tell. the pip install stdout/stderr don't seem to be echoed in the log19:39
ayoungI see it in the tox and venv, but it doesn't seem to be explicitly required in tools19:39
ayoungfungi, I can wipe mine out and regen, and captuer the output..1 sec19:39
fungithanks. that's what i was about to resort to doing anyway19:40
apevecayoung, ipv6 was disabled recently in https://review.openstack.org/#/q/Ib7a5a5f0dcac82229b7b7fb3df4b1ef141586c5f,n,z19:40
apevecayoung, but looks like not completely enough?19:40
*** jbresnah has joined #openstack-dev19:40
ayoungapevec, I thik the problem is happening starting up the server to run the unit tests that come in via a web request19:41
*** PaulM has joined #openstack-dev19:41
apevecfungi, what about disabling ipv6 on slave machines?19:42
stevemartermie: ping19:42
termiestevemar: not here19:42
termiestevemar: i replied to your email man, ISN'T THAT ENOUGH19:42
termiestevemar: what's up19:42
stevemartermie: :) had some more q's19:42
termiestevemar: pro-tip, work from the version before the last commit19:43
termiestevemar: whatever that means in this WIP world19:43
*** rnirmal has joined #openstack-dev19:44
stevemartermie: the first part of the email - so for the consumer, UUID is the key, but in create_request_token you were passing in the consumer_key to get the consumer19:44
stevemartermie: shouldn't that be the UUID?19:44
*** mkollaro has quit IRC19:44
stevemartermie: but from the request, we only know the consumer_key, not the UUID19:45
*** sandywalsh has joined #openstack-dev19:46
topoldolphm, QQ regarding spzala's most recent patch.   He can work on how to fix the reduce test coverage issue.  But for my own edification, if this is mainly for a backport, doesnt that imply that the code is fairly stable at this point and thus won't change much?  And since his change only touches the LDAP backend its scope is reduced?  So if you put both of those together why is the reduced...19:46
topol...test coverage so scary at this point in the already released code's lifecycle?19:46
*** digitalsanctum has quit IRC19:47
*** salv-orlando has joined #openstack-dev19:48
clarkbkeystone folks it would be super duper if https://review.openstack.org/#/c/28592/ and its child change could get reviewed19:48
termiestevemar: they are the same thing19:49
termiestevemar: the keys are just strings19:49
termiestevemar: it should be the uuid19:49
* stevemar facepalm19:50
termiestevemar: do what i say not what i code :p19:50
stevemartermie: 2nd q in the email - i was worried i didn't explain it well enough, i'll try again here :)19:50
*** salv-orlando_ has joined #openstack-dev19:51
termiestevemar: the first part of the answer seems like it should be obvious: no backend exists19:51
termiestevemar: unless you've written one19:51
stevemartermie: yep yep19:52
*** pabelanger has quit IRC19:52
stevemartermie: i was wondering how these guys are generated:19:52
stevemar1c. ... and is given a generated string with a funny format::19:52
stevemar19:52
stevemar     oauth_token=1234-{suitably_secure_hash:saltybits+'1234'+'5678'}19:52
stevemar     &oauth_token_secret=bar-secret19:52
stevemartermie: just some hash, on the consumer key? anything specific?19:53
*** salv-orlando has quit IRC19:53
*** salv-orlando_ is now known as salv-orlando19:53
termiestevemar: can be whatever you'd like at the moment, just hash a uuid or the like19:53
stevemartermie: ahh okay19:53
termiestevemar: not related to the consumer key19:53
stevemartermie: okay, that is do-able19:54
*** salv-orlando_ has joined #openstack-dev19:54
stevemartermie: i think that is cool for now, i can make some head way19:55
stevemartermie: ty!19:55
termiestevemar: what it will eventually be is: some salt, probably based on the user, a string that is public like 1234, and the request-verifier19:55
termiestevemar: so the hash is a combination of those three things, when they try to get the access token they need to hand us the verifier so we can regenerate the whole string19:56
termiestevemar: and verify that the user actually gave them the verifier19:57
termiestevemar: for the simple case right now you can ignore the verifier or just make the verifier a big uuid that you've hashed19:57
*** salv-orlando has quit IRC19:57
*** salv-orlando_ is now known as salv-orlando19:57
stevemartermie: hmmm, okay, i'll try the latter for now19:58
*** savid has joined #openstack-dev19:59
*** pcm__ has quit IRC20:00
savidHi guys, just curious, has the possibility of being able to modify nova policy via API call ever been considered by the community?20:00
*** HenryG has quit IRC20:03
*** vipul|away is now known as vipul20:03
ayoungfungi, out of curiosity, on that machine, is there a pip log file?  Mine wants to put it under /root/.pip20:05
fungiayoung: well, those slaves get reused for all sorts of tests so any pip log for the jenkins user is likely to be really, really long... but i'll take a look20:05
*** dprince has quit IRC20:05
*** topol has quit IRC20:06
*** asgreene has joined #openstack-dev20:06
*** lloydde has quit IRC20:07
*** glikson has quit IRC20:07
ayoungfungi, I have it running now, but my last run was on the wrong machine, and it failed out...jusrt wondering if it was faster for you to look...parallelizing20:07
ayoungDownloading/unpacking dnspython>=1.10.0 (from swift->-r /opt/stack/keystone/tools/test-requires (line 30))20:07
ayoung  Downloading dnspython-1.10.0.zip (212Kb): 212Kb downloaded20:07
ayoung  Running setup.py egg_info for package dnspython20:07
fungiayoung: yup, ~jenkins/.pip/pip.log has quite a bit of history. i should be able to find the job timestamp for a point of reference20:08
fungibut it'll probably be the same as what you're finding. let's see20:08
ayoungfungi, it is all swift's fault20:09
ayoungline 30 is20:09
ayounghttp://tarballs.openstack.org/swift/swift-master.tar.gz#egg=swift20:09
*** yidclare has joined #openstack-dev20:09
clarkbis keystone installing swift? o_O20:09
ayoungclarkb, # swift_auth test dependencies20:10
*** [1]glikson has joined #openstack-dev20:10
fungiincestuous dependencies indeed20:10
clarkbayoung: seems like a good place for a mock :)20:11
*** [1]glikson is now known as glik20:11
*** bmclaughlin has quit IRC20:12
fungimmm, actually i can't tell for sure that's what's happening on the slave, since pip running in the jobs seems not to log to pip.log after all (~jenkins/.pip/pip.log was last modified april 29 for the slave where that periodic job ran today)20:12
*** toanster has quit IRC20:12
fungibut if swift started depending on dnspython around may 3rd or 4th, that would also be telling20:13
* fungi fires up the ol' git blamethrower20:13
*** AlanClark has quit IRC20:13
*** glik has quit IRC20:15
fungiayoung: https://review.openstack.org/#/c/27428/8/tools/pip-requires20:15
swifterdarrellfungi: ayoung: Swift did add a dep on dnspython20:15
*** spzala has joined #openstack-dev20:16
ayoungswifterdarrell, yes, we see that20:16
fungilanded in master on the 3rd, corresponding with when these tests started to fail20:16
swifterdarrellthis bit/will bite packages (including myself a day or so ago)20:16
openstackgerritA change was merged to openstack-dev/grenade: Fix setup-grenade to pass user and host as parameter  https://review.openstack.org/2808120:16
swifterdarrell*bite20:16
*** drewlander has quit IRC20:16
swifterdarrellHere's me trying to roll that dep back to at least what Ubuntu Oneiric/Precise system packages provide: https://review.openstack.org/#/c/28356/   (not sure if that's relevant to your problems, exactly)20:17
openstackgerritA change was merged to openstack-dev/grenade: Fix: kill screen on stop-base  https://review.openstack.org/2863720:17
*** anniec has joined #openstack-dev20:17
swifterdarrellcorresponding openstack/requirements change:  https://review.openstack.org/#/c/28653/20:17
fungiayoung: i want to say the ipv6 breakage stemmed from dnspython/eventlet monkey-patch interactions, and may depend on which gets imported first20:18
ayoungfungi, not really20:18
*** toanster has joined #openstack-dev20:18
ayoungfungi, it is IPV620:18
ayoungbut I think that it is explicitly eventlet20:18
fungik20:19
ayoungbut...the questions is whether we can somehow fix this.20:19
fungimy steel sieve memory may also be conflating a couple of tangentially-related issues there20:19
openstackgerritA change was merged to openstack-dev/grenade: Ensure default flavor is set when upgrading tempest  https://review.openstack.org/2869320:19
*** toanster has quit IRC20:19
ayoungswifterdarrell, what happens if we remove the dnspython dependency for swift?  What breaks?20:19
ayoungI assume IPv6 on later platforms, right?20:20
swifterdarrellayoung: if your Swift doesn't try to use the cname_lookup middleware, it's completely irrelevant20:20
ayoungswifterdarrell, so one option is to remove it from the pip-requires for swift20:21
*** cdsvrd has quit IRC20:21
ayoungthat is a little draconian20:21
ayoungI suspect that Keystone without it will work OK20:21
swifterdarrellayoung: so, there was a bugfix for the cname_lookup middleware--pretty small.  But the cname_lookup middleware was both functionally optional (safe-conditionally imported) and its dependency, dnspython was not listed as a dependency of Swift anywhere20:21
ayoungswifterdarrell, somehow that assumption got falsificated20:22
swifterdarrellayoung: some folks didn't like that implicit dependency and made it a first-class dependency (which also allows previously-optional tests covering cname_lookup to start running for everyone instead of whoever had actually manually installed dnspython)20:22
swifterdarrellayoung: that being said, it's a dep that was already in openstack/requirements, so outside of packaging, I'm not sure why it's causing other consternation20:22
*** jamespage_ has joined #openstack-dev20:23
swifterdarrellayoung: personally, I got pissed off that more fanfare wasn't made when that patch landed (/confession I reviewed it and the guy who wanted it made a first-class dep sits right next to me; but quite a few of us missed the ramifications of that new dep)20:23
ayoungswifterdarrell, OK, think we should roll that back to start, and then have a more thought out plan to deal with dnspython, IPv6 and eventlet.  Sound about right?20:24
*** salv-orlando has quit IRC20:24
ayoungThinking Eventlet should get the Old Yeller treatment myself.20:24
*** salv-orlando has joined #openstack-dev20:25
fungii think a lot of people feel that way, but the bullet will require a fair amount of code :/20:25
*** megha is now known as baba20:25
swifterdarrellayoung: I don't understand enough about how dnspython is causing trouble for you, but I'm pretty sure getting rid of eventlet is not the cheapest solution20:25
*** lloydde has joined #openstack-dev20:26
*** toanster has joined #openstack-dev20:26
torgomaticso here's a crazy thought: what if the requirements gate were made less strict, so that Swift could specify a dnspython requirement that works with distro packages, but the stuff in openstack-requirements would also work?20:26
swifterdarrellit *was* already in openstack/requirements20:26
swifterdarrelltorgomatic: I already linked to the two patchsets which (will) do that (when merged)20:27
torgomaticthat is, change the check from "is exactly equal to openstack/requirements" to "is satisfied by what's in openstack/requirements"20:27
fungii don't think there's anything specific to openstack/requirements which prevents that now20:27
torgomaticswifterdarrell: ah, I must not have seen that20:27
ayoungswifterdarrell, dnspython is somehow failing on ::1 lookups for IPv6.  We commented it out of our unit tests, but looks like it is getting triggered anyway.20:27
swifterdarrellayoung: can you give more details on what's breaking?20:27
swifterdarrellayoung: (thx)20:27
swifterdarrellhrm20:27
ayoungswifterdarrell, there was a bug for it..I'll link20:27
openstackgerritA change was merged to openstack/nova: Imported Translations from Transifex  https://review.openstack.org/2860020:28
ayoungswifterdarrell, https://bugs.launchpad.net/keystone/+bug/117620420:28
uvirtbotLaunchpad bug 1176204 in keystone/grizzly "keystone ipv6 tests fail" [Undecided,Fix committed]20:28
*** jclift has quit IRC20:28
notmynameayoung: swifterdarrell: I remember I know that mikal was having some issues with dnspython and IPv6, but I don't know the resolution. mikal added it to openstack/requirements, so I'm assuming there was some sort of resolution20:29
*** jamespage_ has quit IRC20:29
ayoungswifterdarrell, https://bitbucket.org/eventlet/eventlet/issue/105/name-resolution-needs-to-support-ipv620:29
*** drewlander has joined #openstack-dev20:29
ayoungnotmyname, I would state that "resolution" is too optimisitic a term.20:30
notmynameayoung: all I'm saying is that I don't know what it was20:30
swifterdarrellI'm currently at a lost as to how Swift's requirements.txt started making DNS resolution for keystone unit tests fail for IPv6...20:31
swifterdarrellthat's quite a bit of action-at-a-distance20:31
notmynamese we added the dependency on dnspython (and subsequently are trying to bump the version requirement back to match LTS version). dnspython was already in openstack/requirements, so whatis the issue?20:32
mordredso ...20:32
ayoungswifterdarrell, swift's tarball started making keystone's venv code pull it in20:32
notmynameand I see in -infra what the issue actually is?20:32
mordredwell, that's a thing that's up for discussion20:32
ayoungthe issue is that dnspython breaks when monkeypatched and trying to do IPv620:32
notmynamewhy is keystone pulling in swift?20:33
mordredone take on this is that keystone/swift integration should be in tempest, not in keystone unittests. BUT...20:33
*** reed_ has joined #openstack-dev20:33
mordredalso, dnspython+eventlet turns out to be broken20:33
notmyname...for ipv620:33
mordredmikal spent a good amount of time trying to figure out how it can work or not20:33
*** lloydde has quit IRC20:33
mordredyeah20:33
mordredmikal: around?20:34
clarkbis he awake yet?20:34
mordredlifeless is. I expect mikal to be awake with lifeless is20:34
lifelessnup20:34
lifelessmikal is 2 hours behind me20:34
mordredin any case - the 'easy' answer might be for keystone to not do unittests on swift20:34
mordredbut I tihnk that's a false fix20:35
mordredbecause devstack would still install both keystone and swfit20:35
ayoungI'm OK for yanking swift auth from Keystone.20:35
*** sacharya has quit IRC20:35
ayounggoing once20:35
mordredwhich would mean that dnspython would be pulled in20:35
mordredanywya20:35
ayounggoing twice20:35
swifterdarrellI'm dubious of unit tests which start breaking when deps in openstack/requirements get installed... is that crazy?20:35
clarkbmordred: yeah there are definitely two different problems here20:35
clarkb(at least 2 :) )20:35
mordredswifterdarrell: python dependencies are a complete disaster - so don't be too dubious20:35
ayoungswifterdarrell, no, we in Keystone are paranoid.  And we end up catching things like this.20:35
mordredayoung: is it a problem in keystone if dnspython is installed? or just in your unittests?20:36
clarkbmordred: its a problem if ipv6 is involved. the unittests just happen to catch it20:36
jeblairswifterdarrell: the unit tests are correctly failing because eventlet's monkeypatching code paths change when dnspython is installed.  so, yeah, there's something to be dubious of here, but i think it's eventlet and monkeypatching, not the unit tests.  :)20:36
mordredso the problem is real20:36
mordredjeblair: ++20:36
mordredit just takes a certain combination of things to trigger it20:37
swifterdarrellokie doke20:37
mordredso all removing swift dep from keystone will do is cover up the problem again, yeah?20:37
mordredI think the real solution is to rewrite swift in go20:38
*** rkukura has quit IRC20:38
fungiunfortunately it also means that unless some specific resolution is reached, our effort to switch to python26 unit test jobs on centos6 will cause keystone master and grizzly to instabreak in the gate20:38
mordredkeystone in perl and nova in objective-c - that way we can be assured that we won't step on each other with dependencies20:38
mordredfungi: sad panda20:38
ayoungI think the real solution is dropping eventlet20:38
mordredyeah. clearly. it seems to be the cause of most of our problems20:39
swifterdarrellayoung: that'll be an interesting patch20:39
* mordred not kidding20:39
fungione distasteful option would be to make keystone python26 unit tests non-voting in the gate, but ew20:39
mordredno, I don't think that's the right choice20:39
fungiagreed20:39
ayoungswifterdarrell, we can already do it for Keystone20:39
mordredayoung: oh yeah?20:39
ayounghttps://review.openstack.org/#/c/28387/20:39
ayoungmordred, jamielennox is kicking ass and taking UUIDs20:40
ayoungthat patch is overkill, as he was learning the tree and refactoring as he was going, but the idea was that we can extract all of the eventlet code and run using webtest20:40
ayoungmordred, I had keystone/apache HTTPD running a year ago.20:41
mordredayoung: awesome. I am fully in favor of that as a path20:41
*** lloydde has joined #openstack-dev20:41
mordredayoung: so, let me summarize - there is a problem deep in here, but it's a weird enough one that in a production deployment it's avoidable for now20:41
mordredayoung: if we remove swift from keystone unittests for the moment, we unbreak the current issue20:41
mordredayoung: then, we make the issue go away long term by removing eventlet from keystone20:42
mordredfirst in the tests20:42
mikallifeless: mikal is actually awake, just still booting up20:42
mordredand eventually just by having apache/httpd handle the good stuff20:42
mordredyeah?20:42
fungikeep in mind this is affecting keystone stable/grizzly too, so whatever is done to work around or fix this on master needs a backport as well20:42
mordredayoung: do you think removing the test-requires on swift in stable/grizzly is ok?20:43
notmynameit seems that the issue is that if dnspython is used, eventlet ends up patching getaddrinfo(). which causes ipv6 resolutions to break. and since keystone is pulling in swift (I don't understand why on that yet) and swift is now pulling in dnspython, keystone's tests (which probably do some getarrdinfo calls with ipv6) break.20:43
ayoungI'll try it morazi20:43
notmynameis that correct?20:43
ayoungmordred,20:43
fungiprobably as simple as pinning test-requires on swift in stable/grizzly to a version before the dnspython dependency was declared20:43
mordrednotmyname: yes. but I think we've got a plan forward that removes many of those pieces20:44
mordredayoung: awesome20:44
fungiif there's concern over ripping swift out of the test-requires for keystone stable/grizzly altogether i mean20:44
*** zul has joined #openstack-dev20:44
mordredayoung: I could love it if keystone+apache became a new model forward for how we don't have to solve the concurrency issues in pure python20:45
mordreds/could/would/20:45
lifelessmikal: heh20:45
lifelessmikal: of course in a few months you'll be awake 24x7 again20:46
notmynamemordred: some people, when confronted with a problem, say, "I know! I'll use C!"....20:46
mikalYay!20:46
mikalNot really looking forward to that20:46
lifelessmikal: but I can mail you a staplegun if you like20:46
lifelessmikal: so you can fix your problem to the wall and get some shuteye :)20:46
mikalHeh20:46
mordrednotmyname: yup.20:46
vishyjgriffith: ping20:47
mordrednotmyname: some people, when confronted with a problem, say "I know! I'll use a thing that's already been written and powers over half of the internet"20:47
fungipfe20:47
notmynamemordred: http://www.slideshare.net/GrahamDumpleton/pycon-us-2013-making-apache-suck-less-for-hosting-python-web-applications20:48
*** adalbas has quit IRC20:48
jgriffithvishy: hey ya20:48
*** drewlander has quit IRC20:48
clarkbone thing we may have overlooked: leaving dnspython in swift may bork ipv6 there20:48
vishyjgriffith: considering my obvious lack of cinder reviews you probably should just remove me from core :)20:49
clarkbnot sure if that is a concern but I don't think anyone brought it up20:49
jgriffithvishy: or you could join the dark side forever :)20:49
vishyjgriffith: a bit unlikely i feel :)20:49
jgriffithvishy: :)20:49
jgriffithvishy: sure, but I'm hoping to still be able to get your advice and help (you can't hide)20:50
clarkbalso do we know why only centos is affected?20:51
clarkbthe other ipv6 + dnspython stuff we ran into was affecting ubuntu too20:51
vishyjgriffith: i will still happily give advice and help20:51
jgriffithvishy: I had assumed, thanks20:53
fungiclarkb: might be a question for dprince... he's a bit more intimate with the other weirdness related to having to use red hat's eventlet package to work with the modified python they ship. not sure if there's some special interaction there (eventlet present when dnspython is imported?)20:55
*** alop has quit IRC20:56
*** salv-orlando has quit IRC20:56
*** Mapleh has quit IRC20:57
*** bswartz has quit IRC20:59
*** nunosantos has joined #openstack-dev21:00
*** rerngvit has joined #openstack-dev21:00
*** lloydde has quit IRC21:00
*** rerngvit has left #openstack-dev21:00
*** markmcclain has quit IRC21:01
ayoungfungi, I'm running it now.  I'll let you know what happens, but I expect some of the tests to fail.  We can probably disable those tests, and I would be OK with migrating them to Tempest. I'm looking now to see what I'd expect to fail21:02
*** johnthetubaguy has joined #openstack-dev21:02
ayoungtest_s3_token_middleware.py is the obvious one21:02
*** alop has joined #openstack-dev21:03
fungiayoung: okay. should i hold off from making centos tests active in the gate, or do you think you have a sufficient handle on patching this up?21:03
*** spzala has quit IRC21:03
*** mikal has quit IRC21:03
ayoungfungi, if you make it active in gate, it will break everything.  DO IT!21:03
*** sacharya has joined #openstack-dev21:04
ayounglets get everyone pinging on this!21:04
fungithe impetus being today is the last day ubuntu has security support for oneiric, and they don't have later releases with python 2.621:04
*** mikal has joined #openstack-dev21:05
fungiso centos is basically our python 2.6 life raft now21:05
openstackgerritA change was merged to openstack/keystone: A minor refactor in wsgi.py  https://review.openstack.org/2865721:05
*** johnthetubaguy1 has joined #openstack-dev21:05
ayoungfungi, patch: **** Can't open patch file contrib/redhat-eventlet.patch : No such file or directory  how come I didn't see that error before?21:07
*** johnthetubaguy has quit IRC21:07
*** johnthetubaguy1 is now known as johnthetubaguy21:07
*** lauria has quit IRC21:08
fungiayoung: mmm, good question. the extent of my knowledge there is that rhel backported something from later python to 2.6, which in turn broke eventlet... so then they use a patched eventlet to work around their nonstandard python... it's honestly a mess21:08
*** ashwini has quit IRC21:08
*** adrian_otto has joined #openstack-dev21:10
ayoungfungi, yeah, but I ran the unit tests on this machine already, and didn't see it the first time around..maybe I am on the wrong machine?21:10
ayoungNope, it is one of dprince's at rax running centos....def the right machine.  Trying again with tox now21:11
funginot entirely sure what to suggest in that case21:11
ayoungtox still failed the same way21:12
ayoungactually...no21:12
ayoungKcMaster now passes21:12
ayoungKc11TestCase  fails...21:12
ayoungImportError: No module named contrib.ec221:13
ayoungok, I think that is a red herring, saw that before21:13
ayoungfungi, I think we are good with no swift.  I'll run the whole tox thing to confirm that only the swift middleware test fails21:13
fungiayoung: awesome21:14
fungiayoung: i'm going to cook dinner quickly, and then get behind the centos change and start pushing21:14
*** armax has quit IRC21:14
ayoungfungi, sounds good.  I just saw the Kc11Test case pass when running the full tox suite21:15
*** cloudchimp has quit IRC21:16
ayoungswifterdarrell, this would mean yanking the swift middleware test from keystone.21:17
*** pabelanger_ has quit IRC21:18
swifterdarrellayoung: been working on something else for a bit; to clarify, you're talking about not continuing to test a piece of code which lives in the Swift repo in the unit tests of the Keystone repo?21:18
swifterdarrellayoung: that seems entirely reasonable21:19
*** pabelanger has joined #openstack-dev21:19
ayoungswifterdarrell, we have th3e middleware in keystone21:19
openstackgerritA change was merged to openstack/python-glanceclient: Expand HACKING with commit message guidelines  https://review.openstack.org/2758921:19
*** zul has quit IRC21:19
ayoungswifterdarrell, https://github.com/openstack/keystone/blob/master/keystone/middleware/s3_token.py21:19
*** chuck_ has joined #openstack-dev21:19
swifterdarrellah, so it'd be not continuing to test a piece of code which lives in the Keystone repo (which runs inside of Swift) in the unit tests of the Keystone repo?21:20
*** radez is now known as radez_g0n321:22
ayoungswifterdarrell,  so the keystone code pulls in code from swift. Kindof round-about21:22
*** malini_ has quit IRC21:22
ayoungswifterdarrell, that code was origianlly in swift, I think21:23
swifterdarrellayoung: *nod*21:23
ayoungswifterdarrell, OK, that code totally does not need swift21:23
ayoungswift_utils.split_path(req.path, 1, 4, True)21:23
swifterdarrellayoung: IIRC it also didn't used to scale well until it was mated with Swift's concurrency framework (the oft-maligned eventlet)21:23
ayoungand  self.logger = swift_utils.get_logger(conf, log_route='s3token')21:23
*** jamespage_ has joined #openstack-dev21:24
swifterdarrellayoung: the first one could be inlined easily, but I think deployers are going to expect this keystone middleware to handle logging the way all the rest of Swift does21:25
*** erfanian has joined #openstack-dev21:25
ayoungswifterdarrell, it is possible that eventlet maps well to someones use case in Open Stack, just not Keystone's.  But I doubt it.21:25
swifterdarrellayoung: duly noted21:25
*** diogogmt has quit IRC21:26
ayoungswifterdarrell, there is very little in here that is keystone specific, either.  It more likely belongs in common21:26
ayoungbut as for logging, is swift really doing something so different?21:27
*** giulivo has joined #openstack-dev21:29
ayoungOK, just looked at the get_logger code.  Yes, yes you are21:29
*** jasondotstar has quit IRC21:29
notmynameayoung: :-)21:29
*** alop_ has joined #openstack-dev21:29
*** alop has quit IRC21:30
*** alop_ is now known as alop21:30
*** jamespage_ has quit IRC21:31
*** mlavalle has quit IRC21:31
*** lloydde has joined #openstack-dev21:32
*** pabelanger has quit IRC21:32
*** pabelanger has joined #openstack-dev21:32
*** FatDarrel has quit IRC21:33
*** esheffield has quit IRC21:34
*** SergeyLukjanov has quit IRC21:35
*** openstackstatus has quit IRC21:36
*** cdsvrd has joined #openstack-dev21:38
*** chuck_ has quit IRC21:38
termiei really shouldn't try to schedule work after code reviews21:39
*** soleblaze has quit IRC21:39
*** lloydde has quit IRC21:41
*** athomas has quit IRC21:41
*** gyee has quit IRC21:41
*** jjmb has joined #openstack-dev21:43
*** openstackstatus has joined #openstack-dev21:44
ayoungfungi, OK,  I deleted:    tests/test_s3_token_middleware.py and modified:   tools/test-requires to remove swift, and now the tests run 100%21:44
*** jbresnah has quit IRC21:44
ayoungI'm going to post that for review.  We can argu over it there.21:44
*** lpmulligan has quit IRC21:44
swifterdarrellayoung: maybe your test code could import eventlet, monkeypatch eventlet.monkey_patch to NOOP, then continue with normal imports?   I'm only half-joking21:46
*** alop has quit IRC21:47
*** eglynn has quit IRC21:48
ayoungswifterdarrell, only half?21:49
*** bswartz has joined #openstack-dev21:49
*** salv-orlando has joined #openstack-dev21:49
fungiayoung: i'm about to make that review very popular, when it becomes the only change in keystone which can pass the gate ;)21:50
ayoungfungi, schweeet!21:51
apevecayoung, fungi - what about stable ?21:53
fungiapevec: ayoung needs to post a backport of it to keystone stable/grizzly too21:53
fungistable/folsom is unaffected by the issue though21:53
*** dolphm has joined #openstack-dev21:54
*** dolphm has quit IRC21:54
*** READ10 has quit IRC21:54
apevecfungi, how come, it also has http://tarballs.openstack.org/swift/swift-master.tar.gz#egg=swift in test-requires ?21:55
ayoungapevec, to get the s3 middleware test to pass21:55
*** redbeard2 has quit IRC21:55
ayoungapevec, https://github.com/openstack/keystone/blob/master/keystone/middleware/s3_token.py21:56
apevecfungi, ayoung - what about changing that test dep to http://tarballs.openstack.org/swift/swift-stable-folsom.tar.gz21:56
apevecand http://tarballs.openstack.org/swift/swift-stable-grizzly.tar.gz21:56
apevecon respective keystone stable branches?21:56
*** dolphm has joined #openstack-dev21:56
*** stevemar has quit IRC21:56
ayoungapevec, lets hand this off to dolphm , as I need to catch a bus.  dolphm ...21:57
fungii expect that would work too, but it's up to ayoung. i'm really just the ci messenger trying not to get shot21:57
ayoungdolphm, we are pulling in swift, which is pulling in dns21:57
ayoungdolphm, and dns gets up IPv6 nasitness, which is why the gate is breaking21:58
dolphmoh really?21:58
fungi(or will be anyway)21:58
dolphmin folsom?21:58
ayoungdolphm, look in test-requires21:58
openstackgerritA change was merged to openstack-dev/devstack: Add pbr to devstack.  https://review.openstack.org/2859021:58
ayoungdolphm, master21:58
fungipost-folsom. grizzly and master only21:58
ayoungdolphm, https://github.com/openstack/keystone/blob/master/keystone/middleware/s3_token.py#L4121:58
ayoungthat needs swift code which we get from21:58
*** martine has quit IRC21:58
ayounghttps://github.com/openstack/keystone/blob/master/tools/test-requires#L3021:59
bknudsonayoung: which tests are failing?21:59
ayoungbknudson, all the ones that start up a server.  Keystone client21:59
dolphmayoung: is there a gate failure we can look at?21:59
bknudsonMy patch just disabled the ones where it started keystone with ::121:59
ayoungOK, I'm out...bus is leaving in 221:59
apevecdolphm, I think fungi has logs22:00
fungidolphm: ayoung: i can get you logs and a redux momentarily22:00
*** alunduil has quit IRC22:00
*** alop has joined #openstack-dev22:00
*** radsy has joined #openstack-dev22:00
*** jjmb has quit IRC22:01
*** jecarey has quit IRC22:01
bknudsonso keystone doesn't start at all with eventlet + dnspython on centos?22:02
bknudsonI thought it was just IPv6 address resolving that was broken.22:02
clarkbbkero: there is apparently another reaction when run on centos22:03
clarkbpotentially because rhel and friends need a patches eventlet to accomodate python2.6 on rhel22:03
*** ayoung has quit IRC22:03
apevecfungi, ayoung - I've put https://review.openstack.org/28728 just as a test, to see what jenkins will say22:03
*** aelkikhia has joined #openstack-dev22:04
*** portante is now known as portante|afk22:04
clarkbs/patches/patched/22:04
*** brunnhilde has quit IRC22:04
*** lloydde has joined #openstack-dev22:05
*** sacharya has quit IRC22:05
*** FatDarrel has joined #openstack-dev22:06
*** kbringard has quit IRC22:06
*** johnthetubaguy has quit IRC22:06
fungibknudson: dolphm: to catch you up, immediately after swift landed a change in master to start depending on dnspython, keystone master and stable/grizzly unit tests began doing this on centos... http://logs.openstack.org/periodic/periodic-keystone-python26-stable-grizzly-centos6/4/console.html.gz22:07
*** savid has quit IRC22:08
dolphmfungi: i assume dnspython is monkey patching something?22:08
bknudsonfungi: at the same time, keystone master otherwise failed, too, because it couldn't start the server with an IPv6 address.22:08
bknudsonfungi: so this is weirder. I might be able to put together a centos vm to debug on.22:09
fungibknudson: dolphm: note that http://logs.openstack.org/periodic/periodic-keystone-python26-stable-grizzly-centos6/3/console.html.gz passed on the same exact keystone gitsha the night before, and the only difference according to pip freeze was that dnspython was suddenly present22:09
bknudsonsee why the server doesn't start.22:09
*** zul has joined #openstack-dev22:09
bknudsonfungi: in the ipv6 address case, we found the line of code in eventlet that said "eventlet doesn't support IPv6 address resolution"22:10
fungibknudson: yes, that's a (at least for some of us) well-known situation with eventlet22:11
*** maoy has quit IRC22:11
bknudsonalthough it's maybe not worth it to figure out what the problem is if removing the swift dependency is better anyways.22:11
fungibknudson: add to the mix, eventlet and dnspython do not play well together, possibly dependent on which is loaded into the namespace first (they both monkey-patch some of the same bits of stdlib)22:11
bknudsonthe output says this: "keystone.common.wsgi: DEBUG: Starting .tox/py26/bin/nosetests on ::1:0" ?? why's it say ::1:0?22:12
bknudsonis localhost ::1?22:12
*** soleblaze has joined #openstack-dev22:12
bknudsonis hostname ::1?22:12
clarkbbknudson: yes, ::1 is the loopback address for ipv622:12
*** anniec has quit IRC22:12
fungiyes, ::1 is the ipv6 equivalent of 127.0.0.0/822:12
clarkb:0 is listen on a random high port22:12
*** Ryan_Lane1 has joined #openstack-dev22:13
bknudsonbut on other systems, does it say ::1 or 127.0.0.1 ?22:13
*** Ryan_Lane has quit IRC22:13
fungiclarkb: you're thinking on a different osi layer there ;)22:13
*** Ryan_Lane1 is now known as Ryan_Lane22:13
*** Ryan_Lane has joined #openstack-dev22:13
*** giulivo has quit IRC22:13
*** giulivo has joined #openstack-dev22:13
fungibknudson: it usually says both on the modern linus distributions with which i'm familiar. some of them may say localhost6 for ::1 instead22:14
*** soleblaze has quit IRC22:14
*** mlavalle has joined #openstack-dev22:14
bknudsonoh, I was looking at the wrong log, never mind.22:14
*** yidclare has quit IRC22:14
fungier, s/linus/linux/22:14
*** soleblaze has joined #openstack-dev22:14
apevecjd__, is ceilo stable/grizzly good to go?22:15
clarkbfungi: am I? ifconfig says lo has address ::1/12822:15
*** yidclare has joined #openstack-dev22:16
apevecjd__, I guess yes, nothing is open: https://review.openstack.org/#/q/status:open+project:openstack/ceilometer+branch:stable/grizzly,n,z22:16
fungiclarkb: your :0 comment about listening on a random port is layer 4-ish (and a linux tcp/ip stack detail anyway)22:16
clarkboh that bit22:16
fungioh, nevermind. you were commenting on the :0 part of ::1:022:16
* fungi looks sheepish22:17
bknudsonall this seems to indicate that keystone isn't going to work on centos, if they happen to have dnspython installed.22:17
dolphmbknudson: clarkb: related? https://bugs.launchpad.net/nova/+bug/116482222:18
uvirtbotLaunchpad bug 1164822 in nova "eventlet monkey patches getaddrinfo if dnspython is installed" [Medium,Fix committed]22:18
clarkbbknudson: correct, but they only need dnspython for a small subset of functionality so it doesn't appear to bother them too much22:18
fungibknudson: i can't entirely disagree with that assertion, as i don't know a ton about keystone's network stack22:18
openstackgerritA change was merged to openstack/nova: Record smoketest dependency on gFlags  https://review.openstack.org/2785222:18
openstackgerritA change was merged to openstack/glance: Add documentation for query parameters  https://review.openstack.org/2854822:18
openstackgerritA change was merged to openstack/nova: Fixes KeyError bug with network api associate  https://review.openstack.org/2849722:18
openstackgerritA change was merged to openstack/nova: Fix zookeeper import and tests  https://review.openstack.org/2855522:18
clarkbdolphm: yes, and that appears to be a potential fix22:18
dolphmclarkb: adding keystone to that bug then22:18
clarkbso maybe you can do that and get ipv6 testing back too22:18
*** belmoreira has quit IRC22:19
bknudsondolphm: I don't know if it's just getaddrinfo, though. Since the getaddrinfo problem was only about IPv6 address resolution.22:20
bknudsonand I don't think that's what's happening on the failing centos tests.22:20
dolphmbknudson: only about ipv6 resolution? is that not the problem here?22:21
bknudsondolphm: that was the problem on the other systems, and we worked around it by not testing ipv6. So all these tests that are now failing on centos pass on other systems with dnspython.22:22
*** datsun180b has quit IRC22:22
bknudsonthat's why I think it's not the IPv6 getaddrinfo problem... I don't see how IPv6 is involved in the current failing tests.22:23
*** koolhead17 has quit IRC22:23
bknudsonbut I haven't got a centos system handy22:23
clarkbI think it may not be the resolution necessarily but may be the monkey patching22:23
clarkbmy understanding of what mikal said in that bug is you avoid monkeypatching with that fix22:24
*** hartsocks has quit IRC22:25
*** hartsocks has joined #openstack-dev22:25
bknudsonclarkb: maybe it would be worth a try to do the same fix as https://review.openstack.org/#/c/26325/ ?22:25
*** adrian_otto has quit IRC22:25
clarkbyeah you can whip that up fairly quick and jenkins will let us know if it works or not22:26
bknudsonI'll give it a shot22:26
dolphmclarkb: all he's doing is setting an environment variable as early as possible; you can test that yourself without a patch22:29
dolphmbknudson: ^22:29
clarkbI thought he was importing eventlet before anything else22:29
clarkbso that monkey patching does not occur22:30
fungidolphm: bknudson: clarkb: some of the difference may have to do with rhel vs debuntu conventions in /etc/hosts... http://paste.openstack.org/show/37073/22:30
apevecfungi, you said earlier that folsom is not affected, but isn't that just b/c you don't have centos jobs there? e.g. https://review.openstack.org/#/c/28679/22:30
fungiworth keeping in mind anyway22:30
dolphmclarkb: more importantly he's setting an environment variable before eventlet has a chance to monkeypatch, so that it won't do so22:30
dolphmexport EVENTLET_NO_GREENDNS=yes22:30
clarkbah22:30
clarkbI missed that. currently doing far too much mutlitasking22:30
fungiapevec: no, we're doing bitrot jobs of stable/folsom on centos as well, and those are consistently passing22:30
apevecfungi, interesting22:31
clarkbthey may not include those tests22:31
fungiapevec: it may be that keystone didn't start to depend on swift until grizzly22:31
*** hartsocks has quit IRC22:31
bknudsonclarkb: it does look like it kinda worked.22:32
dolphmfungi: apevec: we've had that import from swift for as long as i can remember22:32
apevecyeah, folsom keystone also has swift-master.tar.gz in test-requires22:32
fungidolphm: apevec: so there's something else making the folsom unit tests immune to this issue then, but they're definitely passing22:32
fungidolphm: apevec: success still this morning... http://logs.openstack.org/periodic/periodic-keystone-python26-stable-folsom-centos6/9/console.html.gz22:33
*** alop_ has joined #openstack-dev22:33
dolphmfungi: i don't think swift does backports, so whatever version of swift keystone stable/folsom is testing against definitely wouldn't be pulling in dnspython22:33
*** eharney has quit IRC22:33
*** alop_ has quit IRC22:34
*** rnirmal has quit IRC22:34
*** alop_ has joined #openstack-dev22:34
dolphm"Greendns: if dnspython is installed, Eventlet will automatically use it to provide non-blocking DNS queries.  Set the environment variable 'EVENTLET_NO_GREENDNS' if you don't want greendns but have dnspython installed."22:34
apevecdolphm, it's testing against swift-master22:34
*** alop has quit IRC22:34
*** alop_ is now known as alop22:34
apevechttps://github.com/openstack/keystone/blob/stable/folsom/tools/test-requires#L2622:35
openstackgerritA change was merged to openstack/nova: Enable live block migration when using iSCSI volumes  https://review.openstack.org/2541922:35
dolphmapevec: hmm, i don't think keystone stable/folsom has ipv6 support22:35
apevecah that's it then22:35
*** danwent has quit IRC22:35
*** Aarti has quit IRC22:37
bknudsondolphm: I slapped this together, we'll see what happens: https://review.openstack.org/#/c/28737/22:37
*** vkmc has joined #openstack-dev22:37
*** Aarti has joined #openstack-dev22:37
dolphmbknudson: lol give still credit for his comment22:37
fungidolphm: bknudson: just to reiterate, the impetus here is that we're switching from ubuntu oneiric (which is officially eol upstream today) to centos 6.3 for python 2.6 unit testing22:38
bknudsonI guess I'll show up in the blame anyways, so didn't have to change the name.22:38
dolphmbknudson: nah22:38
dolphmbknudson: i can't upvote plagiarism anyway22:38
*** lloydde has quit IRC22:39
bknudsonI could try to put my own spin on it.22:39
dolphmbknudson: why put it in those entry points instead of keystone/__init__.py?22:39
dolphmbknudson: oh, nevermind22:39
bknudsonI have to somehow catch it before "import eventlet"22:39
dolphmyeah, nova's got tests/__init__.py for that, etc22:40
termiei remember when we had one of those22:41
* termie still does22:41
*** epim has quit IRC22:42
dolphmtermie: that's step 1 of keystoneliter22:42
termiedolphm: __init__.py's EVERYWHERE22:43
termiedocs/__init__.py22:43
*** Aarti has quit IRC22:44
bknudsondolphm: Here's the next rev https://review.openstack.org/#/c/28737/22:46
dolphmbknudson: so the gate-keystone-python26-centos6 job should pass now?22:48
apevecfungi, stable/grizzly centos worked with swift tarball locked to grizzly (swift 1.8.0) https://review.openstack.org/#/c/28728/22:49
bknudsondolphm: If the problem was eventlet address resolution, then it'll pass22:49
*** lloydde has joined #openstack-dev22:49
*** terryh has quit IRC22:49
apevecfungi, granade failed though, but that's unrelated I guess22:49
bknudsondolphm: although it might be useful to take this change anyways since a) added back in ipv6 tests, and b) consistent with nova.22:49
*** terryh has joined #openstack-dev22:50
dolphmbknudson: cool, if it passes, merge it! i'm about to run off as well22:50
fungiapevec: maybe. grenade has been mostly passing on master, but maybe not on stable22:50
*** reed_ is now known as reed22:50
clarkbfungi: yeah I noticed in the last dayish it has failed semi randomly22:50
bknudsondolphm: I can check in later.22:50
*** jimfehlig has quit IRC22:50
*** jjmb has joined #openstack-dev22:51
*** Ryan_Lane has quit IRC22:51
fungiokay, just be aware the .*-centos6 jobs will shortly disappear, and the python26 jobs will be running on centos6 slaves22:51
*** terryh has quit IRC22:52
fungithose non-voting jobs were our dry-run to make sure everything would be okay. this swift->dnspython->keystone situation came up out of nowhere and jumped us at the end, or else we'd have given a little more of a heads-up22:53
*** zul has quit IRC22:53
dolphmno worries22:54
*** galstrom is now known as galstrom_zzz22:54
*** jjmb1 has joined #openstack-dev22:56
*** michchap_ has quit IRC22:56
*** michchap has joined #openstack-dev22:56
*** bknudson has quit IRC22:57
*** jjmb has quit IRC22:58
*** JonnyNomad_ has joined #openstack-dev22:59
*** matiu has joined #openstack-dev23:00
*** matiu has quit IRC23:00
*** matiu has joined #openstack-dev23:00
*** JonnyNomad has quit IRC23:00
*** alunduil has joined #openstack-dev23:00
*** yidclare has quit IRC23:00
*** michchap_ has joined #openstack-dev23:03
*** yidclare has joined #openstack-dev23:03
*** dolphm has quit IRC23:04
*** michchap_ has quit IRC23:05
*** michchap_ has joined #openstack-dev23:05
*** michchap has quit IRC23:05
*** rcleere has quit IRC23:07
*** zul has joined #openstack-dev23:07
*** soleblaze has quit IRC23:08
*** danwent has joined #openstack-dev23:08
*** soleblaze has joined #openstack-dev23:09
*** mdenny has quit IRC23:10
*** apevec has quit IRC23:12
*** pcm_ has joined #openstack-dev23:15
*** katylava has quit IRC23:16
*** ayoung has joined #openstack-dev23:17
openstackgerritA change was merged to openstack/cinder: Remove outdated cinder test doc  https://review.openstack.org/2872623:19
*** jayg is now known as jayg|g0n323:19
*** jamespage_ has joined #openstack-dev23:20
*** anniec has joined #openstack-dev23:21
*** FunnyLookinHat has quit IRC23:24
*** ashwini has joined #openstack-dev23:24
*** pixelbeat_ has joined #openstack-dev23:25
ayoungfungi, all resolved?23:25
fungiayoung: prolly. dolphm and bknudson were all over parallel reimplementations of your fix23:26
fungithough they both seem to have disappeared since23:27
ayoungfungi, I'll look in the evesdrop23:27
fungiredundant fixes are always welcome anyway23:27
*** giulivo has quit IRC23:29
ayoungfungi, https://review.openstack.org/#/c/28728/23:29
ayoungapproved and merging23:29
fungiright on. zuul is waiting for running tests to complete before it reloads to make centos the python26 default node type23:30
*** jvrbanac has quit IRC23:30
fungiso your merge will (hopefully) go through shortly thereafter23:31
*** bknudson has joined #openstack-dev23:32
*** dolphm has joined #openstack-dev23:35
*** michchap has joined #openstack-dev23:36
*** michchap_ has quit IRC23:36
*** Vinay_ has joined #openstack-dev23:37
*** bknudson has quit IRC23:37
Vinay_Hello is anyone here?23:37
Vinay_I have a question about instance_info_cache23:37
comstudVinay_: I can probably help you23:39
Vinay_Hi comstud thanks23:40
Vinay_We want to know what is instance_info_cache23:40
Vinay_and how it is used in nova23:40
comstudit's a table used to store a copy of data that is normally outside of nova23:40
comstudspecifically right now it's used to store networking information23:40
comstudso we don't have to query quantum for it when someone does a 'nova show' or 'nova list'23:40
comstudas that would add a lot of latency to the API call23:41
fungiayoung: oh, actually 28728 is already enqueued in the gate, so it'll go in before the reload (assuming it passes)23:41
*** lloydde has quit IRC23:41
Vinay_What  we are seeing is this table is storing an outdated version of our gateway (for a particular instance) and the fixed_ips table gets corrupted23:41
comstudVinay_: (So we stash a copy in the nova DB)23:41
comstudIf you're using nova-network, I can only guess somehow the gateway is misconfigured23:42
Vinay_so which process does this stashing of a copy in the db..23:42
Vinay_Yes we are using nova-network23:42
comstudnova-compute tends to do this23:42
Vinay_so we modified the gateway in networks table to some other value23:43
comstudthe code is likely in nova/network/api.py23:43
comstudwhich nova-compute calls23:43
Vinay_then restarted everything23:43
comstudAh ok..23:43
comstudso your instances were built with the old gateway23:43
comstudright?23:43
Vinay_yes23:43
Vinay_instances were built with old gateway , then we changed it23:43
comstudI don't think there's anything that'll fix the cache automatically. because...23:44
comstudbecause your instances were built with the old gateway23:44
comstudSolution is to build new instances with the corrected gateway :)23:44
comstudOr if you've manually fixed things.. you'll have to manually fix the cache too23:44
Vinay_ok how do I manually fix the cache23:45
comstudThere is a periodic task on compute that tries to heal the cache... over time.23:45
*** cdsvrd has quit IRC23:45
comstudVinay_: creating a script to do SQL queries :)23:45
*** PaulM1 has joined #openstack-dev23:45
*** stevemar has joined #openstack-dev23:46
comstudmodify the network_info column23:46
*** PaulM has quit IRC23:46
Vinay_but we restarted the nova-compute process..23:46
Vinay_So even a restart of nova-compute doesnt cause it to take the new values?23:47
comstudyeah, but that doesn't change your instnaces' network info23:47
comstudThey were built with wrong network info23:47
comstudright23:47
*** esheffield has joined #openstack-dev23:47
comstudit should fix NEW builds23:47
comstudnew instances23:47
comstudnot old ones23:47
Vinay_actually right now I am seeing that it corrected itself after a few hours23:47
comstudyeah, ok... that is good.  the periodic healing is working then23:47
comstudSo that's the other solution... just wait a period of time23:48
comstudand it'll auto-correct23:48
Vinay_okay do you know what that period is ?23:48
comstudit's a conf setting23:48
comstudsec23:48
Vinay_or if there is a nova.con setting23:48
Vinay_yeah23:48
comstudit defaults to 1 instance per minute23:49
comstudie, default is 60s... and it only does 1 on each run23:49
comstudyou could try dropping it down to 1, but it's going to create a lot of traffic.23:49
comstudheal_instance_info_cache_interval=123:50
comstudetc23:50
comstudyou could try that temporarily.. until all instances are updated23:50
comstudand then reset it to 6023:50
comstudhow many instances do you have?23:50
Vinay_ok.. I dont have the conf in my nova.conf .. so the default is 60sec ?23:50
comstudyea23:50
*** aelkikhia has left #openstack-dev23:51
Vinay_we have 95 vms23:51
comstudok, so with the default, it'll take 95 minutes to update them all, i guess23:51
comstudif you drop the setting down to 1, it should take 95 seconds23:52
Vinay_but on this particular compute host i have only 123:52
comstudthen you can reset it to 6023:52
comstudah yeah. you have to update all compute nodes23:52
comstudeach compute node updates its own instances23:52
comstudi would think that everything would be updated within 10 minutes if you have like 10 compute nodes.23:53
*** bknudson has joined #openstack-dev23:53
Vinay_so this compute node has 1 instance and my flag is default setting=60s so can I expect it to self-correct after 60 secs.. ????23:53
Vinay_on that host23:53
*** reed has quit IRC23:53
comstudit should , yes23:54
*** pixelbeat_ has quit IRC23:54
comstudfor compute nodes that have 10 instances, it would take 10 minutes to get them all23:54
comstudetc.23:54
Vinay_ye.p .. but we have seen this happen after 2-3 hours !23:54
comstudstrange!23:54
comstudi'm not sure why that would be, then.23:55
*** michchap has quit IRC23:55
Vinay_so we do see the heal happen every minute but it was not getting the right dhcp server address. the gateway address was right though...23:55
Vinay_After 2-3 hours we now see it getting both addresses right23:55
*** michchap has joined #openstack-dev23:56
comstudah23:56
comstudno idea23:56
Vinay_what is the idea behind this heal_instance_info_cache ... can I read it somewhere?23:57
comstudThe idea is... we re-query the network info to update the cache in case it gets out of sync23:57
*** jog0 has quit IRC23:57
comstudperiodically.. so we're not slamming nova-network with a lot of calls23:57
comstudthe code is in: def _heal_instance_info_cache(self, context):23:58
comstudin nova/compute/manager.py23:58
Vinay_ok so the cache is instance_info_cache table.. but where do you read the network info from ?23:58
*** PaulM1 has quit IRC23:58
comstudit asks nova-network for it23:58
comstudand in your case.. nova-network gets it from the DB23:58
comstudin different tables23:59
Vinay_ok cool.. which tables would those be23:59
comstudfor the quantum case, it asks quantum, and quantum gives it23:59
comstudi think it's the virtual_interfaces and fixed_ips tables.23:59
comstudand networks table23:59
Vinay_no quantum .. only nova-network with vlan manager (folsom)23:59
comstudnetworks table contains gateway, and general info for each network23:59
comstudvirtual_interfaces I think contains the instance <-> virtual interface mapping23:59
« Wednesday, 2013-05-08 Index Friday, 2013-05-10 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!