| *** ncoghlan has joined #openstack-keystone | 00:14 | |
| *** r-daneel__ has quit IRC | 00:24 | |
| *** stevemar has joined #openstack-keystone | 00:25 | |
| *** ncoghlan is now known as ncoghlan_afk | 00:27 | |
| *** ncoghlan_afk is now known as ncoghlan | 00:28 | |
| *** openstackgerrit has quit IRC | 00:31 | |
| *** openstackgerrit has joined #openstack-keystone | 00:31 | |
| *** soulxu_ has quit IRC | 00:35 | |
| morganfainberg | dstanek, ping | 00:38 |
|---|---|---|
| morganfainberg | or stevemar | 00:38 |
| stevemar | morganfainberg, yo | 00:39 |
| morganfainberg | ok so the memcache pool | 00:39 |
| dstanek | morganfainberg: pong | 00:39 |
| morganfainberg | what is the case for a maximum timeout? | 00:39 |
| morganfainberg | and how... do we handle that? | 00:39 |
| dstanek | timeout for that wait? | 00:39 |
| morganfainberg | dstanek, yeah | 00:40 |
| dstanek | the safest thing would be to just fail with a 500 - that way there is less of a chance to hammer the DB | 00:41 |
| morganfainberg | well i mean, we have a lot of options here | 00:41 |
| dstanek | it's a highly exceptional condition that probably needs immediate attention | 00:41 |
| morganfainberg | do we log a "oh hey we're blocked" and let it re-attempt? | 00:41 |
| morganfainberg | do we want it to raise up a 500? | 00:41 |
| morganfainberg | do we want to try 5 times then 500? | 00:41 |
| morganfainberg | this is way deep in the internals of dogpile, and we're getting some very strange things happening if we're blocking here. (e.g. all memcache servers are down) | 00:42 |
| dstanek | i guess having a timeout and retries in the config file would give the most flexibility | 00:42 |
| morganfainberg | dstanek, i think the only real case we're hitting here is all memcache servers are un-reachable | 00:43 |
| morganfainberg | dstanek, and this is going to need a string freeze exception | 00:43 |
| dstanek | really? i though this is because we reached the max of the pool and for some reason we're not releasing connections | 00:43 |
| dstanek | *thought* | 00:44 |
| morganfainberg | dstanek, it is, and afaict the only way that occurs is if we're being hammered and hitting socket timeout (3s) for every request | 00:44 |
| morganfainberg | dstanek, this is extremely deep in the internals between dogpile and the memcache lib. | 00:44 |
| morganfainberg | dstanek, i'm not saying we shouldn't have a timeout, i agree, just trying to figure out what that timeout looks like | 00:45 |
| dstanek | i think the simpest thing is to blowup and not do any retries - hmmm...i wonder how nofify works under eventlet | 00:48 |
| *** morgan_remote_ has quit IRC | 00:50 | |
| morganfainberg | dstanek, stupid eventlet. | 00:52 |
| dstanek | a shorter answer is don't use eventlet in produciton | 00:52 |
| morganfainberg | dstanek, afaict notify in eventlet just means that when we swap to execute on the greenthread it doesn't immidiately yeild | 00:53 |
| morganfainberg | dstanek, a .wait() is equivalent to .sleep(0) while condition | 00:53 |
| morganfainberg | and notify sets condition to False | 00:53 |
| openstackgerrit | Steve Martinelli proposed a change to openstack/python-keystoneclient: Handle federated tokens https://review.openstack.org/121146 | 00:54 |
| morganfainberg | dstanek, ++ to "never use eventlet in production" oh wait... everything but keystone wont run then | 00:55 |
| morganfainberg | dstanek, this is what i came up with | 01:03 |
| morganfainberg | dstanek, http://paste.openstack.org/show/112361/ | 01:03 |
| openstackgerrit | Kui Shi proposed a change to openstack/keystone: Add memcached_backend configuration https://review.openstack.org/122037 | 01:11 |
| *** rodrigods_ has quit IRC | 01:13 | |
| morganfainberg | YorikSar, ping you here? | 01:16 |
| dstanek | morganfainberg: so i'm back to wondering about the max connections again | 01:18 |
| morganfainberg | i've changed it like this | 01:18 |
| morganfainberg | dstanek, http://pasteraw.com/2r9bvhlw4e2th6r6yhc91qb97agcr3 | 01:18 |
| dstanek | the docs say that cond.wait() will release the lock and then that would mean multiple greenthreads are waiting. when we run notify_all they will all get notified and not check the count right? | 01:19 |
| morganfainberg | dstanek, that should *really* prevent us yielding out somewhere in the memcache lib and accidently ending up with too many connections | 01:19 |
| dstanek | having trouble making test scenarios | 01:19 |
| morganfainberg | dstanek, aw crap we can't raise out on a timeout | 01:20 |
| morganfainberg | dstanek, you release the lock and re-enter the while loop on the next coroutine switch to that thread | 01:20 |
| dstanek | yeah, i don't know how wait tells you it was a timeout | 01:20 |
| morganfainberg | dstanek, it doesn't | 01:20 |
| morganfainberg | dstanek, it just returns out just like a notify does. | 01:20 |
| morganfainberg | dstanek, so we can't raise out. | 01:20 |
| morganfainberg | we rely on the while loop working indefinitely to block extra connections from being created. | 01:21 |
| morganfainberg | since if another connection grabbed the lock, and pushed us to max connections, the while loop would send us back into the wait state | 01:21 |
| morganfainberg | and remember only one greenthread is running at a time in the eventlet process (worker) | 01:22 |
| dstanek | it feels like we should just be using a real queue here | 01:23 |
| dstanek | is that while loop effectively a busy wait on a loaded server? | 01:24 |
| morganfainberg | dstanek, well, except we still need to wait for the response, so a queue will still run into the same issues, always waiting | 01:25 |
| morganfainberg | dstanek, i think it is a busy wait on a loaded server, but not *always* the same way we're synchronous grabbing data from the db | 01:26 |
| dstanek | morganfainberg: i think you can do it much better with http://eventlet.net/doc/modules/queue.html | 01:26 |
| morganfainberg | this just forces the same types of restrictions | 01:26 |
| morganfainberg | dstanek, except we can't just queue we need to get specific values back and expect a response for that value not "a" value | 01:27 |
| *** diegows has quit IRC | 01:27 | |
| morganfainberg | this isn't straight FIFO, because each request is tied to the specific greenthread | 01:27 |
| morganfainberg | and internal coroutine | 01:27 |
| morganfainberg | or you mean to use the queue for the connections instead of a list? | 01:28 |
| dstanek | what you would do is make free pool a queue - you'd still have to manage how many things are out of the queue (because we want to kill older connections) - and use the context manager approach to pop a connection and push it back | 01:29 |
| morganfainberg | dstanek, i think we end up with the same mechanism really. | 01:29 |
| dstanek | morganfainberg: very similar except that a timeout on a wait is sane and i think there are other small benefits - i can whip un an example in a minute or two | 01:31 |
| morganfainberg | dstanek, we still need to manage the max number of connections outstanding with a while loop though | 01:31 |
| morganfainberg | dstanek, since the queue doesn't know about connections not in it | 01:32 |
| morganfainberg | dstanek, so the wait/timeout would net us the same issue as the busy wait | 01:32 |
| dstanek | yeah, i was thinking that i would use two queues to eliminate the problem, but that could cause locking issues | 01:34 |
| morganfainberg | dstanek, yeah. :( | 01:34 |
| morganfainberg | we're doing the inverse of a queue, we're tracking a maximum number of items and tossing the items not in use on a queue (list, whatever) | 01:35 |
| morganfainberg | if an active "thing" was being placed on a queue to be acted on, it would be different | 01:35 |
| morganfainberg | i guess we could do a runner queue and a result queue, but you're likely filtering every value in the result queue to find yours when you wakeup | 01:36 |
| morganfainberg | i think we're trying to effecitvely re-implement AMPQ here | 01:37 |
| morganfainberg | get me value for X, i'll be waiting and listening on Y for a result. | 01:37 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/keystone: Add a pool of memcached clients https://review.openstack.org/119452 | 01:39 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/keystone: Add a pool of memcached clients https://review.openstack.org/119452 | 01:40 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/keystone: Add a pool of memcached clients https://review.openstack.org/119452 | 01:41 |
| morganfainberg | somehow "im" ended at the end of one of the ASL license headers | 01:41 |
| morganfainberg | dstanek, ok that should address all but your comment about the timeout. | 01:41 |
| morganfainberg | still needs testing though | 01:42 |
| *** ncoghlan is now known as ncoghlan_afk | 01:44 | |
| *** amerine has quit IRC | 01:48 | |
| *** andreaf has quit IRC | 01:48 | |
| *** kevinbenton has quit IRC | 01:48 | |
| *** nonameentername has quit IRC | 01:48 | |
| *** cyeoh has quit IRC | 01:48 | |
| *** mitz has quit IRC | 01:48 | |
| *** zhiyan has quit IRC | 01:48 | |
| *** xianghui has quit IRC | 01:48 | |
| *** amakarov_away has quit IRC | 01:48 | |
| *** csd has quit IRC | 01:48 | |
| *** jamielennox|away has quit IRC | 01:48 | |
| *** lsmola______ has quit IRC | 01:48 | |
| *** rushiagr_away has quit IRC | 01:48 | |
| openstackgerrit | A change was merged to openstack/python-keystoneclient: Versioned Endpoint hack for Sessions https://review.openstack.org/90632 | 01:51 |
| *** dims has joined #openstack-keystone | 01:53 | |
| *** diegows has joined #openstack-keystone | 01:53 | |
| *** amerine has joined #openstack-keystone | 01:53 | |
| *** andreaf has joined #openstack-keystone | 01:53 | |
| *** kevinbenton has joined #openstack-keystone | 01:53 | |
| *** nonameentername has joined #openstack-keystone | 01:54 | |
| *** cyeoh has joined #openstack-keystone | 01:54 | |
| *** mitz has joined #openstack-keystone | 01:54 | |
| *** zhiyan has joined #openstack-keystone | 01:54 | |
| *** xianghui has joined #openstack-keystone | 01:54 | |
| *** amakarov_away has joined #openstack-keystone | 01:54 | |
| *** csd has joined #openstack-keystone | 01:54 | |
| *** jamielennox|away has joined #openstack-keystone | 01:54 | |
| *** lsmola______ has joined #openstack-keystone | 01:54 | |
| *** rushiagr_away has joined #openstack-keystone | 01:54 | |
| *** mitz has quit IRC | 01:54 | |
| *** mitz has joined #openstack-keystone | 01:55 | |
| *** dims is now known as Guest9788 | 01:55 | |
| *** alex_xu has joined #openstack-keystone | 01:56 | |
| *** jraim has quit IRC | 01:58 | |
| *** amerine_ has joined #openstack-keystone | 01:58 | |
| *** diegows has quit IRC | 01:59 | |
| *** amerine has quit IRC | 02:00 | |
| *** jraim has joined #openstack-keystone | 02:02 | |
| *** marcoemorais has quit IRC | 02:05 | |
| *** amerine has joined #openstack-keystone | 02:07 | |
| *** amerine_ has quit IRC | 02:11 | |
| *** wanghong has quit IRC | 02:15 | |
| *** lbragstad1 has joined #openstack-keystone | 02:15 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/keystone: Add a pool of memcached clients https://review.openstack.org/119452 | 02:15 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/keystone: Add a pool of memcached clients https://review.openstack.org/119452 | 02:18 |
| morganfainberg | dstanek, some basic tests. but still not sure how to test the concurrency stuff | 02:19 |
| dstanek | is the requirement to trim down the number of connections worth it? | 02:22 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/keystone: Add a pool of memcached clients https://review.openstack.org/119452 | 02:24 |
| morganfainberg | dstanek, the point is valid that if you have 5x keystones you may not want to consume 100 FDs/Sockets to memcached at low traffic times. | 02:26 |
| morganfainberg | dstanek, but i dunno. | 02:26 |
| morganfainberg | dstanek, are we going to have a lot of benefit there? not in keystone. though i could see a bigger win in say auth_token | 02:26 |
| morganfainberg | dstanek, some endpoints may have lower traffic. | 02:27 |
| morganfainberg | and this does need to be ported over to auth_token as well. | 02:27 |
| dstanek | i dunno. if you can have up to 100 by the configuration then you have to be prepared to have 100 at any time | 02:27 |
| morganfainberg | dstanek, right, but if you *can* have 100 from every endpoint, should you legitimately always expect to have 100 from every endpoint? | 02:28 |
| morganfainberg | in keystone it's probably a moot point. we will likely always be at the capacity | 02:28 |
| dstanek | in any event the operator will have to size the hardware/infrastructure to support that number along with all of the other expected traffic | 02:29 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 02:31 | |
| *** ncoghlan_afk is now known as ncoghlan | 02:31 | |
| morganfainberg | dstanek, unfortunately the cleanup bits are not the complex parts of this pool | 02:32 |
| *** harlowja is now known as harlowja_away | 02:32 | |
| morganfainberg | the base concurrency support is. | 02:32 |
| dstanek | they actually are because the make the rest of the impl suck | 02:33 |
| *** richm has joined #openstack-keystone | 02:33 | |
| morganfainberg | dstanek, not really, the whole impl sucks. not because YorikSar did a bad job, but because we're working around a major short coming of combining eventlet plus thread.local | 02:33 |
| morganfainberg | dstanek, the cleanup bit really is a single function call in release now. removing that part doesn't change the landscape that much | 02:34 |
| morganfainberg | we'd still have acquire count tracking, still need to use the context manager, etc. | 02:35 |
| dstanek | morganfainberg: this is a very simple pool similar to ones i have used in the past - i just can't implement _trim without creating my own queue impl | 02:38 |
| dstanek | morganfainberg: does mysql manage it's own connection pool? | 02:38 |
| morganfainberg | dstanek, sqla does some work on that i think, but mysqldb is swig, so C bindings meaning we don't get eventlet issues | 02:39 |
| morganfainberg | dstanek, any call to mysql is blocking via mysqldb | 02:39 |
| dstanek | hmm...single connection? | 02:40 |
| morganfainberg | dstanek, i think we can have a pool, but it wouldn't benefit us a lot | 02:40 |
| morganfainberg | that is the idea behind moving to anothe rmysql connector (mysql-connector?) or running multiple workers (eventlet multi-worker mode, or mod_wsgi) | 02:40 |
| *** ncoghlan is now known as ncoghlan_afk | 02:41 | |
| dstanek | i've actually tried a pure python driver and it wasn't noticably faster when using rally | 02:41 |
| *** ncoghlan_afk is now known as ncoghlan | 02:41 | |
| morganfainberg | dstanek, it's not, the pure python connector itself is way way slower, so the benefits form eventlet we get are probably lost | 02:42 |
| morganfainberg | oh unrelated, i'll be here tomorrow. | 02:42 |
| morganfainberg | thursday is the day i'll be spotty | 02:42 |
| morganfainberg | misread the date on the appointment | 02:43 |
| dstanek | the profiling of both didn't show a significant differnence in driver performance - i think we are just doing some silly stuff | 02:43 |
| morganfainberg | dstanek, oh i know we are. | 02:44 |
| morganfainberg | dstanek, that was part of the idea behind: https://review.openstack.org/#/c/103304/ | 02:44 |
| morganfainberg | as a starting place | 02:44 |
| *** achampion has quit IRC | 02:49 | |
| *** achampion has joined #openstack-keystone | 02:51 | |
| *** ncoghlan is now known as ncoghlan_afk | 02:59 | |
| *** ayoung has joined #openstack-keystone | 03:06 | |
| *** ncoghlan_afk is now known as ncoghlan | 03:08 | |
| *** Guest9788 has quit IRC | 03:12 | |
| openstackgerrit | Bob Thyne proposed a change to openstack/keystone: Update Endpoint Filter API https://review.openstack.org/122046 | 03:15 |
| openstackgerrit | Bob Thyne proposed a change to openstack/keystone: Update Endpoint Filter API https://review.openstack.org/122046 | 03:15 |
| *** richm has quit IRC | 03:16 | |
| *** gokrokve has quit IRC | 03:17 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/keystone: Add a pool of memcached clients https://review.openstack.org/119452 | 03:18 |
| morganfainberg | ^ fixing doc bug | 03:18 |
| *** amerine_ has joined #openstack-keystone | 03:21 | |
| *** amerine has quit IRC | 03:23 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 03:24 | |
| stevemar | morganfainberg, thx, i was wondering about that | 03:28 |
| openstackgerrit | Bob Thyne proposed a change to openstack/identity-api: Update Endpoint Filter API https://review.openstack.org/122048 | 03:29 |
| morganfainberg | stevemar, also sorry for being > 400 lines of change :( | 03:29 |
| morganfainberg | stevemar, this fix is already kindof unwieldy | 03:29 |
| stevemar | morganfainberg, tis a bit | 03:30 |
| *** alex_xu has quit IRC | 03:31 | |
| *** gordc has joined #openstack-keystone | 03:32 | |
| *** ayoung has quit IRC | 03:37 | |
| *** alex_xu has joined #openstack-keystone | 03:44 | |
| *** vhoward has joined #openstack-keystone | 03:46 | |
| *** ayoung has joined #openstack-keystone | 03:50 | |
| *** achampio1 has joined #openstack-keystone | 04:01 | |
| *** achampion has quit IRC | 04:02 | |
| *** nonameentername has quit IRC | 04:03 | |
| *** wanghong has joined #openstack-keystone | 04:11 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/keystonemiddleware: Add composite authentication support https://review.openstack.org/108384 | 04:18 |
| morganfainberg | This was a nasty rebase, please look carefully at the code when reviwing ^ | 04:20 |
| *** nonameentername has joined #openstack-keystone | 04:20 | |
| stevemar | morganfainberg, i *just* rebased it | 04:23 |
| morganfainberg | stevemar, beat ya to submitting it | 04:23 |
| stevemar | haven't uploaded the changes yet, but ya nasty rebase | 04:23 |
| morganfainberg | stevemar, it wasn't a pretty rebase :( | 04:23 |
| stevemar | i'll see if we got the same result :) | 04:23 |
| morganfainberg | stevemar, ++ | 04:23 |
| morganfainberg | stevemar, 5 open bugs till RC | 04:25 |
| morganfainberg | none not inprogress | 04:25 |
| stevemar | morganfainberg, yeah, in the __call__ method, i think you caught them all | 04:25 |
| stevemar | move remove_auth out of try, add the log debugs, and take the returns out of the try's | 04:25 |
| morganfainberg | the remove_auth got left in the try for mine. *doh* | 04:26 |
| morganfainberg | but the returns were caught plus the change to _call_app from _app | 04:26 |
| morganfainberg | and the addition of the reject_line where needed | 04:26 |
| *** gokrokve has joined #openstack-keystone | 04:27 | |
| *** gordc has quit IRC | 04:28 | |
| stevemar | morganfainberg, i think this one is super close https://review.openstack.org/#/c/121146/ i addressed brant's comments | 04:30 |
| stevemar | and it had some +2 love from lbragstad | 04:30 |
| stevemar | and from jamielennox | 04:30 |
| morganfainberg | stevemar, yeah circling through those now. | 04:30 |
| stevemar | the trust['enabled'] one needs a test :( but i'm not sure where to add it | 04:31 |
| *** gokrokve has quit IRC | 04:31 | |
| stevemar | morganfainberg, do you have write access to https://gist.github.com/dolph/651c6a1748f69637abd0 or something? | 04:32 |
| morganfainberg | stevemar, if I star a review it is added | 04:32 |
| morganfainberg | stevemar, same with lbragstad and dstanek | 04:32 |
| stevemar | ohh | 04:32 |
| morganfainberg | stevemar, dolph did magic :) | 04:33 |
| stevemar | fancy | 04:33 |
| stevemar | i was wondering how it was changed to (approved) so quickly | 04:33 |
| stevemar | fyi, might want to add bobt's stuff | 04:34 |
| stevemar | https://review.openstack.org/#/c/122046/2/keystone/contrib/endpoint_filter/routers.py | 04:34 |
| stevemar | well - https://review.openstack.org/#/c/122046/ | 04:34 |
| morganfainberg | stevemar, +2 on that federated tokens one | 04:34 |
| stevemar | and https://review.openstack.org/#/c/122048 | 04:34 |
| stevemar | yee haw | 04:34 |
| stevemar | that'll make marek happy :) | 04:34 |
| morganfainberg | added bob's reviews | 04:34 |
| morganfainberg | identity-api isn't really included in release blockers | 04:35 |
| morganfainberg | stevemar, btw, https://review.openstack.org/#/q/starredby:mdrnstm+is:open,n,z | 04:35 |
| morganfainberg | that is the list of reviews i've actively starred | 04:35 |
| morganfainberg | stevemar, i can almost taste RC ;) | 04:37 |
| stevemar | i have the same ones open in a bunch of tabs | 04:37 |
| stevemar | indeed | 04:37 |
| *** achampio1 has quit IRC | 04:41 | |
| *** achampion has joined #openstack-keystone | 04:44 | |
| *** russo3999 has quit IRC | 04:50 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 04:51 | |
| *** ajayaa has joined #openstack-keystone | 05:17 | |
| morganfainberg | stevemar, ugh endpoint filter extension may violate HTTP | 05:28 |
| morganfainberg | stevemar, the HEAD/GET issue again | 05:29 |
| stevemar | o/ | 05:29 |
| stevemar | you mean the old stuff, not the new endpoint grouping? | 05:29 |
| morganfainberg | i'm too tired to verify it tonight but remind me tomorrow (before we accept bob's patches) so we know if we need to open another bug on it | 05:29 |
| morganfainberg | the identity-api is claiming 204 on HEAD request | 05:29 |
| morganfainberg | and looking at the code it has separate methods for get/head in cases | 05:30 |
| morganfainberg | so.. | 05:30 |
| morganfainberg | i thin | 05:30 |
| * morganfainberg is like i said too tired | 05:30 | |
| morganfainberg | if we need to fix it, we'll just make it a followup to bob's patches | 05:30 |
| stevemar | its probably the old ep filter stuff | 05:36 |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 05:40 | |
| YorikSar | morganfainberg: pong | 05:49 |
| YorikSar | morganfainberg: I guess it's too late :) | 05:49 |
| *** stevemar has quit IRC | 05:50 | |
| *** ukalifon has joined #openstack-keystone | 05:57 | |
| *** YorikSar has quit IRC | 06:00 | |
| *** YorikSar has joined #openstack-keystone | 06:02 | |
| openstackgerrit | Bob Thyne proposed a change to openstack/keystone: Update Endpoint Filter API https://review.openstack.org/122046 | 06:10 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/keystone: Imported Translations from Transifex https://review.openstack.org/120695 | 06:11 |
| *** rushiagr_away is now known as rushiagr | 06:12 | |
| *** henrynash has joined #openstack-keystone | 06:12 | |
| *** k4n0 has joined #openstack-keystone | 06:26 | |
| *** andreaf has quit IRC | 06:29 | |
| openstackgerrit | henry-nash proposed a change to openstack/keystone: Ensure identity sql driver supports domain-specific configuration. https://review.openstack.org/121246 | 06:35 |
| *** henrynash has quit IRC | 06:36 | |
| *** BAKfr has joined #openstack-keystone | 07:06 | |
| openstackgerrit | Marcos Fermín Lobo proposed a change to openstack/keystone: Implement group related methods for LDAP backend https://review.openstack.org/102244 | 07:13 |
| openstackgerrit | A change was merged to openstack/python-keystoneclient: Allow retrying some failed requests https://review.openstack.org/118004 | 07:31 |
| openstackgerrit | Qin Zhao proposed a change to openstack/python-keystoneclient: Fix the condition expression for ssl_insecure https://review.openstack.org/112232 | 07:46 |
| *** GARNAV has joined #openstack-keystone | 08:00 | |
| *** henrynash has joined #openstack-keystone | 08:01 | |
| *** bdossant has joined #openstack-keystone | 08:03 | |
| *** lsmola______ is now known as lsmola | 08:07 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 08:07 | |
| *** ncoghlan has quit IRC | 08:15 | |
| *** ajayaa has quit IRC | 08:18 | |
| *** amerine_ has quit IRC | 08:26 | |
| *** amerine has joined #openstack-keystone | 08:27 | |
| *** alex_xu has quit IRC | 08:31 | |
| *** achampion has quit IRC | 08:32 | |
| *** alex_xu has joined #openstack-keystone | 08:39 | |
| *** rushiagr is now known as rushiagr_away | 08:42 | |
| *** rushiagr_away is now known as rushiagr | 08:43 | |
| *** henrynash has quit IRC | 08:52 | |
| *** andreaf_ is now known as andreaf | 09:01 | |
| *** ajayaa has joined #openstack-keystone | 09:07 | |
| *** amakarov_away is now known as amakarov | 09:22 | |
| *** jaosorior has joined #openstack-keystone | 09:26 | |
| *** aix has joined #openstack-keystone | 09:57 | |
| openstackgerrit | Qin Zhao proposed a change to openstack/python-keystoneclient: Fix the condition expression for ssl_insecure https://review.openstack.org/112232 | 10:17 |
| *** dguerri has joined #openstack-keystone | 10:29 | |
| *** ajayaa has quit IRC | 10:30 | |
| *** henrynash has joined #openstack-keystone | 10:31 | |
| *** rushiagr is now known as rushiagr_away | 10:35 | |
| *** dims has joined #openstack-keystone | 10:37 | |
| *** dims has quit IRC | 10:39 | |
| *** dims has joined #openstack-keystone | 10:39 | |
| *** rushiagr_away is now known as rushiagr | 10:39 | |
| dguerri | hello there :) got a quick question for you. | 10:39 |
| dguerri | I am using keystone with AD as a backend for identification. | 10:39 |
| dguerri | I need to keep assignment local, so I set the [assignment] section accordingly. | 10:39 |
| dguerri | Now, I would like to assign a default role for a specific tenant to all users. Is that possible? or do I need to authorize each user? | 10:39 |
| *** ajayaa has joined #openstack-keystone | 10:42 | |
| *** YorikSar_ has joined #openstack-keystone | 10:44 | |
| *** YorikSar has quit IRC | 10:48 | |
| *** alex_xu has quit IRC | 10:50 | |
| *** henrynash has quit IRC | 11:22 | |
| *** gordc has joined #openstack-keystone | 11:38 | |
| *** stevemar has joined #openstack-keystone | 11:44 | |
| *** gordc has quit IRC | 11:54 | |
| *** stevemar has quit IRC | 11:56 | |
| *** stevemar has joined #openstack-keystone | 11:57 | |
| *** henrynash has joined #openstack-keystone | 12:04 | |
| *** stevemar has quit IRC | 12:05 | |
| *** k4n0 has quit IRC | 12:14 | |
| *** richm has joined #openstack-keystone | 12:20 | |
| *** diegows has joined #openstack-keystone | 12:28 | |
| marekd | just a question about gerrit tests - why some testsuites are named gate-* and some of them check-* ? | 12:35 |
| *** Tahmina has joined #openstack-keystone | 12:45 | |
| *** dims has quit IRC | 12:45 | |
| *** gordc has joined #openstack-keystone | 12:45 | |
| *** dims has joined #openstack-keystone | 12:45 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 12:53 | |
| *** aix has quit IRC | 13:00 | |
| rodrigods | marekd, I think that I've read somewhere that "check" are the tests run before the patch is approved and "gate" afterworlds | 13:01 |
| rodrigods | infra tests | 13:01 |
| marekd | rodrigods: i see | 13:02 |
| marekd | thanks. | 13:02 |
| ayoung | marekd, which version of AD supports the SAML Federation portal? What did you test against? | 13:02 |
| marekd | ADFS2.0 | 13:03 |
| marekd | ayoung: need some more specific version? | 13:03 |
| marekd | ayoung: i may ask people who run it. | 13:03 |
| *** nkinder has quit IRC | 13:12 | |
| ayoung | marekd, do you know if that works with 2008 or if it needs 2012? | 13:15 |
| ayoung | judging by the number of posts that say 2011 on them, I'm going to guess that 2008 is sufficient | 13:16 |
| *** joesavak has joined #openstack-keystone | 13:21 | |
| *** stevemar has joined #openstack-keystone | 13:25 | |
| *** ayoung has quit IRC | 13:26 | |
| *** aix has joined #openstack-keystone | 13:27 | |
| *** ayoung has joined #openstack-keystone | 13:36 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 13:39 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:39 | |
| *** bknudson has joined #openstack-keystone | 13:43 | |
| *** achampion has joined #openstack-keystone | 13:44 | |
| marekd | ayoung: let me ask. | 13:46 |
| ayoung | marekd, thanks....the AD guy for our team is in transit today anyway, but I'd like to give your code a test run. | 13:46 |
| marekd | ayoung: it'd be awesome. | 13:50 |
| ayoung | marekd, I'll let you know | 13:50 |
| marekd | ayoung: my guy is on the phone so I will have an answer in few minutes, but I'd say it's 2012 | 13:51 |
| ayoung | marekd, then we'll need a way to scare up a copy for OpenStack testing. I'll ask the Prime Minister... | 13:51 |
| *** topol has joined #openstack-keystone | 13:55 | |
| marekd | It's ADFS2.1 on w2012. Same as adfs2.0 on w2008. | 13:58 |
| ayoung | marekd, so you think we can test against thew 2008 version. Good to know. I just want to minimize the amount of hunting we have to do | 13:59 |
| marekd | ayoung: i can rerun the code again on my adfs | 13:59 |
| marekd | and make sure it works | 13:59 |
| ayoung | marekd, always a good idea | 13:59 |
| ayoung | marekd, but not if its going to take away from other things. I suspect I'll be bothering you once we have a setup to test against | 14:00 |
| marekd | ayoung: sure thing. | 14:01 |
| marekd | i was fearing i'd be the only person who actually ran the code with a adfs . | 14:01 |
| *** GARNAV has quit IRC | 14:01 | |
| *** nkinder has joined #openstack-keystone | 14:02 | |
| marekd | ayoung: i also made a separate piece of code for cli adfs authentication. Needs some polishing, but it does its core job - authenticates with ADFS: https://github.com/zaccone/pyadfsclient | 14:02 |
| ayoung | marekd, looking | 14:03 |
| marekd | this is basically what's already in keystoneclient. | 14:03 |
| *** radez_g0n3 is now known as radez | 14:03 | |
| ayoung | marekd, you really need to start doing some shortening of those constants | 14:04 |
| ayoung | {http://www.w3.org/2005/08/addressing} | 14:04 |
| marekd | that's lxmls philosphy.... | 14:04 |
| marekd | but you are right | 14:04 |
| ayoung | marekd, BTW, did you seem my mailing list message about CORS, Horizon etc? Its specifically targetting the SAML use case. Does it make sense to you? | 14:14 |
| marekd | ayoung: i did. and yes, the idea with JS makes lots of sense to me. | 14:17 |
| marekd | stevemar: LOL: "It is interesting that this latter comment came from the | 14:17 |
| marekd | academic/science world, whereas the supportive one came from the | 14:17 |
| marekd | business world | 14:17 |
| marekd | :-) | 14:17 |
| ayoung | marekd, Ah, mail client just caught up | 14:17 |
| marekd | " | 14:17 |
| ayoung | marekd, just responded to that point. Short answer some Idps are public, the rest private | 14:18 |
| marekd | ayoung: ++ | 14:18 |
| ayoung | marekd, did I show you my proof of concept? | 14:18 |
| stevemar | marekd, i thought that was very funny! | 14:18 |
| marekd | stevemar: it was :-) | 14:19 |
| marekd | ayoung: JS code? yes. | 14:19 |
| ayoung | stevemar, same question; did I show you guys my proof of concept? | 14:19 |
| ayoung | marekd, good | 14:19 |
| marekd | ayoung: I saw it few weeks back but didn't bookmark the URL. Could you paste it again? Or, even better do you have some repo with the code? | 14:20 |
| ayoung | marekd, I can do both | 14:20 |
| ayoung | first the repo | 14:20 |
| ayoung | https://github.com/admiyo/keystone-cops | 14:21 |
| ayoung | marekd, the code is running at | 14:21 |
| stevemar | ayoung, i saw it a few weeks ago, not recently | 14:21 |
| ayoung | https://keystone.younglogic.net/keystone/cops/# | 14:21 |
| marekd | ayoung: thanks! | 14:22 |
| ayoung | Good. I've started moving it over to Angular, and focused a bit more on trusts, but the concepts are the same. I'd love to get a SAML example up there as well; I'll have to talk to simo about getting his SAML provider installed | 14:22 |
| *** david-lyle has joined #openstack-keystone | 14:22 | |
| ayoung | marekd, the older version, the straight JQuery is at the same url but + old.html | 14:23 |
| ayoung | that has the role management stuff | 14:23 |
| *** bambam1 has joined #openstack-keystone | 14:24 | |
| stevemar | ayoung, now do you trust us with a set of credentials :) | 14:24 |
| marekd | stevemar: :D | 14:24 |
| ayoung | stevemar, of course | 14:25 |
| ayoung | stevemar, something is making my machines connection to that crawl. I think it is here on my end, as I've been having network issues on and off all morning | 14:27 |
| *** mflobo has joined #openstack-keystone | 14:27 | |
| *** ajayaa has quit IRC | 14:27 | |
| ayoung | stevemar, ah, I never gave either of you guys accounts in the LDAP server backing it...hold on | 14:29 |
| ayoung | https://ipa.cloudlab.freeipa.org/ipa/ui/ | 14:31 |
| ayoung | stevemar, marekd hit the IPA server and change your passwords, assuming it responds....My IPA webui is still hanging. | 14:33 |
| *** jaosorior has quit IRC | 14:36 | |
| *** zhiyan has quit IRC | 14:36 | |
| *** serverascode has quit IRC | 14:36 | |
| *** ctracey_ has quit IRC | 14:36 | |
| marekd | ayoung: in a minute. wanted to finish something. | 14:37 |
| *** topol has quit IRC | 14:44 | |
| *** richm has quit IRC | 14:44 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 14:47 | |
| *** zhiyan has joined #openstack-keystone | 14:48 | |
| ayoung | stevemar, so, in the Federated workflow, at what point should I be handing a SAML document to Keystone? In the call to get an unscoped Federated token? | 14:49 |
| *** jaosorior has joined #openstack-keystone | 14:50 | |
| stevemar | ayoung, yep, it should be handed to the the idp/protocol specific protected URL | 14:50 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:50 | |
| marekd | ayoung: yes. | 14:50 |
| ayoung | marekd, stevemar, is there a "prep" call to Keystone that provides some way of keeping the SAML assertion from being a bearer token? | 14:51 |
| ayoung | also, where does the SAML assertion go in the request? | 14:51 |
| *** ctracey_ has joined #openstack-keystone | 14:52 | |
| marekd | ayoung: you are asking about websso or ecp? | 14:52 |
| ayoung | marekd, hmmm, good question. I guess this would be websso | 14:52 |
| ayoung | ecp should be similary, but I'm trying to understand the horizon use case first | 14:53 |
| ayoung | marekd, is it like this: | 14:53 |
| *** serverascode has joined #openstack-keystone | 14:53 | |
| ayoung | post to OS-FEDERATION/identity_providers/{idp}/prtocols/saml2 and get a redirect... | 14:54 |
| marekd | ayoung: ecp eorkflow is described in docstring in ecp impl. in keystoneclient. I tried to make is verbose so ppl can understand what's going on: https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/contrib/auth/v3/saml2.py#L318 | 14:54 |
| ayoung | and then when the redirect returns, we would have the SAML assertion handed to keystone, and keystone would have handed back a token? | 14:54 |
| marekd | ayoung: yes. you can try this thing with your browser even today. | 14:55 |
| *** Tahmina has quit IRC | 14:55 | |
| marekd | browser will print XMLized unscoped token. | 14:55 |
| marekd | ayoung: and you don't POST, you GET OS-FEDERATION/identity_providers/{idp}/prtocols/saml2 | 14:56 |
| ayoung | marekd, I see that it is POST/GET in our docs https://github.com/openstack/identity-api/blob/master/v3/src/markdown/identity-api-v3-os-federation-ext.md#request-an-unscoped-os-federation-token-getpost-os-federationidentity_providersidentity_providerprotocolsprotocolauth | 14:56 |
| marekd | ayoung: err, should rather be GET. | 14:57 |
| marekd | at least in our case. | 14:57 |
| marekd | but anyway | 14:58 |
| marekd | you want to send some data with this call? | 14:58 |
| ayoung | marekd, I wrote up the flow in the email as I understand it. I'd be interested to see how ksiu/dchadwick made it work for the Horizon case already, but I suspect it was by Horizon just playing man in the middle | 15:02 |
| ayoung | marekd, for the Javascript apporach GET vs POST is not a problem as we can support either | 15:02 |
| marekd | ayoung: yes, i was simply wonderin if you *need* to send some data (hence use POST) with this call... | 15:02 |
| marekd | ayoung: reading the thread post. | 15:03 |
| ayoung | Nope, just picked the first off the list | 15:03 |
| marekd | ayoung: ok. | 15:03 |
| *** ukalifon has quit IRC | 15:09 | |
| marekd | ayoung: i am responding to the e-mail. | 15:10 |
| ayoung | cool | 15:10 |
| lbragstad | bknudson: around? | 15:16 |
| morganfainberg | YorikSar_, I've added what tests I can come up with for the pool. if you have any ideas on further testing, it would (of course) be appreciated. But all in all, I'm fairly happy with the pool at this point. | 15:17 |
| Daviey | ayoung: Whilst talking about auth... did anyone work on adding requests_kerberos support into keystoneclient? | 15:19 |
| ayoung | Daviey, yes, it is going into a separate repo | 15:20 |
| ayoung | Daviey, https://review.openstack.org/#/c/120310/ | 15:20 |
| *** cjellick has joined #openstack-keystone | 15:21 | |
| morganfainberg | ayoung, for the enumerate unscoped tokens let me know when you have a sec, want to check with you on the domain stuff, because the rest of the code looked pretty good. | 15:23 |
| ayoung | morganfainberg, you mean "we should also enumerate domains with unscoped tokens" yes, but it is less pressing, as it is a different use case | 15:24 |
| Daviey | ayoung: Thanks! | 15:24 |
| morganfainberg | ayoung, no | 15:25 |
| ayoung | morganfainberg, as you need to know your domain a-priori in order to log in | 15:25 |
| morganfainberg | ayoung, i think you dropped support for filtering on domain completly | 15:25 |
| ayoung | looking | 15:25 |
| morganfainberg | ayoung, unless you pass domain_id. | 15:25 |
| *** topol has joined #openstack-keystone | 15:25 | |
| ayoung | morganfainberg, OK, I think I have dropped it. It was unintentional. I'll fix | 15:26 |
| morganfainberg | ayoung, ++ other than that i think that looks good | 15:26 |
| ayoung | thanks | 15:26 |
| ayoung | good catch, too. It is clear now that you pointed it out, but I missed it in previous passes | 15:27 |
| *** aix has quit IRC | 15:28 | |
| *** rushiagr is now known as rushiagr_away | 15:29 | |
| Daviey | ayoung: Hmm, the external repo hasn't been created yet - right? | 15:32 |
| *** dhellmann is now known as dhellmann_ | 15:32 | |
| ayoung | Daviey, yes, but the code is not yet submitted. I have it in my github, though | 15:33 |
| Daviey | ayoung: Ok, thanks | 15:33 |
| ayoung | Daviey, and the original patch is still posted for review in python-keystoneclient | 15:33 |
| ayoung | links in a second | 15:33 |
| Daviey | ayoung: Jose's patchset? | 15:35 |
| ayoung | Daviey, yes | 15:35 |
| ayoung | Daviey, I've reworked it slightly | 15:36 |
| ayoung | Daviey, https://github.com/admiyo/python-keystoneclient-kerberos | 15:36 |
| bknudson | lbragstad: I'm around now. | 15:36 |
| Daviey | ayoung: ok, i'll give it a go. Thanks | 15:37 |
| ayoung | Daviey, cool. I'm working on integrating it into Horizon: Django-Openstack-Auth to be exact | 15:38 |
| lbragstad | bknudson: hey, so I had a quick question on your review | 15:38 |
| lbragstad | I played around with passing in a parameter to a decorated function, | 15:38 |
| bknudson | to the decorator? | 15:38 |
| lbragstad | but it never reaches _check_grant_protection | 15:38 |
| lbragstad | bknudson: the kwargs are set by https://github.com/openstack/keystone/blob/ae22900713ba902c116d296f1c237ce1f5092945/keystone/assignment/routers.py#L102-L104 | 15:39 |
| Daviey | ayoung: Yeah, been following your threads on Horizon | 15:39 |
| *** aix has joined #openstack-keystone | 15:40 | |
| ayoung | Daviey, cool. You understand the S4U2Setup necessary to make it work? | 15:42 |
| bknudson | lbragstad: _check_grant_protection is referenced in @controller.protected(callback=_check_grant_protection) | 15:48 |
| bknudson | lbragstad: if you want to pass an extra parameter to it, you can use functools.partial | 15:48 |
| bknudson | e.g., @controller.protected(callback=functools.partial(_check_grant_protection, allow_no_user=True) | 15:49 |
| bknudson | ) | 15:49 |
| *** cjellick has quit IRC | 15:50 | |
| *** cjellick has joined #openstack-keystone | 15:54 | |
| *** diegows has quit IRC | 15:55 | |
| * lbragstad thanks bknudson | 16:03 | |
| *** wwriverrat has joined #openstack-keystone | 16:04 | |
| *** jsavak has joined #openstack-keystone | 16:04 | |
| *** joesavak has quit IRC | 16:04 | |
| *** rushiagr_away is now known as rushiagr | 16:05 | |
| *** radez is now known as radez_g0n3 | 16:08 | |
| marekd | i cannot even resolve this hostname ipa.cloudlab.freeipa.org :( | 16:09 |
| marekd | ayoung: ^^ | 16:09 |
| marekd | ayoung: are you sure it works from the internet ? | 16:09 |
| ayoung | marekd, heh, did I send you the wrong one? | 16:09 |
| ayoung | it should be | 16:09 |
| ayoung | ipa.younglogic.net | 16:09 |
| bknudson | lbragstad: another option is to have a new function, _check_grant_protection_delete() and use that instead. | 16:09 |
| ayoung | that might be the problem...let me check | 16:09 |
| ayoung | Yep...that was my problem too, sorry marekd | 16:10 |
| marekd | ayoung: no worries. | 16:10 |
| lbragstad | bknudson: I got it working with _check_grant_protection | 16:10 |
| ayoung | the cloudlab one is my internal dev system | 16:10 |
| marekd | ayoung: that's what i thought. | 16:10 |
| ayoung | marekd, actually it is arelief | 16:10 |
| lbragstad | bknudson: thanks for the functools.partial suggestion | 16:10 |
| *** zzzeek has joined #openstack-keystone | 16:10 | |
| bknudson | lbragstad: maybe @controller.protected() could accept more kwargs and pass them on to the callback function. | 16:11 |
| bknudson | but that seems like overkill when you can just do functools.partial. | 16:11 |
| lbragstad | bknudson: right | 16:11 |
| *** dhellmann_ is now known as dhellmann | 16:11 | |
| lbragstad | we don't ever pass in additional parameters | 16:11 |
| lbragstad | no other call does that | 16:11 |
| ayoung | marekd, at least I created the accounts in the right system | 16:11 |
| lbragstad | at the very most, they pass in a callback, | 16:12 |
| marekd | ayoung: lol, did you set some localization options (language etc?)? | 16:12 |
| lbragstad | so this seems to be the unique case | 16:12 |
| ayoung | marekd, yes I did | 16:12 |
| openstackgerrit | Lance Bragstad proposed a change to openstack/keystone: Allow users to clean up role assignments. https://review.openstack.org/119843 | 16:12 |
| marekd | ayoung: ah,ok. | 16:12 |
| ayoung | marekd, specifically, it uses the locale specified by your browser | 16:12 |
| ayoung | marekd, I tested it with Hebrew, Russian and Chinese | 16:12 |
| openstackgerrit | Lance Bragstad proposed a change to openstack/keystone: Allow users to clean up role assignments https://review.openstack.org/119843 | 16:13 |
| ayoung | I took a phrase, passed it to google translate and put the Russian in. When I showed it to my boss (A native Russian speaker) he was immobilized with laughter for about a minute | 16:13 |
| marekd | ayoung: so it's now tested with Polish. | 16:14 |
| ayoung | marekd, Doskonałe | 16:15 |
| marekd | ayoung: some parts are simply not translated. The rest looks cool. | 16:16 |
| ayoung | marekd, getting translations for all languages is a huge undertaking. I'm psyched that we even have a subset in Polish | 16:16 |
| marekd | ayoung: understood. | 16:16 |
| ayoung | marekd, its also a Kerberos IdP. Once you get the password changed, if you want, I can help you go through the steps to get a TGT | 16:17 |
| *** BAKfr has quit IRC | 16:17 | |
| marekd | ayoung: ok, so i am in you PoC (/cops). I simply put my user/pass, and what options shall i choose for getting a token? | 16:18 |
| morganfainberg | lbragstad, the latest patch looks good. waiting for jenkins | 16:18 |
| ayoung | marekd, ah | 16:18 |
| lbragstad | morganfainberg: awesome | 16:18 |
| marekd | ayoung: ayoung Token Fetch Failed with status 401. | 16:18 |
| ayoung | marekd, the domain should be YOUNGLOGIC.NET | 16:18 |
| *** wwriverrat has left #openstack-keystone | 16:18 | |
| marekd | ayoung: https://keystone.younglogic.net/keystone/cops/# | 16:19 |
| ayoung | yep | 16:19 |
| marekd | aaa | 16:19 |
| marekd | ok ok | 16:19 |
| marekd | i see | 16:19 |
| ayoung | Default is what the installer sets up, and works for admin and demo | 16:19 |
| ayoung | if I could remember the passwords | 16:19 |
| marekd | ayoung: which authn method? | 16:20 |
| *** bdossant has quit IRC | 16:20 | |
| marekd | password? | 16:20 |
| ayoung | marekd, yeah, until you set up Kerberos | 16:20 |
| ayoung | start with password, you should be able to get a token | 16:20 |
| marekd | ayoung: ok, need to start in incognito mode. | 16:20 |
| marekd | i have a password. | 16:20 |
| marekd | tfu | 16:21 |
| marekd | token | 16:21 |
| ayoung | marekd, now you can do list projects | 16:21 |
| ayoung | once that is done, the drop down is populated, select one, and also the token radio button | 16:21 |
| ayoung | next "Get token" should get you a scoped one | 16:21 |
| marekd | ayoung: my project list i empty. | 16:22 |
| ayoung | of course it is...one sec | 16:22 |
| marekd | maybe i don't have any roles assigned | 16:22 |
| ayoung | marekd, what did it generate for your user id? | 16:22 |
| marekd | mdenis | 16:22 |
| *** bjornar_ has joined #openstack-keystone | 16:23 | |
| ayoung | marekd, no, the SHA256 one | 16:23 |
| marekd | ayoung: a, sorry: 88cd1c7c4618371137f59d4bbdadafb618617f1b368087a5e76ece04e1c0228f | 16:23 |
| ayoung | marekd, try again | 16:24 |
| ayoung | you should have "Member" role on "Demo" project | 16:24 |
| marekd | ayoung: so long story short: your JS scripts simply mimic keystoneclient and are smart enough to send JSON to /auth/tokens and handle them later, right? | 16:24 |
| openstackgerrit | Lance Bragstad proposed a change to openstack/keystone: Allow users to clean up role assignments https://review.openstack.org/119843 | 16:24 |
| marekd | ayoung: works | 16:24 |
| openstackgerrit | Lance Bragstad proposed a change to openstack/keystone: Allow users to clean up role assignments https://review.openstack.org/119843 | 16:25 |
| ayoung | marekd, you can now create a trust, if you want | 16:28 |
| marekd | ayoung: ok | 16:28 |
| *** cjellick has quit IRC | 16:28 | |
| ayoung | you will be both trustor and trustee, and it is based on your token | 16:28 |
| ayoung | Proof of concept limitations | 16:28 |
| ayoung | but then you can use that trust + your token to get another token | 16:29 |
| ekarlso | For you keystone folks: http://localhost:8000/introducing-servizor.html just as a fun tool using ksclient :) | 16:30 |
| ekarlso | https://cloudistic.net/blog/introducing-servizor.html rather | 16:32 |
| morganfainberg | ekarlso, hehe | 16:32 |
| morganfainberg | ekarlso, was going to say "localhost" ;) | 16:32 |
| ekarlso | found it as a pretty fun way to load endpoints quickly :) | 16:32 |
| ayoung | ekarlso, looking | 16:34 |
| ayoung | neat...anything that should find its way into the openstack common client, or just a debuggin tool? | 16:35 |
| ekarlso | ayoung: i'm using it for changing from http / https atm | 16:37 |
| ayoung | Nice | 16:37 |
| ekarlso | but could probably be added to the client as a contrib thing | 16:37 |
| ayoung | ekarlso, I've needed that on occasion | 16:37 |
| *** cjellick has joined #openstack-keystone | 16:37 | |
| ekarlso | doing keystone shell commands to do it is way too tedious | 16:38 |
| *** bjornar_ has quit IRC | 16:38 | |
| *** andreaf is now known as andreaf_ | 16:38 | |
| raildo | henrynash, ping | 16:45 |
| *** jasonsb has quit IRC | 16:46 | |
| *** jaosorior has quit IRC | 17:00 | |
| *** nkinder has quit IRC | 17:00 | |
| *** palendae has quit IRC | 17:00 | |
| *** bjornar has quit IRC | 17:00 | |
| *** DavidHu_ has quit IRC | 17:00 | |
| *** fifieldt_ has quit IRC | 17:00 | |
| *** jimbaker has quit IRC | 17:00 | |
| *** jaosorior has joined #openstack-keystone | 17:00 | |
| *** nkinder has joined #openstack-keystone | 17:00 | |
| *** palendae has joined #openstack-keystone | 17:00 | |
| *** DavidHu_ has joined #openstack-keystone | 17:00 | |
| *** bjornar has joined #openstack-keystone | 17:00 | |
| *** fifieldt_ has joined #openstack-keystone | 17:00 | |
| *** jimbaker has joined #openstack-keystone | 17:00 | |
| *** bambam1 has quit IRC | 17:00 | |
| *** rodrigods has quit IRC | 17:00 | |
| *** rwsu has quit IRC | 17:00 | |
| *** toddnni has quit IRC | 17:00 | |
| *** palendae has quit IRC | 17:01 | |
| *** amcrn has joined #openstack-keystone | 17:01 | |
| *** gyee has joined #openstack-keystone | 17:02 | |
| *** bambam1 has joined #openstack-keystone | 17:02 | |
| *** rodrigods has joined #openstack-keystone | 17:02 | |
| *** rwsu has joined #openstack-keystone | 17:02 | |
| *** toddnni has joined #openstack-keystone | 17:02 | |
| *** palendae has joined #openstack-keystone | 17:02 | |
| *** topol has quit IRC | 17:02 | |
| *** ctracey_ has quit IRC | 17:02 | |
| *** henrynash has quit IRC | 17:02 | |
| *** amerine has quit IRC | 17:02 | |
| *** harlowja_away has quit IRC | 17:02 | |
| *** thiagop has quit IRC | 17:02 | |
| *** htruta has joined #openstack-keystone | 17:02 | |
| *** topol has joined #openstack-keystone | 17:02 | |
| *** ctracey_ has joined #openstack-keystone | 17:02 | |
| *** henrynash has joined #openstack-keystone | 17:02 | |
| *** amerine has joined #openstack-keystone | 17:02 | |
| *** harlowja_away has joined #openstack-keystone | 17:02 | |
| *** thiagop has joined #openstack-keystone | 17:02 | |
| raildo | henrynash, we have created this document that shows all our patches about hierarchical projects, if you want to review :) http://paste.openstack.org/raw/112539/ | 17:04 |
| *** jsavak has quit IRC | 17:06 | |
| *** joesavak has joined #openstack-keystone | 17:08 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:18 | |
| YorikSar_ | morganfainberg: Hi | 17:18 |
| YorikSar_ | morganfainberg: Unfortunately I couldn't find time to work on this during the day... | 17:19 |
| morganfainberg | YorikSar_, no worries | 17:19 |
| morganfainberg | YorikSar_, i've addressed most of the concerns and added some basic testing. | 17:19 |
| YorikSar_ | morganfainberg: I've seen your tests though, they look fine. I'd like to add some concurrency tests if you don't mind. | 17:19 |
| *** YorikSar_ is now known as YorikSar | 17:19 | |
| *** radez_g0n3 is now known as radez | 17:19 | |
| dstanek | YorikSar: that would be awesome | 17:20 |
| morganfainberg | YorikSar, yeah some concurrency tests would be great | 17:20 |
| YorikSar | morganfainberg: Although I totally don't understand what was the issue with _create_connection and eventlet's yielding... | 17:20 |
| dstanek | YorikSar: it doesn't look like it's a problem now, but could be in the future | 17:20 |
| YorikSar | morganfainberg: It all is done under lock, no two threads can modify pool at one time... | 17:20 |
| YorikSar | dstanek: ^ | 17:21 |
| morganfainberg | YorikSar, so... we've serialized everything accessing memcached. | 17:21 |
| morganfainberg | YorikSar, ick. that is likely bad. | 17:21 |
| YorikSar | morganfainberg: No, not everything. | 17:21 |
| YorikSar | morganfainberg: Only everything related to pool itself. | 17:21 |
| YorikSar | morganfainberg: acquire and release | 17:22 |
| dstanek | we've only serialized getting stuff in and out of the queue | 17:22 |
| YorikSar | dstanek: Basically, yes. | 17:22 |
| morganfainberg | dstanek, which is most everything since an acquire/release is done on each and every action to memcache | 17:22 |
| YorikSar | morganfainberg: We release lock after acquire | 17:22 |
| YorikSar | morganfainberg: And now other threads can access this pool. | 17:23 |
| *** dhellmann is now known as dhellmann_ | 17:23 | |
| dstanek | morganfainberg: yes, sorta - it definitely doesn't help performance there | 17:23 |
| morganfainberg | YorikSar, but it's eventlet, you're not really running full threads. | 17:23 |
| morganfainberg | YorikSar, anyway. | 17:23 |
| morganfainberg | dstanek, thats my thought | 17:23 |
| YorikSar | morganfainberg: With monkeypatch all primitives are patched | 17:23 |
| morganfainberg | whatever though | 17:23 |
| YorikSar | morganfainberg: So locks are greenthread-aware | 17:23 |
| dstanek | why hasn't this been reported before? | 17:24 |
| morganfainberg | YorikSar, i really wish i could just delete eventlet | 17:24 |
| morganfainberg | YorikSar, it is awful | 17:24 |
| morganfainberg | dstanek, because no one was paying attention to the number of sockets consumed by python-memcached under eventlet before | 17:24 |
| morganfainberg | dstanek, actually it was reported there is an ancient bug saying we need an eventlet safe memcache lib | 17:25 |
| YorikSar | morganfainberg: Ah... You just need to forget about it. As long as your code don't use C libraries, everything works just as with usual threads. | 17:25 |
| marekd | morganfainberg: you don't like concept of single threaded + nonblocking operations application or just eventlet? :-) | 17:25 |
| dstanek | but if it is a leak wouldn't it grow indefinitely until it reached the ulimit of fds per process? | 17:25 |
| morganfainberg | marekd, eventlet is a trainwreck | 17:25 |
| morganfainberg | marekd, coroutines are generally awful to work with. | 17:25 |
| dstanek | morganfainberg: in this case thought it's not evenlets fault | 17:25 |
| morganfainberg | dstanek, it is thread.local + eventlet | 17:25 |
| marekd | morganfainberg: threads can also be tricky :( | 17:26 |
| YorikSar | morganfainberg, dstanek: Our guys ran rally on 20-nodes deployment of OPenStack. And they found out that there's awfully huge number of connections on controllers. | 17:26 |
| morganfainberg | marekd, well with the GIL threading is just a bad idea all around | 17:26 |
| marekd | morganfainberg: Python :( | 17:26 |
| dstanek | morganfainberg: from what i can tell doing some testing by hand i was having 1 memcached connection per greenthread, which is what i expected | 17:26 |
| morganfainberg | dstanek, it's not a "leak" it's a every request gets a new client object and they aren't cleaned up quickly in all cases, sometimes load/contention causes it to be even worse | 17:26 |
| YorikSar | dstanek: There's no leak... | 17:26 |
| *** harlowja_away is now known as harlowja | 17:27 | |
| morganfainberg | dstanek, in a simple tight loop of requesting tokens i could get a devstack to run out of FDs or starve memcached | 17:27 |
| *** marcoemorais has joined #openstack-keystone | 17:27 | |
| morganfainberg | dstanek, 400-600 active sockets to memcached | 17:27 |
| YorikSar | dstanek: But with thousands of open connections memcached and Python processes become very slow. | 17:27 |
| dstanek | hmm...that's interesting. i wasn't able to duplicate that | 17:28 |
| morganfainberg | dstanek, it's a known issue and thread.local is a bad idea with eventlet. the probelm is python-memcache and dogpile both use thread.local | 17:28 |
| morganfainberg | dstanek, it just opens us up to being DOSed simply by forcing connections to memcached | 17:29 |
| YorikSar | dstanek: Just run Keystone with devstack and run 'ab -c 100 -n 1000 -H "X-Auth-Token: $token" http://127.0.0.1:5000/v2.0/users' | 17:29 |
| morganfainberg | since everything would block on memcached (e.g. auth_token, keystone validating a token, etc) | 17:29 |
| YorikSar | dstanek: Then you can see in 'lsof -i4:11211' huge number of connections. | 17:29 |
| morganfainberg | it's one of the main reasons memcached should probably not be used as a token backend ever. | 17:30 |
| morganfainberg | the othe rbeing that using memcached as stable storage is an awful idea | 17:30 |
| YorikSar | dstanek: Now if you split load producer and have 3 controllers, you'll get the issue. | 17:30 |
| dstanek | YorikSar: that's basically how i was testing. i wonder if i was somehow reusing greenthreads instead of spawing a new one for each request | 17:30 |
| YorikSar | dstanek: Nah... There will be a lot of connections waiting to be closed... | 17:31 |
| dstanek | morganfainberg: yeah, that is explicitly called out everywhere as a huge no-no | 17:31 |
| morganfainberg | YorikSar, if you want to rever the bit of stuff around the acquired thats fine | 17:32 |
| YorikSar | morganfainberg: I want to rearrange things a bit there to make it clear when happens what. | 17:32 |
| YorikSar | morganfainberg: btw, I keep wondering, why do you post so many patchsets in a row? :) | 17:33 |
| *** marcoemorais has quit IRC | 17:33 | |
| morganfainberg | YorikSar a few were typos / forgot to git add a file | 17:33 |
| morganfainberg | YorikSar fixing a doc bug. | 17:34 |
| *** marcoemorais has joined #openstack-keystone | 17:34 | |
| morganfainberg | YorikSar a couple things that cropped up along the way that I missed in local testing | 17:34 |
| *** marcoemorais has quit IRC | 17:34 | |
| *** marcoemorais has joined #openstack-keystone | 17:34 | |
| *** marcoemorais has quit IRC | 17:34 | |
| *** marcoemorais has joined #openstack-keystone | 17:35 | |
| *** marcoemorais has quit IRC | 17:35 | |
| morganfainberg | YorikSar, in either case lets get some concurrency testing in and see if we can get this gating. | 17:35 |
| *** marcoemorais has joined #openstack-keystone | 17:36 | |
| YorikSar | morganfainberg: Yes, sure. I'm already working on acquire, will switch to tests soon | 17:36 |
| *** marcoemorais has quit IRC | 17:36 | |
| morganfainberg | YorikSar, thanks. | 17:36 |
| lbragstad | morganfainberg: is this related to the bug you just opened? https://bugs.launchpad.net/keystone/+bug/1370492 | 17:36 |
| uvirtbot | Launchpad bug 1370492 in keystone "calling curl "HEAD" ops time out on /v3/auth/tokens" [Undecided,New] | 17:36 |
| *** marcoemorais has joined #openstack-keystone | 17:37 | |
| morganfainberg | lbragstad, don't think that is the same thing | 17:41 |
| morganfainberg | lbragstad, the issue i reported is just that in some cases a HEAD will result in a 404 where a get will result in 2XX | 17:41 |
| lbragstad | gotcha | 17:41 |
| YorikSar | morganfainberg, dstanek: Should I add that busylooping to acquire to log smth like "Still waiting for a slot..."? | 17:42 |
| *** packet has joined #openstack-keystone | 17:42 | |
| stevemar | bknudson, i think i addressed your comments here: https://review.openstack.org/#/c/121146/ should be a quick review | 17:43 |
| morganfainberg | YorikSar, nah, lets not add more logging. | 17:47 |
| morganfainberg | YorikSar, so, what is the major downside if we move away from the auto-scaling pools? | 17:48 |
| morganfainberg | YorikSar, don't we need to be able to handle the maximum capacity anyway? | 17:48 |
| morganfainberg | YorikSar, (this is just makeing sure we're not over-engineering here) | 17:48 |
| *** bjornar_ has joined #openstack-keystone | 17:49 | |
| YorikSar | morganfainberg: Ah, yes, I caught some of your discussion yesterday... | 17:49 |
| morganfainberg | YorikSar, could we just pre-allocate the connections and use a straight eventlet.queue ( dstanek brought this up last night ) it is worth considering | 17:49 |
| *** dhellmann_ is now known as dhellmann | 17:50 | |
| morganfainberg | YorikSar, it would be a lot simpler code, removes the explicit busy wait scenarios. | 17:50 |
| morganfainberg | and we don't need the extra locking for reaping the connections, a simple get/put on the queue in the context manager | 17:51 |
| morganfainberg | locking is implicit vs explicit then | 17:51 |
| lbragstad | bknudson: the token api doesn't use json_home? | 17:51 |
| YorikSar | morganfainberg: Well, I think having 100 unused open connecions is a waste... | 17:52 |
| bknudson | lbragstad: I don't think anything uses json home at this point. | 17:52 |
| YorikSar | morganfainberg: And we'll still need all this code around, except cleaner. | 17:52 |
| morganfainberg | YorikSar, but you still need to support up to 100 connections at any given time. | 17:52 |
| morganfainberg | YorikSar, so what is the difference between having the connections and spending cycles ramping up/reaping connections. | 17:52 |
| openstackgerrit | Steve Martinelli proposed a change to openstack/python-keystoneclient: SAML2 federated authentication for ADFS. https://review.openstack.org/111771 | 17:52 |
| *** rushiagr is now known as rushiagr_away | 17:53 | |
| *** amakarov is now known as amakarov_away | 17:53 | |
| *** zzzeek has quit IRC | 17:53 | |
| YorikSar | morganfainberg: Time needed to open/close a connection is negligible compared to unused_timeout. | 17:54 |
| morganfainberg | YorikSar, that isn't answering my question. | 17:54 |
| *** zzzeek has joined #openstack-keystone | 17:54 | |
| morganfainberg | YorikSar, if we need to support 100 active connections, what is the real benefit of removing them vs. just keeping them around. | 17:54 |
| morganfainberg | YorikSar, again this is not about saying the reaping is a bad idea, just making sure we're not overengineering this and making it harder to maintain | 17:55 |
| YorikSar | morganfainberg: Just to release resources to let them be used elsewhere, I think... | 17:57 |
| *** radez is now known as radez_g0n3 | 17:57 | |
| YorikSar | morganfainberg: It's rather cheap (see prev. message) and don't require much code (15 lines of code in cleanup). | 17:58 |
| YorikSar | morganfainberg: So not holding to unused resources should be enough to justify having this very small overhead. | 18:00 |
| dstanek | YorikSar: i think it's the complexity of managing it at all - the busy loop, etc.; not the lines of code | 18:02 |
| *** radez_g0n3 is now known as radez | 18:02 | |
| dstanek | the operator has to scale to handle the number of expected connections anyway - why not get into a steady state early? | 18:02 |
| YorikSar | dstanek: That loop is not so busy and is needed to limit total number of connections, no cleanup them. | 18:02 |
| *** packet has quit IRC | 18:03 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:03 | |
| dstanek | that can be done without a loop thought right? assuming you don't have to clean them up | 18:03 |
| morganfainberg | YorikSar, dstanek, http://paste.openstack.org/show/112561/ | 18:04 |
| morganfainberg | something much much simpler without the need to "cleanup" | 18:04 |
| *** stevemar has quit IRC | 18:04 | |
| *** stevemar has joined #openstack-keystone | 18:04 | |
| morganfainberg | it's not complete, just a quick example | 18:04 |
| dstanek | morganfainberg: :-) i was just looking for the paste i sent you, but wasn't having any luck | 18:04 |
| morganfainberg | dstanek, hehe | 18:05 |
| morganfainberg | dstanek, yeah couldn't find it so i recreated something close | 18:05 |
| morganfainberg | dstanek, close-ish | 18:05 |
| YorikSar | morganfainberg: Oh, please, dont do explicit imports from eventlet :) | 18:05 |
| dstanek | morganfainberg: found it! http://paste.openstack.org/show/112371/ | 18:06 |
| morganfainberg | dstanek, ++ much better | 18:06 |
| dstanek | YorikSar: we could use Python's queue, but in this case i don't think this backend should be used outside of eventlet | 18:06 |
| YorikSar | morganfainberg: Otherwise yes, this hides all this stuff in stdlib... | 18:06 |
| YorikSar | (which still have that loop inside) | 18:07 |
| morganfainberg | sure. i'm looking at maintainability, do we need to re-implment it | 18:07 |
| dstanek | i don't think we'll hit any loops in the stdlib version because we'll only be putting things that fit under maxsize | 18:08 |
| YorikSar | dstanek: Let me show you... | 18:09 |
| dstanek | it should just be for getting items which is a simpler loop | 18:09 |
| YorikSar | dstanek: https://hg.python.org/cpython/file/7a4d960fc801/Lib/threading.py#l461 | 18:10 |
| *** saipandi has joined #openstack-keystone | 18:10 | |
| YorikSar | dstanek: Queue uses semaphore, semaphore uses condition, condition _requires_ loop aroud waiting on it. | 18:10 |
| morganfainberg | YorikSar, but we're not needing to re-implement stdlib stuff in our code. | 18:10 |
| YorikSar | morganfainberg: Yes, I agree. | 18:10 |
| dstanek | right, but we're not adding our own loop - either solution uses that | 18:11 |
| YorikSar | morganfainberg: I'm thinking about how we could use semaphore here. | 18:11 |
| dstanek | my comment about removing the loop was our own busy loop | 18:11 |
| morganfainberg | YorikSar, which was the whole point of this conversation, do we get a benefit from re-implementing the loop? | 18:11 |
| YorikSar | morganfainberg: Wait... We weren't talking about loop. We were talking about cleaning. | 18:12 |
| morganfainberg | YorikSar, they are closely related | 18:12 |
| dstanek | morganfainberg: the reason you see that factory in my paste is that i started to play with a _trim method to removed idle connections | 18:12 |
| morganfainberg | YorikSar, without the need for cleaning do we need our own loop | 18:12 |
| YorikSar | morganfainberg: Nah... | 18:12 |
| YorikSar | morganfainberg: It seems we don't' need that loop in any case. | 18:12 |
| morganfainberg | YorikSar since we don't need to reach into the list to clean it up, we don't. | 18:13 |
| dstanek | a Queue has a queue property that is a deque so it's possible to remove idle, but slightly complicated and needs extra locking | 18:13 |
| YorikSar | morganfainberg: If we need cleaning or not is a separate question. | 18:13 |
| morganfainberg | YorikSar, queue is more opaque than a list, it largely looks like the loop is meant to support a list instead of queue object | 18:13 |
| morganfainberg | so we can clean it up without needing to pop everything off the queue, inspect and re add | 18:13 |
| YorikSar | morganfainberg: Ok, I don't understand you... With queue we can't have cleanup logic. | 18:14 |
| YorikSar | morganfainberg: To have cleanup logic we heed stack (-ish). | 18:14 |
| morganfainberg | YorikSar, right | 18:14 |
| YorikSar | morganfainberg: So if we don't need cleanup, we can leave all locking to queue. | 18:15 |
| morganfainberg | YorikSar, the re-implementation of the loop appears to be related to using a list as a stack instead of a full queue. | 18:15 |
| YorikSar | morganfainberg: But we won't need any explicit loops even with cleanup. | 18:16 |
| YorikSar | morganfainberg: no.... | 18:16 |
| dstanek | you can cleanup with a queue | 18:16 |
| morganfainberg | dstanek, you indicated it was painful, so i'd say we shouldn't | 18:16 |
| YorikSar | morganfainberg: Loop is there just because I didin't think about using semaphore here.. | 18:16 |
| YorikSar | dstanek: No, you can't. | 18:16 |
| dstanek | morganfainberg: it's painful, but possible | 18:17 |
| morganfainberg | dstanek, yeah lest assume it's not possible for our implementation's sake :) | 18:17 |
| dstanek | YorikSar: why not? i had a poc that did it (unsafely because i didn't implement locking) | 18:17 |
| YorikSar | dstanek: queue is FIFO. One thread with 6 memcached requests will use 6 different connections from the queue. | 18:17 |
| YorikSar | dstanek: So all elements of the queue will get constantly renewed. | 18:18 |
| *** packet has joined #openstack-keystone | 18:18 | |
| YorikSar | dstanek: You need to replace queue with stack (or better well) | 18:18 |
| morganfainberg | YorikSar, ok so for this implementation's sake (and because it's not worth it) lets assume we can't cleanup a queue. | 18:19 |
| YorikSar | dstanek: So that connections are used from the top of it, and on the bottom we have old unused connections rotting. | 18:19 |
| dstanek | YorikSar: isn't that what we have now and we just the cleanup to find them? | 18:19 |
| morganfainberg | dstanek, we actually append recently used connections to the end, it's not really a "stack" it's a FIFO with some sliceing cleanup logic. | 18:20 |
| YorikSar | dstanek: Now we acquire connections from the top of the stack (.pop()) and release them to the top of the stack (.append) while cleaner works from the bottom of the stack. | 18:21 |
| morganfainberg | oh crap wait i misread that yeah we use it as a stack | 18:21 |
| morganfainberg | blah brain .. not working today | 18:21 |
| dstanek | YorikSar: not worth discussing now; but my poc uses a Queue subclass that overrides ._get to .pop and extends .put to cleanup | 18:22 |
| dstanek | YorikSar: i was tempted to use LifoQueue, but i like the efficiency of using a deque | 18:24 |
| YorikSar | dstanek: I don't think deque is faster than working on the left end of the list. | 18:27 |
| dstanek | YorikSar: it should be much faster because it doesn't have to move the list or create a new one | 18:28 |
| dstanek | i haven't timed it though | 18:28 |
| lbragstad | morganfainberg: I can confirm https://bugs.launchpad.net/keystone/+bug/1370492 | 18:29 |
| uvirtbot | Launchpad bug 1370492 in keystone "calling curl "HEAD" ops time out on /v3/auth/tokens" [Low,Confirmed] | 18:29 |
| lbragstad | morganfainberg: want to include it in RC1? | 18:29 |
| YorikSar | dstanek: After you fill the list, it'll stay fixed in memory as long as you don't add 2x more items to it. | 18:29 |
| morganfainberg | lbragstad, hm. | 18:30 |
| morganfainberg | lbragstad, we use HEAD for "check" on the token right? | 18:30 |
| YorikSar | morganfainberg, dstanek: Looking and 'implementation' (4 overloaded methods) of LifoQueue I think we can use that, add some cleanup logic and proper size accounting and have the same pool, but without explicit concurrency outside stdlib. | 18:31 |
| *** marcoemorais has quit IRC | 18:31 | |
| morganfainberg | lbragstad, means we probably need to look at why that is occuring. | 18:31 |
| morganfainberg | lbragstad, and hit it before RC | 18:31 |
| *** marcoemorais has joined #openstack-keystone | 18:32 | |
| *** marcoemorais has quit IRC | 18:32 | |
| *** marcoemorais has joined #openstack-keystone | 18:32 | |
| YorikSar | morganfainberg, dstanek: Should I try that or do you think it's not worth it at all? | 18:32 |
| morganfainberg | YorikSar, it would probably make things a lot more readable. | 18:35 |
| morganfainberg | YorikSar, but i'm still not sure we *need* the cleanup logic | 18:35 |
| lbragstad | morganfainberg: that *should* be doing https://github.com/openstack/keystone/blob/master/keystone/auth/controllers.py#L507 | 18:35 |
| morganfainberg | lbragstad, hm. i wonder | 18:36 |
| YorikSar | morganfainberg: Ok, I'll implement it and we'll see... | 18:36 |
| *** amcrn has quit IRC | 18:38 | |
| *** amcrn has joined #openstack-keystone | 18:38 | |
| dstanek | YorikSar: i'm curious now about the implementation of lists (for my own knowledge) - where can i find info on why what we are doing won't shift the elements? | 18:39 |
| dstanek | YorikSar: i thought the l[:1] = [] was effectively the same as l.pop(0) | 18:39 |
| YorikSar | dstanek: Yes, that part is slow. | 18:40 |
| lbragstad | morganfainberg: ... it's looping in routes/middleware.py | 18:40 |
| morganfainberg | lbragstad, ah | 18:40 |
| dstanek | YorikSar: ah, ok; that's why i used the deque | 18:40 |
| YorikSar | dstanek: But with deque we have constantly allocating and deallocating chunks when we put and fetch from it. | 18:40 |
| YorikSar | dstanek: So with list common operations are fast (pop/append), rare operations are slow (popleft). | 18:42 |
| YorikSar | dstanek: Ah, wait... Did you put items to the left end of deque? | 18:42 |
| dstanek | i change get to pop so that they came from the right and the put already appended to the right | 18:43 |
| dstanek | my cleanup did a popleft and checked for idle timeout - if it was idle i would trash the connection and popleft again | 18:43 |
| YorikSar | dstanek: Oh, in that case deque will just add fast cleanup, yes... | 18:43 |
| dstanek | once i reached a non-idle connection i put it back | 18:43 |
| dstanek | let me see if i still have that - i was a trivial amount of code | 18:44 |
| *** stevemar has quit IRC | 18:44 | |
| YorikSar | dstanek: mb you can put it to that CR? | 18:45 |
| dstanek | YorikSar: i don't have it anymore, but i can recreate a post it | 18:47 |
| YorikSar | dstanek: btw, did you account for acquired connections? | 18:48 |
| YorikSar | dstanek: I guess if you don't have the code I can write it then. | 18:49 |
| dstanek | YorikSar: yeah, i had to keep a count in the cleanup version, but it's not safe yet - i have see about using the existing queue lock for it | 18:52 |
| dstanek | YorikSar: that's why i like the no-cleanup version - much simpler | 18:52 |
| YorikSar | dstanek: If you do cleanup in _put, you're already protected by queue's mutex. | 18:53 |
| morganfainberg | dstanek, and the no-cleanup version would hit our needs for Juno and middleware | 18:53 |
| *** Apsu has joined #openstack-keystone | 18:54 | |
| dstanek | YorikSar: it's not the cleanup that bothers me it's adding more items when we need more | 18:54 |
| dstanek | morganfainberg: i'll throw something together for you guys to look at | 18:54 |
| YorikSar | dstanek: That should happen in _get - still guarded by mutex | 18:54 |
| *** aix has quit IRC | 18:55 | |
| *** cjellick has quit IRC | 18:55 | |
| dstanek | YorikSar: that's not the current design though; i could make the Pool a subclass of queue and maybe do it like that | 18:55 |
| YorikSar | dstanek: Yes. | 18:56 |
| YorikSar | dstanek: Ok, looks like now we both are doing the same thing :) | 18:56 |
| YorikSar | dstanek: I guess I'll leave it to you then. | 18:56 |
| *** packet has quit IRC | 18:56 | |
| dstanek | YorikSar: did you already start working on it? | 18:56 |
| YorikSar | dstanek: Just a little bit. | 18:57 |
| YorikSar | dstanek: Alhough I'd be happy to leave it to you if you don't mind. Its getting late here :) | 18:57 |
| dstanek | YorikSar: i'm happy to let you continue this; there's still a few RC reviews to get through today; just let me know what you want to do | 18:57 |
| *** jasonsb has joined #openstack-keystone | 18:58 | |
| YorikSar | dstanek: Oh, ok. I'll subclass queue and add cleanup logic to it. | 18:58 |
| YorikSar | dstanek: I'll post my version in some time then. | 18:58 |
| dstanek | YorikSar: ok, if you need to bail because it's late just let me know i can help pick up where you left off | 18:59 |
| YorikSar | dstanek: Ok, sure | 18:59 |
| *** gyee has quit IRC | 19:02 | |
| *** packet has joined #openstack-keystone | 19:04 | |
| *** amcrn has quit IRC | 19:07 | |
| *** packet has quit IRC | 19:11 | |
| *** joesavak has quit IRC | 19:11 | |
| *** marcoemorais has quit IRC | 19:11 | |
| *** packet has joined #openstack-keystone | 19:12 | |
| *** marcoemorais has joined #openstack-keystone | 19:12 | |
| *** joesavak has joined #openstack-keystone | 19:12 | |
| *** marcoemorais has quit IRC | 19:12 | |
| *** radez is now known as radez_g0n3 | 19:12 | |
| *** marcoemorais has joined #openstack-keystone | 19:12 | |
| *** amcrn has joined #openstack-keystone | 19:16 | |
| *** marcoemorais has quit IRC | 19:16 | |
| *** marcoemorais has joined #openstack-keystone | 19:16 | |
| *** stevemar has joined #openstack-keystone | 19:17 | |
| *** vdreamarkitex has joined #openstack-keystone | 19:17 | |
| YorikSar | dstanek: damn eventlet... | 19:23 |
| YorikSar | dstanek: it doesn't provide a way to override _qsize. | 19:23 |
| dstanek | YorikSar: in eventlet's queue implementation? | 19:24 |
| YorikSar | dstanek: yep | 19:24 |
| YorikSar | dstanek: I can add rather ugly hack-around though... | 19:25 |
| *** packet has quit IRC | 19:25 | |
| YorikSar | dstanek: can=will | 19:25 |
| dstanek | you can't override qsize()? | 19:25 |
| YorikSar | dstanek: I can, but we shouldn't do that in stdlib's version of queue. | 19:26 |
| YorikSar | dstanek: I'll add an if: to cover eventlet... | 19:27 |
| dstanek | are you using both? | 19:27 |
| YorikSar | dstanek: I just ran tests created by morganfainberg. We seem to have eventlet around in tests. | 19:28 |
| morganfainberg | YorikSar, we likely do as we start full versions of keystone in the functional tests | 19:29 |
| *** cjellick has joined #openstack-keystone | 19:30 | |
| YorikSar | morganfainberg: Yeah... That means we don't run unittests w/o eventlet. | 19:31 |
| morganfainberg | YorikSar, sounds about right. | 19:31 |
| morganfainberg | YorikSar you can set an ENV arg to use standard threads | 19:31 |
| *** jaosorior has quit IRC | 19:32 | |
| YorikSar | morganfainberg: Just 'ENV= tox -e py27'? | 19:32 |
| morganfainberg | YorikSar, looking for it | 19:32 |
| *** openstackgerrit_ has joined #openstack-keystone | 19:34 | |
| morganfainberg | YorikSar, 'STANDARD_THREADS' | 19:34 |
| YorikSar | morganfainberg: Oh, thanks | 19:34 |
| morganfainberg | YorikSar, so STANDARD_THREADS=True tox -epy27 | 19:34 |
| morganfainberg | should do it | 19:34 |
| YorikSar | morganfainberg: Good. Will verify with usual thread later then. | 19:34 |
| morganfainberg | YorikSar, ++ | 19:35 |
| *** radez_g0n3 is now known as radez | 19:36 | |
| openstackgerrit | Steve Martinelli proposed a change to openstack/python-keystoneclient: Handle federated tokens https://review.openstack.org/121146 | 19:46 |
| stevemar | bknudson, ^ round 2! fight! | 19:47 |
| *** jsavak has joined #openstack-keystone | 19:50 | |
| *** packet has joined #openstack-keystone | 19:51 | |
| henrynash | dstanek, lbragstad: if either of you have a moment, looking for additional +2/A on https://review.openstack.org/#/c/121246/14 | 19:51 |
| *** joesavak has quit IRC | 19:51 | |
| lbragstad | henrynash: checking | 19:51 |
| henrynash | lbragstad: thx | 19:52 |
| *** aix has joined #openstack-keystone | 19:53 | |
| stevemar | henrynash, do you deserve a +2 on that! | 19:56 |
| stevemar | .... looking | 19:56 |
| henrynash | stevemar: far be it for me to be so bold as to suggest anything so outrageous… | 19:56 |
| henrynash | stevemar: but if a couple of pints of english “old thumper” ale will swing your vote let me know :-) | 19:58 |
| morganfainberg | stevemar and lbragstad, henrynash's patch looks good to me, but i'm happy to let you continue reviewing it. | 19:58 |
| stevemar | henrynash, well you're off to a good start, passed the bknudson test | 19:58 |
| morganfainberg | the tests too me a while to get through | 19:58 |
| stevemar | morganfainberg, go ahead and +A/+2, i'm still going to review it to get to know the code | 19:58 |
| lbragstad | henrynash: morganfainberg stevemar I stepped through the tests again... looks all good | 19:58 |
| lbragstad | my comments were addressed | 19:58 |
| morganfainberg | lbragstad, go for the +2/+A then. | 19:58 |
| dstanek | lbragstad: ++ | 19:59 |
| * lbragstad picks up hammer | 19:59 | |
| stevemar | *slam!* | 19:59 |
| * lbragstad good Mjölnir... good... | 19:59 | |
| henrynash | thank y’all kindly | 20:00 |
| morganfainberg | really winding down the RC list | 20:00 |
| lbragstad | henrynash: thank you for being so prompt! | 20:00 |
| morganfainberg | woot | 20:00 |
| *** andreaf has joined #openstack-keystone | 20:02 | |
| dstanek | to that end...does anyone know the author of https://review.openstack.org/#/c/119345/ ? they haven't been back in a while | 20:03 |
| *** radez is now known as radez_g0n3 | 20:04 | |
| bknudson | only thor can lift Mjölnir. | 20:05 |
| stevemar | dstanek, was thinking the same, maybe one of us should add a test | 20:06 |
| stevemar | bknudson, maybe lbragstad is Thor | 20:06 |
| morganfainberg | dstanek, i actually saw that recently | 20:06 |
| bknudson | stevemar: I've never seen them both in the same place at the same time... | 20:06 |
| morganfainberg | it should be an easy test case | 20:07 |
| lbragstad | inception! | 20:07 |
| *** jasonsb has quit IRC | 20:07 | |
| stevemar | bknudson, I think you scared off the authors of https://review.openstack.org/#/c/119345/ | 20:07 |
| lbragstad | like Clark and his glasses | 20:07 |
| bknudson | stevemar: I'm waiting for them to email my manager. | 20:07 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack/keystone: Add a pool of memcached clients https://review.openstack.org/119452 | 20:08 |
| YorikSar | dstanek, morganfainberg: ^ | 20:08 |
| morganfainberg | YorikSar, looking, it's looking easier to read already. | 20:08 |
| stevemar | morganfainberg, if you want to reapply your +2 here: https://review.openstack.org/#/c/121146/ I just moved some things around to make future federation fixtures easier | 20:09 |
| *** topol has quit IRC | 20:09 | |
| YorikSar | dstanek, morganfainberg: It passed Morgan's tests and couple of ab runs with dying and resurrecting memcached backends | 20:09 |
| *** amcrn has quit IRC | 20:09 | |
| morganfainberg | stevemar, done | 20:09 |
| YorikSar | dstanek, morganfainberg: I'm falling asleep, so I'm looking forward to seeing your comments/fixes in the morning. | 20:10 |
| dstanek | YorikSar: awesome, thanks! i'll take a look | 20:10 |
| morganfainberg | YorikSar, generalyl speaking looking good! sleep well! | 20:10 |
| *** amcrn has joined #openstack-keystone | 20:13 | |
| *** jasonsb has joined #openstack-keystone | 20:15 | |
| stevemar | bknudson, morganfainberg dstanek i'm looking at tempest now, and according to their docs - they perform CLI tests, but apparently the tests can't 'change the cloud', so i guess no create tests? | 20:15 |
| bknudson | stevemar: that's what I assumed... so no token_flush | 20:16 |
| morganfainberg | dstanek, going to post a fix with a couple minor changes to the recent patch by YorikSar unless you are currently reviewing | 20:16 |
| stevemar | is there a way to test the CLI so that it can change content in the cloud? | 20:16 |
| morganfainberg | dstanek, which case i'll post my 2 comments. | 20:16 |
| bknudson | stevemar: ask on -qa | 20:16 |
| dstanek | morganfainberg: nope, looking at another review now | 20:16 |
| morganfainberg | k will post with the 2 minor fixes. | 20:16 |
| stevemar | bknudson, yeah, figured i'd ask here first to see what we've done in the past | 20:17 |
| bknudson | stevemar: if we'd done anything in the past we wouldn't have so many bugs | 20:17 |
| stevemar | lol | 20:17 |
| stevemar | the amount of hate bknudson has for bugs is too damn high | 20:17 |
| *** jasonsb has quit IRC | 20:19 | |
| *** aix has quit IRC | 20:23 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/keystone: Add a pool of memcached clients https://review.openstack.org/119452 | 20:25 |
| *** gyee has joined #openstack-keystone | 20:35 | |
| *** jsavak has quit IRC | 20:37 | |
| *** jasonsb has joined #openstack-keystone | 20:38 | |
| *** jasonsb has quit IRC | 20:43 | |
| *** _cjones_ has joined #openstack-keystone | 20:44 | |
| *** bjornar_ has quit IRC | 20:47 | |
| morganfainberg | stevemar, lol | 20:48 |
| *** radez_g0n3 is now known as radez | 20:54 | |
| *** cjellick has quit IRC | 20:58 | |
| *** marcoemorais has quit IRC | 21:02 | |
| *** marcoemorais has joined #openstack-keystone | 21:03 | |
| *** zzzeek has quit IRC | 21:04 | |
| *** cjellick has joined #openstack-keystone | 21:05 | |
| *** nkinder has quit IRC | 21:09 | |
| *** gordc has quit IRC | 21:10 | |
| *** stevemar has quit IRC | 21:12 | |
| *** stevemar has joined #openstack-keystone | 21:18 | |
| *** david-lyle has quit IRC | 21:19 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 21:20 | |
| dstanek | so what's the point of this bug? https://bugs.launchpad.net/keystone/+bug/1362245 | 21:23 |
| uvirtbot | Launchpad bug 1362245 in keystone "Update Endpoint Filter APIs" [Low,In progress] | 21:23 |
| dstanek | the "why" we need this or "why" it's currently wrong is missing | 21:23 |
| bknudson | dstanek: I don't think that should be a release blocker. | 21:26 |
| bknudson | it looks like a nice-to-have | 21:27 |
| bknudson | especially since the date for stability of the API is already passed | 21:27 |
| bknudson | and the keystoneclient already has an implementation using the old api | 21:27 |
| morganfainberg | bknudson, agree. it should not be a release blocker, but it's also something that we could easily get in within the next day or so. | 21:27 |
| morganfainberg | bknudson, as long as the old impl still works i'm fine with this going in. if it slips to K1, i wont be upset though | 21:27 |
| dstanek | bknudson: yes, i definitely agree that it shouldn't be a blocker | 21:27 |
| dstanek | why switch the order of the URL though? | 21:28 |
| bknudson | prettier | 21:28 |
| morganfainberg | dstanek, more consistent with our other APIs | 21:28 |
| bknudson | does anybody still use the URLs? just use JSON Home. | 21:28 |
| morganfainberg | lol | 21:29 |
| dstanek | ++ | 21:29 |
| dstanek | morganfainberg: the near duplicate options here is confusing https://review.openstack.org/#/c/119452/28/keystone/common/config.py | 21:30 |
| morganfainberg | dstanek, yeah it is because cache != kvs | 21:30 |
| morganfainberg | dstanek, i added some extra details into the help text and they were removed in a subsequent patch | 21:31 |
| morganfainberg | dstanek, i can add the info back in easily | 21:31 |
| dstanek | morganfainberg: ah i'll go back and look for those; like what's the difference between cache.memcache_socket_timeout and memcache.socket_timeout | 21:32 |
| morganfainberg | dstanek, the [cache] options are for the caching layer | 21:32 |
| morganfainberg | dstanek, the [memcache] options affect the kvs (e.g. token persistence driver) | 21:32 |
| morganfainberg | dstanek, i had added some information for that into the [memcache] option help text. | 21:33 |
| morganfainberg | easy to add it back in. | 21:33 |
| morganfainberg | if needed | 21:33 |
| morganfainberg | (probably should be added back in) | 21:33 |
| morganfainberg | or we could move it to the KVS config section | 21:34 |
| *** stevemar has quit IRC | 21:34 | |
| *** dims has quit IRC | 21:35 | |
| lbragstad | morganfainberg: six reviews from RC1 | 21:35 |
| *** dims has joined #openstack-keystone | 21:35 | |
| morganfainberg | lbragstad i count it as 4 (2 are approved) | 21:36 |
| morganfainberg | ;) | 21:37 |
| lbragstad | s/six/six (including keystonemiddleware)/ | 21:37 |
| morganfainberg | hmm. | 21:37 |
| morganfainberg | weird https://jenkins05.openstack.org/job/gate-keystone-python27/1679/consoleFull | 21:37 |
| morganfainberg | something looks like it isn't getting cleaned up on that patch. | 21:38 |
| morganfainberg | how did this pass check. | 21:39 |
| morganfainberg | *blink* | 21:40 |
| *** dims has quit IRC | 21:40 | |
| dstanek | morganfainberg: i'm looking at the pool patch again and i don't see how we enforce max size - it looks to me like things could just deadlock in the put() when there are traffic spikes | 21:42 |
| dstanek | morganfainberg: hmmm...wait...tracing this throught the Queue and it looks like that may not be the case | 21:42 |
| morganfainberg | dstanek, isn't _put behind the lock? | 21:42 |
| morganfainberg | dstanek, where put() isn't | 21:43 |
| dstanek | morganfainberg: i was more worried about the get not properly checking the size and then the puts would block because the queue would be full | 21:45 |
| *** rkofman has quit IRC | 21:45 | |
| morganfainberg | dstanek, i think he is solving that with _qsize | 21:46 |
| morganfainberg | which factors in .acquired | 21:46 |
| dstanek | morganfainberg: why isn't _qsize defined as the parent's queue size plus acquired clients? | 21:46 |
| *** rkofman has joined #openstack-keystone | 21:47 | |
| morganfainberg | dstanek, i don't know why he isn't calling super() there. but he is returning .maxsize which is a property of the parent i *think* | 21:47 |
| dstanek | in the Queue the _qsize is the count of things in the queue - ours is the available slots? | 21:47 |
| morganfainberg | ther eis also the hasattr line at 93 | 21:47 |
| morganfainberg | dstanek, yes that sounds correct. and it looks like ._get isn't checking anything | 21:49 |
| morganfainberg | so we'll just run out of connections. | 21:49 |
| morganfainberg | erm, run over limit and block | 21:49 |
| dstanek | it's get() that does the checking | 21:49 |
| morganfainberg | *looks like* | 21:49 |
| morganfainberg | oh | 21:49 |
| dstanek | yeah, something is bogus there because pool.full() would be True even if there was nothing in the queue | 21:52 |
| morganfainberg | dstanek, while self._qsize() == self.maxsize: | 21:54 |
| morganfainberg | dstanek, we only block on the condition of that | 21:54 |
| dstanek | morganfainberg: ok, i'm confused now. i don't see how anything can be added to the queue at all. the put checks to see if there is room by seeing if qsize is maxsize and isn't that the starting state? | 21:55 |
| morganfainberg | whoopse wrong line. sec looking at Queue.Queue | 21:56 |
| morganfainberg | ah ok so .get in Queue.Queue blocks if .qsize is 0 | 21:56 |
| morganfainberg | if we have maxsize - acquired being .qsize we block as expected because we're "full" even though we're really not | 21:57 |
| *** meker12 has joined #openstack-keystone | 21:57 | |
| morganfainberg | and the only time the put fails is if ,_qsize == maxsize which, based on the .get should never happen | 21:57 |
| *** jasonsb has joined #openstack-keystone | 21:58 | |
| morganfainberg | i think i can write some unit tests to cover those cases since we can do timeouts now. | 21:58 |
| dstanek | morganfainberg: isn't qsize == maxsize the initial state? | 21:59 |
| *** jamielennox|away is now known as jamielennox | 21:59 | |
| morganfainberg | dstanek, hm, yeah. you're right | 21:59 |
| *** cjellick has quit IRC | 21:59 | |
| *** bknudson has quit IRC | 22:00 | |
| *** cjellick has joined #openstack-keystone | 22:00 | |
| morganfainberg | dstanek, but it isn't failing in a "real" test | 22:00 |
| dstanek | get should fail weirdly because the queue is "full", but nothing is in it and put should block adding to the queue because the queue is "full" | 22:00 |
| dstanek | going to pull this version down and start messing with it :-) just looking at the code has me stumped | 22:01 |
| *** nkinder has joined #openstack-keystone | 22:01 | |
| *** david-lyle has joined #openstack-keystone | 22:03 | |
| morganfainberg | i'm doing the same atm | 22:04 |
| morganfainberg | this *shouldn't* work afaict | 22:04 |
| dstanek | this can't work anyway - it's using super with old style classes | 22:05 |
| morganfainberg | dstanek, eventlet | 22:06 |
| morganfainberg | dstanek, eventlet changes the behavior of the queue this would break under apache | 22:06 |
| morganfainberg | dstanek, {"error": {"message": "must be type, not classobj", "code": 400, "title": "Bad Request"}}[stack@localhost ~]$ | 22:07 |
| morganfainberg | dstanek, eventlet mucks with things in a way that "fixes" the brokenness | 22:07 |
| *** cjellick has quit IRC | 22:08 | |
| *** zzzeek has joined #openstack-keystone | 22:08 | |
| *** cjellick has joined #openstack-keystone | 22:08 | |
| dstanek | eventlet monkey patches it's own version from eventlet.queue me thinks | 22:08 |
| morganfainberg | dstanek, yep, just confirmed it | 22:08 |
| morganfainberg | dstanek, look at eventlet.queue.LightQueue | 22:08 |
| *** jamielennox is now known as jamielennox|away | 22:09 | |
| *** marcoemorais has quit IRC | 22:10 | |
| *** marcoemorais has joined #openstack-keystone | 22:10 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack/keystone: Ensure identity sql driver supports domain-specific configuration. https://review.openstack.org/121246 | 22:11 |
| dstanek | morganfainberg: this does appear to be working somehow | 22:12 |
| *** richm1 has joined #openstack-keystone | 22:12 | |
| morganfainberg | dstanek, under eventlet the whole _qsize logic is very very different | 22:13 |
| morganfainberg | dstanek, i think it works in eventlet but wont limit max size how it's supposed to | 22:13 |
| morganfainberg | and under non-eventlet it blatently wont work because we're classobj vs new-style classes | 22:13 |
| dstanek | yeah i fixed the supers locally and it appears to enforce the max size | 22:14 |
| morganfainberg | henrynash, booted your SQL fix out of the gate, it was failing py27 each time i ran it locally and saw the failure in the gate | 22:15 |
| morganfainberg | henrynash, i'm not sure how it passed check. but commented on it can provide the traceback if needed | 22:15 |
| henrynash | hmmm…ouch….ok..let me investiagte | 22:15 |
| morganfainberg | henrynash, http://paste.openstack.org/show/112617/ | 22:16 |
| morganfainberg | henrynash, that for a *ton* of tests | 22:16 |
| henrynash | morganfainberg: ok, I’m on it | 22:17 |
| morganfainberg | henrynash, :) i was shocked when i saw the error cause it passed check just fine! | 22:17 |
| *** Tahmina has joined #openstack-keystone | 22:17 | |
| dstanek | morganfainberg: get doesn't check against maxsize at all which is the get works | 22:19 |
| morganfainberg | dstanek, ah | 22:19 |
| *** thiagop has quit IRC | 22:19 | |
| morganfainberg | dstanek, oh i see it in the non-eventlet one, it looks at _qsize not maxsize | 22:20 |
| dstanek | the logic is kind of backward but it always says it full until it acutally is | 22:20 |
| morganfainberg | which would be an issue in the put case? | 22:21 |
| morganfainberg | under non-eventlet | 22:21 |
| morganfainberg | under eventlet i'm sure it'll behave reasonably well. | 22:21 |
| dstanek | no, because as they are actually created the acquired gets incremented - so it doesn't actually stay full like i thought | 22:23 |
| morganfainberg | dstanek, oh a side effect of get solves the put issue | 22:27 |
| morganfainberg | aha | 22:27 |
| *** wanghong has quit IRC | 22:29 | |
| *** meker12 has quit IRC | 22:30 | |
| *** meker12 has joined #openstack-keystone | 22:31 | |
| dstanek | took me a few minutes to piece it together | 22:32 |
| morganfainberg | dstanek, hmm. so looks like the big issue is super() | 22:32 |
| *** packet has quit IRC | 22:33 | |
| *** achampion has quit IRC | 22:33 | |
| *** meker12 has quit IRC | 22:35 | |
| dstanek | yeah. since we are recommending this new pooled version instead of dogpile's | 22:35 |
| *** david-lyle has quit IRC | 22:40 | |
| *** david-ly_ has joined #openstack-keystone | 22:40 | |
| *** meker12 has joined #openstack-keystone | 22:41 | |
| *** meker12 has joined #openstack-keystone | 22:42 | |
| morganfainberg | dstanek ok i'll remove the super calls | 22:43 |
| morganfainberg | dstanek, and repost unless you want to. | 22:43 |
| dstanek | morganfainberg: sure, just got done dinnering | 22:47 |
| morganfainberg | dstanek, ok i'll get it posted once i run tests | 22:47 |
| morganfainberg | dstanek, almost done. | 22:47 |
| dstanek | cool | 22:47 |
| morganfainberg | it feels weird not using super() :P | 22:48 |
| *** david-ly_ has quit IRC | 23:06 | |
| *** richm1 has quit IRC | 23:07 | |
| morganfainberg | dstanek, should we put a maximum timeout on .get() ? | 23:14 |
| morganfainberg | dstanek, since i'm in here i can do that now. | 23:14 |
| morganfainberg | dstanek, and if so, what do we raise out / is it worth the string freeze exception for this? | 23:14 |
| dstanek | morganfainberg: can we use an exising string and log the stacktrace? | 23:16 |
| morganfainberg | hm... | 23:16 |
| dstanek | "An unexpected error prevented the server from fulfilling your request." -- seems vaguely correct | 23:17 |
| dstanek | at least more so than 'User not in domain: %s' | 23:17 |
| morganfainberg | i could just use UnexpectedError() with no string i guess | 23:18 |
| morganfainberg | dstanek, should this be a config option or just some static value we set (like 300s?) | 23:19 |
| morganfainberg | i'm thinking just some static value, maybe 120s? | 23:19 |
| dstanek | we should LOG.exception though so there is an actual record of what happened | 23:19 |
| dstanek | 120s means that an API request could block for that long without getting a response - i would think it would have to be configurable | 23:20 |
| morganfainberg | dstanek, log exception wont help we need a new string, the exception will be 'Empty' | 23:22 |
| morganfainberg | :( | 23:22 |
| morganfainberg | ok i'll make it a config option as well. | 23:22 |
| dstanek | where i used to work our timeouts were mostly under 10s - because user's won't wait that long for a web browser | 23:22 |
| morganfainberg | right | 23:22 |
| henrynash | morganfainberg: think I’m too tired to debug this tonight…I’ll get some kip and re-attack early in the mornig | 23:24 |
| morganfainberg | henrynash, sounds good. yeah it was kindof icky | 23:24 |
| morganfainberg | henrynash, no worries :) | 23:24 |
| *** henrynash has quit IRC | 23:25 | |
| *** achampion has joined #openstack-keystone | 23:26 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:32 | |
| openstackgerrit | A change was merged to openstack/keystone: Allow users to clean up role assignments https://review.openstack.org/119843 | 23:33 |
| *** meker12 has quit IRC | 23:36 | |
| *** meker12 has joined #openstack-keystone | 23:37 | |
| *** alex_xu has joined #openstack-keystone | 23:41 | |
| *** achampio1 has joined #openstack-keystone | 23:41 | |
| *** achampion has quit IRC | 23:44 | |
| *** saipandi has quit IRC | 23:52 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!