| *** chlong has joined #openstack-keystone | 00:19 | |
| *** jorge_munoz has joined #openstack-keystone | 00:50 | |
| * jamielennox hates when I discover notmorgan might be right | 00:50 | |
| notmorgan | jamielennox: might be right about? | 00:51 |
|---|---|---|
| notmorgan | also... why is it a problem when I might be right? | 00:51 |
| jamielennox | notmorgan: only cause i disagreed with you for ages | 00:51 |
| notmorgan | jamielennox: ok what did you disagree with me on this time? | 00:51 |
| jamielennox | notmorgan: i'm looking at keystone and the user/service token validation stuff | 00:51 |
| jamielennox | about passing only user_id, project_id etc as headers | 00:51 |
| jamielennox | and - it's going to be super painful | 00:52 |
| *** jorge_munoz has quit IRC | 00:52 | |
| notmorgan | heh | 00:52 |
| jamielennox | i forgot how messy the actual auth layers are | 00:52 |
| notmorgan | yah | 00:52 |
| jamielennox | but i'm currently looking at having to do a new provider | 00:52 |
| jamielennox | anyway - you're right in that if i just pass everything that auth_token passes as headers from service->service then i don't need to validate any of it via keystone | 00:53 |
| jamielennox | and it can all be done as auth_token middleware | 00:53 |
| jamielennox | and it feels.. icky but there is so much pain doing it the other way | 00:53 |
| notmorgan | jamielennox: i might have been thinking about this for a long time :( | 00:54 |
| jamielennox | yea, it's nice to have a purist point of view when discussing stuff, but at some point sucumb to the practical | 00:55 |
| jamielennox | which to be clear - does not mean i'm agreeing with the oslo.messaging thread local stuff yet | 00:55 |
| notmorgan | jamielennox: oslo.messaging thread local? | 00:56 |
| notmorgan | jamielennox: oslo_context you mean? | 00:56 |
| jamielennox | yea, kinda, we discussed how to make sure all services send the right stuff via rpc and your idea was to just make oslo.messaging look at the thread local data and do it without the services input | 00:57 |
| notmorgan | oh yeah | 00:57 |
| notmorgan | i still think that is the right answer | 00:57 |
| jamielennox | it might be the only practical answer unfortunately | 00:57 |
| jamielennox | but one revelation at a time | 00:57 |
| *** itlinux has quit IRC | 01:08 | |
| *** itlinux has joined #openstack-keystone | 01:18 | |
| *** itlinux has quit IRC | 01:23 | |
| *** itlinux has joined #openstack-keystone | 01:25 | |
| *** jamielennox is now known as jamielennox|away | 01:26 | |
| *** itlinux has quit IRC | 01:26 | |
| *** jamielennox|away is now known as jamielennox | 01:28 | |
| *** itlinux has joined #openstack-keystone | 01:35 | |
| *** EinstCrazy has joined #openstack-keystone | 01:48 | |
| *** clenimar_ has joined #openstack-keystone | 01:49 | |
| *** clenimar_ has quit IRC | 01:49 | |
| *** clenimar_ has joined #openstack-keystone | 01:52 | |
| *** chlong has quit IRC | 01:56 | |
| clenimar_ | maybe it's time to remove the block on https://review.openstack.org/#/c/282377/ | 01:58 |
| patchbot | clenimar_: patch 282377 - keystoneauth - Add is_domain to keystoneauth token | 01:58 |
| clenimar_ | as the server side got merged :) | 01:58 |
| clenimar_ | jamielennox ^ | 01:59 |
| jamielennox | clenimar_: block removed, i'll try and get to the review today | 01:59 |
| clenimar_ | jamielennox: thank you sir :) | 01:59 |
| jamielennox | sorry about putting the -2 on originally, it caught me the first time when it had 2 +As and wasn't merging | 02:00 |
| *** EinstCrazy has quit IRC | 02:03 | |
| clenimar_ | no problem, it was the right thing to do | 02:03 |
| *** itlinux has quit IRC | 02:04 | |
| *** EinstCrazy has joined #openstack-keystone | 02:04 | |
| *** itlinux has joined #openstack-keystone | 02:05 | |
| *** clenimar_ has quit IRC | 02:10 | |
| *** itlinux has quit IRC | 02:10 | |
| *** dave-mccowan has joined #openstack-keystone | 02:20 | |
| *** EinstCrazy has quit IRC | 02:26 | |
| *** EinstCrazy has joined #openstack-keystone | 02:28 | |
| *** EinstCrazy has quit IRC | 02:34 | |
| *** EinstCrazy has joined #openstack-keystone | 02:36 | |
| *** EinstCra_ has joined #openstack-keystone | 02:40 | |
| *** EinstCrazy has quit IRC | 02:43 | |
| *** itlinux has joined #openstack-keystone | 03:03 | |
| *** jamielennox is now known as jamielennox|away | 03:06 | |
| *** hoonetorg has quit IRC | 03:35 | |
| *** jamielennox|away is now known as jamielennox | 03:36 | |
| *** darren-wang has joined #openstack-keystone | 03:39 | |
| darren-wang | Hi, I'm recently research the access control of openstack, I wonder can we get 'domain_id' from project-scoped token? | 03:42 |
| darren-wang | when we use 'domain_id' in policy.v3cloudsample.json, does it specifically mean 'a domain-scoped user' ? | 03:43 |
| darren-wang | And more accurate, I think I want to know, when we use the 'domain_id' in policy.v3cloudsample.json, are we saying that this user 'BELONGS to this domain' or 'SCOPES to this domain'? | 03:47 |
| *** links has joined #openstack-keystone | 03:49 | |
| *** hoonetorg has joined #openstack-keystone | 03:51 | |
| darren-wang | So this is confusing because each user has attribute domain_id, meaning this user belongs to this domain, but when it comes to access control, 'domain_id' in policy language meaning the user is scoping to this domain | 04:01 |
| darren-wang | Am I correct? | 04:01 |
| *** fangxu has quit IRC | 04:57 | |
| *** fangxu has joined #openstack-keystone | 04:58 | |
| *** hoonetorg has quit IRC | 05:11 | |
| *** rcernin has joined #openstack-keystone | 05:16 | |
| *** rcernin has quit IRC | 05:21 | |
| *** hoonetorg has joined #openstack-keystone | 05:24 | |
| *** TxGVNN has joined #openstack-keystone | 05:32 | |
| *** rcernin has joined #openstack-keystone | 05:38 | |
| *** GB21 has joined #openstack-keystone | 05:43 | |
| darren-wang | Any body could help me with my questions about 'domain_id' in policy file? | 05:46 |
| *** naresht has joined #openstack-keystone | 06:09 | |
| naresht | Hi stevemar | 06:10 |
| naresht | Stevemar: Hi | 06:10 |
| *** sheel has joined #openstack-keystone | 06:12 | |
| *** furface has quit IRC | 06:15 | |
| *** jaosorior has joined #openstack-keystone | 06:18 | |
| *** fangxu has quit IRC | 06:33 | |
| *** fangxu has joined #openstack-keystone | 06:34 | |
| naresht | I am trying to do Keystone Google Federation. I got this error http://paste.openstack.org/show/497176/ | 06:44 |
| naresht | Any help is highly appreciated | 06:44 |
| *** josecastroleon has joined #openstack-keystone | 06:44 | |
| *** jaosorior has quit IRC | 06:55 | |
| *** jaosorior_ has joined #openstack-keystone | 06:55 | |
| *** jaosorior_ is now known as jaosorior | 07:00 | |
| *** GB21 has quit IRC | 07:03 | |
| *** josecastroleon has quit IRC | 07:14 | |
| *** GB21 has joined #openstack-keystone | 07:19 | |
| naresht | I am trying to do Keystone Google Federation. | 07:21 |
| naresht | I getting this error oidc_authenticate_user: the URL hostname (federation.dams.com) of the configured OIDCRedirectURI does not match the URL hostname of the URL being accessed (xxx.xx.x.xxx): the "state" and "session" cookies will not be shared between the two! | 07:22 |
| naresht | Any one tried this before ? | 07:22 |
| naresht | Any help is highly appreciated | 07:23 |
| *** GB21 has quit IRC | 07:35 | |
| *** vnogin has quit IRC | 07:43 | |
| *** vnogin has joined #openstack-keystone | 07:45 | |
| *** GB21 has joined #openstack-keystone | 07:52 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-keystone | 08:00 | |
| *** jaosorior has quit IRC | 08:03 | |
| *** jaosorior has joined #openstack-keystone | 08:03 | |
| *** GB21 has quit IRC | 08:03 | |
| *** henrynash has quit IRC | 08:03 | |
| *** markvoelker has joined #openstack-keystone | 08:15 | |
| *** GB21 has joined #openstack-keystone | 08:16 | |
| *** markvoelker has quit IRC | 08:23 | |
| *** markvoelker has joined #openstack-keystone | 08:24 | |
| *** jistr has joined #openstack-keystone | 08:30 | |
| *** dmk0202 has joined #openstack-keystone | 08:32 | |
| *** zzzeek has quit IRC | 08:56 | |
| *** zzzeek has joined #openstack-keystone | 08:58 | |
| *** yiorgos_272 has quit IRC | 09:15 | |
| *** mvk has quit IRC | 09:28 | |
| *** mvk has joined #openstack-keystone | 09:57 | |
| *** daemontool has joined #openstack-keystone | 10:01 | |
| *** GB21 has quit IRC | 10:20 | |
| *** GB21 has joined #openstack-keystone | 10:33 | |
| *** fangxu has quit IRC | 10:33 | |
| *** fangxu has joined #openstack-keystone | 10:33 | |
| *** EinstCra_ has quit IRC | 10:36 | |
| *** rodrigods has quit IRC | 10:48 | |
| *** rodrigods has joined #openstack-keystone | 10:48 | |
| samueldmq | morning keystone | 10:56 |
| *** markvoelker has quit IRC | 11:06 | |
| *** _fortis has quit IRC | 11:13 | |
| *** pnavarro has joined #openstack-keystone | 11:15 | |
| *** GB21 has quit IRC | 11:24 | |
| *** pnavarro has quit IRC | 11:27 | |
| *** pnavarro has joined #openstack-keystone | 11:36 | |
| *** GB21 has joined #openstack-keystone | 11:37 | |
| *** sdake has joined #openstack-keystone | 11:55 | |
| *** raildo-afk is now known as raildo | 12:06 | |
| *** markvoelker has joined #openstack-keystone | 12:08 | |
| *** sdake has quit IRC | 12:10 | |
| *** fangxu has quit IRC | 12:11 | |
| *** fangxu has joined #openstack-keystone | 12:12 | |
| *** markvoelker has quit IRC | 12:13 | |
| *** harbor has joined #openstack-keystone | 12:15 | |
| *** harbor has quit IRC | 12:16 | |
| *** harbor has joined #openstack-keystone | 12:17 | |
| *** harbor is now known as heyhty | 12:19 | |
| *** heyhty has quit IRC | 12:19 | |
| *** heyhty has joined #openstack-keystone | 12:19 | |
| *** heyhty has left #openstack-keystone | 12:19 | |
| *** GB21 has quit IRC | 12:20 | |
| *** heyhty has joined #openstack-keystone | 12:20 | |
| *** heyhty is now known as harbor2 | 12:20 | |
| *** iurygregory has joined #openstack-keystone | 12:29 | |
| *** pnavarro has quit IRC | 12:32 | |
| *** edmondsw has joined #openstack-keystone | 12:32 | |
| *** zqfan has quit IRC | 12:33 | |
| *** julim has joined #openstack-keystone | 12:40 | |
| *** notmorgan has quit IRC | 12:53 | |
| *** ninag has joined #openstack-keystone | 12:54 | |
| *** links has quit IRC | 12:55 | |
| *** markvoelker has joined #openstack-keystone | 13:00 | |
| *** pauloewerton has joined #openstack-keystone | 13:01 | |
| dstanek | samueldmq: morning | 13:03 |
| *** markvoelker has quit IRC | 13:04 | |
| *** markvoelker has joined #openstack-keystone | 13:04 | |
| *** josecastroleon has joined #openstack-keystone | 13:16 | |
| *** ramishra has joined #openstack-keystone | 13:17 | |
| ramishra | ayoung: hi | 13:20 |
| ayoung | ramishra, hey, you had a policy quesiont? | 13:20 |
| ramishra | yep, I'm trying to use is_admin_project thing | 13:20 |
| ramishra | Not sure why this policy is working. | 13:21 |
| ramishra | https://review.openstack.org/#/c/316627/1/etc/heat/policy.json | 13:21 |
| patchbot | ramishra: patch 316627 - heat - Allow admin super user across projects | 13:21 |
| ramishra | I thought https://review.openstack.org/#/c/312443/ is the problem, but after changing that loally it does not seem to work either | 13:25 |
| patchbot | ramishra: patch 312443 - openstack-dev/devstack - Change the domain name in keystone.conf | 13:25 |
| ramishra | any idea what's wrong? | 13:25 |
| ayoung | ramishra, so it turns out that most of the services use oslo-context to enforce policy, and we would need a couple other patches to pass on that info | 13:25 |
| ayoung | I'll get you the patches | 13:25 |
| ayoung | https://review.openstack.org/#/c/295870/ | 13:27 |
| patchbot | ayoung: patch 295870 - oslo.context - Add is_admin_project check | 13:27 |
| ramishra | ah! thanks ayoung! | 13:28 |
| ayoung | ramishra, I'm not certain if you only need that patch, or one to middleware to populate it yet. | 13:28 |
| *** markvoelker has quit IRC | 13:28 | |
| ayoung | ramishra, jamielennox (who should be asleep right now) was taking this and running with it | 13:28 |
| ramishra | yeah, I realised that I don't need that | 13:29 |
| ramishra | I'm not using is_admin_projct from the context | 13:29 |
| ayoung | ramishra, is the token object available? | 13:29 |
| *** darosale has joined #openstack-keystone | 13:29 | |
| ramishra | I don't see that attribute in the token | 13:31 |
| ayoung | ramishra, then the config options in keystone are not properly set | 13:32 |
| ayoung | ramishra, you need admin_project_domain_name and admin_project_name | 13:32 |
| ayoung | ramishra, check me on the exact names...and make sure they are in the right subsection of the file. Also, make sure you are a running Mitaka | 13:32 |
| ramishra | yeah, it's there [resource] | 13:33 |
| ramishra | admin_project_name = admin | 13:33 |
| ramishra | admin_project_domain_name = default | 13:33 |
| ramishra | driver = sql | 13:33 |
| ramishra | That's the correct subsection I assume? | 13:34 |
| *** tonytan4ever has joined #openstack-keystone | 13:35 | |
| ayoung | ramishra, I could say yes, but I'd be lying. I don't remember. Always check everything I say | 13:35 |
| *** rderose has joined #openstack-keystone | 13:37 | |
| dstanek | ayoung: ++ | 13:37 |
| *** gordc has joined #openstack-keystone | 13:40 | |
| *** edmondsw has quit IRC | 13:43 | |
| *** sheel has quit IRC | 13:45 | |
| *** spzala has joined #openstack-keystone | 13:50 | |
| *** TxGVNN has quit IRC | 13:52 | |
| openstackgerrit | Victor Stinner proposed openstack/keystone: Port test_v2 unit test to Python 3 https://review.openstack.org/312060 | 13:54 |
| openstackgerrit | Victor Stinner proposed openstack/keystone: Port test_v3_auth unit test to Python 3 https://review.openstack.org/312061 | 13:54 |
| *** zzzeek has quit IRC | 13:56 | |
| *** ametts has joined #openstack-keystone | 13:57 | |
| *** zzzeek has joined #openstack-keystone | 13:58 | |
| *** josecastroleon has quit IRC | 13:59 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:04 | |
| *** josecastroleon has joined #openstack-keystone | 14:05 | |
| *** sdake has joined #openstack-keystone | 14:08 | |
| *** doug-fish has joined #openstack-keystone | 14:08 | |
| *** pushkaru has joined #openstack-keystone | 14:10 | |
| lbragstad | samueldmq morning | 14:11 |
| lbragstad | samueldmq re: calling the revocation api directly instead of using notifications. Don't we do this already? | 14:14 |
| *** links has joined #openstack-keystone | 14:15 | |
| *** rbridgeman has quit IRC | 14:18 | |
| *** rbridgeman has joined #openstack-keystone | 14:18 | |
| *** phalmos has joined #openstack-keystone | 14:18 | |
| *** wxy has quit IRC | 14:20 | |
| *** stingaci has quit IRC | 14:24 | |
| *** markvoelker has joined #openstack-keystone | 14:25 | |
| *** edtubill has joined #openstack-keystone | 14:27 | |
| *** markvoelker has quit IRC | 14:30 | |
| *** woodburn1 has quit IRC | 14:30 | |
| *** jorge_munoz has joined #openstack-keystone | 14:32 | |
| *** rbridgeman_ has joined #openstack-keystone | 14:34 | |
| *** josecastroleon has quit IRC | 14:35 | |
| *** edmondsw has joined #openstack-keystone | 14:35 | |
| *** rbridgeman has quit IRC | 14:38 | |
| *** slberger has joined #openstack-keystone | 14:38 | |
| *** stingaci has joined #openstack-keystone | 14:43 | |
| *** raddaoui has joined #openstack-keystone | 14:46 | |
| ayoung | lbragstad, no | 14:47 |
| ayoung | we want that to die | 14:47 |
| lbragstad | ayoung yeah, we need to sync on what samueldmq and I found out over the weekend | 14:48 |
| lbragstad | samueldmq around? | 14:48 |
| ayoung | lbragstad, what did you find? | 14:48 |
| lbragstad | cc dstanek also wants to be a part of that conversation | 14:48 |
| lbragstad | ayoung wrapping up a meeting quick | 14:48 |
| lbragstad | ayoung are you available in 10 - 15 minutes? | 14:48 |
| dims | has anyone tried pyldap? https://review.openstack.org/#/c/315793/ | 14:48 |
| patchbot | dims: patch 315793 - requirements - Add pyldap to g-r | 14:48 |
| *** stingaci_ has joined #openstack-keystone | 14:49 | |
| ayoung | lbragstad, I'll be here | 14:49 |
| ayoung | dims, stevemar was working on it | 14:49 |
| ayoung | dims, the issue he found was that ldappool was not ported | 14:49 |
| ayoung | and we were discussing whether we could kill LDAP pool now that we are out from evently | 14:50 |
| ayoung | eventlet | 14:50 |
| ayoung | dims, but he has a WIP patch up for it | 14:50 |
| *** stingaci has quit IRC | 14:52 | |
| *** josecastroleon has joined #openstack-keystone | 14:52 | |
| openstackgerrit | Andrew Laski proposed openstack/oslo.policy: Add policy registration https://review.openstack.org/313141 | 14:52 |
| openstackgerrit | Andrew Laski proposed openstack/oslo.policy: Add authorize method to Enforcer https://review.openstack.org/313142 | 14:52 |
| *** henrynash has joined #openstack-keystone | 14:53 | |
| *** ChanServ sets mode: +v henrynash | 14:53 | |
| *** notmorgan has joined #openstack-keystone | 14:53 | |
| dims | ayoung : thanks | 14:53 |
| notmorgan | . | 14:54 |
| openstackgerrit | Andrew Laski proposed openstack/oslo.policy: Add sample file generation script https://review.openstack.org/314244 | 14:55 |
| openstackgerrit | Andrew Laski proposed openstack/oslo.policy: Add helper methods for generating policy info https://review.openstack.org/314774 | 14:55 |
| openstackgerrit | Andrew Laski proposed openstack/oslo.policy: Add __str__ to PolicyOpt https://review.openstack.org/315712 | 14:55 |
| *** jaosorior has quit IRC | 14:55 | |
| *** stingaci has joined #openstack-keystone | 14:55 | |
| lbragstad | ping dolphm dstanek ayoung samueldmq jorge_munoz i'm going to reload coffee quick and then I'll come brain dump what samueldmq and i found last week | 14:55 |
| *** jaosorior has joined #openstack-keystone | 14:55 | |
| *** timcline has joined #openstack-keystone | 14:56 | |
| *** stingaci_ has quit IRC | 14:57 | |
| *** mou has quit IRC | 15:00 | |
| *** naresht has quit IRC | 15:00 | |
| *** mou has joined #openstack-keystone | 15:01 | |
| *** ninag has quit IRC | 15:01 | |
| *** julim has quit IRC | 15:01 | |
| *** jaosorior has quit IRC | 15:01 | |
| *** jaosorior has joined #openstack-keystone | 15:02 | |
| *** itlinux has quit IRC | 15:02 | |
| *** gordc has quit IRC | 15:03 | |
| lbragstad | ayoung ok - so samueldmq submitted a patch to keystone to add logging to the revocation API https://review.openstack.org/#/q/topic:token-revocation-issue | 15:03 |
| *** julim has joined #openstack-keystone | 15:03 | |
| *** ninag has joined #openstack-keystone | 15:03 | |
| lbragstad | there are duplicates so that we would hopefully get a better failure rate | 15:03 |
| ayoung | What? | 15:03 |
| lbragstad | ayoung for some reason - no one has been able to recreate this locally | 15:04 |
| lbragstad | despite using the same devstack configs | 15:04 |
| *** ninag_ has joined #openstack-keystone | 15:04 | |
| *** tonytan4ever has quit IRC | 15:04 | |
| lbragstad | some patches were added to tempest to log the x-auth-token and some patches were added to keystone to log the revocation api | 15:05 |
| notmorgan | stevemar, lbragstad, dstanek, ayoung: https://review.openstack.org/#/c/315793/ | 15:05 |
| patchbot | notmorgan: patch 315793 - requirements - Add pyldap to g-r | 15:05 |
| *** yarkot has quit IRC | 15:05 | |
| notmorgan | this needs your eyes to land. | 15:05 |
| *** ninag__ has joined #openstack-keystone | 15:05 | |
| lbragstad | ayoung here is one of the failures | 15:06 |
| lbragstad | http://logs.openstack.org/56/316256/4/check/gate-tempest-dsvm-postgres-full/6956621/console.html#_2016-05-13_23_39_15_618 | 15:06 |
| ayoung | File Not Found | 15:06 |
| ayoung | lbragstad, one sec | 15:07 |
| *** yarkot has joined #openstack-keystone | 15:07 | |
| ayoung | http://logs.openstack.org/56/316256/4/check/gate-tempest-dsvm-postgres-full/6956621/console.html.gz#_2016-05-13_23_39_15_618 | 15:07 |
| lbragstad | ayoung yep - so this is the test that is failing https://github.com/openstack/tempest/blob/master/tempest/api/identity/admin/v2/test_roles_negative.py#L64-L73 | 15:08 |
| ayoung | raise mismatch_error | 15:08 |
| *** ninag has quit IRC | 15:08 | |
| lbragstad | yep - we get a token, delete that token, and then try to assert that we can't create a role with that token | 15:08 |
| ayoung | self.assertRaises(lib_exc.Unauthorized, | 15:08 |
| lbragstad | the assertion is failing because keystone still thinks that token is valid | 15:08 |
| ayoung | and that mismatches with the actual value | 15:09 |
| *** ninag_ has quit IRC | 15:09 | |
| lbragstad | despite the fact that keystone returns a 204 when the token is deleted | 15:09 |
| ayoung | lbragstad, I wonder if it has to do with the delete call | 15:09 |
| lbragstad | the delete call returns a 204 as expected | 15:09 |
| ayoung | some sort of interaction with the previous test | 15:09 |
| *** ninag__ has quit IRC | 15:10 | |
| *** pushkaru has quit IRC | 15:10 | |
| lbragstad | this is the request ID of the call that should have failed the role create (because the token was deleted) | 15:10 |
| ayoung | lbragstad, um, wouldn't the client be smart enough to request a new token? | 15:10 |
| lbragstad | http://pastebin.com/tRdTUA6n | 15:10 |
| *** jorge_munoz has quit IRC | 15:10 | |
| lbragstad | ayoung that's why we added logging using the x-auth-token in temepst | 15:11 |
| ayoung | lbragstad, please just merge my "replace tree with list" | 15:11 |
| ayoung | its not worth debugging the tree code | 15:11 |
| *** haplo37 has joined #openstack-keystone | 15:11 | |
| lbragstad | we wanted to make sure the clients weren't being smart about token | 15:11 |
| *** jorge_munoz has joined #openstack-keystone | 15:11 | |
| ayoung | https://review.openstack.org/#/c/311652/ | 15:11 |
| patchbot | ayoung: patch 311652 - keystone - Replace revoke tree with linear search | 15:11 |
| ayoung | seriously, madness will ensure. We are talkin Lovecraftian stuff here, if you pursue the Tree. | 15:11 |
| lbragstad | ayoung one sec | 15:11 |
| *** rbridgeman_ has quit IRC | 15:12 | |
| lbragstad | if you look at http://pastebin.com/tRdTUA6n you see that the revocation event that should have been persisted isn't in the revocation backend | 15:12 |
| *** rbridgeman_ has joined #openstack-keystone | 15:12 | |
| lbragstad | ^ that logging is the logging of the request ID for the create role call | 15:12 |
| *** itlinux has joined #openstack-keystone | 15:18 | |
| lbragstad | the revocation tree in that log should have an event that matches the token (either the user_id 4e5920ae32c048c3acc9e567cd116480 or audit_id 0sZ2HIxOSJGJw7oPC6zt6w ) | 15:18 |
| *** zqfan has joined #openstack-keystone | 15:19 | |
| lbragstad | ayoung will switching to a list still be prone to that ^ problem? | 15:20 |
| lbragstad | problem/race condition? | 15:20 |
| samueldmq | lbragstad: hi | 15:21 |
| lbragstad | samueldmq o/ | 15:21 |
| samueldmq | o/ | 15:21 |
| *** josecastroleon has quit IRC | 15:21 | |
| lbragstad | samueldmq I did some poking over the weekend | 15:22 |
| samueldmq | lbragstad: I think I have a strategy to debug this issue ... let's see what you think about it | 15:22 |
| samueldmq | lbragstad: nice | 15:22 |
| samueldmq | lbragstad: a good strategy* | 15:22 |
| ayoung | lbragstad, no idea, but it will be a heckofalot easier to debug | 15:22 |
| openstackgerrit | ayoung proposed openstack/keystone: Replace revoke tree with linear search https://review.openstack.org/311652 | 15:22 |
| lbragstad | samueldmq the only thing I can come up with is that the revocation event isn't persisted when we go to validate the token | 15:22 |
| lbragstad | * when we go to validate the "invalid" token* | 15:23 |
| samueldmq | lbragstad: what if we logged all the trace between request arriving in keystone (token delete) -> reaching the managers etc -> emitting revocation event -> registering in the tree | 15:23 |
| ayoung | samueldmq, lbragstad lets please use ^^ before we spend anymore cycles on the tree | 15:23 |
| *** richm has joined #openstack-keystone | 15:24 | |
| samueldmq | ayoung: issue is not just the tree ... it's not even reaching the tree we think | 15:24 |
| notmorgan | bknudson: ping looks like i found a bug in bandit. | 15:24 |
| notmorgan | bknudson: https://review.openstack.org/#/c/311133/10/keystoneauth1/fixture/hooks.py not detected as a mutable default. | 15:24 |
| patchbot | notmorgan: patch 311133 - keystoneauth - Use betamax hooks to mask fixture results | 15:24 |
| ayoung | samueldmq, maybe the revoke event cache needs to be invalidated? | 15:25 |
| samueldmq | ayoung: not sure ... event the call to add_event is delayed | 15:25 |
| samueldmq | ayoung: looks like it happens seconds later | 15:25 |
| bknudson | notmorgan: I thought we had a pep8 check for that? | 15:26 |
| notmorgan | bknudson: i thought it was bandit :P | 15:26 |
| samueldmq | ayoung: what I was suggesting to lbragstad was to LOG all the trace of the request (DELETE token) within keystone, so we will see where it stops/fails/whatevers | 15:26 |
| notmorgan | bknudson: in either case.. mutable default check fails there. | 15:26 |
| bknudson | notmorgan: http://git.openstack.org/cgit/openstack/keystone/tree/keystone/tests/hacking/checks.py#n68 | 15:27 |
| ayoung | samueldmq, what if delete token ran a check to ensure the token was deleted? | 15:27 |
| ayoung | internally? | 15:27 |
| bknudson | notmorgan: not sure if bandit would have a check for that... is it a security issue? | 15:28 |
| samueldmq | ayoung: not true for fernet | 15:28 |
| samueldmq | ayoung: we can't do that | 15:28 |
| bknudson | I guess it could lead to a security problem, but then so could most coding issues. | 15:28 |
| notmorgan | bknudson: mutable defaults lead that way iirc. | 15:28 |
| ayoung | samueldmq, it better be true for Fernet. | 15:29 |
| *** stingaci has quit IRC | 15:29 | |
| samueldmq | ayoung: tokens be deleted ? fernet tokens aren't stored, right? | 15:29 |
| lbragstad | right | 15:30 |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Add password table columns to meet PCI-DSS change password requirements https://review.openstack.org/314284 | 15:30 |
| notmorgan | bknudson: or we ned to fix pep8 :) | 15:30 |
| notmorgan | either wfm | 15:30 |
| *** doug-fish has quit IRC | 15:30 | |
| lbragstad | we would have to ensure the token that was deleted has the proper/matching revocation event stored in the revocation api | 15:30 |
| samueldmq | exactly | 15:32 |
| lbragstad | this isn't an issue with uuid because even though a revocation event is stored when we use uuid - we delete the token | 15:32 |
| lbragstad | so when we go to validate a uuid token, we fail on the backend/sql look up before comparing the uuid token context to the revocation api | 15:33 |
| bknudson | notmorgan: that check is only on keystone and not on keystoneauth. Might want to move it upstream. | 15:33 |
| notmorgan | bknudson: ah. | 15:34 |
| samueldmq | lbragstad: ++ so even if our revocation API code was failing, the result was the same | 15:34 |
| *** diazjf has joined #openstack-keystone | 15:37 | |
| *** pushkaru has joined #openstack-keystone | 15:38 | |
| lbragstad | ayoung you're proposing that when we do a DELETE token in keystone, we do something like: http://cdn.pasteraw.com/8a9z6lrebclm2y5jdvm9a1fe7e31tq1 | 15:39 |
| lbragstad | essentially we would be polling the revocation api until we can confirm that the revocation event is stored | 15:41 |
| lbragstad | before returning the 204 of the DELETE token call | 15:41 |
| *** fangxu has quit IRC | 15:41 | |
| ayoung | lbragstad, nope | 15:42 |
| ayoung | in the controller for delete tokens | 15:42 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/auth/controllers.py#n547 | 15:42 |
| lbragstad | ayoung ok - but same pattern right? | 15:42 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/auth/routers.py#n31 | 15:43 |
| ayoung | lbragstad, yep | 15:43 |
| ayoung | lbragstad, and if it fails...what should we do? | 15:43 |
| lbragstad | ayoung if what fails? | 15:44 |
| lbragstad | the validate call? | 15:44 |
| ayoung | lbragstad, yes | 15:44 |
| *** dmk0202 has quit IRC | 15:44 | |
| samueldmq | HTTP 5000 dump the world | 15:44 |
| ayoung | samueldmq, nope | 15:44 |
| lbragstad | so - we get a token id and say revoke this token, then we validate the token and it comes back as valid | 15:44 |
| ayoung | samueldmq, it might be legit to fail. | 15:44 |
| ayoung | lbragstad, yeah...sleep (1) ? | 15:45 |
| samueldmq | ayoung: fail to delete a token? | 15:45 |
| ayoung | we probably build a DOS mech in to Keystione if we do that, though | 15:45 |
| lbragstad | yeah | 15:45 |
| samueldmq | we just need to figure out where the workflow is broken | 15:45 |
| samueldmq | I will try to get a change to LOG all the DELETE token workflow | 15:45 |
| samueldmq | and we'll see where is the issue | 15:46 |
| lbragstad | technically - keystone returns a 204 on the DELETE token call | 15:46 |
| samueldmq | makes sense? | 15:46 |
| lbragstad | so why isn't the revocation event being stored? | 15:46 |
| samueldmq | lbragstad: yes, so it emits the token revocation event sucessfully | 15:46 |
| samueldmq | lbragstad: looks like the issue is in the revocation logic | 15:46 |
| lbragstad | samueldmq another thing | 15:47 |
| lbragstad | it could be getting stored - but maybe it isn't being pulled out properly? | 15:47 |
| *** josecastroleon has joined #openstack-keystone | 15:47 | |
| * lbragstad is grabbing at straw | 15:47 | |
| lbragstad | s | 15:47 |
| samueldmq | lbragstad: I don't think so, the logs don't show a consistent 'Persisting event .. ' thing | 15:48 |
| lbragstad | oh - good point | 15:48 |
| *** rbridgeman_ has quit IRC | 15:50 | |
| samueldmq | lbragstad: ayoung: hmmmm, I found something interesting | 15:51 |
| *** gordc has joined #openstack-keystone | 15:51 | |
| samueldmq | lbragstad: ayoung https://github.com/openstack/keystone/search?utf8=%E2%9C%93&q=invalidate_individual_token_cache | 15:52 |
| samueldmq | the call to invalidate_individual_token_cache in the token provider only happens in the persist layer | 15:53 |
| samueldmq | which is something that isn't called for fernet, so invalidate_individual_token_cache is never called for fernet ? | 15:53 |
| samueldmq | does this make sense? ^ | 15:54 |
| lbragstad | samueldmq that's strange | 15:54 |
| lbragstad | cc notmorgan ^ | 15:54 |
| *** henrynash has quit IRC | 15:54 | |
| lbragstad | I thought I remember seeing that last week | 15:54 |
| *** sheel has joined #openstack-keystone | 15:56 | |
| samueldmq | lbragstad: you agree a cache issue may be the cause? | 16:00 |
| *** furface has joined #openstack-keystone | 16:00 | |
| lbragstad | samueldmq it could be | 16:01 |
| lbragstad | samueldmq but if the cache was never being invalidated for fernet - how would it work sometimes but not others? | 16:01 |
| samueldmq | lbragstad: does fernet need token/persistence/core.py at all? | 16:01 |
| lbragstad | samueldmq no | 16:02 |
| *** henrynash has joined #openstack-keystone | 16:02 | |
| *** ChanServ sets mode: +v henrynash | 16:02 | |
| lbragstad | samueldmq we have logic in the token provider to explicitly ignore persistence if using fernet | 16:03 |
| samueldmq | lbragstad: could you point me to the code where a fernet token is "deleted"? i.e the revocation notification is sent? | 16:04 |
| samueldmq | lbragstad: also to where a fernet tken is validated ? | 16:04 |
| *** julim has quit IRC | 16:04 | |
| lbragstad | samueldmq this is where a token is revoked - https://github.com/openstack/keystone/blob/996b0c7e1019235d384648f74de24542b0a736d7/keystone/token/provider.py#L431 | 16:04 |
| lbragstad | samueldmq that *all* happens in the keystone.token.provider.py module | 16:05 |
| lbragstad | and a revoke token call doesn't go down to the provider in the fernet case | 16:05 |
| *** stingaci has joined #openstack-keystone | 16:05 | |
| *** julim has joined #openstack-keystone | 16:06 | |
| samueldmq | lbragstad: so what I said is true, invalidate_individual_token_cache is never called in fernet case | 16:07 |
| samueldmq | lbragstad: https://github.com/openstack/keystone/blob/996b0c7e1019235d384648f74de24542b0a736d7/keystone/token/provider.py#L223 and | 16:08 |
| samueldmq | lbragstad: https://github.com/openstack/keystone/blob/996b0c7e1019235d384648f74de24542b0a736d7/keystone/token/provider.py#L267 | 16:08 |
| samueldmq | lbragstad: validate fernet tokens right ? | 16:08 |
| lbragstad | samueldmq yes - but there is also a validate_token method | 16:08 |
| *** gyee has joined #openstack-keystone | 16:09 | |
| *** ChanServ sets mode: +v gyee | 16:09 | |
| lbragstad | samueldmq https://github.com/openstack/keystone/blob/996b0c7e1019235d384648f74de24542b0a736d7/keystone/token/provider.py#L204 | 16:09 |
| *** rderose has quit IRC | 16:09 | |
| lbragstad | which attempts to figure out which version token you have | 16:09 |
| lbragstad | and validates it accordingly | 16:09 |
| samueldmq | lbragstad: will take a look, thx | 16:10 |
| *** ninag has joined #openstack-keystone | 16:12 | |
| *** ninag has quit IRC | 16:12 | |
| *** henrynash has quit IRC | 16:13 | |
| *** josecastroleon has quit IRC | 16:16 | |
| *** ninag has joined #openstack-keystone | 16:17 | |
| *** ninag has quit IRC | 16:18 | |
| lbragstad | samueldmq should we move https://github.com/openstack/keystone/blob/f7b33213f1cb8313d2cb81225e8530ebbc37ce18/keystone/token/persistence/core.py#L193 in to the token_provider? | 16:21 |
| samueldmq | lbragstad: I can't find where the fernet token is checked against the revocation tree at all | 16:21 |
| samueldmq | lbragstad: I think so | 16:22 |
| *** rderose has joined #openstack-keystone | 16:22 | |
| *** julim has quit IRC | 16:23 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystoneauth: Updated from global requirements https://review.openstack.org/316978 | 16:25 |
| *** agrebennikov has joined #openstack-keystone | 16:26 | |
| *** lhcheng has joined #openstack-keystone | 16:27 | |
| *** ChanServ sets mode: +v lhcheng | 16:27 | |
| lbragstad | samueldmq it is in keystone/token/provider.py | 16:29 |
| lbragstad | samueldmq https://github.com/openstack/keystone/blob/996b0c7e1019235d384648f74de24542b0a736d7/keystone/token/provider.py#L260-L265 | 16:30 |
| samueldmq | lbragstad: oh I think that is the issue | 16:30 |
| *** josecastroleon has joined #openstack-keystone | 16:30 | |
| samueldmq | lbragstad: https://github.com/openstack/keystone/blob/996b0c7e1019235d384648f74de24542b0a736d7/keystone/token/provider.py#L204 is the starting point to validate tokens | 16:30 |
| samueldmq | lbragstad: and it calls self._validate_token, which is cached! | 16:31 |
| *** darosale has quit IRC | 16:31 | |
| samueldmq | lbragstad: but _is_valid_token | 16:31 |
| samueldmq | lbragstad: is the part that cehcks against the revoction tree... | 16:32 |
| lbragstad | eyah | 16:32 |
| notmorgan | Yeah. For best caching we had to split the two things. | 16:34 |
| *** julim has joined #openstack-keystone | 16:34 | |
| *** TxGVNN has joined #openstack-keystone | 16:38 | |
| lbragstad | notmorgan for best caching? | 16:39 |
| notmorgan | Yes, so we didn't need to invalidate as much. | 16:40 |
| notmorgan | Invalidations get expensive the more you need to do them since they force cache misses. | 16:40 |
| lbragstad | samueldmq I could see the issue we're seeing a cache invalidation problem over a revocation event persistence problem | 16:40 |
| *** timcline_ has joined #openstack-keystone | 16:41 | |
| notmorgan | So checking the revoke tree is independent of looking up the token data. | 16:41 |
| notmorgan | And it should be. | 16:41 |
| *** fangxu has joined #openstack-keystone | 16:41 | |
| *** mou has quit IRC | 16:42 | |
| *** timcline has quit IRC | 16:42 | |
| amakarov | hi! Can anybody explain why keystone rewrites project ID on creation for v3 and does not for v2.0? | 16:43 |
| agrebennikov | I'd even ask it in a bit different way - v2 allows to specify project ID upon creation, while v3 strictly generates it | 16:44 |
| agrebennikov | is there a reason behind it? | 16:44 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ldappool: Updated from global requirements https://review.openstack.org/316985 | 16:45 |
| *** links has quit IRC | 16:47 | |
| *** dan_nguyen has joined #openstack-keystone | 16:51 | |
| *** diazjf has quit IRC | 16:53 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack/keystone: Invalidate token cache after token delete https://review.openstack.org/316991 | 16:55 |
| samueldmq | lbragstad: ^ | 16:55 |
| lbragstad | samueldmq sweet | 16:57 |
| lbragstad | samueldmq I wonder if we should base all the other changes on that to see if we see *less* occurrences of bug 1578866 | 16:57 |
| openstack | bug 1578866 in OpenStack Identity (keystone) "test_user_update_own_password failing intermittently" [High,In progress] https://launchpad.net/bugs/1578866 - Assigned to Lance Bragstad (lbragstad) | 16:57 |
| samueldmq | lbragstad: ++ | 17:00 |
| samueldmq | lbragstad: I will put depends-on on the temepst patches | 17:01 |
| samueldmq | lbragstad: they're failing more often | 17:01 |
| lbragstad | samueldmq awesome | 17:01 |
| ayoung | amakarov, "rewrites?" | 17:01 |
| amakarov | ayoung, unconditionally generates new ID | 17:02 |
| *** jistr has quit IRC | 17:02 | |
| ayoung | amakarov, V2 wsas not doing that, or just the driver was not? | 17:02 |
| amakarov | ayoung, https://github.com/openstack/keystone/blob/master/keystone/resource/controllers.py#L87 vs https://github.com/openstack/keystone/blob/master/keystone/resource/controllers.py#L240 | 17:03 |
| *** furface has quit IRC | 17:03 | |
| *** josecastroleon has quit IRC | 17:03 | |
| ayoung | amakarov, that is in the controller, hidden from the end user of the web api, though | 17:04 |
| ayoung | amakarov, for v2 we generate the id on line 102 | 17:04 |
| ayoung | https://github.com/openstack/keystone/blob/master/keystone/resource/controllers.py#L102 | 17:05 |
| amakarov | ayoung, right but only if it isn't in tenant_ref | 17:05 |
| ayoung | amakarov, that might be a bug | 17:06 |
| amakarov | on line 240 existing id is just ignored | 17:06 |
| amakarov | ayoung, or a feature ) | 17:06 |
| ayoung | amakarov, well, I would consider it a feature. I filed a feature request for it. | 17:06 |
| *** stingaci has quit IRC | 17:06 | |
| ayoung | amakarov, https://review.openstack.org/#/c/203852/ | 17:07 |
| patchbot | ayoung: patch 203852 - keystone - Specify ID for Project or domain creation (ABANDONED) | 17:07 |
| *** pnavarro has joined #openstack-keystone | 17:07 | |
| ayoung | aas you can see, It was denied | 17:07 |
| *** stingaci has joined #openstack-keystone | 17:07 | |
| amakarov | agrebennikov, ^^ | 17:07 |
| *** doug-fish has joined #openstack-keystone | 17:08 | |
| *** ninag has joined #openstack-keystone | 17:10 | |
| *** doug-fis_ has joined #openstack-keystone | 17:10 | |
| amakarov | ayoung, as I see all the objections was about spec absence | 17:10 |
| ayoung | amakarov, not quite | 17:10 |
| ayoung | his point was that he wanted it as a service admin type operations | 17:10 |
| *** ninag_ has joined #openstack-keystone | 17:11 | |
| *** ninag_ has quit IRC | 17:11 | |
| amakarov | ayoung, looks like a thing to discuss tomorrow | 17:12 |
| *** doug-fi__ has joined #openstack-keystone | 17:12 | |
| ayoung | amakarov, please. | 17:12 |
| agrebennikov | ayoung, so technically if I apply your patch to my current deployment it will not be a dirty hack then? ;) | 17:12 |
| *** david-lyle_ has quit IRC | 17:12 | |
| agrebennikov | if you guys negotiated on pushing it forward | 17:13 |
| *** doug-fish has quit IRC | 17:13 | |
| *** david-lyle has joined #openstack-keystone | 17:13 | |
| *** doug-fis_ has quit IRC | 17:14 | |
| *** ninag has quit IRC | 17:15 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: WIP - PCI-DSS 8.2.4: User must change their password requirements https://review.openstack.org/317007 | 17:15 |
| *** doug-fi__ has quit IRC | 17:16 | |
| *** real56 has joined #openstack-keystone | 17:17 | |
| *** tonytan4ever has joined #openstack-keystone | 17:21 | |
| *** ninag has joined #openstack-keystone | 17:23 | |
| notmorgan | ayoung: meeting invite thing you forwarded didn't end up on my calendar :( sorry | 17:24 |
| *** josecastroleon has joined #openstack-keystone | 17:24 | |
| notmorgan | ayoung: it got lost somewhere in the zimbra -> google move. | 17:24 |
| ayoung | notmorgan, not a big deal. We just touched on the CI issue for a minute at the end, but there is other Keystone issues we discussed that you are full aware of. | 17:25 |
| *** dims_ has joined #openstack-keystone | 17:25 | |
| ayoung | Fernet, revoke events, python3/ldap etc | 17:25 |
| notmorgan | okie | 17:25 |
| notmorgan | ayoung: did you see my ping re the review to add pyldap to g-r? | 17:26 |
| ayoung | notmorgan, yes, and dims came by earlier askin about it | 17:26 |
| *** dims has quit IRC | 17:26 | |
| notmorgan | ayoung: ok, so that is going to hold up ldappool py3 stuff. | 17:26 |
| ayoung | notmorgan, I think we can kill ldappool and just drive on with pyldap | 17:26 |
| notmorgan | ayoung: lets just do a release of ldapool that is py3 compat and look at extricating it from keystone down the line. | 17:27 |
| notmorgan | really review if we can remove it. | 17:27 |
| dstanek | uggg...it looks like dogpile doesn't allow expiration to be provided to the set() :-( | 17:28 |
| notmorgan | but it will be a lot shorter to do pyldap+ldappool in keystone ot hit py3 | 17:28 |
| notmorgan | dstanek: nope, configured at the region level | 17:28 |
| stevemar | notmorgan: yes it will | 17:28 |
| notmorgan | ayoung: and we need to land pyldap in g-r *anyway* | 17:28 |
| notmorgan | unless we're only going with ldap3. | 17:28 |
| notmorgan | and the feeling i got was... that is going to be a lot of discussion | 17:29 |
| notmorgan | even though i'd much prefer it. | 17:29 |
| dstanek | notmorgan: super not cool! | 17:29 |
| *** anush has joined #openstack-keystone | 17:30 | |
| *** anush has quit IRC | 17:32 | |
| *** doug-fish has joined #openstack-keystone | 17:32 | |
| *** anush has joined #openstack-keystone | 17:33 | |
| *** mvk has quit IRC | 17:33 | |
| ayoung | notmorgan, I thought we were going for pyldap short term, ldap3 longer term. | 17:33 |
| *** TxGVNN has quit IRC | 17:33 | |
| notmorgan | ayoung: doesn't matter what way we go | 17:33 |
| notmorgan | ayoung: justsaying we need to land pyldap in g-r unless we only did ldap3 | 17:33 |
| ayoung | notmorgan, agreed, and I did add a note to that effect in the review just now | 17:34 |
| ayoung | stevemar, did you test pyldap with ldappool removed? | 17:34 |
| dstanek | notmorgan: once i get this capstone stuff pushed i'll be looking at the reviews you posted | 17:34 |
| notmorgan | ayoung: ldappool has a fix to use pyldap and pending a release as soon as the g-r lands fwiw | 17:35 |
| notmorgan | ayoung: since we had it transferred to us. | 17:36 |
| *** doug-fish has quit IRC | 17:38 | |
| *** anush has quit IRC | 17:43 | |
| ayoung | notmorgan, that works, too, but I wonder if we need it? | 17:43 |
| *** anush has joined #openstack-keystone | 17:45 | |
| *** rcernin has quit IRC | 17:47 | |
| *** pnavarro has quit IRC | 17:48 | |
| *** doug-fish has joined #openstack-keystone | 17:53 | |
| *** anush has quit IRC | 17:55 | |
| *** doug-fish has quit IRC | 17:58 | |
| *** josecastroleon has quit IRC | 17:58 | |
| *** doug-fish has joined #openstack-keystone | 17:59 | |
| *** josecastroleon has joined #openstack-keystone | 18:01 | |
| *** roxanaghe has quit IRC | 18:02 | |
| *** anush has joined #openstack-keystone | 18:03 | |
| *** julim has quit IRC | 18:07 | |
| *** roxanaghe has joined #openstack-keystone | 18:07 | |
| *** julim has joined #openstack-keystone | 18:08 | |
| *** diazjf has joined #openstack-keystone | 18:10 | |
| *** ninag has quit IRC | 18:13 | |
| *** ninag has joined #openstack-keystone | 18:13 | |
| *** ninag has quit IRC | 18:14 | |
| *** ninag has joined #openstack-keystone | 18:14 | |
| *** doug-fis_ has joined #openstack-keystone | 18:15 | |
| *** doug-fish has quit IRC | 18:18 | |
| *** doug-fis_ has quit IRC | 18:21 | |
| *** doug-fish has joined #openstack-keystone | 18:21 | |
| *** real56 has quit IRC | 18:23 | |
| *** ninag has quit IRC | 18:24 | |
| *** doug-fish has quit IRC | 18:26 | |
| *** josecastroleon has quit IRC | 18:30 | |
| *** ninag has joined #openstack-keystone | 18:36 | |
| *** ninag has quit IRC | 18:36 | |
| *** ninag has joined #openstack-keystone | 18:37 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Move the oauth1 abstract base class out of core https://review.openstack.org/317045 | 18:38 |
| *** josecastroleon has joined #openstack-keystone | 18:41 | |
| *** ninag has quit IRC | 18:42 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Move the oauth1 abstract base class out of core https://review.openstack.org/317045 | 18:44 |
| *** stingaci has quit IRC | 18:47 | |
| lbragstad | notmorgan i think i missed it, but was there a reason for not doing https://github.com/openstack/keystone/blob/f7b33213f1cb8313d2cb81225e8530ebbc37ce18/keystone/token/persistence/core.py#L193 in the provider.py module? | 18:53 |
| notmorgan | lbragstad: it was in persistence already -- why are you needing to explicitly invalidate on revoke? | 18:54 |
| lbragstad | notmorgan just thinking about the race condition we're seeing in the gate and trying to rule that out as a possibility | 18:54 |
| notmorgan | the race is "token validates, but revoke event has been issued"? | 18:55 |
| notmorgan | and is the race against the keystone api? | 18:55 |
| notmorgan | or against a service api | 18:55 |
| lbragstad | notmorgan a revocation event has been issued and tempest goes to validate the token against keystone, expecting a 401, but gets a valid authentication response instead | 18:55 |
| notmorgan | lbragstad: if you validate that "successful" token again, does it still stay valid? | 18:56 |
| notmorgan | like does a sleep(1) after the revoke is issued solve it | 18:56 |
| *** rbridgeman has joined #openstack-keystone | 18:56 | |
| lbragstad | notmorgan this workflow https://github.com/openstack/tempest/blob/c69d06dbb4c14eec305ed6f0db947af77427c04c/tempest/api/identity/admin/v2/test_roles_negative.py#L66-L73 | 18:56 |
| notmorgan | [i am assuming not] | 18:56 |
| lbragstad | at line 69 tempest does a DELETE token | 18:57 |
| notmorgan | and it intermittantly still validates | 18:57 |
| lbragstad | and at line 72 they attempt to do something with the token they just deleted and that operation is successful | 18:57 |
| lbragstad | yes | 18:57 |
| lbragstad | notmorgan that led us to adding the logging for the revocation api | 18:58 |
| lbragstad | we hit the same failure with the logs (which logs the entire revocation tree) | 18:58 |
| lbragstad | and noticed that when the request comes in to do some operation that should be invalid - the revocation isn't matching anything because the revocation event isn't ther e | 18:59 |
| lbragstad | notmorgan I have an example | 18:59 |
| *** anush has quit IRC | 18:59 | |
| lbragstad | notmorgan here we have a failure http://logs.openstack.org/56/316256/4/check/gate-tempest-dsvm-postgres-full/6956621/console.html.gz#_2016-05-13_23_39_15_618 | 18:59 |
| *** ninag has joined #openstack-keystone | 18:59 | |
| lbragstad | keystone should have returned a 401 for that token | 18:59 |
| *** ninag has quit IRC | 18:59 | |
| lbragstad | because it has been revoked | 19:00 |
| *** doug-fis_ has joined #openstack-keystone | 19:00 | |
| *** ninag has joined #openstack-keystone | 19:00 | |
| notmorgan | but the revoke event for that token doesn't exist? | 19:00 |
| lbragstad | this is the logging of the request ID of the create role call http://pastebin.com/tRdTUA6n | 19:00 |
| lbragstad | if you search for the audit_id or user_id of the token that should be revoked, you don't see it in the revocation tree | 19:00 |
| notmorgan | oh wow | 19:02 |
| notmorgan | that is a bug. | 19:02 |
| lbragstad | notmorgan absolutely - but it's also a race condition | 19:02 |
| notmorgan | i think. | 19:02 |
| *** edmondsw has quit IRC | 19:02 | |
| lbragstad | so - how is it that it sometimes passes and sometimes fails? | 19:02 |
| *** doug-fis_ has quit IRC | 19:02 | |
| notmorgan | so i think i know exactly why | 19:02 |
| notmorgan | i'm looking to try and chase it down though. | 19:03 |
| lbragstad | we return a 204 on the DELETE token call, which should mean that we successfully stored a revocation event | 19:03 |
| notmorgan | doesn't mean we saved a revoke event. | 19:03 |
| notmorgan | means we claim we have ;) | 19:03 |
| lbragstad | notmorgan do we cache revocation events? | 19:03 |
| *** woodster_ has joined #openstack-keystone | 19:03 | |
| notmorgan | we do | 19:03 |
| lbragstad | we also cache tokens. | 19:03 |
| notmorgan | which should be fine | 19:04 |
| lbragstad | which is what I think samueldmq was getting to with https://github.com/openstack/keystone/blob/f7b33213f1cb8313d2cb81225e8530ebbc37ce18/keystone/token/persistence/core.py#L193 ? | 19:04 |
| notmorgan | caching a token is 100% ok. in fact, not caching the token would have no impact but to make it a bit slower to validate | 19:04 |
| notmorgan | because with fernet -- we would rebuild the token | 19:04 |
| *** tqtran has joined #openstack-keystone | 19:04 | |
| notmorgan | invalidating the token cache isn't super relevant, with UUID/PKI we *have* to invalidate the token validate because we lean on the revocation_list (a column in the token table) as a way to check validity | 19:05 |
| lbragstad | notmorgan it seems we could be hitting one of two things. We are caching the revocation events and not invalidating that cache when a new revocation event is added. Or we are caching tokens and not invalidating that cache when a new revocation event is added (but I doubt that case). | 19:07 |
| *** josecastroleon has quit IRC | 19:13 | |
| *** rderose has quit IRC | 19:19 | |
| *** mdurrant has joined #openstack-keystone | 19:23 | |
| mdurrant | I'm trying to find an outstanding bug on the current devstack-killing oslo.config 3.9.0 issue... | 19:25 |
| mdurrant | keystone-middleware 4.4.0 installation bombs and complains that it can't find oslo.config 3.9.0 | 19:26 |
| *** tonytan4ever has quit IRC | 19:27 | |
| *** josecastroleon has joined #openstack-keystone | 19:31 | |
| *** rderose has joined #openstack-keystone | 19:33 | |
| lbragstad | notmorgan it sounded like you had a suggestion but I don't think I got it? | 19:34 |
| notmorgan | lbragstad: still looking at code | 19:34 |
| *** rcernin has joined #openstack-keystone | 19:42 | |
| samueldmq | lbragstad: we came to a conclusion that, as persisted tokens were effectivelly deleted, everything was fine for UUID | 19:43 |
| notmorgan | this is almost 100% sure to be an issue with revocation events | 19:44 |
| samueldmq | lbragstad: but ... the error was also happening in password change | 19:44 |
| notmorgan | either the storing or caching of them. | 19:44 |
| samueldmq | notmorgan: agreed | 19:44 |
| notmorgan | so lets not look at token caching unless we eliminate rev. event issues 100% | 19:44 |
| *** edmondsw has joined #openstack-keystone | 19:44 | |
| samueldmq | notmorgan: we were loggingthe calls to add_event in the manager | 19:44 |
| samueldmq | notmorgan: and even them were delayed | 19:44 |
| *** roxanaghe has quit IRC | 19:44 | |
| samueldmq | notmorgan: maybe it's something wrong in the notification system? | 19:45 |
| notmorgan | unlikely | 19:45 |
| *** sheel has quit IRC | 19:45 | |
| notmorgan | because you can't return 204 unless the notification passes internally | 19:45 |
| notmorgan | it's a blocking call | 19:45 |
| notmorgan | the whole delete process is a blocking call | 19:45 |
| notmorgan | unless tempest isn't waiting for the delete to finish - which would be a bigger issue. | 19:46 |
| notmorgan | and the SQL backends wont ACK until it's been stored | 19:46 |
| *** doug-fish has joined #openstack-keystone | 19:46 | |
| notmorgan | the other question | 19:46 |
| notmorgan | with https://review.openstack.org/#/c/311652/ does this continue to fail | 19:47 |
| patchbot | notmorgan: patch 311652 - keystone - Replace revoke tree with linear search | 19:47 |
| lbragstad | we have seen this issue with revocation events that have been sent via the notification system as well as using the revoke api directly | 19:47 |
| samueldmq | notmorgan: maybe not ... this is what ayoung was talking earlier, maybe he fixed it with his refactoring ? | 19:47 |
| ayoung | notmorgan, could it be a cache issue? | 19:47 |
| lbragstad | i'm not sure that https://review.openstack.org/#/c/311652/ does anything with the storing of the event | 19:47 |
| patchbot | lbragstad: patch 311652 - keystone - Replace revoke tree with linear search | 19:47 |
| notmorgan | ayoung: possibly it is also just as likely to be an issue with the whole magic bundle of things in the tree | 19:48 |
| lbragstad | it removes the caching though | 19:48 |
| ayoung | can we just merge that, you know, because it is sane, as compared to the tree, which is clown posse levels of not sane? | 19:48 |
| notmorgan | ayoung: where the tree somewhere identifies two events to be the same when thehy aren't | 19:48 |
| notmorgan | ayoung: that patch should still cache the events coming out of the backend fwiw. | 19:50 |
| notmorgan | just not the whole built tree | 19:50 |
| ayoung | notmorgan, I thought the backend cached it already | 19:50 |
| notmorgan | no | 19:50 |
| notmorgan | caching is at the manager layer | 19:50 |
| ayoung | calling through the driver does not cache? | 19:50 |
| notmorgan | always | 19:50 |
| ayoung | er, manager | 19:50 |
| notmorgan | unless you have @memoize, nope | 19:50 |
| notmorgan | and you call self.driver.list_events() | 19:51 |
| lbragstad | the @MEMOIZE decorator is removed | 19:51 |
| notmorgan | so you're explicitly circumventing the manager | 19:51 |
| notmorgan | and eliminating caching | 19:51 |
| ayoung | I don't think that is different than what was doing before, which again leads me to think it is a cache problem | 19:52 |
| ayoung | we were memo-izing the tree | 19:53 |
| notmorgan | and the tree did magic matching on events | 19:53 |
| notmorgan | to see if they were duplicated | 19:53 |
| *** tonytan4ever has joined #openstack-keystone | 19:53 | |
| notmorgan | vs a strict run through, iirc. | 19:53 |
| notmorgan | this new code does a strict run through the event list | 19:53 |
| lbragstad | that would make sense - because the new revocation event wasn't in the tree | 19:54 |
| lbragstad | in the failures we've been seeing | 19:54 |
| ayoung | notmorgan, OK, so need a memoize, with the cache invalidation on each revoke call again | 19:54 |
| notmorgan | yep | 19:54 |
| notmorgan | basically create .list_events on the manader that just calls driver.list_events() | 19:54 |
| notmorgan | @memoize it | 19:54 |
| notmorgan | and make the invalidate happen on revoke but for self.list_events | 19:54 |
| notmorgan | instead of self._get_revoke_tree | 19:55 |
| notmorgan | there is future optimizations to be done there. | 19:55 |
| *** stingaci has joined #openstack-keystone | 19:56 | |
| *** jaosorior has quit IRC | 19:56 | |
| notmorgan | but that is the first step in the right direction, eventually we'll just push the "is_revoked" down to the driver. and the SQL driver will lean on the RDBMS to search if the token values match. | 19:56 |
| lbragstad | when we list events in the revocation api - is it possible to index on user id or something like that? | 19:56 |
| notmorgan | lbragstad: the next step is to not need to pull the event list into ram | 19:57 |
| lbragstad | notmorgan that's pretty much what you just described ^ | 19:57 |
| notmorgan | lbragstad: but we'll need to index the columns | 19:57 |
| lbragstad | notmorgan so making it so that we can ask the backend for a specific set of events | 19:57 |
| lbragstad | instead of "get me all the haystacks so I can look for a one needle" | 19:58 |
| notmorgan | pretty much don't even ask for "certain" events | 19:58 |
| ayoung | notmorgan, easy enough. Testing now | 19:58 |
| notmorgan | just ask RDBMS "hey does an event match any of these things?!" | 19:58 |
| ayoung | TypeError: No serialization handler registered for type 'RevokeEvent | 19:58 |
| ayoung | notmorgan, we need to do that as JSON, right...some other patch you had? | 19:58 |
| notmorgan | ayoung: well, sortof.. we can't strictly do that because we assume everyting is datetime objects not strs | 19:59 |
| notmorgan | ayoung: and json does datetime->str | 19:59 |
| *** doug-fis_ has joined #openstack-keystone | 20:00 | |
| ayoung | dforget caching. Who needs caching? | 20:00 |
| *** BjoernT has joined #openstack-keystone | 20:00 | |
| notmorgan | ayoung: see https://review.openstack.org/#/c/314188/ errors | 20:00 |
| patchbot | notmorgan: patch 314188 - keystone - Change to use json instead of msgpack in request_l... | 20:00 |
| ayoung | notmorgan, so, back to msgpack? | 20:01 |
| dolphm | dstanek: you don't have a review up for your pysaml work, do you? | 20:01 |
| notmorgan | basically you'll need to kep w/ msgpack for now. | 20:01 |
| *** doug-fi__ has joined #openstack-keystone | 20:01 | |
| *** josecastroleon has quit IRC | 20:01 | |
| *** phalmos has quit IRC | 20:01 | |
| *** doug-f___ has joined #openstack-keystone | 20:02 | |
| *** ninag has quit IRC | 20:03 | |
| *** phalmos has joined #openstack-keystone | 20:04 | |
| *** doug-fish has quit IRC | 20:04 | |
| notmorgan | ayoung: i'll bet fixing http://logs.openstack.org/88/314188/1/check/gate-keystone-python27-db/b5e6607/console.html.gz#_2016-05-09_16_18_45_242 is all that is needed to make the move to json | 20:04 |
| *** doug-fis_ has quit IRC | 20:05 | |
| *** doug-fi__ has quit IRC | 20:05 | |
| ayoung | File "/home/jenkins/workspace/gate-keystone-python27-db/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/timeutils.py", line 102, in normalize_time | 20:06 |
| ayoung | 2016-05-09 16:17:44.477 | offset = timestamp.utcoffset() | 20:06 |
| ayoung | oslo issue? | 20:06 |
| notmorgan | no, we assume the thing is a DATETIME object | 20:06 |
| notmorgan | json serialization makes the datetime object into a string | 20:06 |
| *** ninag_ has joined #openstack-keystone | 20:08 | |
| *** max__ has joined #openstack-keystone | 20:08 | |
| *** max__ is now known as Guest53289 | 20:09 | |
| *** ninag_ has quit IRC | 20:09 | |
| ayoung | notmorgan, where does the fix need to go? | 20:09 |
| notmorgan | in _is_valid i think | 20:10 |
| notmorgan | make sure expires is a DATETIME object | 20:10 |
| notmorgan | (for now) - future looking, fix keystone to not expect everything is a datetime object | 20:10 |
| *** ninag has joined #openstack-keystone | 20:11 | |
| openstackgerrit | Merged openstack/keystoneauth: Updated from global requirements https://review.openstack.org/316978 | 20:11 |
| samueldmq | ayoung: so you basically put the revocation events in a list, and see if any element in the list match with the info from the token? | 20:12 |
| ayoung | samueldmq, yes | 20:12 |
| samueldmq | ayoung: nice, and the idea is to (in the future) make the match in the database query right? | 20:12 |
| ayoung | samueldmq, no. | 20:13 |
| notmorgan | samueldmq: yes | 20:13 |
| *** ninag has quit IRC | 20:13 | |
| samueldmq | ? | 20:13 |
| samueldmq | :) | 20:13 |
| ayoung | I don't want to hit the database | 20:13 |
| notmorgan | ayoung: this should hit the database instead. | 20:13 |
| notmorgan | it will be *WAY* faster | 20:13 |
| notmorgan | this is the kind of thing DBs are good at | 20:13 |
| ayoung | notmorgan, hmmm | 20:13 |
| notmorgan | but i figured we'd iterate on that next | 20:14 |
| notmorgan | since it was an interface change for the driver | 20:14 |
| lbragstad | if it's a db read it should be pretty quick | 20:14 |
| samueldmq | I tend to agree with notmorgan, and the code will also be even easier to understand | 20:14 |
| *** dmk0202 has joined #openstack-keystone | 20:14 | |
| ayoung | let me mull it over | 20:14 |
| samueldmq | basically, store revocations and, when a token arrives, check against the db | 20:14 |
| samueldmq | that means much less logic in the managers ,etc | 20:15 |
| notmorgan | ayoung: select event from db where user_id=($USER_ID) or audit_it=($AUDIT_ID) | 20:15 |
| ayoung | before we tackle that, we should reduce the spurious events | 20:15 |
| lbragstad | ayoung ++ agree | 20:15 |
| notmorgan | with a few other permutations | 20:15 |
| notmorgan | the way i see it is: liniar search, reduce event types, ask DB directly | 20:15 |
| samueldmq | I could test ayoung's patch against my tempest changes | 20:15 |
| samueldmq | but looks like some people don't agree with how I am trying to reproduce the race condition | 20:16 |
| samueldmq | :B | 20:16 |
| *** roxanaghe has joined #openstack-keystone | 20:16 | |
| ayoung | notmorgan, so, my gut says that, with fewer revoke events, it is not going to be worth hitting the DB. We can try it, but I'd suspect that in process would beat out of process, especially for really short list, and mfisch's data was that, already, we don't have many revoke events | 20:16 |
| *** ninag has joined #openstack-keystone | 20:16 | |
| mfisch | just keep them in memory? | 20:16 |
| mfisch | memcache? | 20:17 |
| notmorgan | ayoung: i'm going to go a step further and say it will be because we don't have to do things in memory/python | 20:17 |
| notmorgan | which will be slower | 20:17 |
| mfisch | how would you distribute them? | 20:17 |
| ayoung | mfisch, cached on each server | 20:17 |
| mfisch | anyway I need to drive to the mechanic I will catch up when I get back | 20:17 |
| ayoung | changes still write trhough to the datqabase | 20:17 |
| ayoung | but...then we have the issue of cache revocation | 20:17 |
| notmorgan | being able to ask quickly on an indexed table if a token is valid is going to win over caching in most cases. | 20:18 |
| ayoung | notmorgan, I'm wondering then if we should do that now | 20:18 |
| notmorgan | ayoung: we could, it's a lot more work with the volume of types we have | 20:18 |
| samueldmq | ayoung: you addressing morgan's comment on patch 311652 already? | 20:18 |
| patchbot | samueldmq: https://review.openstack.org/#/c/311652/ - keystone - Replace revoke tree with linear search | 20:18 |
| lbragstad | ayoung testing your patch with http://cdn.pasteraw.com/enp4yjepsw08lgf5kzfuq5zg0qteyl6 locally | 20:18 |
| notmorgan | and i wasn't sure we'd land it in newton. | 20:18 |
| ayoung | but it would be a pain with the larger set of events | 20:18 |
| ayoung | lbragstad, cool | 20:19 |
| samueldmq | ayoung: I will take a look at it now too ... let's get that merged and see if it makes things easier | 20:19 |
| lbragstad | samueldmq ayoung http://cdn.pasteraw.com/enp4yjepsw08lgf5kzfuq5zg0qteyl6 is addresses some of notmorgan's comments | 20:19 |
| notmorgan | ayoung: i mean, i already know what the SQL driver ends up looking like. it's just a bunch of filter ors for the different columns | 20:19 |
| ayoung | lbragstad, necessary but not sufficient | 20:19 |
| samueldmq | lbragstad: could you post it as a new patchset? | 20:20 |
| notmorgan | and if anyone column matches - AND the rev. time matches, boom | 20:20 |
| ayoung | I also need to code I wriopped out of _cointexct_cache but modified for the events instead of the map | 20:20 |
| lbragstad | samueldmq i think ayoung has some changes locally too - i don't want to stomp on them | 20:20 |
| ayoung | lbragstad, samueldmq yeah, don't post. I'm working on it | 20:20 |
| *** ninag has quit IRC | 20:20 | |
| *** josecastroleon has joined #openstack-keystone | 20:20 | |
| notmorgan | but basically we could just *delete* the revoke model. | 20:20 |
| *** rderose has quit IRC | 20:21 | |
| notmorgan | and lean on SQL. | 20:21 |
| notmorgan | requires a migration to index the table | 20:21 |
| *** ninag has joined #openstack-keystone | 20:21 | |
| lbragstad | notmorgan that would make sense if we didn't have many revocation cases | 20:21 |
| lbragstad | i.e. once we remove all the extra events we get for free from the fernet format | 20:21 |
| notmorgan | lbragstad: pretty easy to do even now. | 20:21 |
| notmorgan | lbragstad: just would rather make the .is_revoked() change be smarter | 20:22 |
| notmorgan | and just ask for the values it needs rather than **kwargs | 20:22 |
| notmorgan | and two interface changes is ickier than 1 | 20:22 |
| ayoung | OK, think I've got it. | 20:22 |
| dstanek | dolphm: no, but i can pretty soon | 20:23 |
| openstackgerrit | ayoung proposed openstack/keystone: Replace revoke tree with linear search https://review.openstack.org/311652 | 20:24 |
| lbragstad | dstanek o/ | 20:24 |
| *** rderose has joined #openstack-keystone | 20:24 | |
| ayoung | still a net reduction in code | 20:24 |
| notmorgan | yeh | 20:25 |
| ayoung | notmorgan, so....if we don't do the "reduce spuriours revoke events" it might actually be faster | 20:26 |
| ayoung | we'll have more events, but we won't rebuild the token ever time, and just read them from cache | 20:26 |
| lbragstad | dstanek we're all iterating on https://review.openstack.org/#/c/311652/10 | 20:26 |
| patchbot | lbragstad: patch 311652 - keystone - Replace revoke tree with linear search | 20:26 |
| ayoung | if possilble | 20:26 |
| * notmorgan taps foot waiting for inspection folks to arrive. | 20:27 | |
| dstanek | lbragstad: to figure out the bug? | 20:28 |
| notmorgan | oh. well crap they are doing the inspection between 2 and 5pm... damn it. means i screwed up my lunch plans :( | 20:28 |
| lbragstad | dstanek I think we've narrowed it down to a revocation event cache bug | 20:28 |
| dstanek | lbragstad: neato | 20:29 |
| *** ninag has quit IRC | 20:29 | |
| *** doug-f___ has quit IRC | 20:29 | |
| *** rbridgeman has quit IRC | 20:29 | |
| *** iurygregory has quit IRC | 20:30 | |
| *** ericksonsantos has quit IRC | 20:30 | |
| *** clenimar has quit IRC | 20:30 | |
| *** pauloewerton has quit IRC | 20:30 | |
| *** doug-fish has joined #openstack-keystone | 20:31 | |
| notmorgan | lbragstad: i think it's not even cache | 20:31 |
| *** Guest53289 has quit IRC | 20:31 | |
| notmorgan | lbragstad: i think it's the .add_event logic mis-matching duplicated events | 20:32 |
| notmorgan | (.add_event on the tree) | 20:32 |
| notmorgan | not ... yah anyway | 20:32 |
| *** clenimar has joined #openstack-keystone | 20:32 | |
| *** raildo is now known as raildo-afk | 20:33 | |
| *** raildo-afk is now known as raildo | 20:33 | |
| *** iurygregory has joined #openstack-keystone | 20:34 | |
| *** ericksonsantos has joined #openstack-keystone | 20:34 | |
| *** pauloewerton has joined #openstack-keystone | 20:34 | |
| lbragstad | notmorgan oh - here? https://github.com/openstack/keystone/blob/master/keystone/models/revoke_model.py#L142 | 20:34 |
| knikolla | do we want separate devstack plugins for the k2k idp and k2k sp? or only one and have a bool value to distinguish which is which? | 20:36 |
| notmorgan | lbragstad: i think so. | 20:37 |
| *** sdake has quit IRC | 20:37 | |
| *** sdake has joined #openstack-keystone | 20:37 | |
| notmorgan | lbragstad: or somewhere in the tree logic itself. | 20:37 |
| samueldmq | notmorgan: hmm, if that's the case, ayoung patch will fix it | 20:38 |
| *** rbridgeman has joined #openstack-keystone | 20:38 | |
| *** pauloewerton has quit IRC | 20:41 | |
| *** ericksonsantos has quit IRC | 20:41 | |
| *** pauloewerton has joined #openstack-keystone | 20:42 | |
| *** ericksonsantos has joined #openstack-keystone | 20:42 | |
| *** ninag has joined #openstack-keystone | 20:46 | |
| *** ninag has quit IRC | 20:46 | |
| *** ninag has joined #openstack-keystone | 20:47 | |
| *** tonytan4ever has quit IRC | 20:47 | |
| *** tonytan4ever has joined #openstack-keystone | 20:47 | |
| *** jorge_munoz has quit IRC | 20:48 | |
| *** roxanaghe has quit IRC | 20:49 | |
| *** josecastroleon has quit IRC | 20:50 | |
| *** CIA has quit IRC | 20:51 | |
| *** fangxu has quit IRC | 20:51 | |
| *** ninag has quit IRC | 20:51 | |
| *** doug-fis_ has joined #openstack-keystone | 20:51 | |
| *** stingaci has quit IRC | 20:52 | |
| *** josecastroleon has joined #openstack-keystone | 20:52 | |
| *** doug-fish has quit IRC | 20:54 | |
| samueldmq | ayoung: you still around ? | 20:55 |
| ayoung | samueldmq, would you believe me if I said "no?" | 20:55 |
| samueldmq | ayoung: I just have two comments I need to confirm they're valid in patch 311652 | 20:55 |
| patchbot | samueldmq: https://review.openstack.org/#/c/311652/ - keystone - Replace revoke tree with linear search | 20:55 |
| *** julim has quit IRC | 20:55 | |
| samueldmq | ayoung: maybe, cuz you're ayoung :-) | 20:55 |
| samueldmq | ayoung: it's looking pretty good imo | 20:56 |
| ayoung | samueldmq, so I hate comments | 20:56 |
| ayoung | I think we spend way too much time quibbling over comments, and in doing so break the flow | 20:56 |
| ayoung | so, to your second question, no, I won;'t put a comment there | 20:56 |
| ayoung | "I assume all these attributes are mandatory in the token data. Is this right?" Yep | 20:56 |
| ayoung | samueldmq, :) | 20:57 |
| samueldmq | ayoung: you added comments everywhere else lol | 20:57 |
| ayoung | samueldmq, all of the rules are "fall through" | 20:57 |
| ayoung | and the return false | 20:57 |
| ayoung | is the way of short circuiting the logic. | 20:58 |
| samueldmq | ayoung: I could understand it, looks clear as it is | 20:58 |
| ayoung | the fact that the date comes at the end is not really relevant, no specific order in thei code. In the Tree code, it was important that it be last | 20:58 |
| *** chrisshattuck has joined #openstack-keystone | 20:58 | |
| samueldmq | ayoung: ah, got it | 21:00 |
| *** anush has joined #openstack-keystone | 21:01 | |
| samueldmq | ayoung: how are events removed from that list? | 21:02 |
| ayoung | samueldmq, they aren't | 21:02 |
| samueldmq | ayoung: never ? wow | 21:02 |
| ayoung | samueldmq, the query just returns all that are relevant | 21:02 |
| notmorgan | on new event being issued old events are pruned from the db | 21:03 |
| samueldmq | ayoung: ah ok so the manager only gets relevant data | 21:03 |
| ayoung | oh, right, that | 21:03 |
| notmorgan | s/old/expired | 21:03 |
| samueldmq | notmorgan: nice | 21:03 |
| samueldmq | ayoung: notmorgan: I am a +2 on that, looking good enough for me | 21:03 |
| ayoung | lets see if it passes tempest etc with the caching turned on before we get too optimisitc | 21:04 |
| samueldmq | I don't see a reason it wouldn't pass because of that caching, since it's how we do it everywhere else | 21:04 |
| samueldmq | but ok | 21:05 |
| ayoung | samueldmq, feel free to +2 it. It won't go in if it fails tests anyway | 21:05 |
| samueldmq | ayoung: again? | 21:06 |
| samueldmq | :) | 21:06 |
| *** roxanaghe has joined #openstack-keystone | 21:09 | |
| *** haplo37 has quit IRC | 21:11 | |
| samueldmq | ayoung: TypeError: can't compare datetime.datetime to str | 21:12 |
| samueldmq | ayoung: http://logs.openstack.org/52/311652/10/check/gate-keystone-python27-db/01f7702/console.html#_2016-05-16_21_00_10_788 | 21:12 |
| *** doug-fis_ has quit IRC | 21:17 | |
| *** doug-fish has joined #openstack-keystone | 21:18 | |
| *** fangxu has joined #openstack-keystone | 21:21 | |
| *** josecastroleon has quit IRC | 21:22 | |
| *** doug-fish has quit IRC | 21:22 | |
| *** raildo is now known as raildo-afk | 21:22 | |
| ayoung | samueldmq, OK, I think I have that. Running unit tests now | 21:23 |
| *** stingaci has joined #openstack-keystone | 21:24 | |
| *** doug-fish has joined #openstack-keystone | 21:25 | |
| *** doug-fish has quit IRC | 21:26 | |
| *** doug-fish has joined #openstack-keystone | 21:27 | |
| *** doug-fish has quit IRC | 21:32 | |
| *** doug-fish has joined #openstack-keystone | 21:33 | |
| *** doug-fish has quit IRC | 21:37 | |
| *** doug-fish has joined #openstack-keystone | 21:38 | |
| *** doug-fish has quit IRC | 21:38 | |
| *** doug-fish has joined #openstack-keystone | 21:38 | |
| *** josecastroleon has joined #openstack-keystone | 21:42 | |
| *** spzala has quit IRC | 21:46 | |
| *** roxanaghe has quit IRC | 21:51 | |
| openstackgerrit | Dolph Mathews proposed openstack/keystonemiddleware: Fix D202: No blank lines allowed after function docstring (PEP257) https://review.openstack.org/317102 | 21:52 |
| notmorgan | lbragstad: https://review.openstack.org/#/c/311886/5 is incorrect btw | 21:53 |
| patchbot | notmorgan: patch 311886 - keystone - Fix fernet audit ids for v2.0 (MERGED) | 21:53 |
| *** pauloewerton has quit IRC | 21:54 | |
| notmorgan | lbragstad: basically you end up with a non-unique set of audit ids on rescope in v2 | 21:54 |
| notmorgan | stevemar: ^ cc | 21:54 |
| *** phalmos has quit IRC | 21:54 | |
| notmorgan | cc dolphm ^ | 21:55 |
| notmorgan | lbragstad: i'm worried you totally fixed the wrong thing. | 21:56 |
| openstackgerrit | Dolph Mathews proposed openstack/keystonemiddleware: Fix D200: One-line docstring should fit on one line with quotes (PEP257) https://review.openstack.org/317103 | 21:56 |
| *** roxanaghe has joined #openstack-keystone | 21:57 | |
| *** markvoelker has joined #openstack-keystone | 21:57 | |
| *** edmondsw has quit IRC | 21:58 | |
| *** edtubill has quit IRC | 21:58 | |
| *** rderose has quit IRC | 22:00 | |
| lbragstad | notmorgan what happened? | 22:00 |
| notmorgan | lbragstad: basically you fixed the issue by maintaining the same audit ids across rescopes | 22:01 |
| *** markvoelker has quit IRC | 22:01 | |
| notmorgan | lbragstad: which is incorrect. | 22:01 |
| *** tonytan4ever has quit IRC | 22:01 | |
| notmorgan | t leas that is what it looks like | 22:01 |
| *** markvoelker has joined #openstack-keystone | 22:01 | |
| notmorgan | the main audit_id should be unique per token. | 22:01 |
| notmorgan | the parent id stays the same or is empty | 22:01 |
| notmorgan | lbragstad: at least that is what it looks like is happening now. | 22:02 |
| notmorgan | this is a problem with the split code paths, it's *really* hard to be sure we don't mess one of these up | 22:03 |
| lbragstad | notmorgan isn't that what these tested? | 22:04 |
| lbragstad | https://github.com/openstack/keystone/blob/master/keystone/tests/unit/test_auth.py#L524 | 22:04 |
| notmorgan | lbragstad: should be, but as you know fernet is only partially tested | 22:04 |
| *** anush has quit IRC | 22:04 | |
| lbragstad | notmorgan I inherited that class and ran it with fernet https://github.com/openstack/keystone/blob/master/keystone/tests/unit/test_auth.py#L675 | 22:05 |
| *** timcline_ has quit IRC | 22:06 | |
| *** doug-fish has quit IRC | 22:07 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:09 | |
| *** rderose has joined #openstack-keystone | 22:10 | |
| notmorgan | lbragstad: hmm/ | 22:10 |
| *** dmk0202 has quit IRC | 22:10 | |
| notmorgan | lbragstad: something still looks really wrong here. | 22:10 |
| *** josecastroleon has quit IRC | 22:12 | |
| openstackgerrit | Dolph Mathews proposed openstack/keystonemiddleware: Fix D105: Missing docstring in magic method (PEP257) https://review.openstack.org/317110 | 22:12 |
| *** doug-fis_ has joined #openstack-keystone | 22:13 | |
| lbragstad | notmorgan http://cdn.pasteraw.com/gbofv598z0penydq22js3wx52f8mwtf | 22:14 |
| lbragstad | ^ inspecting the audit ids in that test | 22:14 |
| lbragstad | with fernet | 22:14 |
| *** doug-fi__ has joined #openstack-keystone | 22:16 | |
| notmorgan | lbragstad: does this not impact liberty or kilo? | 22:17 |
| *** doug-fis_ has quit IRC | 22:18 | |
| lbragstad | notmorgan i'd have to check but liberty and kilo didn't have the consolidation of the code path | 22:19 |
| notmorgan | ok | 22:19 |
| ayoung | notmorgan, we got rid of "revoked_at" didn't we? | 22:19 |
| lbragstad | notmorgan this was the piece that was broken https://github.com/openstack/keystone/blob/996b0c7e1019235d384648f74de24542b0a736d7/keystone/token/providers/fernet/core.py#L159-L172 | 22:20 |
| *** diazjf has quit IRC | 22:20 | |
| notmorgan | ayoung: uh | 22:20 |
| lbragstad | notmorgan which is something we did as a hook to get data from the request in order to create the token | 22:20 |
| notmorgan | ayoung: don't think os? | 22:20 |
| *** ninag has joined #openstack-keystone | 22:20 | |
| lbragstad | ayoung on the revocation event? | 22:20 |
| notmorgan | lbragstad: yeah | 22:20 |
| lbragstad | no - revoked_at should still be there | 22:21 |
| lbragstad | a revocation event has three datetime entries | 22:21 |
| lbragstad | issued_before, revoked_at, and expires_at | 22:21 |
| *** doug-fi__ has quit IRC | 22:21 | |
| lbragstad | or at least it did the last time I checked | 22:21 |
| *** edtubill has joined #openstack-keystone | 22:22 | |
| *** ninag_ has joined #openstack-keystone | 22:22 | |
| notmorgan | issued_before was the important one | 22:22 |
| notmorgan | revoked_at was important for (i think) expiring the event? | 22:22 |
| lbragstad | yeah - that's what we use in the comparison | 22:22 |
| lbragstad | we compare the token's issued_at time to the revocation events issued_before time | 22:23 |
| notmorgan | lbragstad: also did this bad audit_id just mean we didn't maintain the parent_audit_id? or did we fail to rescope the token? or ... | 22:23 |
| notmorgan | lbragstad: or we just couldn't revoke the entire chain? | 22:23 |
| notmorgan | it *looks* like we still put an audit_id into the token | 22:24 |
| lbragstad | notmorgan yes - i believe the issue was that we didn't carry over the audit id from the parent token when rescoping the token | 22:24 |
| notmorgan | ok so we just couldn't revoke the chain | 22:24 |
| notmorgan | whihc... i don't think we every actually use in keystone | 22:24 |
| lbragstad | so when you got an unscoped token you'd get ['<audit_id>'] | 22:24 |
| notmorgan | yeah ok | 22:24 |
| notmorgan | that is not the worst thing. | 22:25 |
| *** ninag has quit IRC | 22:25 | |
| lbragstad | and then when you rescoped you'd get ['<new_audit_id>', '<newer_audit_id>'] | 22:25 |
| notmorgan | thats still fine | 22:25 |
| notmorgan | because revoke by audit_id would still work | 22:25 |
| notmorgan | just can't revoke by chain | 22:25 |
| lbragstad | right | 22:26 |
| lbragstad | that makes sense | 22:26 |
| *** ninag_ has quit IRC | 22:26 | |
| ayoung | so the to_dict() function was not ever setting the revoked_at value | 22:27 |
| lbragstad | nice | 22:28 |
| lbragstad | "this is not the revoked_at you're looking for" | 22:28 |
| ayoung | see http://git.openstack.org/cgit/openstack/keystone/tree/keystone/models/revoke_model.py#n97 | 22:28 |
| ayoung | and... | 22:28 |
| ayoung | does anything else check that? It is not in the tree, I think | 22:28 |
| ayoung | yeah, that is what we use to prune | 22:30 |
| ayoung | guessing pruning is broke | 22:30 |
| lbragstad | lol | 22:30 |
| ayoung | so that means I have a few more unit tests to fix, too | 22:31 |
| ayoung | lbragstad, but that is not supposed to be a nullable column | 22:31 |
| ayoung | should have blown up all over the place | 22:31 |
| ayoung | Maybe to_dict was not used | 22:32 |
| lbragstad | ayoung revoked_at isn't suppose to be a nullable column? | 22:33 |
| notmorgan | ayoung: it was not used in the tree afaik | 22:34 |
| ayoung | just for cleanup | 22:34 |
| *** ninag has joined #openstack-keystone | 22:34 | |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/revoke/backends/sql.py#n34 | 22:35 |
| lbragstad | ah | 22:36 |
| lbragstad | ayoung I assume issued_before is nullable since it is conditional depending on the revocation event? | 22:36 |
| lbragstad | or is issued_before something that should always be set? | 22:36 |
| ayoung | lbragstad, no idea. Brain now fried from revokcatiojanjiasd | 22:38 |
| lbragstad | ayoung i know right? | 22:38 |
| *** ninag has quit IRC | 22:39 | |
| lbragstad | staring at that code last week made me question my life choices | 22:39 |
| *** ninag has joined #openstack-keystone | 22:40 | |
| *** doug-fish has joined #openstack-keystone | 22:40 | |
| lbragstad | ayoung if i remember right - the revoke model populated both the revoked_at and issued_before fields | 22:41 |
| *** ametts has quit IRC | 22:41 | |
| lbragstad | ayoung actually - not revoked_at just expires | 22:41 |
| lbragstad | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/models/revoke_model.py#n114 | 22:41 |
| edtubill | Hi, I was just wondering if anyone here knew off the bat if horizon sends a request to keystone to invalidate a token when a user logs out? | 22:43 |
| david-lyle | edtubill: on logout, yes | 22:44 |
| notmorgan | edtubill: what david-lyle said | 22:44 |
| *** doug-fish has quit IRC | 22:44 | |
| notmorgan | edtubill: he'd know. | 22:44 |
| *** ninag has quit IRC | 22:44 | |
| edtubill | david-lyle: thx! | 22:44 |
| *** rderose has quit IRC | 22:45 | |
| *** pushkaru has quit IRC | 22:46 | |
| *** tmcpeak has joined #openstack-keystone | 22:51 | |
| tmcpeak | o/ edtubill | 22:51 |
| edtubill | tmcpeak: hey, so I'm trying to figure out if https://wiki.openstack.org/wiki/OSSN/OSSN-0017 is still valid. | 22:53 |
| tmcpeak | so in the note it says: "It should be noted that Horizon does request that Keystone invalidate the token upon user logout, but this has not been implemented for the Identity API v3. Token invalidation may also fail if the Keystone service is unavailable." | 22:54 |
| tmcpeak | but if an attacker is able to copy the token, either via MITM or local access, I still don't see how the server can tell whether it's valid or not without maintaining some state on the server side | 22:55 |
| *** BjoernT has quit IRC | 22:56 | |
| tmcpeak | the problem seems to be that without a way to revoke tokens server side there is no way to guarantee a token is revoked | 22:57 |
| edtubill | tmcpeak: wouldn't keystone store if a token is valid or not? But I guess if keystone is unreachable for a few seconds and the user logs out, the attacker could use the token because it wasn't invalidated? | 22:57 |
| tmcpeak | ok yeah, I'm trying to untangle what of this is a Horizon behavior and what's Keystone | 22:58 |
| tmcpeak | so Keystone is going to maintain a token independent of Horizon | 22:58 |
| tmcpeak | Keystone tokens can be revoked | 22:59 |
| tmcpeak | so the note says that token invalidation on user logout hasn't been implemented in Identity V3 API | 22:59 |
| tmcpeak | I guess we need to confirm if it was implemented sometime between Juno-Mitaka | 22:59 |
| tmcpeak | there is this: http://developer.openstack.org/api-ref-identity-v3.html#revokeTokens | 23:01 |
| *** doug-fish has joined #openstack-keystone | 23:02 | |
| *** lhcheng has quit IRC | 23:02 | |
| *** lhcheng has joined #openstack-keystone | 23:02 | |
| *** ChanServ sets mode: +v lhcheng | 23:02 | |
| tmcpeak | if Horizon is calling that it might work… Keystone experts, any thoughts? | 23:02 |
| david-lyle | support for v3 token revoke in horizon was added in 2014 | 23:02 |
| david-lyle | https://github.com/openstack/django_openstack_auth/commit/cad8def073222618857bebf9a18bb4d8dd098bfc | 23:02 |
| david-lyle | Dec 2014 | 23:03 |
| *** pushkaru has joined #openstack-keystone | 23:03 | |
| david-lyle | version 1.1.9 of django_openstack_auth and forward | 23:03 |
| david-lyle | https://bugs.launchpad.net/django-openstack-auth/+bug/1331978 is the closed bug link | 23:05 |
| openstack | Launchpad bug 1331978 in django-openstack-auth "Revoke v3 token on logout" [Medium,Fix released] - Assigned to Lin Hua Cheng (lin-hua-cheng) | 23:05 |
| tmcpeak | david-lyle: ok awesome, thank you! | 23:05 |
| *** markvoelker has quit IRC | 23:06 | |
| edtubill | david-lyle: yes thx again! | 23:06 |
| david-lyle | no problem, hope that helps | 23:07 |
| *** rcernin has quit IRC | 23:08 | |
| *** pushkaru has quit IRC | 23:10 | |
| *** gordc has quit IRC | 23:16 | |
| *** edtubill has quit IRC | 23:22 | |
| *** edtubill has joined #openstack-keystone | 23:25 | |
| *** tqtran has quit IRC | 23:26 | |
| *** ninag has joined #openstack-keystone | 23:30 | |
| *** edtubill has quit IRC | 23:31 | |
| *** ninag has quit IRC | 23:35 | |
| *** roxanaghe has quit IRC | 23:39 | |
| *** spandhe has joined #openstack-keystone | 23:41 | |
| *** jamielennox is now known as jamielennox|away | 23:51 | |
| *** rdo has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!