| *** markvoelker has joined #openstack-keystone | 00:05 | |
| *** markvoelker has quit IRC | 00:10 | |
| *** markvoelker has joined #openstack-keystone | 01:05 | |
| *** markvoelker has quit IRC | 01:09 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 01:20 | |
| *** imacdonn has quit IRC | 01:22 | |
| *** imacdonn has joined #openstack-keystone | 01:22 | |
| *** Emine has quit IRC | 01:22 | |
| *** Emine has joined #openstack-keystone | 01:23 | |
| *** Dinesh_Bhor has quit IRC | 01:53 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 02:03 | |
| openstackgerrit | ayoung proposed openstack/keystone-specs master: Explicit Domain Ids https://review.openstack.org/611201 | 02:05 |
|---|---|---|
| *** ayoung has joined #openstack-keystone | 02:11 | |
| *** felipemonteiro has quit IRC | 02:16 | |
| *** ayoung has quit IRC | 02:17 | |
| *** Emine has quit IRC | 02:23 | |
| *** Dinesh_Bhor has quit IRC | 02:38 | |
| openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: Set min and max length for resource_name https://review.openstack.org/611484 | 02:45 |
| *** Dinesh_Bhor has joined #openstack-keystone | 02:47 | |
| *** dave-mccowan has quit IRC | 02:56 | |
| *** rcernin has quit IRC | 03:02 | |
| *** bnemec has joined #openstack-keystone | 03:05 | |
| *** bnemec has quit IRC | 03:10 | |
| *** deepak_mourya__ has joined #openstack-keystone | 03:12 | |
| *** rcernin has joined #openstack-keystone | 03:28 | |
| *** Dinesh_Bhor has quit IRC | 04:05 | |
| openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: Adding 'date' for trust_flush https://review.openstack.org/607897 | 04:32 |
| *** Dinesh_Bhor has joined #openstack-keystone | 04:44 | |
| openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: Set min and max length for resource_name https://review.openstack.org/611484 | 04:51 |
| vishakha | ayoung : Hello. Regarding https://review.openstack.org/#/c/606912/ | 05:03 |
| vishakha | ayoung: I have removed the return from cli. As there are 3 prints in the mock_call containing Json object. It is difficult to match the expected with the actual as you can see in http://paste.openstack.org/show/731879/. Due to the different key positions the matching is failed. | 05:10 |
| vishakha | So I asserted a mock_count in code to confirm that all 3 prints are there but not able to validate stdout | 05:10 |
| *** Dinesh_Bhor has quit IRC | 05:13 | |
| *** felipemonteiro has joined #openstack-keystone | 05:24 | |
| *** lbragstad_503 has quit IRC | 05:27 | |
| *** lbragstad_503 has joined #openstack-keystone | 05:27 | |
| *** ChanServ sets mode: +o lbragstad_503 | 05:27 | |
| *** deepak_mourya__ has quit IRC | 05:30 | |
| *** felipemonteiro has quit IRC | 06:07 | |
| *** Emine has joined #openstack-keystone | 06:37 | |
| *** Emine has quit IRC | 06:42 | |
| openstackgerrit | Jose Castro Leon proposed openstack/keystone master: Add caching on trust role validation to improve performance https://review.openstack.org/608963 | 06:56 |
| *** rcernin has quit IRC | 07:07 | |
| *** xek has joined #openstack-keystone | 07:12 | |
| *** openstackgerrit has quit IRC | 07:35 | |
| *** rdopiera has joined #openstack-keystone | 07:52 | |
| *** xek has quit IRC | 07:58 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 08:37 | |
| *** pooja-jadhav has joined #openstack-keystone | 08:46 | |
| *** pooja_jadhav has quit IRC | 08:46 | |
| *** Dinesh_Bhor has quit IRC | 09:03 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 09:28 | |
| *** Dinesh_Bhor has quit IRC | 09:31 | |
| *** pooja-jadhav is now known as pooja_jadhav | 09:33 | |
| *** openstackgerrit has joined #openstack-keystone | 10:16 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/keystone master: DNM: Test jobs running on bionic instead of xenial https://review.openstack.org/611563 | 10:16 |
| *** dave-mccowan has joined #openstack-keystone | 10:18 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/keystone master: DNM: Test jobs running on bionic instead of xenial https://review.openstack.org/611563 | 10:18 |
| openstackgerrit | Colleen Murphy proposed openstack/keystone-specs master: Update spec template https://review.openstack.org/611583 | 12:07 |
| *** raildo has joined #openstack-keystone | 12:13 | |
| *** dims has quit IRC | 12:30 | |
| *** dims has joined #openstack-keystone | 12:33 | |
| *** jrist has quit IRC | 13:11 | |
| *** jrist has joined #openstack-keystone | 13:13 | |
| *** mvkr has quit IRC | 13:27 | |
| *** felipemonteiro has joined #openstack-keystone | 13:28 | |
| *** lbragstad_503 is now known as lbragstad | 13:32 | |
| *** bnemec has joined #openstack-keystone | 13:43 | |
| *** mchlumsky has joined #openstack-keystone | 13:52 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Implement scope_type checking for credentials https://review.openstack.org/594547 | 13:53 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Pass context objects to policy enforcement https://review.openstack.org/605539 | 13:53 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Remove obsolete credential policies https://review.openstack.org/597187 | 13:54 |
| *** mvkr has joined #openstack-keystone | 13:57 | |
| *** mchlumsky has quit IRC | 13:57 | |
| *** mchlumsky has joined #openstack-keystone | 13:59 | |
| *** felipemonteiro has quit IRC | 14:42 | |
| *** jmlowe has quit IRC | 14:57 | |
| *** itlinux has quit IRC | 15:01 | |
| *** dklyle has joined #openstack-keystone | 15:06 | |
| openstackgerrit | Lance Bragstad proposed openstack/oslo.policy master: Add domain scope support for scope types https://review.openstack.org/611443 | 15:17 |
| *** gyee has joined #openstack-keystone | 15:43 | |
| *** rook has joined #openstack-keystone | 15:48 | |
| rook | kmalloc: hey - curious w/ Rocky is there a way to disable the flask bits, so keystone only uses wsgi? | 15:48 |
| *** itlinux has joined #openstack-keystone | 15:51 | |
| *** dklyle has quit IRC | 16:16 | |
| *** dklyle has joined #openstack-keystone | 16:16 | |
| *** dklyle has quit IRC | 16:37 | |
| lbragstad | rook there isn't a way to disable flask | 16:55 |
| rook | lbragstad: ack - between Queens and Rocky where there any big chnages to keystone that could of impacted performance? | 16:58 |
| rook | the reason I asked about flask is because it was flooding the log when debug is enabled. | 16:58 |
| *** mvkr has quit IRC | 16:58 | |
| lbragstad | rook you can disable debug if you'd like - which is what we recommend if you're not actively debugging an issue in production | 16:58 |
| rook | lbragstad: but I am activly debugging :) | 16:59 |
| lbragstad | rook ah - well in that case :) | 16:59 |
| rook | trying to determine why we see a performance difference between rocky and queens :) | 16:59 |
| lbragstad | we did land a pretty large refactor to the token API | 16:59 |
| rook | the flask log puke is the only thing i see | 16:59 |
| lbragstad | and it did affect performance, we have a patch for it upstream that we're going to be backporting as soon as it lands in master | 16:59 |
| rook | let me see if I can share what i am seeing | 16:59 |
| rook | ooo got a link handy to the patchset? | 17:00 |
| lbragstad | https://review.openstack.org/#/c/608963/ is the fix - not sure if that impacts you | 17:00 |
| lbragstad | it's specific to trust scoped tokens | 17:00 |
| lbragstad | rook https://launchpad.net/bugs/1796887 is the bug report | 17:00 |
| openstack | Launchpad bug 1796887 in OpenStack Identity (keystone) "Validation of tokens degraded after upgrade to Rocky" [High,In progress] - Assigned to Jose Castro Leon (jose-castro-leon) | 17:00 |
| lbragstad | if that's not what you're seeing, please share what exactly is underperforming and we can help debug | 17:01 |
| rook | https://docs.google.com/spreadsheets/d/1EAAT7lFQkJsxwGCboMv2p74_opmMVoLkNIU_thfaLjM/edit?usp=sharing | 17:01 |
| rook | can you see that lbragstad ? | 17:01 |
| lbragstad | looks like i need permission | 17:01 |
| * lbragstad requests access | 17:01 | |
| rook | gd google | 17:02 |
| rook | try again :) | 17:02 |
| rook | so, this does seem relevant | 17:03 |
| rook | since it is the validation code. | 17:03 |
| lbragstad | interesting | 17:03 |
| lbragstad | looks like neutron is the biggest offender? | 17:04 |
| rook | the textual output is easier to read IMHO. | 17:04 |
| rook | Right | 17:04 |
| rook | validating neutron | 17:04 |
| rook | however, if you look at neutron (different tab) it seems to be performing better in rocky :) | 17:04 |
| lbragstad | do you know what you're doing in neutron when that happens? | 17:04 |
| rook | this is the rally scenario - authenicate.validate_neutron | 17:05 |
| lbragstad | how easy is it for you to apply patches to your env? | 17:05 |
| rook | hm - this is against master :/ not sure how easy/hard it would be apply to rocky | 17:06 |
| rook | i would just monkey patch it in. | 17:06 |
| rook | unless hrybacki could help me | 17:06 |
| lbragstad | that's what i would do (`git ready stable/rocky; git review -x 608963; pip install -e keystone`) | 17:09 |
| openstackgerrit | Merged openstack/keystone master: Replace JSON Body middleware with flask-native func https://review.openstack.org/609535 | 17:10 |
| lbragstad | rook or i suppose you could do the same thing with master | 17:12 |
| lbragstad | monkey patch master with the same patch and rerun the tests? | 17:12 |
| rook | lbragstad: ack - let me see what sort of pain that involves. | 17:13 |
| *** rdopiera has quit IRC | 17:15 | |
| openstackgerrit | Merged openstack/keystone master: Cleanup keystone.server.flask.application https://review.openstack.org/609548 | 17:15 |
| rook | lbragstad: I must be missing how this patch is actually caching... i think ayoung asked the same question. | 17:16 |
| lbragstad | rook https://review.openstack.org/#/c/608963/5/keystone/assignment/core.py@132 | 17:17 |
| lbragstad | ^ that decorator implements caching | 17:17 |
| rook | ack | 17:17 |
| lbragstad | prior to that patch, building roles for trust scoped tokens wasn't using a memoized method, so the result of the method wasn't being cached | 17:17 |
| *** dklyle has joined #openstack-keystone | 17:30 | |
| *** mvkr has joined #openstack-keystone | 17:41 | |
| *** dklyle has quit IRC | 17:42 | |
| *** dklyle has joined #openstack-keystone | 17:50 | |
| openstackgerrit | Magnus Lööf proposed openstack/keystone master: Fix example for getting system scoped token https://review.openstack.org/611685 | 17:54 |
| *** dklyle has quit IRC | 18:02 | |
| *** lbragstad has quit IRC | 18:17 | |
| *** dklyle has joined #openstack-keystone | 18:20 | |
| *** dklyle has quit IRC | 18:29 | |
| *** lbragstad has joined #openstack-keystone | 18:48 | |
| *** ChanServ sets mode: +o lbragstad | 18:48 | |
| *** dklyle has joined #openstack-keystone | 19:07 | |
| *** dklyle has quit IRC | 19:14 | |
| *** itlinux has quit IRC | 19:16 | |
| *** lbragstad has quit IRC | 19:21 | |
| *** lbragstad has joined #openstack-keystone | 19:24 | |
| *** ChanServ sets mode: +o lbragstad | 19:24 | |
| rook | lbragstad: that patched helped slightly in my tests... | 19:29 |
| rook | however, i have re-run with newer Rocky bits, and keystone looks better overall.. | 19:30 |
| lbragstad | nice! | 19:35 |
| lbragstad | are you still seeing performance degrade with neutron explicitly? | 19:35 |
| rook | nope | 19:35 |
| rook | actually performing better than Queens | 19:36 |
| rook | :D | 19:36 |
| lbragstad | woot! | 19:36 |
| lbragstad | that's awesome | 19:36 |
| lbragstad | just by applying that one patch? | 19:36 |
| rook | newer rocky bits + the patch | 19:40 |
| rook | I have multiple iterations | 19:40 |
| rook | I am also looking at something else too, might have impacted things | 19:40 |
| rook | nova changed (scheduler) so by default it deploys # of cores / # nova-scheduler workers... | 19:40 |
| rook | which also puts additonal pressure on keystone | 19:40 |
| lbragstad | interesting | 19:49 |
| rook | lbragstad: yeah... i mean, these newer processors have tons of threads... | 19:54 |
| rook | so dual socket machines could easily have 96 threads | 19:54 |
| rook | and if you have ha, that is a ton of workers asking for a lot out of keystone | 19:54 |
| rook | :) | 19:54 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Implement scope_type checking for credentials https://review.openstack.org/594547 | 20:12 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Pass context objects to policy enforcement https://review.openstack.org/605539 | 20:12 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Implement scope_types for user API https://review.openstack.org/611179 | 20:12 |
| *** openstackgerrit has quit IRC | 20:36 | |
| *** pcaruana has quit IRC | 20:44 | |
| *** itlinux has joined #openstack-keystone | 21:06 | |
| *** raildo has quit IRC | 21:09 | |
| *** bnemec is now known as bnemec-bbl | 21:20 | |
| *** dklyle has joined #openstack-keystone | 21:36 | |
| *** dklyle has quit IRC | 21:45 | |
| kmalloc | lbragstad: see my comment on https://review.openstack.org/#/c/611121/ | 22:02 |
| kmalloc | basically... this is not something testable | 22:02 |
| kmalloc | it's code flow. | 22:02 |
| kmalloc | we could try a decorator...but i think it'd be unwieldy | 22:02 |
| lbragstad | yeah - i figured | 22:03 |
| kmalloc | lbragstad: any concerns on the current flask (final) stack? | 22:06 |
| kmalloc | asking before i go dig into KSM adjustments (mostly extracting a couple things into isolated functions) | 22:06 |
| *** mchlumsky has quit IRC | 22:21 | |
| *** openstackgerrit has joined #openstack-keystone | 22:25 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Support KWARGS in the cache key generating function(s). https://review.openstack.org/611120 | 22:25 |
| *** rcernin has joined #openstack-keystone | 22:33 | |
| *** dnguyen has joined #openstack-keystone | 22:35 | |
| openstackgerrit | Merged openstack/keystone-specs master: Update spec template https://review.openstack.org/611583 | 22:42 |
| openstackgerrit | Adrian Turjak proposed openstack/keystone master: Implement auth receipts spec https://review.openstack.org/611230 | 22:47 |
| *** dnguyen has left #openstack-keystone | 22:47 | |
| adriant | kmalloc: when you get a chance, can you give me a little bit of feedback on ^ | 22:48 |
| adriant | at the very least around the new auth API bits with flask | 22:49 |
| kmalloc | sure. | 22:49 |
| kmalloc | looking now. | 22:49 |
| kmalloc | adriant: i hope flask made this somewhat easier | 22:49 |
| adriant | kmalloc: yes and no :P | 22:49 |
| adriant | kmalloc: because authenticate_for_token no longer returns a response, I need to handle some stuff elsewhere | 22:50 |
| adriant | and that's the part I'm not happy with yet or not sure of | 22:50 |
| kmalloc | hehe | 22:50 |
| kmalloc | yeah, it's a bit more explicit | 22:50 |
| adriant | mostly my changes in keystone/api/auth.py | 22:50 |
| openstackgerrit | Adrian Turjak proposed openstack/keystone master: Implement auth receipts spec https://review.openstack.org/611230 | 22:53 |
| adriant | kmalloc: are we doing any custom error handling for the flask APIs? As in, when we raise an Unauthorised error, where is that caught and returns a 401 to the user? | 22:56 |
| adriant | part of my was thinking that rather than handling that error and returning the receipt in auth.py, we could do it in that same layer we catch the other errors | 22:56 |
| adriant | but that may in turn making it harder to find/debug | 22:56 |
| kmalloc | adriant: https://review.openstack.org/#/c/609796/ | 22:58 |
| kmalloc | adriant: it will merge *soon*ish to move to flask handling exceptions | 22:58 |
| adriant | OH, so until that is merged the exceptionss via flask aren't really being handled? | 23:01 |
| adriant | or did we have some weird flask+webob handling? | 23:02 |
| adriant | ah yes | 23:02 |
| adriant | URL Normalizing Middleware | 23:02 |
| adriant | alright, that makes more sense to me. Just not sure if having auth receipt creation there is a good idea :/ | 23:05 |
| kmalloc | you should add a specific handling case to the error handler, like we have for the Unauthorized. | 23:13 |
| kmalloc | imo | 23:13 |
| kmalloc | once that merges. | 23:13 |
| kmalloc | it makes sense to go in the handler | 23:14 |
| adriant | works for me, and makes it nicer than making auth.py ugly | 23:14 |
| kmalloc | it modifies the response in some cases rather than in the auth view. | 23:14 |
| kmalloc | ++ | 23:14 |
| kmalloc | i have lots of comments on your code. | 23:14 |
| kmalloc | btw. | 23:14 |
| adriant | as expected! and it's still missing unit tests. | 23:15 |
| kmalloc | almost done with the first pass review, and should be enough to justify another patchset and enough changes to allow for another pass on a future patchset | 23:15 |
| adriant | plus I might rebase my change on the exception handling one so I can get that code written before it gets merged | 23:16 |
| kmalloc | comments on patchset 3 | 23:18 |
| kmalloc | lots of comments, like i said ;) | 23:19 |
| adriant | kmalloc: I'm ok with lots of comments, it means someone at least cares about the code! | 23:20 |
| kmalloc | :) | 23:21 |
| kmalloc | the biggest comment is | 23:21 |
| kmalloc | i would just lean on all the token invalidation queues | 23:21 |
| adriant | will try and address them tonight and hopefully have a patch soon. Plus I need to rewrite the unit tests and port those over. | 23:21 |
| kmalloc | if you need to invalidate the token cache, you need to invalidate the receipt cache (probably) | 23:21 |
| * adriant nods | 23:21 | |
| kmalloc | (global invalidate) | 23:21 |
| adriant | that makes sense | 23:22 |
| kmalloc | it's tough to cache the receipts since they aren't issued with concrete values like tokens e.g. user_id | 23:23 |
| kmalloc | so you can't really invalidate tokens for a user | 23:23 |
| kmalloc | or receipts | 23:23 |
| kmalloc | you might want to lean on the revocation events as well | 23:23 |
| kmalloc | so user password change invalidates the receipts as well as the tokens. | 23:23 |
| kmalloc | lbragstad: you here? | 23:24 |
| kmalloc | have some questions on scope_type bits | 23:24 |
| *** felipemonteiro has joined #openstack-keystone | 23:48 | |
| *** bnemec has joined #openstack-keystone | 23:50 | |
| lbragstad | kmalloc go ahead and ask, i should be back a bit later | 23:50 |
| kmalloc | lbragstad: commented | 23:50 |
| lbragstad | ok | 23:50 |
| kmalloc | a minor change needed really | 23:50 |
| kmalloc | the try/excepts can all be collapsed to a build_target function | 23:50 |
| kmalloc | i gave you an example | 23:50 |
| kmalloc | keeps the API / resource code cleaner | 23:51 |
| kmalloc | lbragstad: beyond that, the code looks good, i need to more closely review the tests | 23:51 |
| kmalloc | but nothing looked broken that side | 23:52 |
| *** bnemec-bbl has quit IRC | 23:52 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Support KWARGS in the cache key generating function(s). https://review.openstack.org/611120 | 23:54 |
| *** markvoelker has joined #openstack-keystone | 23:57 | |
| *** gyee has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!