| *** rcernin_ has joined #openstack-keystone | 00:04 | |
| *** rcernin has quit IRC | 00:04 | |
| *** linkmark has quit IRC | 00:13 | |
| * jamielennox can't believe flask is actually happening | 01:01 | |
| kmalloc | jamielennox: hehe | 01:08 |
|---|---|---|
| kmalloc | mnaser: hold on. let me read that question | 01:08 |
| kmalloc | mnaser: both keystones need to be able to validate the token. | 01:09 |
| kmalloc | mnaser: it's totally fine to have a local replica/master that can issue tokens and that you can then use to speak to $_ENDPOINT | 01:09 |
| kmalloc | mnaser: keystoneauth should handle that just fine as long as the token is valid | 01:10 |
| kmalloc | jamielennox: i was sick of dealing with webob weirdness | 01:10 |
| jamielennox | not having v2 will have helped | 01:10 |
| kmalloc | yes | 01:10 |
| kmalloc | that made it a lot easier | 01:10 |
| jamielennox | i had it prototyped out at one point, but the v2 controller was a disaster | 01:10 |
| kmalloc | but... realistically, it's not that much more work to wire up APIs | 01:10 |
| jamielennox | v3 controller, also not great | 01:11 |
| jamielennox | but yea | 01:11 |
| kmalloc | what is awesome, the controller is dying | 01:11 |
| kmalloc | what is not awesome, a lot of that code is not dying...has been "ported" to flask | 01:11 |
| kmalloc | what is most awesome... @protected is finally on it's last leg | 01:11 |
| jamielennox | so the blocker there was always unwrapping policy | 01:12 |
| jamielennox | :) | 01:12 |
| kmalloc | took me ~8 days to rewrite @protected | 01:12 |
| jamielennox | again, prototypes but getting that to work across v2/v3 was just a mess | 01:12 |
| kmalloc | but we have docstrings now. | 01:12 |
| kmalloc | and a generally better interface | 01:12 |
| kmalloc | still pretty opaque | 01:12 |
| kmalloc | but, way more usable | 01:12 |
| jamielennox | yea, that's what i found with protected, the layers went so deep | 01:13 |
| jamielennox | and no one was willing to touch the reviews for it | 01:13 |
| kmalloc | jamielennox: https://review.openstack.org/#/c/576639/23 | 01:14 |
| kmalloc | somehow i got that landed. | 01:14 |
| kmalloc | now i am just fighting with circular imports that don't show up in unit tests. | 01:14 |
| kmalloc | which is kindof driving me batty | 01:14 |
| jamielennox | lol, that there is just trust | 01:14 |
| jamielennox | can't validate that | 01:14 |
| openstackgerrit | Vu Cong Tuan proposed openstack/ldappool master: Switch to stestr https://review.openstack.org/581307 | 01:14 |
| kmalloc | jamielennox: well, i wrote a bunch of tests for it too | 01:15 |
| kmalloc | and it seemed to mirror @protected | 01:15 |
| jamielennox | return flask.request.environ.get(context.REQUEST_CONTEXT_ENV, None) - feels like that should be easier | 01:16 |
| kmalloc | so far so good | 01:16 |
| jamielennox | but at least it's using context | 01:16 |
| kmalloc | jamielennox: yeah, well flask doesn't really talk "oslo context" | 01:16 |
| jamielennox | right | 01:16 |
| kmalloc | i figured that was better than the alternative... subsclassing flask.request or some such | 01:16 |
| kmalloc | sorry flask.Request | 01:16 |
| jamielennox | was just thinking i did a bunch of stuff to get ksm in front of keystone and it was passing down a usable context | 01:16 |
| kmalloc | yep, and it does. | 01:17 |
| jamielennox | but yea, i've no idea what that would look like in flask | 01:17 |
| kmalloc | totally doable | 01:17 |
| kmalloc | but flask's contexts are a bit less structured. | 01:17 |
| kmalloc | i might revisit and squash all the middleware to "before_request"functions once the apis are moved | 01:17 |
| kmalloc | so we only legit load middleware from non-local packages | 01:17 |
| jamielennox | eh, i trust you, and you're aware of all that stuff so i'm not criticizing from the sidelines | 01:18 |
| kmalloc | hehe, i didn't take it as criticism | 01:18 |
| kmalloc | once all the APIs are in-fact flask native, we'll be solid to do stuff like that. but right now we're mostly doing weird dispatching to the old mappers for everything but discovery (/ and /v3) and /credentials once trhe newest patch lands. | 01:19 |
| *** stewie925 has quit IRC | 01:30 | |
| lbragstad | +1 for the context bits | 01:32 |
| lbragstad | i'm at least happy that the policy enforcement stuff is easier to grok post-flask | 01:34 |
| kmalloc | lbragstad: ok so... i have no idea how in the hell this is getting a circular import | 01:39 |
| kmalloc | and that it doesn't happen in test | 01:39 |
| *** sapd has joined #openstack-keystone | 01:39 | |
| kmalloc | heck, i can't duplicate this with raw import in a venv | 01:39 |
| *** mvk_ has quit IRC | 01:39 | |
| lbragstad | is this still in the credentials patch? | 01:40 |
| *** mvk_ has joined #openstack-keystone | 01:42 | |
| kmalloc | yepo | 01:43 |
| lbragstad | hmm | 01:44 |
| kmalloc | lbragstad: http://logs.openstack.org/50/582450/4/check/tempest-full/d8c524c/controller/logs/screen-keystone.txt.gz#_Jul_16_22_31_06_040276 | 01:44 |
| lbragstad | so - it's affecting tempest and not unit tests? | 01:44 |
| kmalloc | yep | 02:02 |
| *** mvk_ has quit IRC | 02:02 | |
| kmalloc | and i can't even make it fail with direct imports of the modules | 02:02 |
| kmalloc | and my local commit is the same as the one in gerrit | 02:03 |
| * kmalloc shrugs. | 02:03 | |
| kmalloc | i'll poke at it more in a little bit | 02:03 |
| *** mvk_ has joined #openstack-keystone | 02:03 | |
| *** dave-mccowan has joined #openstack-keystone | 02:07 | |
| *** dave-mcc_ has joined #openstack-keystone | 02:24 | |
| *** dave-mccowan has quit IRC | 02:25 | |
| lbragstad | that's weird | 02:27 |
| openstackgerrit | Merged openstack/keystone master: Filter by entity_type in get_domain_mapping_list https://review.openstack.org/572446 | 02:38 |
| openstackgerrit | Merged openstack/keystone master: Increase test coverage of entity_type id mapping query https://review.openstack.org/582692 | 02:38 |
| *** flwang1 has quit IRC | 02:41 | |
| *** flwang1 has joined #openstack-keystone | 02:42 | |
| kmalloc | Yep, totally weird. | 02:58 |
| *** lbragstad has quit IRC | 03:05 | |
| *** abhi89 has joined #openstack-keystone | 03:14 | |
| *** dave-mcc_ has quit IRC | 03:30 | |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Strict two level limit model https://review.openstack.org/557696 | 04:00 |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Add project_id filter for listing limit https://review.openstack.org/579330 | 04:00 |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Add include_limits filter https://review.openstack.org/579331 | 04:00 |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Update project depth check https://review.openstack.org/580258 | 04:00 |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Add project hierarchical tree check when Keystone start https://review.openstack.org/580331 | 04:00 |
| *** flwang1 has quit IRC | 04:56 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Move Credentials API to Flask Native https://review.openstack.org/582450 | 04:58 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Allow class-level definition of API URL Prefix https://review.openstack.org/582726 | 04:58 |
| *** nelsnelson has quit IRC | 05:05 | |
| kmalloc | lbragstad[m]: i think i found the issue. looks like it is working now. | 05:20 |
| *** rcernin_ has quit IRC | 05:25 | |
| *** rcernin has joined #openstack-keystone | 05:26 | |
| *** links has joined #openstack-keystone | 05:33 | |
| *** nelsnelson has joined #openstack-keystone | 05:41 | |
| *** alex_xu has quit IRC | 05:59 | |
| *** alex_xu has joined #openstack-keystone | 06:02 | |
| *** s10 has joined #openstack-keystone | 06:35 | |
| *** alex_xu has quit IRC | 06:36 | |
| *** alex_xu has joined #openstack-keystone | 06:36 | |
| *** martinus__ has joined #openstack-keystone | 06:37 | |
| *** s10 has quit IRC | 06:38 | |
| *** tesseract has joined #openstack-keystone | 06:48 | |
| *** ispp has joined #openstack-keystone | 06:59 | |
| *** peereb has joined #openstack-keystone | 07:04 | |
| *** rcernin has quit IRC | 07:11 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 07:22 | |
| *** ispp has quit IRC | 07:28 | |
| openstackgerrit | Merged openstack/ldappool master: Switch to stestr https://review.openstack.org/581307 | 07:32 |
| *** tosky has joined #openstack-keystone | 07:38 | |
| *** gongysh has joined #openstack-keystone | 07:57 | |
| *** ispp has joined #openstack-keystone | 08:15 | |
| *** alex_xu has quit IRC | 08:45 | |
| *** alex_xu has joined #openstack-keystone | 08:46 | |
| *** hoonetorg has quit IRC | 08:47 | |
| openstackgerrit | Dirk Mueller proposed openstack/ldappool master: Bump to hacking 1.1.x https://review.openstack.org/583162 | 08:52 |
| *** flwang1 has joined #openstack-keystone | 08:53 | |
| openstackgerrit | Dirk Mueller proposed openstack/ldappool master: Switch to python-ldap again https://review.openstack.org/583164 | 08:55 |
| *** ispp has quit IRC | 09:02 | |
| *** hoonetorg has joined #openstack-keystone | 09:04 | |
| openstackgerrit | Tuan Do Anh proposed openstack/keystone master: Change "a SQL" to "an SQL" https://review.openstack.org/579432 | 09:16 |
| *** annp has quit IRC | 09:18 | |
| *** annp has joined #openstack-keystone | 09:26 | |
| *** mvk_ has quit IRC | 09:56 | |
| *** abhi89 has quit IRC | 10:09 | |
| *** mvk_ has joined #openstack-keystone | 10:26 | |
| *** gongysh has quit IRC | 10:36 | |
| openstackgerrit | Dao Cong Tien proposed openstack/keystone master: Adds doc8 check to pep8 https://review.openstack.org/583196 | 10:44 |
| *** ispp has joined #openstack-keystone | 10:55 | |
| *** sapd has quit IRC | 11:03 | |
| *** mvk_ has quit IRC | 11:09 | |
| *** mvk_ has joined #openstack-keystone | 11:09 | |
| *** d0ugal has quit IRC | 11:15 | |
| *** dave-mccowan has joined #openstack-keystone | 11:18 | |
| *** d0ugal has joined #openstack-keystone | 11:18 | |
| *** d0ugal has quit IRC | 11:18 | |
| *** d0ugal has joined #openstack-keystone | 11:18 | |
| *** sapd has joined #openstack-keystone | 11:20 | |
| *** abhi89 has joined #openstack-keystone | 11:33 | |
| openstackgerrit | wangxiyuan proposed openstack/keystoneauth master: [WIP]Add netloc and version check for version discovery https://review.openstack.org/583215 | 11:46 |
| *** edmondsw has joined #openstack-keystone | 12:06 | |
| *** ispp has quit IRC | 12:22 | |
| *** gongysh has joined #openstack-keystone | 12:30 | |
| *** lbragstad has joined #openstack-keystone | 12:31 | |
| *** ChanServ sets mode: +o lbragstad | 12:31 | |
| *** nels has joined #openstack-keystone | 12:34 | |
| *** nelsnelson has quit IRC | 12:36 | |
| *** raildo has joined #openstack-keystone | 12:40 | |
| *** lbragstad has quit IRC | 12:41 | |
| *** raildo has quit IRC | 12:45 | |
| *** raildo has joined #openstack-keystone | 12:46 | |
| *** raildo has quit IRC | 12:50 | |
| *** raildo has joined #openstack-keystone | 12:52 | |
| *** peereb has quit IRC | 12:56 | |
| *** tosky has quit IRC | 12:56 | |
| *** tosky has joined #openstack-keystone | 12:56 | |
| *** raildo has quit IRC | 13:00 | |
| *** raildo has joined #openstack-keystone | 13:00 | |
| *** raildo has quit IRC | 13:03 | |
| *** raildo has joined #openstack-keystone | 13:03 | |
| *** ispp has joined #openstack-keystone | 13:17 | |
| *** aning_ has left #openstack-keystone | 13:24 | |
| *** aning_ has joined #openstack-keystone | 13:24 | |
| *** raildo has quit IRC | 13:35 | |
| *** raildo has joined #openstack-keystone | 13:36 | |
| *** raildo has quit IRC | 13:37 | |
| knikolla | o/ | 13:37 |
| evrardjp | hello folks | 13:38 |
| *** raildo has joined #openstack-keystone | 13:39 | |
| devx | Hola! | 13:39 |
| *** raildo has quit IRC | 13:41 | |
| evrardjp | so I have a question about the state of keystone tempest testing... | 13:42 |
| evrardjp | where are we in the migration from https://github.com/openstack/tempest/tree/master/tempest/api/identity to https://github.com/openstack/keystone-tempest-plugin/tree/master/keystone_tempest_plugin/services/identity | 13:42 |
| evrardjp | is it the plan to not port everything to the keystone tempest plugin (and keep things in tempest)? or is there a plan to move things around? | 13:43 |
| evrardjp | I am asking this because I see commits like https://github.com/openstack/tempest/commit/7d2b636a30057ed8db8cfd4fe2248f509b3570f1#diff-5c9acbc10dc9d27b47985cd74ab100f6 happening . | 13:44 |
| cmurphy | evrardjp: the plan was never to remove any of keystone's tests from tempest | 13:45 |
| cmurphy | the plugin is just for the weird keystone things like ldap and saml testing | 13:45 |
| evrardjp | ok | 13:46 |
| evrardjp | thanks for the clarification! | 13:46 |
| cmurphy | no problem | 13:46 |
| evrardjp | .buffer 8 | 13:46 |
| evrardjp | woops | 13:46 |
| *** ispp has quit IRC | 13:48 | |
| *** ispp has joined #openstack-keystone | 13:50 | |
| *** gongysh has quit IRC | 13:52 | |
| *** ispp has quit IRC | 13:52 | |
| mordred | lbragstad[m], cmurphy: ftr - the ksa bug about internal urls and version discovery is 100% caused by the fact that none of the client libs ever actually did version discovery but instead just hardcoded url manipulations | 14:00 |
| kmalloc | mordred: ouch | 14:00 |
| *** gongysh has joined #openstack-keystone | 14:01 | |
| kmalloc | mordred: sigh. | 14:01 |
| *** ispp has joined #openstack-keystone | 14:01 | |
| mordred | kmalloc: yah. | 14:01 |
| kmalloc | mordred: that makes me... sad. | 14:01 |
| kmalloc | like... super sad | 14:01 |
| mordred | we have a workaround/fix in sdk | 14:01 |
| mordred | that I think my suggestion is going to be pushing down into ksa | 14:01 |
| mordred | even though it's ... sad | 14:01 |
| *** ispp has quit IRC | 14:02 | |
| cmurphy | I think wxy already proposed a potential fix https://review.openstack.org/583215 | 14:02 |
| mordred | oh good | 14:04 |
| mordred | cmurphy: yes - that's basically the approach we have in sdk | 14:05 |
| mordred | hrm. I say we have a fix - I think we defer to ksa now :) | 14:07 |
| * kmalloc needs to wake up. | 14:07 | |
| *** raildo has joined #openstack-keystone | 14:07 | |
| * cmurphy puts a cat near kmalloc's dog | 14:08 | |
| kmalloc | we have not introduced nori to ... cats yet | 14:10 |
| kmalloc | we don't want her to lose that first interaction quite yet :P | 14:10 |
| kmalloc | and... with the cats near by... it wont go well for her :P | 14:10 |
| * kmalloc tags wxy's change ^ for review. | 14:11 | |
| kmalloc | looks pretty straightforward. | 14:11 |
| mordred | kmalloc, cmurphy: I like it - and I think the tests that are broken are actually broken tests (or tests that show the brokenness) - so I think just fixing them is correct | 14:12 |
| cmurphy | wxy: ^ | 14:12 |
| kmalloc | yep. | 14:13 |
| kmalloc | that was my quick gander | 14:13 |
| kmalloc | i'll prob. pick it up and play clenaup if wxy doesn't have time. otherwise I'll provide review for it. | 14:13 |
| *** quackrabbit has joined #openstack-keystone | 14:18 | |
| *** spilla has joined #openstack-keystone | 14:20 | |
| *** jmlowe has joined #openstack-keystone | 14:36 | |
| *** s10 has joined #openstack-keystone | 14:37 | |
| *** d0ugal has quit IRC | 14:37 | |
| *** ispp has joined #openstack-keystone | 14:38 | |
| *** ispp has quit IRC | 14:39 | |
| *** d0ugal has joined #openstack-keystone | 14:44 | |
| *** abhi89 has quit IRC | 14:55 | |
| *** lbragstad has joined #openstack-keystone | 14:57 | |
| *** ChanServ sets mode: +o lbragstad | 14:57 | |
| *** josecastroleon has joined #openstack-keystone | 14:58 | |
| *** josecastroleon has quit IRC | 14:59 | |
| *** ispp has joined #openstack-keystone | 15:03 | |
| *** d0ugal has quit IRC | 15:03 | |
| *** wxy| has joined #openstack-keystone | 15:06 | |
| *** abhi89 has joined #openstack-keystone | 15:08 | |
| lbragstad | kmalloc: looks like you figured out the import thing? | 15:09 |
| lbragstad | was it https://review.openstack.org/#/c/582450/4..5/keystone/server/flask/__init__.py ? | 15:09 |
| *** jmlowe has quit IRC | 15:09 | |
| kmalloc | I don't know why I couldn't dupe it outside of tempest | 15:09 |
| lbragstad | if not - i do have a devstack ready to go | 15:09 |
| kmalloc | But it works now. | 15:09 |
| lbragstad | huh... | 15:09 |
| kmalloc | I know what was broken... But being unable to dupe it was weird. | 15:12 |
| lbragstad | and it was failing keystone tempest only though? | 15:13 |
| kmalloc | yep | 15:13 |
| kmalloc | well it was failking to standup keystone in tempest | 15:13 |
| kmalloc | i am sure i could have duplicated if i was standing up a whole keystone | 15:14 |
| *** d0ugal has joined #openstack-keystone | 15:14 | |
| kmalloc | but the fact that straight import statements didn't duplicate it was weird. | 15:14 |
| kmalloc | and that unit tests worked | 15:14 |
| lbragstad | yeah... | 15:15 |
| *** dgonzalez has left #openstack-keystone | 15:15 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: [WIP] Move Trusts to Flask Native https://review.openstack.org/583278 | 15:15 |
| kmalloc | lbragstad: ^ checkpoint for trusts moving | 15:15 |
| kmalloc | just so i don't lose my place. it was a lot of stash/stash pop :P | 15:16 |
| *** jmlowe has joined #openstack-keystone | 15:18 | |
| kmalloc | lbragstad: but short of a few minor things trusts should be ready soon too | 15:18 |
| lbragstad | going through the pre-requisite patches right now | 15:18 |
| lbragstad | leading up to the credentials API | 15:19 |
| kmalloc | yep. | 15:19 |
| kmalloc | these are a LOT less dense to go through now that we landed the bulk of the base code work | 15:19 |
| *** errr has joined #openstack-keystone | 15:22 | |
| errr | when using shibboleth federation and having websso enabled in horizon what generates the url for when I select my idp from the horizon dashboard vs logging in with a local account? | 15:24 |
| kmalloc | lbragstad: i'm thinking application credentials, catalog, s3, ec2, revoke, auth, domain, project, group, user is about the order for conversion | 15:24 |
| kmalloc | toss in oauth1 and policy somewhere in there | 15:24 |
| lbragstad | kmalloc: that seems reasonable... | 15:25 |
| kmalloc | limit may move last just since we're iterating on it right now | 15:25 |
| lbragstad | errr: i believe the IDP url is setup via keystone's configuration file | 15:25 |
| *** d0ugal has quit IRC | 15:26 | |
| errr | lbragstad: I have grepped for what is "wrong" with my url and cant find it in any of the files | 15:26 |
| lbragstad | hmmm | 15:26 |
| errr | lbragstad: for example mine is adding "okta" there and it should be "saml2" and if I manually change it to that in my browser things work | 15:27 |
| * lbragstad double checks something quick | 15:27 | |
| lbragstad | errr: have you checked shibboleth configs? | 15:32 |
| errr | yes | 15:33 |
| errr | lbragstad: so right now my url is :5000/v3/auth/OS-FEDERATION/websso/okta and it needs to be :5000/v3/auth/OS-FEDERATION/websso/saml2 and I grepped for "okta" and "websso" but I cant find that in there | 15:33 |
| *** s10 has quit IRC | 15:33 | |
| *** s10 has joined #openstack-keystone | 15:35 | |
| *** abhi89 has quit IRC | 15:37 | |
| *** r-daneel has joined #openstack-keystone | 15:37 | |
| lbragstad | errr: what about your horizon configuration? | 15:38 |
| *** jistr is now known as jistr|afk | 15:38 | |
| lbragstad | https://docs.openstack.org/keystone/pike/advanced-topics/federation/federated_identity.html#horizon-changes | 15:38 |
| errr | lbragstad: so this url is generated with out horizon being involved but I have made the relevant changes there | 15:39 |
| lbragstad | what do you idp resources look like in keystone? | 15:40 |
| *** felipemonteiro has joined #openstack-keystone | 15:41 | |
| errr | lbragstad: from openstack identity provider list ? | 15:42 |
| lbragstad | yeah | 15:42 |
| errr | | okta | True | fe2fb222a44a40ca8592b8f8ced6ae15 | None | | 15:42 |
| errr | I have another working system which has the same output for that command but it creates the correct url with the 'saml2' in it instead of 'okta' | 15:43 |
| lbragstad | hmm | 15:43 |
| knikolla | errr: do you have multiple idps? | 15:44 |
| errr | knikolla: no just 1 for now. | 15:44 |
| knikolla | errr: in the horizon config, there is a config for websso choices, with (idp, protocol) pairings | 15:48 |
| knikolla | https://github.com/CCI-MOC/MOCOSPpuppet/blob/3d902da487fb4eac19fb3035c02bf76b3d141782/horizon/templates/local_settings.py.erb#L602-L603 | 15:48 |
| knikolla | horizon uses that to create figure out the url that it redirects you | 15:49 |
| errr | ha! | 15:49 |
| errr | oh man. I totally missed that and I thought I had it | 15:49 |
| errr | but you are right I am missing that from my config | 15:50 |
| knikolla | errr: https://github.com/openstack/horizon/blob/3e0da5b91c28acdf86f5c33278623ac856e223c5/openstack_auth/utils.py#L189-L210 | 15:50 |
| knikolla | full docs here | 15:50 |
| errr | I mean I have it.. but I had okta there and not saml2 which is what my working system has | 15:51 |
| knikolla | cool! | 15:52 |
| errr | bouncing services now. i hope this works :D | 15:52 |
| *** jmlowe has quit IRC | 15:53 | |
| errr | sweet it works. Now I can figure out what else I have wrong. Probably the audience redirect or something but at least now I get my idp login page. Thanks for the help | 15:56 |
| *** d0ugal has joined #openstack-keystone | 15:56 | |
| *** tesseract has quit IRC | 15:57 | |
| lbragstad | thanks knikolla | 15:58 |
| *** jmlowe has joined #openstack-keystone | 15:58 | |
| openstackgerrit | wangxiyuan proposed openstack/keystoneauth master: Add netloc and version check for version discovery https://review.openstack.org/583215 | 15:59 |
| knikolla | :) | 16:00 |
| *** dklyle has quit IRC | 16:00 | |
| wxy| | s10: https://review.openstack.org/583215 a quick fix, could you take a try to check it works or not? | 16:02 |
| mordred | wxy|: that patch lgtm | 16:12 |
| s10 | wxy: It works for the access to the internal endpoints from the controller nodes. I will check, if it works well for public endpoints from outside the cloud. | 16:14 |
| wxy| | s10: sure | 16:17 |
| s10 | wxy: it works for public endpoints. | 16:17 |
| *** d0ugal has quit IRC | 16:18 | |
| *** harlowja has joined #openstack-keystone | 16:18 | |
| wxy| | mordred: :) | 16:19 |
| mordred | \o/ | 16:20 |
| *** links has quit IRC | 16:24 | |
| *** dklyle has joined #openstack-keystone | 16:28 | |
| *** d0ugal has joined #openstack-keystone | 16:30 | |
| *** spilla has quit IRC | 16:31 | |
| *** gongysh has quit IRC | 16:35 | |
| *** spilla has joined #openstack-keystone | 16:41 | |
| *** mvk_ has quit IRC | 16:46 | |
| kmalloc | lol, OS-TRUST != OS-TRUSTS *facepalm* | 16:47 |
| *** jmlowe has quit IRC | 16:48 | |
| *** jmlowe has joined #openstack-keystone | 16:51 | |
| *** AlexeyAbashkin has quit IRC | 16:59 | |
| lbragstad | #startmeeting keystone-office-hours | 17:01 |
| openstack | Meeting started Tue Jul 17 17:01:51 2018 UTC and is due to finish in 60 minutes. The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:01 |
| *** openstack changes topic to " (Meeting topic: keystone-office-hours)" | 17:01 | |
| *** ChanServ changes topic to "Rocky release schedule: https://releases.openstack.org/rocky/schedule.html | Meeting agenda: https://etherpad.openstack.org/p/keystone-weekly-meeting | Bugs that need triaging: http://bit.ly/2iJuN1h | Trello: https://trello.com/b/wmyzbFq5/keystone-rocky-roadmap !!NOTE!! This Channel is Logged ( https://tinyurl.com/OpenStackKeystone )" | 17:01 | |
| openstack | The meeting name has been set to 'keystone_office_hours' | 17:01 |
| lbragstad | i have to step away for lunch | 17:06 |
| *** fiddletwix has joined #openstack-keystone | 17:06 | |
| *** ispp has quit IRC | 17:06 | |
| * gagehugo goes to grab lunch as well | 17:09 | |
| openstackgerrit | wangxiyuan proposed openstack/keystoneauth master: Add netloc and version check for version discovery https://review.openstack.org/583215 | 17:09 |
| *** lbragstad has quit IRC | 17:11 | |
| *** wxy| has quit IRC | 17:11 | |
| * kmalloc just ate | 17:19 | |
| kmalloc | breakfast. | 17:19 |
| *** felipemonteiro_ has joined #openstack-keystone | 17:26 | |
| *** felipemonteiro has quit IRC | 17:30 | |
| *** jistr|afk is now known as jistr | 17:30 | |
| *** mvk_ has joined #openstack-keystone | 17:35 | |
| cmurphy | so with this one https://review.openstack.org/#/c/578008/ I'm unclear on why this option isn't automatically exposed by keystoneauth and wondering if we should be exposing it there rather than registering it in keystonemiddleware | 17:41 |
| openstackgerrit | Merged openstack/ldappool master: Bump to hacking 1.1.x https://review.openstack.org/583162 | 17:44 |
| *** s10 has quit IRC | 17:44 | |
| *** spilla_ has joined #openstack-keystone | 17:46 | |
| *** spilla has quit IRC | 17:46 | |
| *** r-daneel has quit IRC | 17:50 | |
| *** spilla_ has quit IRC | 17:51 | |
| *** mchlumsky has quit IRC | 17:51 | |
| *** mchlumsky has joined #openstack-keystone | 17:51 | |
| *** harlowja has quit IRC | 17:54 | |
| *** spilla has joined #openstack-keystone | 17:58 | |
| mnaser | does the validate token endpoint speak to the db at all when using fernet? | 18:04 |
| *** pcichy has quit IRC | 18:14 | |
| kmalloc | mnaser: yes. | 18:19 |
| kmalloc | mnaser: the fernet data is very limited and relies on the db to look up the values | 18:19 |
| mnaser | ah, i thought they can be validated on their own | 18:19 |
| mnaser | using the private key | 18:19 |
| kmalloc | nope. that was a feature of PKI tokens, but the token data was so large we exploded HTTP request handling | 18:20 |
| mnaser | yeah i remember those times | 18:20 |
| kmalloc | Fernet tokens are "live" validated, meaning direct lookup in the db | 18:20 |
| kmalloc | it also means if a user's roles change, the validation payload would change, it reflects the current state of the DB plus or minus some delta depending on caching | 18:21 |
| * kmalloc kicks the trust controller ... hard. | 18:21 | |
| kmalloc | ok ok.. what in the heck... i am getting a non-iso time back... but afaict i'm only emitting iso time into the dat astruct | 18:22 |
| kmalloc | how am i dropping the 'Z'... | 18:22 |
| kmalloc | *glare* | 18:22 |
| cmurphy | that sounds like a familiar bug | 18:23 |
| kmalloc | yeah | 18:24 |
| kmalloc | i am not seeing how the Z is being dropped | 18:24 |
| kmalloc | it's... weird. | 18:24 |
| *** jistr is now known as jistr|off | 18:25 | |
| *** r-daneel has joined #openstack-keystone | 18:27 | |
| kmalloc | ahhh found it | 18:31 |
| kmalloc | badly name variables | 18:31 |
| kmalloc | cmurphy: "trust" vs "new_trust" *eyeroll* | 18:31 |
| *** r-daneel has quit IRC | 18:32 | |
| cmurphy | kmalloc: badly named variables changed the time format? | 18:32 |
| kmalloc | yeah | 18:32 |
| *** harlowja has joined #openstack-keystone | 18:32 | |
| kmalloc | i was re-normalizing the input data | 18:33 |
| kmalloc | not the "after store in the db" data | 18:33 |
| cmurphy | ah | 18:33 |
| kmalloc | new_trust = providers.trust_api.create_trust() | 18:33 |
| kmalloc | then normalize_expires_at(trust) | 18:33 |
| kmalloc | whoope | 18:33 |
| kmalloc | whoopse* | 18:33 |
| kmalloc | i renamed "new_trust" to "return_trust" | 18:33 |
| kmalloc | just to make it easier to see and behold, normalizing the correct ref makes the difference | 18:34 |
| cmurphy | ++ | 18:34 |
| *** lbragstad has joined #openstack-keystone | 18:47 | |
| *** ChanServ sets mode: +o lbragstad | 18:47 | |
| *** mchlumsky has quit IRC | 18:55 | |
| *** mchlumsky has joined #openstack-keystone | 18:56 | |
| *** imacdonn has joined #openstack-keystone | 18:56 | |
| lbragstad | mnaser: are you having some issues with fernet tokens? | 18:57 |
| mnaser | lbragstad: no, just trying to think of the cleanest way to architect this solution. Our keystone is based out in Montreal and we’re opening a region in the Silicon Valley | 18:59 |
| mnaser | So trying to make sure the latency doesn’t break the world :) | 18:59 |
| *** pcichy has joined #openstack-keystone | 19:00 | |
| lbragstad | oh... | 19:01 |
| lbragstad | sure | 19:01 |
| lbragstad | i assume both are writeable? | 19:02 |
| lbragstad | since token validation is read-only, the validation process should be immediate | 19:05 |
| *** ksavich_ has joined #openstack-keystone | 19:09 | |
| *** s10 has joined #openstack-keystone | 19:10 | |
| kmalloc | lbragstad: damn | 19:15 |
| kmalloc | lbragstad: looks like we need a handler that explicitly does a 404 not a 405 when a method is not implemented =/ | 19:16 |
| kmalloc | lbragstad: since our contract is crappy and 404s in those cases. | 19:16 |
| lbragstad | bah | 19:16 |
| kmalloc | lbragstad: though... realistically that *isnt* really part of our api | 19:16 |
| kmalloc | PATCH /v3/OS-TRUST/trusts/<trust_id> isn't really part of the API. | 19:16 |
| kmalloc | but... | 19:17 |
| kmalloc | it requires me to "change" a test. | 19:17 |
| kmalloc | so... what is your opinion here | 19:17 |
| kmalloc | I'm personally ok with moving to a 405 in this case | 19:17 |
| kmalloc | we just explicitly test for a 404. | 19:17 |
| kmalloc | if someone tries to patch a trust, it's a rando-404 | 19:18 |
| lbragstad | if we end up going in that direction, i'd like to do it all at once for all 404s like that | 19:18 |
| kmalloc | ok i'll add a TODO explicitly in the PATCH implementation | 19:18 |
| lbragstad | i assume you're just talking about trusts? | 19:18 |
| kmalloc | yeah for now | 19:18 |
| kmalloc | since we are migrating apis piece-meal i think a 404->405 for these cases is fine as we go | 19:19 |
| kmalloc | ftr: "put" will 405 for trusts | 19:19 |
| kmalloc | and we don't check for that | 19:19 |
| lbragstad | hmm | 19:19 |
| kmalloc | we're highly inconsistent here | 19:20 |
| *** tosky has quit IRC | 19:20 | |
| kmalloc | and it's not something that is "API" specific | 19:20 |
| kmalloc | it's not like PATCH for trust ever did anything | 19:20 |
| kmalloc | it does mean we need to implement a GET/POST/PUT/PATCH/DELETE for every resource that blindly 404s | 19:21 |
| kmalloc | unless it is overidden. it feels weird to do that, esp. since we test for some of these cases but not really all/many/consistently any of them | 19:21 |
| kmalloc | lbragstad: i'll defer to your call here though. | 19:22 |
| kmalloc | so: quick check on options (pick one) | 19:22 |
| kmalloc | 1) Implement explicit 404 where we test for it | 19:22 |
| lbragstad | the explicity implementation would be nice | 19:23 |
| kmalloc | 2) Implement explicit 404 everywhere | 19:23 |
| kmalloc | for un-defined methods | 19:23 |
| kmalloc | 3) allow 405 to pass through for unimplemented methods | 19:23 |
| * kmalloc prefers #3 | 19:23 | |
| lbragstad | #2 makes things 405 -> 404 | 19:23 |
| kmalloc | #2 is closest to what we have now. | 19:23 |
| lbragstad | how much harder would it be to do #2 over #3? | 19:24 |
| kmalloc | #3 makes some things 404->405, but they aren't part of our API, it happens to be magic it happens | 19:24 |
| kmalloc | #2 is just defining a base class and if someone doesn't use it, it will 405 | 19:24 |
| kmalloc | 405 is the MOST correct error to pass through in these cases. | 19:25 |
| lbragstad | i agree there | 19:25 |
| kmalloc | it mostly was an accident we got 404s because of how our system was implemented | 19:25 |
| lbragstad | i'm wondering what a client is going to do when they've been dealing with 404s and now they get a 405 | 19:25 |
| kmalloc | they've been using an invalid/not part of the API already :P | 19:25 |
| kmalloc | it could have resulted in any number of things. | 19:26 |
| kmalloc | let me check if tempest tries patching trusts. | 19:26 |
| kmalloc | i think that will answer my question on "is this part of the api" | 19:26 |
| kmalloc | yeah tempest doesn't even try to patch a trust | 19:27 |
| kmalloc | so, revised order of preference: #3 -> 405s, #1 -> explicit 404 if we test for it, #2 blanket 404 | 19:28 |
| *** flwang1 has quit IRC | 19:29 | |
| lbragstad | from an API guidelines perspective, going from 404 -> 405 is allowed? | 19:29 |
| kmalloc | i'd contest this isn't part of the API | 19:30 |
| kmalloc | PATCH is not implemented for Trusts. | 19:30 |
| kmalloc | same with PUT | 19:30 |
| kmalloc | if patch was implemented, it wouldn't be allowed | 19:30 |
| kmalloc | but since it's an unimplemented method, it isn't part of the API. | 19:31 |
| kmalloc | it is the responsibility of the underlying server to handle it. | 19:31 |
| *** s10 has quit IRC | 19:32 | |
| lbragstad | ok | 19:33 |
| lbragstad | in that case i think i'm fine with #3 | 19:33 |
| kmalloc | yeah. | 19:34 |
| kmalloc | you know me, i'm pretty strict on the not breaking the contract | 19:34 |
| kmalloc | ;) | 19:34 |
| kmalloc | i'm proposing it as 405 | 19:34 |
| kmalloc | but we can reverse course if needed | 19:34 |
| *** fiddletwix has quit IRC | 19:34 | |
| *** fiddletwix has joined #openstack-keystone | 19:35 | |
| lbragstad | but 405 seems like the most correct thing in this context | 19:35 |
| kmalloc | yep | 19:36 |
| lbragstad | at least based on my interpretation of the RFC | 19:37 |
| kmalloc | exactly | 19:37 |
| kmalloc | i'll make sure to add a note in the review for the reviewers | 19:37 |
| lbragstad | ++ | 19:37 |
| lbragstad | ^ kinda would be nice in a separate patch.. but | 19:37 |
| lbragstad | er - that'd be a reason for it... | 19:37 |
| kmalloc | i have to separate out some patches anyway | 19:37 |
| lbragstad | but calling it out in the review might be fine | 19:37 |
| kmalloc | i'll split that | 19:38 |
| kmalloc | i have a bug in RBACEnforcer, Json_home population, and something else | 19:38 |
| kmalloc | so this is being split into 2-3 patches anyway | 19:38 |
| lbragstad | ok | 19:41 |
| *** fiddletwix has quit IRC | 19:42 | |
| *** fiddletwix has joined #openstack-keystone | 19:43 | |
| *** kfox1111 has joined #openstack-keystone | 19:45 | |
| kfox1111 | question. does keystone support osprofiler and how far back does its support go? | 19:45 |
| lbragstad | yes - we've had that support since like newton i think | 19:48 |
| kfox1111 | ok. cool. thanks. | 19:48 |
| lbragstad | 639e36adbfa0f58ce2c3f31856b4343e9197aa0e | 19:50 |
| lbragstad | https://review.openstack.org/#/c/103368/ | 19:51 |
| kfox1111 | nice. :) | 19:52 |
| kmalloc | lbragstad: bah | 20:05 |
| kmalloc | lbragstad: i found a bug in our json_home test... | 20:05 |
| kmalloc | i think. | 20:05 |
| kmalloc | ah just "obseved" expected is wonky | 20:07 |
| kmalloc | nvm | 20:07 |
| *** quackrabbit has quit IRC | 20:09 | |
| lbragstad | i have to relocate quick | 20:20 |
| mnaser | lbragstad: i think i am okay with only 1 of the keystones being write-able | 20:20 |
| mnaser | so auth happens in that one location, always. | 20:20 |
| lbragstad | ah | 20:20 |
| lbragstad | but validate should be able to happen in both | 20:21 |
| mnaser | yes | 20:21 |
| mnaser | and the idea is validate being able to happen to the closer datacenter (this is really to avoid latency in the openstack apis) | 20:21 |
| lbragstad | well - let me know if there is anything we can do to improve that upstream | 20:21 |
| *** AlexeyAbashkin has joined #openstack-keystone | 20:22 | |
| mnaser | well i'm just wondering how 'bad' it would be if i had a 70ms latency between keystone/openstack (while using memcache anyways) | 20:22 |
| mnaser | memcache being local obviously | 20:22 |
| mnaser | 75ms rtt that is | 20:23 |
| kmalloc | not terrible, but... sub-ideal imo | 20:26 |
| kmalloc | like... nothing should break overtly | 20:26 |
| kmalloc | a keystone validate is not "fast" | 20:27 |
| kmalloc | but i worry about a non-local memcache in general | 20:27 |
| *** lbragstad has quit IRC | 20:28 | |
| mnaser | oh there will be a local memcache | 20:29 |
| mnaser | but i suspect not a lot of clients reuse tokens besides openstack services | 20:29 |
| kmalloc | ah | 20:29 |
| kmalloc | lbragstad[m]: should have trusts pushed up in a moment | 20:30 |
| *** raildo has quit IRC | 20:31 | |
| imacdonn | speaking of memcache, I have a topic for discussion, but I don't want to interrupt, so let me know when you guys are done ;) | 20:32 |
| errr | Im having some trouble getting logged into Horizon using keystone shibboleth federation. When I successfully auth with my IDP I get redirected https://aio.mrice.internal:5000/v3/auth/OS-FEDERATION/websso/saml2?origin=https://aio.mrice.internal/dashboard/auth/websso/ and it tells me 401 The request you have made requires authentication. | 20:35 |
| errr | any idea what I may have missed in my setup thats keeping this from working? | 20:35 |
| cmurphy | errr: if you turn on insecure_debug in keystone.conf it will tell you specifically what went wrong (remember to turn it off in production) | 20:36 |
| errr | cmurphy: thanks | 20:36 |
| *** raildo has joined #openstack-keystone | 20:38 | |
| imacdonn | so my problem has to do with exceeding memcached's maximum connections limit .., caused by neutron-server, which uses keystonemiddleware | 20:39 |
| errr | wow. that helped a ton. Thanks again cmurphy | 20:39 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Move trusts to flask native dispatching https://review.openstack.org/583278 | 20:39 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Correctly pull input data for enforcement https://review.openstack.org/583356 | 20:39 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Allow for 'extension' rel in json home https://review.openstack.org/583357 | 20:39 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Trusts do not implement patch. https://review.openstack.org/583358 | 20:39 |
| kmalloc | imacdonn: ok i can focus now that those patches are pushed | 20:39 |
| kmalloc | imacdonn: yep, i've seen that in the past. the correct answer is, unfortunately, to use the memcache-pool | 20:40 |
| kmalloc | imacdonn: the issue is eventlet creates a new connection per-green-thread | 20:40 |
| kmalloc | imacdonn: and doesn't cleanup it's connections well | 20:40 |
| imacdonn | kmalloc: yes, exactly .. gleaned from comments in https://bugs.launchpad.net/fuel-ccp/+bug/1653071 | 20:40 |
| openstack | Launchpad bug 1653071 in fuel-ccp "Lack of free connections to memcached cause keystone middleware to stall" [High,Fix released] - Assigned to Fuel CCP Bug Team (fuel-ccp-bugs) | 20:40 |
| kmalloc | imacdonn: i'll need to check to make sure memcachepool has been implemented for ksm | 20:40 |
| kmalloc | imacdonn: it might be only in oslo_cache, and ksm is not on oslo_cache (if i remember correctly) yet | 20:41 |
| imacdonn | kmalloc: there's a config option "memcache_use_advanced_pool", but I've not been able to make much sense of it | 20:41 |
| kmalloc | this is one of the major reasons keystone dropped eventlet and all greenlet/greenthread based handling. | 20:41 |
| kmalloc | imacdonn: ah that would be the option. | 20:41 |
| kmalloc | imacdonn: it... is not a good piece of code. (and i apologize for that) | 20:41 |
| kmalloc | imacdonn: python-memcache is sortof a trainwreck on some fronts and we eat it badly because of it. | 20:42 |
| kmalloc | our solution(s): migrate to oslo-cache and implement a better backend for dogpile that is not based on python-memcached | 20:42 |
| kmalloc | it's been a long term goal. | 20:42 |
| imacdonn | kmalloc: Heh. OK, well at least it's good to know I'm not missing something stupid | 20:42 |
| kmalloc | nope. that advanced pool is the only real solution | 20:43 |
| kmalloc | it basically builds a shared set of memcache connections | 20:43 |
| kmalloc | but since python-memcache uses threadlocal natively and we stack on top of that, it is prone to being more fragile than we'd like | 20:43 |
| kmalloc | and we have had to reference internal interfaces | 20:43 |
| imacdonn | so there are a couple of bugs related to that - https://bugs.launchpad.net/keystonemiddleware/+bug/1748160 and https://bugs.launchpad.net/keystonemiddleware/+bug/1747565 | 20:44 |
| openstack | Launchpad bug 1748160 in keystonemiddleware "memcache_use_advanced_pool = True doesn't work when use oslo.cache" [Undecided,Fix released] - Assigned to wangxiyuan (wangxiyuan) | 20:44 |
| openstack | Launchpad bug 1747565 in keystonemiddleware "AttributeError when use memcache_use_advanced_pool = True in Ocata" [Undecided,Fix released] - Assigned to wangxiyuan (wangxiyuan) | 20:44 |
| imacdonn | I tried back-porting the fixes, and also tried updating middleware to 5.x in my Queens environment | 20:44 |
| kmalloc | lbragstad[m], knikolla: damn so close. -333, +337, would have been aweseme if it was +333/-333 | 20:45 |
| imacdonn | but I can't get it to work .. it doesn't seem to make any connections to memcached ... and then things start sporadically hanging | 20:45 |
| kmalloc | imacdonn: updating middleware beyond the release is a recipe for disaster, since ksm needs to lean on the libs in that nova/neutron/etc do | 20:45 |
| imacdonn | kmalloc: yeah, I figured, but had to try it (in a lab env) | 20:45 |
| kmalloc | right | 20:45 |
| kmalloc | hm. | 20:45 |
| aning_ | Hi, for fernet keys, are there any ways to generate them other than keystone-manage fernet_setup? | 20:46 |
| imacdonn | so I guess I can try upping the max connections limit, but it's icky :/ | 20:46 |
| kmalloc | yeah | 20:48 |
| *** martinus__ has quit IRC | 20:48 | |
| kmalloc | imacdonn: you're on Queens? | 20:48 |
| imacdonn | kmalloc: yes | 20:48 |
| kmalloc | imacdonn: hmm. | 20:48 |
| kmalloc | i really want to re-write that | 20:48 |
| * kmalloc wishes he could write more code faster | 20:48 | |
| imacdonn | kmalloc: actually, maybe the problem env is Pike | 20:48 |
| kmalloc | stil. | 20:49 |
| kmalloc | still* | 20:49 |
| imacdonn | yeah, it's Pike ... but I haven't found anything that obviously makes it better in Queens | 20:49 |
| kmalloc | yeah. i just don't know i can offer some "here is an alternative solution" but it likely wont be straightforward | 20:50 |
| kmalloc | and migth require replacing part of ksm's code | 20:50 |
| *** raildo has quit IRC | 20:50 | |
| imacdonn | part of my concern is that I don't have a good handle on how the connections are accumulating, so I don't know what I need to set the limit to | 20:50 |
| imacdonn | (as a workaround) ... or maybe they'll just keep multiplying? :/ | 20:51 |
| *** AlexeyAbashkin has quit IRC | 20:54 | |
| *** d0ugal has quit IRC | 20:58 | |
| *** d0ugal has joined #openstack-keystone | 21:00 | |
| *** d0ugal has quit IRC | 21:00 | |
| *** d0ugal has joined #openstack-keystone | 21:00 | |
| *** spilla has quit IRC | 21:03 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Allow for 'extension' rel in json home https://review.openstack.org/583357 | 21:04 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Move trusts to flask native dispatching https://review.openstack.org/583278 | 21:04 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Use oslo_serialization.jsonutils https://review.openstack.org/583373 | 21:04 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Add pycadf initiator for flask resource https://review.openstack.org/583374 | 21:04 |
| kmalloc | imacdonn: it's just because eventlet and it's backend suck at this part | 21:05 |
| kmalloc | the accumulation is mostly dead connections that haven't been cleaned up | 21:05 |
| kmalloc | the answer is ... set it obnoxiously high | 21:06 |
| kmalloc | if changing on the memcache side | 21:06 |
| imacdonn | yes, I guess that's all I can do .... unless I can shorten the lifetime | 21:06 |
| kmalloc | you can take a look at netstat and see what you have | 21:06 |
| imacdonn | # (Optional) Number of seconds a connection to memcached is held unused in the | 21:06 |
| imacdonn | # pool before it is closed. (integer value) | 21:06 |
| imacdonn | #memcache_pool_unused_timeout = 60 | 21:06 |
| imacdonn | not sure if that comes into play or not | 21:07 |
| kmalloc | that only applies when using the advanced pool | 21:07 |
| imacdonn | ooh | 21:07 |
| *** flwang1 has joined #openstack-keystone | 21:07 | |
| imacdonn | is the advanced pool stuff documented somewhere? I haven't found anything that even mentions if, other than config comments | 21:07 |
| imacdonn | I'm going to try again to patch the two bugs in Queens... but when I tried that before, it seemed it wasn't making any connections to memcached at all | 21:09 |
| imacdonn | side question ... how can I turn on debug logging for this? | 21:09 |
| imacdonn | (from a client like neutron) | 21:09 |
| kmalloc | in pike the memcachepool is here https://github.com/openstack/keystonemiddleware/blob/stable/pike/keystonemiddleware/auth_token/_memcache_pool.py | 21:11 |
| kmalloc | queens moves to oslo_cache | 21:11 |
| kmalloc | patching / backporting is going tombe really hairy | 21:11 |
| kmalloc | totally different code bases | 21:11 |
| imacdonn | if I can get the pool to work on Queens, it'd be good incentive to upgrade ... want to get that done anyway | 21:12 |
| kmalloc | aye | 21:12 |
| kmalloc | so queens is very different | 21:13 |
| kmalloc | man.. our docs suck | 21:14 |
| kmalloc | i'm so very sorry | 21:14 |
| imacdonn | heh | 21:14 |
| kmalloc | so, i think the answer is configure memcache, then do the advanced_pool=true option | 21:18 |
| kmalloc | you have a number of tunables for ksm in the pool | 21:18 |
| kmalloc | most are sane-ish defaults | 21:19 |
| imacdonn | OK, so I went back to the Queens version of ksm, and encountered the two bugs mentioned above ... so applied the patches .... and now I'm back to no connections, and things are hanging | 21:19 |
| kmalloc | weird. | 21:19 |
| kmalloc | very weird. | 21:19 |
| kmalloc | ipv4 or ipv6? | 21:20 |
| imacdonn | the hosts have v6 addresses, but they're not being used | 21:21 |
| kmalloc | right | 21:21 |
| imacdonn | or, at least, nothing it configured to use them (and there are no DNS references to them) | 21:21 |
| kmalloc | memcache has issues in this case with v6 | 21:21 |
| kmalloc | but as long as you're using v4 | 21:21 |
| kmalloc | that is a non-issue | 21:22 |
| imacdonn | ok | 21:22 |
| kmalloc | can you reach memcache server from the neutron server host? | 21:22 |
| imacdonn | yes ... they're actually on the same host | 21:23 |
| kmalloc | telnet should work fine to test | 21:23 |
| imacdonn | # lsof -i TCP:11211 | wc -l | 21:23 |
| imacdonn | 495 | 21:23 |
| imacdonn | # lsof -i TCP:11211 | grep ^neutron | 21:23 |
| imacdonn | # | 21:23 |
| kmalloc | well, more specifically can you reach memcache via telnet using the ip/port specified in neutron config | 21:23 |
| imacdonn | note that it works if the advanced pool is not enabled | 21:23 |
| kmalloc | just making sure it's not just something wonky going on | 21:23 |
| kmalloc | ah | 21:23 |
| kmalloc | ok | 21:23 |
| kmalloc | hm | 21:23 |
| imacdonn | compare to: | 21:24 |
| imacdonn | # lsof -i TCP:11211 | grep -c ^neutron | 21:24 |
| imacdonn | 2866 | 21:24 |
| imacdonn | # | 21:24 |
| imacdonn | (Pike env that has the problem) | 21:24 |
| *** felipemonteiro_ has quit IRC | 21:27 | |
| kmalloc | hm. | 21:28 |
| kmalloc | i don't see how this is not working | 21:28 |
| kmalloc | there is nothing wonky in the code base atm | 21:28 |
| kmalloc | it should just work with the advanced pool | 21:28 |
| imacdonn | yeah. I'm trying to figure out what it's hanging on | 21:29 |
| kmalloc | be back in a few | 21:29 |
| imacdonn | k | 21:29 |
| kmalloc | i need to not look at this for a sec and get some food/another coffee | 21:29 |
| kmalloc | :) | 21:29 |
| imacdonn | :) | 21:29 |
| *** edmondsw has quit IRC | 21:36 | |
| *** edmondsw_ has joined #openstack-keystone | 21:45 | |
| *** mchlumsky has quit IRC | 21:49 | |
| *** edmondsw_ has quit IRC | 21:49 | |
| *** spilla has joined #openstack-keystone | 21:51 | |
| *** dave-mccowan has quit IRC | 22:16 | |
| *** spilla has quit IRC | 22:18 | |
| *** rcernin has joined #openstack-keystone | 22:32 | |
| *** imacdonn has quit IRC | 22:44 | |
| *** imacdonn has joined #openstack-keystone | 22:44 | |
| *** lbragstad has joined #openstack-keystone | 22:50 | |
| *** ChanServ sets mode: +o lbragstad | 22:50 | |
| imacdonn | kmalloc: unsurprisingly, the hang is occurring here (haven't attempted to trace beyond this point yet): https://github.com/openstack/keystonemiddleware/blob/stable/queens/keystonemiddleware/auth_token/__init__.py#L730 | 22:50 |
| lbragstad | #endmeeting | 22:51 |
| *** openstack changes topic to "Rocky release schedule: https://releases.openstack.org/rocky/schedule.html | Meeting agenda: https://etherpad.openstack.org/p/keystone-weekly-meeting | Bugs that need triaging: http://bit.ly/2iJuN1h | Trello: https://trello.com/b/wmyzbFq5/keystone-rocky-roadmap !!NOTE!! This Channel is Logged ( https://tinyurl.com/OpenStackKeystone )" | 22:51 | |
| openstack | Meeting ended Tue Jul 17 22:51:44 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:51 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/keystone_office_hours/2018/keystone_office_hours.2018-07-17-17.01.html | 22:51 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/keystone_office_hours/2018/keystone_office_hours.2018-07-17-17.01.txt | 22:51 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/keystone_office_hours/2018/keystone_office_hours.2018-07-17-17.01.log.html | 22:51 |
| lbragstad | sorry that was a bit late | 22:51 |
| imacdonn | that was a long meeting ;) | 22:53 |
| lbragstad | yeah - technically it was our office hours :) | 22:54 |
| imacdonn | ah | 22:55 |
| *** itlinux has joined #openstack-keystone | 23:03 | |
| lbragstad | cmurphy: thanks for following up on the limits review | 23:04 |
| *** tosky has joined #openstack-keystone | 23:09 | |
| *** itlinux has quit IRC | 23:09 | |
| *** itlinux has joined #openstack-keystone | 23:17 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Correctly pull input data for enforcement https://review.openstack.org/583356 | 23:27 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Use oslo_serialization.jsonutils https://review.openstack.org/583373 | 23:27 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Add pycadf initiator for flask resource https://review.openstack.org/583374 | 23:27 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Allow for 'extension' rel in json home https://review.openstack.org/583357 | 23:27 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Trusts do not implement patch. https://review.openstack.org/583358 | 23:27 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Move trusts to flask native dispatching https://review.openstack.org/583278 | 23:28 |
| *** ksavich_ has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!