| *** tosky has quit IRC | 00:12 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 00:38 | |
| *** Dinesh_Bhor has left #openstack-keystone | 00:38 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 00:39 | |
| *** threestrands has joined #openstack-keystone | 00:40 | |
| *** threestrands has quit IRC | 00:40 | |
| *** threestrands has joined #openstack-keystone | 00:40 | |
| imacdonn | kmalloc: I found that the hang was occurring when trying to reserve() from the the cache pool. Making the pool size larger seems to have fixed it (or delayed it????) .... also, curiously, it seems that the connections to memcached are getting closed immediately after use - if they're getting closed, I'm not sure why the pool needs to be larger ... hmmmm | 00:43 |
|---|---|---|
| *** dave-mccowan has joined #openstack-keystone | 00:57 | |
| *** itlinux has quit IRC | 00:59 | |
| *** dave-mccowan has quit IRC | 01:09 | |
| *** harlowja has quit IRC | 01:13 | |
| *** dave-mccowan has joined #openstack-keystone | 01:17 | |
| *** dave-mcc_ has joined #openstack-keystone | 01:22 | |
| *** dave-mccowan has quit IRC | 01:24 | |
| *** annp has quit IRC | 01:43 | |
| openstackgerrit | wangxiyuan proposed openstack/keystoneauth master: Add netloc and version check for version discovery https://review.openstack.org/583215 | 01:56 |
| openstackgerrit | Dao Cong Tien proposed openstack/keystone master: Adds doc8 check to pep8 https://review.openstack.org/583196 | 02:28 |
| *** alex_xu has quit IRC | 02:31 | |
| *** alex_xu has joined #openstack-keystone | 02:33 | |
| openstackgerrit | ayoung proposed openstack/keystoneauth master: Add netloc and version check for version discovery https://review.openstack.org/583215 | 02:54 |
| *** mguz has quit IRC | 02:57 | |
| *** annp has joined #openstack-keystone | 03:11 | |
| *** flwang1 has quit IRC | 04:57 | |
| *** timburke_ has joined #openstack-keystone | 05:09 | |
| *** openstackgerrit has quit IRC | 05:10 | |
| *** timothyb89_ has joined #openstack-keystone | 05:12 | |
| *** chason_ has joined #openstack-keystone | 05:14 | |
| *** links has joined #openstack-keystone | 05:15 | |
| *** jmlowe has quit IRC | 05:16 | |
| *** wxy has quit IRC | 05:16 | |
| *** robcresswell has quit IRC | 05:16 | |
| *** mwhahaha has quit IRC | 05:16 | |
| *** pas-ha has quit IRC | 05:16 | |
| *** portdirect has quit IRC | 05:16 | |
| *** jamespage has quit IRC | 05:16 | |
| *** NobodyCam has quit IRC | 05:16 | |
| *** wlmbasson has quit IRC | 05:16 | |
| *** jmccrory has quit IRC | 05:16 | |
| *** timothyb89 has quit IRC | 05:16 | |
| *** jamielennox has quit IRC | 05:16 | |
| *** timburke has quit IRC | 05:16 | |
| *** chason has quit IRC | 05:16 | |
| *** tristanC has quit IRC | 05:16 | |
| *** harlowja has joined #openstack-keystone | 05:28 | |
| *** harlowja has quit IRC | 05:32 | |
| *** jmlowe has joined #openstack-keystone | 05:35 | |
| *** wxy has joined #openstack-keystone | 05:35 | |
| *** robcresswell has joined #openstack-keystone | 05:35 | |
| *** mwhahaha has joined #openstack-keystone | 05:35 | |
| *** portdirect has joined #openstack-keystone | 05:35 | |
| *** jamespage has joined #openstack-keystone | 05:35 | |
| *** pas-ha has joined #openstack-keystone | 05:35 | |
| *** NobodyCam has joined #openstack-keystone | 05:35 | |
| *** wlmbasson has joined #openstack-keystone | 05:35 | |
| *** jmccrory has joined #openstack-keystone | 05:35 | |
| *** jamielennox has joined #openstack-keystone | 05:35 | |
| *** martinus__ has joined #openstack-keystone | 05:47 | |
| *** tristanC has joined #openstack-keystone | 06:30 | |
| *** pcaruana has joined #openstack-keystone | 06:35 | |
| *** flwang1 has joined #openstack-keystone | 06:47 | |
| *** flwang1 has quit IRC | 06:49 | |
| *** flwang1 has joined #openstack-keystone | 06:58 | |
| *** ispp has joined #openstack-keystone | 07:00 | |
| *** peereb has joined #openstack-keystone | 07:09 | |
| *** tesseract has joined #openstack-keystone | 07:18 | |
| *** rcernin has quit IRC | 07:28 | |
| *** dklyle has quit IRC | 07:35 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 07:45 | |
| *** tosky has joined #openstack-keystone | 08:06 | |
| *** markvoelker_ has quit IRC | 08:07 | |
| *** threestrands has quit IRC | 08:51 | |
| *** openstackgerrit has joined #openstack-keystone | 08:51 | |
| openstackgerrit | Merged openstack/keystone master: Strict two level limit model https://review.openstack.org/557696 | 08:51 |
| *** gongysh has joined #openstack-keystone | 09:08 | |
| *** hoonetorg has quit IRC | 09:34 | |
| *** hoonetorg has joined #openstack-keystone | 09:48 | |
| *** markvoelker has joined #openstack-keystone | 10:08 | |
| *** kukacz_ has quit IRC | 10:12 | |
| *** kukacz_ has joined #openstack-keystone | 10:13 | |
| *** dmellado has quit IRC | 10:36 | |
| *** gongysh has quit IRC | 10:39 | |
| *** markvoelker has quit IRC | 10:42 | |
| *** dmellado has joined #openstack-keystone | 10:42 | |
| *** pcichy has quit IRC | 10:55 | |
| *** pcichy has joined #openstack-keystone | 10:56 | |
| *** gongysh has joined #openstack-keystone | 10:59 | |
| *** annp has quit IRC | 11:00 | |
| *** gongysh has quit IRC | 11:22 | |
| *** markvoelker has joined #openstack-keystone | 11:39 | |
| *** kevko has joined #openstack-keystone | 11:40 | |
| kevko | Hi guys, just a question, is keystone tested against ldap by tempest ? | 11:40 |
| *** pcichy has quit IRC | 11:52 | |
| *** raildo has joined #openstack-keystone | 11:53 | |
| *** markvoelker has quit IRC | 12:05 | |
| *** markvoelker has joined #openstack-keystone | 12:05 | |
| *** Dinesh_Bhor has quit IRC | 12:06 | |
| *** edmondsw has joined #openstack-keystone | 12:11 | |
| *** vigneshwar has joined #openstack-keystone | 12:23 | |
| vigneshwar | Hi hello,, | 12:23 |
| cmurphy | kevko: sort of, we have a job that sets up a domain-specific ldap backend for devstack but I don't think the tempest tests use that domain as far as i ccan tell | 12:23 |
| vigneshwar | can anyone tell me ..in which location keystone credentials will be stored.. | 12:23 |
| cmurphy | vigneshwar: what do you mean? | 12:26 |
| vigneshwar | cmurphy: after login to keystone, our credential(username & password) , where it will be stored ? | 12:30 |
| cmurphy | vigneshwar: the username and a hash of the password is stored in keystone's database | 12:31 |
| vigneshwar | how can i access that database ? | 12:31 |
| vigneshwar | cmurphy : how can i access that database ? | 12:32 |
| cmurphy | vigneshwar: it's the mysql database that you set up when you install keystone https://docs.openstack.org/keystone/latest/install/keystone-install-ubuntu.html#prerequisites | 12:32 |
| cmurphy | vigneshwar: why do you need to access it? | 12:32 |
| vigneshwar | cmurphy : I thought it wont store any credentials and confidential information details in DB.. | 12:34 |
| vigneshwar | cmurphy : i am new to keystone , just learning | 12:34 |
| cmurphy | vigneshwar: it doesn't store the password, only a hash of the password | 12:35 |
| vigneshwar | is there any mechanism in openstack that will automatically clear those credentials after logout ? | 12:36 |
| cmurphy | vigneshwar: I think you're confusing keystone with horizon, horizon doesn't store any credentials when you log in or out of it | 12:37 |
| vigneshwar | yes,,yes,, | 12:37 |
| cmurphy | you wouldn't want to clear the credentials from keystone because you'd never be able to log back in | 12:38 |
| *** sonuk has joined #openstack-keystone | 12:38 | |
| vigneshwar | cmurphy: about horizon- is it a latest feature ? | 12:38 |
| *** sonuk_ has quit IRC | 12:39 | |
| cmurphy | horizon is an openstack project https://docs.openstack.org/horizon/latest/ | 12:39 |
| cmurphy | it's a web dashboard | 12:39 |
| kevko | cmurphy: because there is issue with python3 ..keystone not working with ldap ... | 12:40 |
| vigneshwar | yes..i meant in the latest openstack release( horizon does'nt store password)..or it is one of the feature of horizon from the begining ? | 12:41 |
| cmurphy | kevko: oh thats sounds likely :( | 12:41 |
| cmurphy | kevko: what issue are you seeing? | 12:41 |
| cmurphy | vigneshwar: horizon has never stored passwords | 12:41 |
| kevko | cmurphy: there is an issue with encode and decode functions in keystone/identity/backends/ldap/common.py | 12:43 |
| cmurphy | kevko: what release? | 12:43 |
| kevko | cmurphy: queens | 12:43 |
| kevko | cmurphy: also in master | 12:44 |
| kevko | cmurphy: https://github.com/openstack/keystone/blob/master/keystone/identity/backends/ldap/common.py | 12:44 |
| kevko | cmurphy: utf8_encode , utf8_decode | 12:44 |
| kevko | cmurphy: also fakeldap.py for testing has to be reworked to really have good tests for py2 and also py3 | 12:45 |
| cmurphy | yeah we do have unit tests for python 3 but i can believe they missed something | 12:46 |
| cmurphy | kevko: can you file a bug and/or propose a patch? | 12:46 |
| *** pcichy has joined #openstack-keystone | 12:46 | |
| kevko | cmurphy: yes, tests are working ... but with real ldap on python3 it is not working ..on py2 it is working | 12:46 |
| cmurphy | oh i see | 12:47 |
| kevko | cmurphy: did you try ? | 13:08 |
| cmurphy | kevko: sorry no I don't have a python3 installation handy at the moment, but could you file a bug and include the traceback? | 13:10 |
| kevko | cmurphy: yes, i will do it .. | 13:11 |
| *** lbragstad has quit IRC | 13:19 | |
| *** mchlumsky has joined #openstack-keystone | 13:19 | |
| *** pcichy has quit IRC | 13:40 | |
| *** pcichy has joined #openstack-keystone | 13:45 | |
| *** pcichy has quit IRC | 13:45 | |
| *** pcichy has joined #openstack-keystone | 13:46 | |
| *** lbragstad has joined #openstack-keystone | 13:46 | |
| *** ChanServ sets mode: +o lbragstad | 13:46 | |
| *** r-daneel has joined #openstack-keystone | 13:48 | |
| *** dklyle has joined #openstack-keystone | 13:50 | |
| *** gongysh has joined #openstack-keystone | 13:51 | |
| *** sonuk_ has joined #openstack-keystone | 13:51 | |
| *** sonuk has quit IRC | 13:54 | |
| *** dklyle has quit IRC | 14:00 | |
| *** spilla has joined #openstack-keystone | 14:03 | |
| errr | Im having touble with getting my mapping correct for my federated users. Keystone is telling me "Could not map any federated user properties to identity values" | 14:10 |
| errr | I have the following mapping: https://gist.github.com/michaelrice/ddabaf70acf118948216452b8172285b | 14:10 |
| errr | Could someone tell me if that mapping would mean that my fed user would map its email to the username in keystone then make them a member of the federated_users group in the federated_domain domain? | 14:11 |
| *** vrv_ has joined #openstack-keystone | 14:14 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Allow for 'extension' rel in json home https://review.openstack.org/583357 | 14:15 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Move trusts to flask native dispatching https://review.openstack.org/583278 | 14:16 |
| knikolla | o/ | 14:16 |
| kmalloc | mornin | 14:16 |
| *** xinran__ has joined #openstack-keystone | 14:18 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Trusts do not implement patch. https://review.openstack.org/583358 | 14:19 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Move trusts to flask native dispatching https://review.openstack.org/583278 | 14:20 |
| cmurphy | errr: is "email" a property that your idp is providing in its assertion? you might see that error if that property doesn't exist | 14:20 |
| errr | cmurphy: yes it is there | 14:20 |
| cmurphy | errr: are you using shibboleth as the service provider? | 14:21 |
| lbragstad | it sounds like you're trying to do auto-provisioning | 14:21 |
| errr | cmurphy: I used the firefox saml tracer to verify its there | 14:21 |
| errr | cmurphy: yes shibboleth | 14:21 |
| cmurphy | errr: you might need to edit /etc/shibboleth/attribute-map.xml and add a line to get it to recognize "email" as a property | 14:21 |
| errr | ah ok | 14:22 |
| errr | thanks | 14:22 |
| knikolla | yup, was just going to suggest that | 14:22 |
| knikolla | shibboleth will only parse what's in attribute-map | 14:22 |
| errr | that is exactly it | 14:23 |
| cmurphy | like this http://www.gazlene.net/demystifying-keystone-federation.html#configure-metadata | 14:23 |
| errr | yall are the greatest :D | 14:23 |
| cmurphy | ^.^ | 14:23 |
| lbragstad | https://docs.openstack.org/keystone/latest/advanced-topics/federation/federated_identity.html#keystone-to-keystone is a short snippet from the docs, but cmurphy's presentation is on point | 14:23 |
| knikolla | errr: do you have control over the idp? | 14:24 |
| cmurphy | lbragstad: that doesn't really cover this problem unfortunately | 14:24 |
| errr | I have watched her talk at the last summit like a dozen times. I wanted a longer more indepth version just on mapping :D | 14:24 |
| errr | knikolla: yes I do | 14:25 |
| lbragstad | cmurphy: the presentation or the doc link? | 14:25 |
| cmurphy | oh shoot that's what i should have submitted | 14:25 |
| cmurphy | lbragstad: the doc link | 14:25 |
| lbragstad | oh - it was the only reference to the attribute-map.xml in our docs | 14:25 |
| errr | cmurphy: you should do a vlog on youtube about that mapping stuff you wanted to cover but ran out of time on | 14:25 |
| knikolla | errr: instead of using just "email" as an attribute you should look into what is the standard URI for referencing email | 14:25 |
| cmurphy | errr: lol | 14:26 |
| errr | for reals :D | 14:26 |
| errr | knikolla: good idea. I will check that out | 14:26 |
| errr | cmurphy: there are literally dozens of us out there having to deal with this federation stuff that would eat that video up | 14:27 |
| cmurphy | well what we should really do is fix our docs so that this stuff isn't so hard | 14:28 |
| errr | it doesnt help that its not as popular as basic stuff | 14:28 |
| lbragstad | ++ we still have two separate sections dedicated to federation in our docs | 14:28 |
| knikolla | then we'd be out of a job if that's too easy, haha | 14:28 |
| errr | haha | 14:28 |
| errr | federation setup is a job Im happy to be out of | 14:29 |
| lbragstad | same - i'll actively automate myself out the door on that one | 14:29 |
| knikolla | federation is all i do, it's not that bad | 14:29 |
| errr | once you get a firm understanding of it Im sure it gets easier | 14:31 |
| errr | Ive been doing a lot of this with the OSA project so these steps are handled by ansible, and now Im having to do it on red hat osp and its all manual | 14:32 |
| errr | so I keep missing things | 14:32 |
| errr | and the only thing red hat even has documented for federation is using mellon but using keycloak and their red hat sso so it turns into a big guessing game when you want any other idp | 14:33 |
| lbragstad | oh - i was going to say... osa should have most of this ready to go in their os_keystone role | 14:33 |
| errr | they do | 14:33 |
| knikolla | errr: have you looked into openid connect? | 14:33 |
| errr | knikolla: no I havent | 14:33 |
| knikolla | keycloak supports both saml and openid connect | 14:34 |
| errr | the customer Im doing this for wants okta | 14:34 |
| knikolla | quick google shows okta supporting openid connect | 14:34 |
| knikolla | but apart from that, things are pretty similar regardless of idp in terms of saml | 14:36 |
| knikolla | and all that needs guessing is in the metadata | 14:36 |
| *** vigneshwar has quit IRC | 14:38 | |
| *** peereb has quit IRC | 14:51 | |
| *** pcaruana has quit IRC | 14:56 | |
| *** pcichy has quit IRC | 14:59 | |
| *** jmlowe has quit IRC | 15:00 | |
| *** gongysh has quit IRC | 15:13 | |
| errr | knikolla: any idea why I would be getting "Group federated_users returned by mapping fico_rules_mapping was not found in the backend." when I log in now? openstack group list shows that group.. | 15:20 |
| errr | oh I think I know what the problem is. never mind | 15:22 |
| *** jmlowe has joined #openstack-keystone | 15:24 | |
| *** ispp has quit IRC | 15:41 | |
| lbragstad | mordred: curious if you'd be able to take another gander at https://review.openstack.org/#/c/583215/5 sometime before tomorrow? | 15:46 |
| *** links has quit IRC | 15:55 | |
| * lbragstad goes to get lunch | 15:58 | |
| *** pcaruana has joined #openstack-keystone | 16:00 | |
| *** lbragstad has quit IRC | 16:02 | |
| kmalloc | o/ | 16:11 |
| *** xinran__ has quit IRC | 16:28 | |
| imacdonn | kmalloc: hi! Will bug you in a bit ... on a conf call now ... see update from last night on the cache pool thing | 16:29 |
| kmalloc | imacdonn: sure. i think i saw it | 16:29 |
| kmalloc | but i was sleepy and didn't read it too closely | 16:29 |
| imacdonn | k | 16:29 |
| * kmalloc does scrollback thing | 16:29 | |
| kmalloc | imacdonn: ahh | 16:31 |
| kmalloc | imacdonn: i think i know what is happening. The pool is too small to handle the concurrency so you're just wedged in a wait-for-a-memcached and/or we have a deadlock bug | 16:32 |
| kmalloc | imacdonn: the immediate closing of connections seems weird. | 16:32 |
| imacdonn | kmalloc: yeah, two weird things ... the immediate close, and the fact that the pool members still seem to be "in use" (???) but there are no connections to memcached | 16:33 |
| imacdonn | kmalloc: Once it gets into the bad state, it seems to stay there (doesn't recover) | 16:34 |
| kmalloc | like i said at the beginning, unfortunately this code is very fragile =/ | 16:35 |
| kmalloc | and a lot of that has to do with layering multiple uses of thread-local and having to reference internal interfaces | 16:35 |
| *** AlexeyAbashkin has quit IRC | 16:36 | |
| kmalloc | wait, in queens you're deadlocked in "reserve()" | 16:37 |
| * kmalloc tries to find this | 16:37 | |
| imacdonn | yeah, so it appears ... I haven't been able to trace that into oslo_cache yet | 16:38 |
| kmalloc | you have some sort of mis-match in code bases | 16:38 |
| kmalloc | i think | 16:38 |
| *** AlexeyAbashkin has joined #openstack-keystone | 16:39 | |
| imacdonn | hmm | 16:39 |
| kmalloc | oh no | 16:39 |
| kmalloc | wow, we did bad things here | 16:39 |
| kmalloc | this is just awful code :( | 16:39 |
| *** AlexeyAbashkin has quit IRC | 16:43 | |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Expose random uuid bug in cadf notifications https://review.openstack.org/580780 | 16:44 |
| *** spilla_ has joined #openstack-keystone | 16:46 | |
| *** spilla has quit IRC | 16:46 | |
| *** lbragstad has joined #openstack-keystone | 16:50 | |
| *** ChanServ sets mode: +o lbragstad | 16:50 | |
| *** nels has quit IRC | 16:52 | |
| imacdonn | kmalloc: so is the pool thing even worse than the eventlet thing? Seems to be a "rock and a hard place" :/ | 16:53 |
| *** jdennis has quit IRC | 17:06 | |
| *** jdennis has joined #openstack-keystone | 17:07 | |
| kmalloc | imacdonn: lol sadly | 17:09 |
| kmalloc | imacdonn: i'm trying to see what could cause it to hang | 17:09 |
| kmalloc | but... i'm thinking there is an issue with not getting the connections back on the queue | 17:10 |
| kmalloc | like we're reaping them while they are active | 17:10 |
| imacdonn | OK ... I can try to add some debug logging if you can help me find the right place | 17:11 |
| kmalloc | oh, no i think i see the issue. | 17:11 |
| * kmalloc sighs. | 17:12 | |
| kmalloc | the pool is just broken. | 17:12 |
| kmalloc | yep... this is untested and i found the issue | 17:12 |
| kmalloc | imacdonn: we don't put the connection back on the queue ever | 17:13 |
| lbragstad | is this an oslo.cache bug? | 17:13 |
| kmalloc | lbragstad: probably more of an ksm bug | 17:13 |
| imacdonn | heh, that seems to make the pool .... of limited value ;) | 17:13 |
| kmalloc | lbragstad: we do self._pool.get() | 17:14 |
| lbragstad | did we move ksm to oslo.cache though? | 17:14 |
| kmalloc | right | 17:14 |
| lbragstad | oh... | 17:14 |
| kmalloc | lbragstad: the issue is accquire does all the logic to get stuff back on the queue | 17:14 |
| kmalloc | .... | 17:14 |
| lbragstad | and we override it? | 17:14 |
| kmalloc | so we basically yeild out all the connections and let them expire | 17:14 |
| kmalloc | no, we used to just do .get() which odes the logic | 17:14 |
| kmalloc | that accquire does (oslo.cache) | 17:14 |
| lbragstad | right | 17:14 |
| kmalloc | when it was moved to oslo.cache we basically hit the queue.Queue "get" function | 17:15 |
| lbragstad | oh - now we're just calling into the pool directly | 17:15 |
| kmalloc | and so we're not doing the context manager that places the connection back into the queue object | 17:15 |
| kmalloc | basically... 100% totally untested code | 17:15 |
| lbragstad | weird | 17:15 |
| lbragstad | i wonder why that happened | 17:15 |
| lbragstad | sounds like the fix should be pretty easy? can't we just call .get() again? | 17:16 |
| kmalloc | no | 17:16 |
| kmalloc | it's... i can fix this | 17:16 |
| kmalloc | but the issue is that the code in ksm was made too simple | 17:16 |
| kmalloc | or... | 17:17 |
| kmalloc | this is an even bigger/longer running bug that i thought | 17:17 |
| * kmalloc is tracing backwards | 17:17 | |
| kmalloc | this has been broken since pike | 17:18 |
| kmalloc | we broke it somewhere post ocata | 17:18 |
| kmalloc | then carried the brokenness forward and into oslo.cache | 17:18 |
| lbragstad | hmm | 17:19 |
| kmalloc | when it was re-written to use queue.Queue | 17:19 |
| lbragstad | do you think this is something we can address by tomorrow? or are we going to wait until stein | 17:19 |
| kmalloc | uhm it's something we're going to have to backport | 17:19 |
| kmalloc | it is probably a KSM only fix.. | 17:20 |
| kmalloc | i think | 17:20 |
| kmalloc | but i want to figure out where accquire is called. | 17:20 |
| lbragstad | do we have a bug report for it yet? | 17:20 |
| kmalloc | and right now i'm not seeing it | 17:20 |
| kmalloc | not yet. | 17:20 |
| kmalloc | imacdonn: so the reason this doesn't work in pike is because it's been broken since pike =/ | 17:20 |
| kmalloc | lbragstad: give me a few minutes and i'll be able to write up a bug report on this. | 17:20 |
| lbragstad | sweet | 17:21 |
| kmalloc | lbragstad: i just want to be sure i'm writing the right info. | 17:21 |
| * kmalloc sighs. | 17:21 | |
| kmalloc | lets plan for a Freeze exception for this bug and backports to follow | 17:21 |
| kmalloc | backports to pike are going to be wonky since it didn't lean on oslo_cache | 17:21 |
| imacdonn | kmalloc: you mean the pool implementation, right? The issue my prod environment has on Pike is the eventlet thing consuming unbounded memcached connections ... separate issue (?) | 17:21 |
| kmalloc | yes | 17:22 |
| lbragstad | knikolla: speaking of ksm, do you have thoughts on https://review.openstack.org/#/c/505396/4 | 17:22 |
| kmalloc | the pool implementation | 17:22 |
| imacdonn | right | 17:22 |
| imacdonn | I've only tried to use that on Queens, in my lab | 17:22 |
| kmalloc | so even if you wanted to make the pool work on pike. it would have the same(ish) issues | 17:22 |
| imacdonn | gotcha | 17:22 |
| kmalloc | because it was re-written poorly and has next to zero testing | 17:22 |
| kmalloc | and in openstack, if it's not tested... it is broken | 17:22 |
| kmalloc | lbragstad: unrelated to this, flask for trusts should be working now. | 17:23 |
| imacdonn | yeah... it sounds like no one is using it, which jeopardises my warm-fuzzies a bit | 17:23 |
| kmalloc | lbragstad: and next after trusts will be auth. | 17:23 |
| kmalloc | lbragstad: because getting into things like users, roles, etc becomes massive patches. | 17:23 |
| lbragstad | kmalloc: do you think we're going to be able to convert all the API prior to stein? | 17:23 |
| imacdonn | I wonder how others are dealing with this .... maybe just allowing memcached to have a gazillion connections ? | 17:23 |
| kmalloc | lbragstad: i hope so. | 17:23 |
| lbragstad | or should we just queue them up and land them first thing in stein? | 17:23 |
| kmalloc | lbragstad: i would expect to have them all done in Rocky as long as no one balks at the code | 17:24 |
| kmalloc | lbragstad: it also has no effect [besides some extra code in the code base] maintaining a mix between the two methods | 17:24 |
| kmalloc | lbragstad: it is 100% ok to have some paths on flask and some on legacy webob | 17:25 |
| kmalloc | the reason i wrote all the supporting code :) | 17:25 |
| kmalloc | imacdonn: that is the easiest solution, but my guess, tight limits on WAIT states and low request numbers | 17:25 |
| lbragstad | if we want an exception for ksm, we'll have to get signoff by the release team (since that's not totally in our control) | 17:27 |
| lbragstad | from an exception perspective | 17:27 |
| kmalloc | well, i'll do what i can, but my guess is we will want to plan for one | 17:27 |
| kmalloc | i don't know if the gate is going to be friendly with us (looking at how slow patches march through atm) | 17:27 |
| kmalloc | for integrated testing that is | 17:28 |
| lbragstad | ok - as soon as we get a bug open, i can swing over to -releases and -requirements and get a feel for things | 17:28 |
| imacdonn | kmalloc: not sure that WAIT states can help .. the connections are all ESTABLISHED | 17:28 |
| kmalloc | imacdonn: but connections drop out as the requests in eventlet drop | 17:29 |
| kmalloc | so probably some tuning there. | 17:29 |
| kmalloc | or... they don't use caching | 17:29 |
| kmalloc | it's amazing how many installs dont | 17:29 |
| kmalloc | OR worse.. they use in-process cache | 17:29 |
| imacdonn | I didn't have it configured properly for a while .. it made quite a difference when I got it right | 17:29 |
| imacdonn | but then I ran into this next hurdle (with my largest deployment) | 17:30 |
| * lbragstad needs to write that doc | 17:30 | |
| kmalloc | lbragstad: yeah ok i found the bug... someone got too clever, looking into the git blame now | 17:32 |
| kmalloc | and will open a bug shortly | 17:32 |
| imacdonn | kmalloc: confused on the tuning thing .... is there some way to manipulate how long the connections are held in the non-pool case? It seems like most of the config options pertain to the pool | 17:32 |
| kmalloc | i am unsure how we will logistically test this. | 17:32 |
| kmalloc | lbragstad: i guess i just need to mock the hell out of things. | 17:32 |
| kmalloc | lbragstad: *grump* | 17:32 |
| kmalloc | imacdonn: i think eventlet can be mucked with in weird ways to force faster reaping of data in the thread locals | 17:33 |
| kmalloc | imacdonn: but honestly, i try not to think about eventlet... this is a MAJOR reason keystone dropped eventlet | 17:33 |
| kmalloc | imacdonn: we almost require caching. | 17:33 |
| *** vrv_ has quit IRC | 17:33 | |
| *** timothyb89_ is now known as timothyb89 | 17:35 | |
| *** tesseract has quit IRC | 17:35 | |
| imacdonn | kmalloc: yeah ... so back from the hard place to the rock :) | 17:37 |
| *** jdennis has quit IRC | 17:37 | |
| kmalloc | so, here is the quick fix to try in your lab | 17:37 |
| imacdonn | k | 17:37 |
| kmalloc | and i'm writing up a bug. | 17:37 |
| *** jdennis has joined #openstack-keystone | 17:37 | |
| kmalloc | change KSM code: change the hanging reserve() method to be: | 17:37 |
| kmalloc | https://www.irccloud.com/pastebin/2gubO3x8/ | 17:37 |
| kmalloc | i don't think .get() is working as a context manager the way we expect | 17:39 |
| imacdonn | ok, working on that | 17:39 |
| *** spilla has joined #openstack-keystone | 17:42 | |
| *** spilla_ has quit IRC | 17:42 | |
| imacdonn | I actually see connections to memcached now ... one connection per neutron-api worker process | 17:42 |
| kmalloc | :) | 17:43 |
| kmalloc | so, i have NO idea how this isn't throwing errors all over the place right now. | 17:44 |
| imacdonn | no lockups so-far ... I have a loop running continually hitting neutron-api with a request that requires auth | 17:44 |
| kmalloc | but... | 17:44 |
| kmalloc | not the fixed one | 17:44 |
| kmalloc | the current code base | 17:44 |
| openstackgerrit | Merged openstack/keystone master: Switch to python-ldap https://review.openstack.org/582977 | 17:44 |
| kmalloc | lbragstad, cmurphy: ^ if that is the case, ldappool also needs an update | 17:45 |
| *** r-daneel has quit IRC | 17:45 | |
| imacdonn | well, the Queens code has a couple of other issues (those two bugs I mentioned yesterday) ... I suspect that no one is actually using the pool implementation :/ | 17:45 |
| kmalloc | imacdonn: as i said, sadly if "it isn't tested it is broken" is the case for most code | 17:46 |
| imacdonn | yeah | 17:46 |
| kmalloc | so i'm checking a few more things, but it seems like acquire is the right entry | 17:46 |
| *** r-daneel has joined #openstack-keystone | 17:47 | |
| lbragstad | kmalloc: https://review.openstack.org/#/c/583164/ | 17:48 |
| kmalloc | lbragstad: looking | 17:50 |
| kmalloc | yes | 17:50 |
| kmalloc | already commented | 17:50 |
| *** clarkb has joined #openstack-keystone | 17:50 | |
| lbragstad | yep - i should have looked before sending that | 17:50 |
| clarkb | github would like keystone to know that pysaml2 should be set to >= 4.5.0 | 17:51 |
| kmalloc | clarkb: in lower-constraints? sure. we can spin that up shortly | 17:51 |
| kmalloc | clarkb: thanks. | 17:51 |
| clarkb | requirements.txt but will affect lower constraints too | 17:51 |
| kmalloc | ah wait... we have minimums in requirements.txt? i can't keep track of what goes in reqs | 17:52 |
| clarkb | hrm | 17:52 |
| kmalloc | vs what goes in *-contstraints | 17:52 |
| clarkb | reading your requirements file doens't really make sense vs what they emailed me about | 17:52 |
| clarkb | I think their new python dep checking must be buggy | 17:52 |
| kmalloc | hehe | 17:52 |
| clarkb | pysaml2>=4.0.2,!=4.0.3,!=4.0.4,!=4.0.5,!=4.0.5rc1,!=4.1.0,!=4.2.0,!=4.3.0,!=4.4.0 is our requirement line | 17:52 |
| kmalloc | right. | 17:53 |
| lbragstad | we explicity exclude a bunch | 17:53 |
| clarkb | oh its that less than 4.5 is buggy | 17:53 |
| kmalloc | that looks sane[ish] | 17:53 |
| clarkb | so they want us to prop up the minimum | 17:53 |
| *** r-daneel_ has joined #openstack-keystone | 17:53 | |
| lbragstad | but ultimately the first bit should cover us, yeah? | 17:53 |
| kmalloc | Oh! | 17:53 |
| kmalloc | got it | 17:53 |
| lbragstad | pysaml2>=4.0.2 | 17:53 |
| *** r-daneel has quit IRC | 17:53 | |
| *** r-daneel_ is now known as r-daneel | 17:53 | |
| kmalloc | right and thye want >=4.5.0 | 17:53 |
| lbragstad | yep | 17:53 |
| lbragstad | to me, what we have should be fine? | 17:54 |
| kmalloc | they want the minimum higher | 17:54 |
| lbragstad | oh | 17:54 |
| lbragstad | nevermind | 17:54 |
| clarkb | yup | 17:54 |
| kmalloc | hehe | 17:54 |
| kmalloc | lbragstad: i did the exact same thing man | 17:54 |
| clarkb | because of moderate security issue in 4.40 | 17:54 |
| clarkb | er 4.4.0 | 17:54 |
| kmalloc | which is fair | 17:54 |
| kmalloc | i'm 100% ok with bumping the minimum as long as it wont break/cause issues with anyone | 17:54 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Bump lower constraint for pysaml2 to 4.5.0 https://review.openstack.org/583678 | 17:55 |
| *** dklyle has joined #openstack-keystone | 17:55 | |
| lbragstad | i need to fill out the commit message with details | 17:55 |
| clarkb | lbragstad: "Known moderate severity security vulnerability detected in pysaml2 <=4.4.0 defined in requirements.txt. " says github | 17:55 |
| lbragstad | wfm | 17:55 |
| kmalloc | imacdonn, lbragstad: https://bugs.launchpad.net/keystonemiddleware/+bug/1782404 | 17:56 |
| openstack | Launchpad bug 1782404 in keystonemiddleware "keystonemiddleware doesn't work with memcachepool" [High,Triaged] - Assigned to Morgan Fainberg (mdrnstm) | 17:56 |
| kmalloc | lbragstad: just proposing to the stable branches is sufficient, ya? [ksm] | 17:56 |
| kmalloc | for the backport | 17:56 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Bump lower constraint for pysaml2 to 4.5.0 https://review.openstack.org/583678 | 17:57 |
| imacdonn | kmalloc: thanks! | 17:57 |
| kmalloc | imacdonn: i think i gave correct credit to you in the bug report | 17:57 |
| kmalloc | feel free to fix that if i did not | 17:57 |
| lbragstad | kmalloc: sweet | 17:57 |
| kmalloc | now let me write a test for this... | 17:58 |
| kmalloc | at least synthetic for the memcachepool | 17:58 |
| imacdonn | I guess I'll try to cherrypick those other two fixes to Queens | 18:00 |
| *** dklyle has quit IRC | 18:01 | |
| kmalloc | if those fixes are needed for queens we should propose backports | 18:01 |
| imacdonn | https://review.openstack.org/#/c/542133/ and https://review.openstack.org/#/c/542134/ | 18:01 |
| imacdonn | yes, that's what I meant by cherrypick | 18:01 |
| kmalloc | :) | 18:01 |
| imacdonn | bah .. merge conflict on the second one .. I'm going to need coffee first ;) | 18:03 |
| imacdonn | do ksm proposed changes not get posted here? | 18:04 |
| * imacdonn pokes openstack in the ribs | 18:04 | |
| kmalloc | not backports | 18:04 |
| kmalloc | only master | 18:04 |
| imacdonn | ok | 18:04 |
| kmalloc | same as with keystone | 18:04 |
| lbragstad | kmalloc: you don't have a patch proposed to master for the ksm bug you just reported, do you? | 18:06 |
| kmalloc | no, working on the tests for it now | 18:06 |
| kmalloc | will be posted shortly | 18:06 |
| lbragstad | ok - not trying to rush you | 18:06 |
| lbragstad | just trying to track things we need to land before tomorrow so we can cut ksm and ksa for rocky | 18:07 |
| *** pcaruana has quit IRC | 18:08 | |
| knikolla | lbragstad: responded (wrt ksm docs patch) | 18:10 |
| lbragstad | knikolla: thanks | 18:11 |
| *** itlinux has joined #openstack-keystone | 18:13 | |
| lbragstad | so far - these are the things i think we should land before tomorrow if possible | 18:15 |
| lbragstad | https://review.openstack.org/#/q/starredby:lbragstad%2540gmail.com+status:open | 18:15 |
| lbragstad | i'll add kmalloc's ksm patch to that list as soon as it is available | 18:15 |
| *** dklyle has joined #openstack-keystone | 18:27 | |
| kmalloc | posting soon. | 18:28 |
| kmalloc | running pep8 | 18:28 |
| gagehugo | lbragstad: looking | 18:29 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystonemiddleware master: Fix KeystoneMiddleware memcachepool abstraction https://review.openstack.org/583695 | 18:29 |
| *** itlinux has quit IRC | 18:31 | |
| imacdonn | kmalloc: urgh, your fix is going to further merge-conflict with this: https://review.openstack.org/#/c/542134/2/keystonemiddleware/auth_token/_cache.py | 18:33 |
| imacdonn | for Queens backports, I mean | 18:34 |
| *** itlinux has joined #openstack-keystone | 18:34 | |
| kmalloc | imacdonn: i am working on backports now | 18:35 |
| kmalloc | imacdonn: checking tests before posting. | 18:35 |
| imacdonn | I guess it won't conflict if I can get that one backported first .. oh, are you backporting the other fixes too ? | 18:35 |
| kmalloc | just mine | 18:35 |
| kmalloc | but the backports can happen in any order | 18:35 |
| imacdonn | I already got this one started: https://review.openstack.org/#/c/583684/ | 18:35 |
| kmalloc | wfm. | 18:35 |
| kmalloc | https://review.openstack.org/583698 | 18:36 |
| kmalloc | for queens | 18:36 |
| kmalloc | https://review.openstack.org/#/c/583699/ for pike | 18:38 |
| kmalloc | lbragstad: ^ | 18:39 |
| lbragstad | sweet | 18:39 |
| kmalloc | lbragstad: i'll leave it to you to setup release reviews to include these fixes. | 18:40 |
| lbragstad | https://review.openstack.org/#/q/starredby:lbragstad%2540gmail.com+status:open | 18:40 |
| openstackgerrit | Merged openstack/keystonemiddleware master: Document endpoint interface and region behavior https://review.openstack.org/505396 | 18:42 |
| lbragstad | kmalloc: a release note for the connection pooling patch would be ++ | 18:43 |
| lbragstad | otherwise the patch looks sane | 18:43 |
| lbragstad | imacdonn: are you able to verify ^ | 18:43 |
| lbragstad | ? | 18:43 |
| kmalloc | hmm. ok | 18:44 |
| kmalloc | i am *not* backporting a release note fwiw. | 18:44 |
| kmalloc | so i guess it was good i did it in this order | 18:44 |
| imacdonn | I'm all tangled up in the other change right now | 18:44 |
| imacdonn | https://review.openstack.org/#/c/542134/2/keystonemiddleware/auth_token/_cache.py <= this removed the use of "with" | 18:45 |
| kmalloc | imacdonn: my fix solves that as well | 18:45 |
| kmalloc | fwiw | 18:45 |
| kmalloc | so that doesn't need to be backported if my fix is landing | 18:45 |
| imacdonn | right, ok .. that's what I was trying to figure out | 18:46 |
| kmalloc | yep :) | 18:46 |
| lbragstad | does _pool still not have an __exit__? | 18:46 |
| kmalloc | it doesn't need one | 18:47 |
| kmalloc | _pool.acquire() is a context manager | 18:47 |
| lbragstad | because it's using .aquire()? | 18:47 |
| lbragstad | oh | 18:47 |
| kmalloc | yep | 18:47 |
| lbragstad | nice | 18:47 |
| kmalloc | *THAT* was the fix that should have been, not just removing with | 18:47 |
| kmalloc | removing with is what got us into this broken-ness | 18:47 |
| kmalloc | instead of fixing to call the context manager | 18:48 |
| lbragstad | thisis only in master https://review.openstack.org/#/c/542134/2 | 18:48 |
| kmalloc | right | 18:48 |
| kmalloc | and the fix made reserve() hang | 18:48 |
| lbragstad | so we're fixing it in rocky, which is good | 18:48 |
| kmalloc | the correct fix would have been -> make it use acquire() not "remove the with" context | 18:49 |
| lbragstad | sure | 18:49 |
| * kmalloc glares at the release note command | 18:49 | |
| lbragstad | but we wont have a gap where we're letting the broken behavior slip through a release | 18:49 |
| kmalloc | right, we just previously had the whole system error when being used | 18:50 |
| kmalloc | i dunno, both seem super broken to me | 18:50 |
| kmalloc | ;) | 18:50 |
| lbragstad | yeah - but we won't have a broken rocky release | 18:50 |
| lbragstad | or a queens release | 18:50 |
| kmalloc | well.. uh | 18:52 |
| kmalloc | re have a broken queens release | 18:52 |
| kmalloc | no __exit__ :P | 18:52 |
| lbragstad | bah | 18:52 |
| kmalloc | and a broken pike release | 18:52 |
| imacdonn | current queens releases are broken ... but kmalloc's patch will fix it .. it's just that the fix looks slightly different for queens vs master | 18:52 |
| lbragstad | but we just need kmalloc's backport | 18:53 |
| imacdonn | due to the other "fix" not having been backported | 18:53 |
| lbragstad | ack | 18:53 |
| *** harlowja has joined #openstack-keystone | 18:54 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystonemiddleware master: Fix KeystoneMiddleware memcachepool abstraction https://review.openstack.org/583695 | 18:56 |
| kmalloc | lbragstad: ^ with release note | 18:57 |
| *** dklyle has quit IRC | 19:23 | |
| *** felipemonteiro_ has joined #openstack-keystone | 19:24 | |
| *** felipemonteiro_ is now known as felipemonteiro | 19:24 | |
| openstackgerrit | Ben Nemec proposed openstack/oslo.policy master: Avoid redundant policy syntax checks https://review.openstack.org/511426 | 19:25 |
| *** felipemonteiro_ has joined #openstack-keystone | 19:30 | |
| *** felipemonteiro has quit IRC | 19:33 | |
| openstackgerrit | Ben Nemec proposed openstack/oslo.policy master: Avoid redundant policy syntax checks https://review.openstack.org/511426 | 19:40 |
| *** felipemonteiro_ has quit IRC | 19:50 | |
| *** felipemonteiro_ has joined #openstack-keystone | 19:50 | |
| *** flwang1 has quit IRC | 19:54 | |
| *** itlinux has quit IRC | 20:09 | |
| openstackgerrit | Gage Hugo proposed openstack/ldappool master: Switch to python-ldap again https://review.openstack.org/583164 | 20:25 |
| gagehugo | cmurphy kmalloc https://review.openstack.org/#/c/583164/ | 20:26 |
| mordred | kmalloc: wanna see a weird bug? | 20:26 |
| mordred | kmalloc: I'm not sure if it's sdk or ksa yet ... | 20:26 |
| mordred | kmalloc: http://paste.openstack.org/show/726235/ | 20:29 |
| *** martinus__ has quit IRC | 20:30 | |
| *** r-daneel has quit IRC | 20:31 | |
| openstackgerrit | Merged openstack/keystone master: Add project_id filter for listing limit https://review.openstack.org/579330 | 20:34 |
| *** jmlowe has quit IRC | 20:36 | |
| mordred | kmalloc: oh - wait - does v3password auth plugin skip discovery? | 20:37 |
| *** itlinux has joined #openstack-keystone | 20:37 | |
| mordred | yup | 20:39 |
| *** dklyle has joined #openstack-keystone | 20:39 | |
| *** r-daneel has joined #openstack-keystone | 20:40 | |
| *** itlinux has quit IRC | 20:42 | |
| *** felipemonteiro__ has joined #openstack-keystone | 20:48 | |
| *** felipemonteiro_ has quit IRC | 20:52 | |
| *** flwang1 has joined #openstack-keystone | 20:58 | |
| *** raildo has quit IRC | 21:06 | |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Expose random uuid bug in cadf notifications https://review.openstack.org/580780 | 21:06 |
| *** felipemonteiro__ has quit IRC | 21:07 | |
| *** felipemonteiro__ has joined #openstack-keystone | 21:07 | |
| *** felipemonteiro__ has quit IRC | 21:28 | |
| lbragstad | related bug https://bugs.launchpad.net/keystoneauth/+bug/1782434 | 21:28 |
| openstack | Launchpad bug 1782434 in keystoneauth "passwordv3 plugin doesn't implement discovery" [Undecided,New] | 21:28 |
| *** dklyle has quit IRC | 21:42 | |
| *** dklyle has joined #openstack-keystone | 21:45 | |
| *** david-lyle has joined #openstack-keystone | 21:53 | |
| *** dklyle has quit IRC | 21:54 | |
| *** edmondsw has quit IRC | 21:54 | |
| *** elibrokeit has quit IRC | 22:00 | |
| kmalloc | mordred: oh fun. | 22:01 |
| kmalloc | mordred: yeah v3 should skip, it should know it's already v3 | 22:01 |
| *** linkmark has joined #openstack-keystone | 22:03 | |
| imacdonn | kmalloc: FYI, trying to enable the "advanced pool" on Pike produces a different failure, prob similar to the "arguments" one for Queens - haven't dug into it too much yet - http://paste.openstack.org/show/726238/ | 22:06 |
| kmalloc | imacdonn: yep | 22:07 |
| kmalloc | imacdonn: that would be my guess | 22:07 |
| kmalloc | something weird for sure | 22:07 |
| imacdonn | looks kinda sorta like https://bugs.launchpad.net/keystone/+bug/1440493 | 22:14 |
| openstack | Launchpad bug 1440493 in keystonemiddleware "Crash with python-memcached==1.5.4" [Undecided,In progress] | 22:14 |
| *** ztrawhcse has joined #openstack-keystone | 22:15 | |
| cmurphy | thanks gagehugo | 22:17 |
| *** ztrawhcse is now known as elibrokeit | 22:17 | |
| *** jmlowe has joined #openstack-keystone | 22:24 | |
| *** jmlowe has quit IRC | 22:25 | |
| *** gongysh has joined #openstack-keystone | 22:25 | |
| *** jmlowe has joined #openstack-keystone | 22:26 | |
| *** lbragstad has quit IRC | 22:29 | |
| *** david-lyle has quit IRC | 22:31 | |
| *** rcernin has joined #openstack-keystone | 22:32 | |
| *** imacdonn has quit IRC | 22:44 | |
| openstackgerrit | Merged openstack/ldappool master: Switch to python-ldap again https://review.openstack.org/583164 | 22:44 |
| *** imacdonn has joined #openstack-keystone | 22:45 | |
| *** jmlowe has quit IRC | 22:58 | |
| *** jmlowe has joined #openstack-keystone | 22:58 | |
| *** itlinux has joined #openstack-keystone | 23:04 | |
| *** jmlowe has quit IRC | 23:05 | |
| *** dklyle has joined #openstack-keystone | 23:09 | |
| imacdonn | kmalloc: It looks like the following didn't make it into ksm. Applying the change seems to work. Thoughts? https://review.openstack.org/#/c/175291/ | 23:11 |
| kmalloc | Most of that is in oslo_cache | 23:13 |
| imacdonn | post-Pike, you mean ? | 23:13 |
| kmalloc | Yeah | 23:13 |
| kmalloc | Looks like we might be doa in Pike. But we can land that still to make it work. | 23:14 |
| imacdonn | I'm contemplating whether I want to make it work with Pike, or use it as leverage to get the Ops team to do the upgrade to Queens | 23:15 |
| *** itlinux has quit IRC | 23:16 | |
| imacdonn | is there process for getting a fix into a "stable" release, when it's not a backport? | 23:18 |
| *** tosky has quit IRC | 23:20 | |
| kmalloc | proposing the fix, usually needs to land in master, but this case a bug and fix specific for the release is fine | 23:20 |
| kmalloc | then the stable team will evaluate it | 23:20 |
| kmalloc | basically lbragstad[m] and myself are stable-core for keystone. | 23:20 |
| imacdonn | right, ok | 23:21 |
| kmalloc | chances are, as long as it doesn't break anything, we can fix it. | 23:21 |
| imacdonn | I applied the same fix as in the above URL, and it seems to be working fine (along with your acquire() fix) | 23:21 |
| imacdonn | I'll do a bug and proposed fix, and you guys can decide if you like it .... | 23:22 |
| kmalloc | wfm | 23:24 |
| *** itlinux has joined #openstack-keystone | 23:28 | |
| *** jmlowe has joined #openstack-keystone | 23:37 | |
| *** threestrands has joined #openstack-keystone | 23:50 | |
| *** threestrands has quit IRC | 23:50 | |
| *** threestrands has joined #openstack-keystone | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!