*** erus has quit IRC | 00:44 | |
*** lbragstad has quit IRC | 00:45 | |
*** gyee has quit IRC | 01:03 | |
*** whoami-rajat has joined #openstack-keystone | 01:12 | |
*** jamesmcarthur has joined #openstack-keystone | 01:20 | |
*** jamesmcarthur has quit IRC | 01:23 | |
*** openstackgerrit has quit IRC | 01:30 | |
cmurphy | hogepodge: i wouldn't usually think of a 503 error as a performance error, are there tracebacks in the logs? | 01:43 |
---|---|---|
hogepodge | cmurphy: tracking things down, I think it's an openstack client issue now | 01:57 |
hogepodge | rolling back to Rocky didn't change anything | 01:57 |
hogepodge | oddly enough if I turn the verbosity of the openstack client up I get much better reliability | 01:58 |
cmurphy | hogepodge: hmm, not sure how openstackclient could cause server side issues | 02:00 |
hogepodge | that's the thing, I'm not seeing any server side logs (I thought maybe it was a uwsgi host thing) | 02:00 |
hogepodge | I'm putting keystone under some mild load using the openstack client across a bunch of different containers, and the number of 503s I'm getting back is alarming. | 02:01 |
cmurphy | hogepodge: not seeing *any* server side logs, or not seeing any errors in the logs? | 02:02 |
hogepodge | I'm not seeing any errors in the keystone logs | 02:05 |
cmurphy | ah hmm | 02:05 |
hogepodge | I actually still don't know if it's uwsgi or docker-proxy causing the issue. I'm going to disable the userland proxy and see if that changes anything. | 02:20 |
*** adriant has joined #openstack-keystone | 02:38 | |
*** erus has joined #openstack-keystone | 02:45 | |
*** openstackgerrit has joined #openstack-keystone | 03:00 | |
openstackgerrit | Colleen Murphy proposed openstack/keystone master: Fix werkzeug imports for version 0.15.x https://review.openstack.org/651430 | 03:00 |
cmurphy | prometheanfire: ^ i like that the "deprecation warning" causes a fatal error | 03:00 |
*** vishakha has joined #openstack-keystone | 03:09 | |
prometheanfire | lol, yep | 03:11 |
*** dave-mccowan has quit IRC | 04:15 | |
*** erus has quit IRC | 04:30 | |
*** openstackstatus has quit IRC | 04:35 | |
*** openstackstatus has joined #openstack-keystone | 04:36 | |
*** ChanServ sets mode: +v openstackstatus | 04:36 | |
*** pcaruana has joined #openstack-keystone | 05:06 | |
*** shyamb has joined #openstack-keystone | 06:01 | |
*** shyamb has quit IRC | 06:40 | |
*** shyamb has joined #openstack-keystone | 06:52 | |
openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: Fix sphinx for requirements.txt https://review.openstack.org/651444 | 06:58 |
*** awalende has joined #openstack-keystone | 07:08 | |
*** shyamb has quit IRC | 07:10 | |
*** shyamb has joined #openstack-keystone | 07:10 | |
*** shyamb has quit IRC | 07:26 | |
*** pcaruana has quit IRC | 07:34 | |
*** pcaruana has joined #openstack-keystone | 07:35 | |
*** shyamb has joined #openstack-keystone | 07:37 | |
*** shyamb has quit IRC | 07:58 | |
*** tkajinam has quit IRC | 08:04 | |
*** phasespace has quit IRC | 08:08 | |
*** evrardjp has quit IRC | 08:18 | |
*** evrardjp has joined #openstack-keystone | 08:19 | |
openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: WIP : Missing packages in lower-contraints.txt https://review.openstack.org/651462 | 08:46 |
openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: Fix sphinx for requirements.txt https://review.openstack.org/651444 | 08:47 |
*** shyamb has joined #openstack-keystone | 09:07 | |
*** phasespace has joined #openstack-keystone | 09:30 | |
*** rcernin has quit IRC | 09:47 | |
*** shyamb has quit IRC | 10:35 | |
openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: WIP : Missing packages in lower-contraints.txt https://review.openstack.org/651462 | 10:44 |
*** shyamb has joined #openstack-keystone | 10:47 | |
*** shyamb has quit IRC | 10:55 | |
*** shyamb has joined #openstack-keystone | 11:12 | |
*** sapd1 has quit IRC | 11:29 | |
*** sapd1 has joined #openstack-keystone | 11:29 | |
*** aloga has quit IRC | 11:35 | |
*** aloga has joined #openstack-keystone | 11:36 | |
*** ybunker has joined #openstack-keystone | 11:55 | |
*** raildo has joined #openstack-keystone | 11:56 | |
ybunker | Hi all, I've a question regarding keystone (Juno) upgrade process,... i'm using keystone with swift, and someone at my team do an upgrade of the swift cluster from Juno to Rocky on the data nodes, and left the Proxy nodes with Queens release, so at this point we have: keystone (Juno), swift-data (rocky) and swift-proxy (queens),.. keystone is using old token process, but i know that on rocky that keystone mechanism is deprecated and has | 11:58 |
ybunker | been removed right?, so at this point I need to upgrade de Keystone from Juno to Rocky, is there a process doc to do this? any advice tips? Thanks in advance all | 11:58 |
*** sapd1 has quit IRC | 12:00 | |
*** sapd1 has joined #openstack-keystone | 12:02 | |
*** shyamb has quit IRC | 12:06 | |
*** dave-mccowan has joined #openstack-keystone | 12:16 | |
*** shyamb has joined #openstack-keystone | 12:19 | |
frickler | so I made a patch in devstack dropping the creation of the admin endpoint, but it's still exploding big time. mostly because ksa still seems to default to using the admin interface. is there any plan to mitigate this? http://logs.openstack.org/92/651492/2/check/devstack/357b6f0/controller/logs/screen-g-api.txt.gz#_Apr_10_11_28_00_915073 | 12:23 |
*** awalende has quit IRC | 12:26 | |
*** awalende has joined #openstack-keystone | 12:26 | |
*** awalende has quit IRC | 12:31 | |
*** lbragstad has joined #openstack-keystone | 12:58 | |
*** ChanServ sets mode: +o lbragstad | 12:58 | |
*** irclogbot_1 has joined #openstack-keystone | 13:03 | |
*** altlogbot_2 has joined #openstack-keystone | 13:07 | |
*** dklyle has joined #openstack-keystone | 13:17 | |
*** mvkr has quit IRC | 13:19 | |
*** shyamb has quit IRC | 13:26 | |
ybunker | anyone? | 13:34 |
*** phasespace has quit IRC | 13:36 | |
lbragstad | ybunker what token format are you using? | 13:52 |
*** mvkr has joined #openstack-keystone | 13:53 | |
ybunker | lbragstad: PKI token | 13:54 |
*** vishakha has quit IRC | 13:54 | |
lbragstad | and you're upgrading to? | 13:55 |
lbragstad | Queens or Rocky? | 13:55 |
ybunker | lbragstad: Rocky | 13:55 |
lbragstad | ok - the only token provider available in that release is Fernet | 13:56 |
lbragstad | https://git.openstack.org/cgit/openstack/keystone/tree/setup.cfg?h=stable/rocky#n146 | 13:56 |
lbragstad | when you mentioned "old token process" in your original ping, were you referencing token providers or something else? http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2019-04-10.log.html#t2019-04-10T11:58:48 | 13:56 |
ybunker | lbragstad: got it, and is possible to move from Juno (PKI) to Rocky (Fernet) is 'minimal' disruption? | 13:56 |
ybunker | lbragstad: provider=keystone.token.providers.pkiz.Provider | 13:58 |
lbragstad | ok - i just wanted to make sure we were both referencing the same thing | 13:58 |
lbragstad | are you familiar with Fernet? | 13:59 |
ybunker | lbragstad: not at all :-( | 14:00 |
lbragstad | ok - that's fine | 14:00 |
lbragstad | we have documentation that introduces the idea and describes how to migrate from old token providers to fernet | 14:01 |
lbragstad | https://www.lbragstad.com/blog/what-you-need-to-know-about-keystones-new-default-token-format | 14:01 |
*** mchlumsky has joined #openstack-keystone | 14:01 | |
lbragstad | keep in mind, that post references ocata a bunch because we made Fernet the default token provider that release, but it should be relevant to your situation if you're upgrading to rocky | 14:01 |
ybunker | lbragstad: oh nice, thanks :-) let me take a look on that | 14:02 |
lbragstad | one you get through that - https://docs.openstack.org/keystone/latest/admin/fernet-token-faq.html will be easier to parse | 14:03 |
ybunker | lbragstad: thanks, also i was hopping to approach with the upg, first creating newly VMs with keystone rocky (setup fernet key repo), take a dump backup of the actual keystone-db, apply that backup on keystone-rocky VMs and run the db_sync.. is that even possible? | 14:05 |
*** phasespace has joined #openstack-keystone | 14:05 | |
ybunker | lbragstad: since i dont want to make an in-place upgrade on the actual keystone VM | 14:06 |
lbragstad | well - one big difference is that fernet tokens aren't persisted | 14:12 |
lbragstad | so they're not actually in the database at all | 14:12 |
*** openstackgerrit has quit IRC | 14:14 | |
lbragstad | ybunker describes some of that, too https://www.youtube.com/watch?v=702SRZHdNW8 | 14:17 |
*** erus has joined #openstack-keystone | 14:18 | |
lbragstad | i'm not sure if you have separate database instances, but you could point your rocky keystone nodes to the database and they should work fine | 14:21 |
lbragstad | granted... you're PKI tokens aren't going to be validateable on rocky nodes, and fernet tokens aren't going to be validateable on your juno nodes | 14:22 |
ybunker | lbragstad: oh i see, let me check the video, thanks! | 14:25 |
*** awalende has joined #openstack-keystone | 14:27 | |
*** sapd1 has quit IRC | 14:27 | |
*** sapd1 has joined #openstack-keystone | 14:27 | |
*** awalende has quit IRC | 14:31 | |
gagehugo | o/ | 14:37 |
erus | o/ | 14:45 |
lbragstad | ybunker yep - let us know if you have additional questions | 14:46 |
ybunker | lbragstad: thanks a lot :-) I really appreciate your help with this | 14:47 |
lbragstad | anytime! | 14:47 |
cmurphy | o/ | 14:57 |
*** sapd1 has quit IRC | 15:09 | |
*** sapd1 has joined #openstack-keystone | 15:09 | |
*** dave-mccowan has quit IRC | 15:23 | |
*** gyee has joined #openstack-keystone | 15:26 | |
mnaser | hmm | 15:26 |
mnaser | is it possible that as part of the upgrade to stein, user:admin/role:admin gets scope:system | 15:27 |
mnaser | so it looks like the user:admin gets the role:admin with system scope = all | 15:28 |
mnaser | however it looks like a few groups we had defined that has role:admin on project:admin didn't get those so im trying to figure out where that happens (and if it should happen or not) | 15:29 |
cmurphy | mnaser: did you re-run keystone-manage bootstrap as part of the upgrade? | 15:29 |
mnaser | cmurphy: yeah, OSA runs bootstrap on every run | 15:29 |
cmurphy | that will do it | 15:29 |
mnaser | cmurphy: I see, did I maybe miss a release note that said 'make sure you update your roles accordingly'? | 15:31 |
cmurphy | mnaser: the admin user should still have the admin role on the admin project | 15:32 |
cmurphy | you shouldn't have to update anything | 15:32 |
*** jamesmcarthur has joined #openstack-keystone | 15:32 | |
*** starborn has joined #openstack-keystone | 15:33 | |
mnaser | cmurphy: right, okay, I guess maybe this is a horizon bug, I have some users that can't access certain portions of horizon with some super cryptic traceback that doesn't give any info at all | 15:34 |
mnaser | only difference between them and the admin user was that the admin user has system_scope:all role:admin setup... but I just did that and it's still oddly not showing things up | 15:35 |
* mnaser goes back to dealing with js | 15:35 | |
mnaser | thanks | 15:35 |
cmurphy | mnaser: hmm okay, let us know if it starts looking more like a keystone issue again | 15:35 |
mnaser | cmurphy: will do.. this is stein so it could be something no one has ever ran into as well, the small price to pay :P | 15:36 |
cmurphy | mnaser: thanks for being our guinea pig :) | 15:36 |
mnaser | :> | 15:37 |
*** shyamb has joined #openstack-keystone | 15:44 | |
*** jamesmcarthur has quit IRC | 16:06 | |
*** jamesmcarthur has joined #openstack-keystone | 16:08 | |
knikolla | o/ | 16:13 |
erus | hi knikolla | 16:18 |
knikolla | hey erus :) | 16:18 |
erus | how are you? :) | 16:18 |
*** sapd1 has quit IRC | 16:32 | |
*** sapd1 has joined #openstack-keystone | 16:32 | |
*** jdennis has quit IRC | 16:35 | |
*** jdennis has joined #openstack-keystone | 16:36 | |
*** shyamb has quit IRC | 16:38 | |
*** altlogbot_2 has quit IRC | 16:45 | |
*** phasespace has quit IRC | 16:47 | |
knikolla | erus: i'm good, how are you? | 17:06 |
knikolla | cmurphy: the federated_domain_name conf option doesn't do anything anymore since we have per idp domains | 17:06 |
*** openstackgerrit has joined #openstack-keystone | 17:06 | |
openstackgerrit | Colleen Murphy proposed openstack/keystone master: Fix werkzeug imports for version 0.15.x https://review.openstack.org/651430 | 17:06 |
cmurphy | knikolla: sorry, context? | 17:08 |
knikolla | cmurphy: https://bugs.launchpad.net/keystone/+bug/1754048 | 17:09 |
openstack | Launchpad bug 1754048 in OpenStack Identity (keystone) "Federated domain is reported when validating a federated token" [Medium,Triaged] | 17:09 |
*** mvkr has quit IRC | 17:09 | |
knikolla | i'm assuming we need to go through the normal deprecation cycle regardless? | 17:10 |
cmurphy | i think so, just to warn people that they can take it out of their config | 17:10 |
openstackgerrit | Kristi Nikolla proposed openstack/keystone master: Deprecate [federation] federated_domain_name https://review.openstack.org/651614 | 17:23 |
knikolla | ^^ | 17:24 |
knikolla | cmurphy: fix-werkzeug-for-real, lol | 17:28 |
knikolla | reminds me of most commit message in most repos | 17:28 |
*** sapd1 has quit IRC | 17:29 | |
*** sapd1 has joined #openstack-keystone | 17:30 | |
cmurphy | i had another local branch that was fix-werkzeug where we just pinned it | 17:33 |
cmurphy | knikolla: do you have any advice for this https://bugs.launchpad.net/keystone/+bug/1823847 | 17:34 |
openstack | Launchpad bug 1823847 in OpenStack Identity (keystone) "Multiple rules in a mapping is not working with type: "local" attribute" [Undecided,Invalid] | 17:34 |
knikolla | cmurphy: hmmm... i guess the only way to handle this with a non breaking change is to have a different type "local_if_exists" | 17:37 |
cmurphy | that's what i was thinking :/ | 17:38 |
cmurphy | hate to add more complexity to the mapping api though | 17:39 |
knikolla | cmurphy: a better approach would be https://bugs.launchpad.net/keystone/+bug/1816076 | 17:40 |
openstack | Launchpad bug 1816076 in OpenStack Identity (keystone) "RFE: Extend user API to support federated attributes" [Wishlist,In progress] | 17:40 |
knikolla | this way they would be able to create ephemeral users ahead of time and just map to them ephemerally | 17:40 |
knikolla | if they don't exist they'll be created | 17:41 |
knikolla | if they do, they'll be mapped | 17:41 |
knikolla | i was thinking of taking that on for train | 17:41 |
knikolla | but just for the ephemeral user creation, none of the other list/update/etc features ron had planned | 17:41 |
cmurphy | that would be awesome | 17:42 |
*** jamesmcarthur has quit IRC | 17:42 | |
knikolla | i'll brush up the spec today | 17:43 |
*** sapd1 has quit IRC | 18:00 | |
*** sapd1 has joined #openstack-keystone | 18:00 | |
*** awalende has joined #openstack-keystone | 18:11 | |
*** jamesmcarthur has joined #openstack-keystone | 18:12 | |
*** awalende has quit IRC | 18:15 | |
erus | knikolla i'm good :) | 18:37 |
*** canori01 has joined #openstack-keystone | 18:49 | |
canori01 | Hello, is it possible to allow non-admin users to create projects via policy.json? | 18:50 |
-openstackstatus- NOTICE: Restarting Gerrit on review.openstack.org to pick up new configuration for the replication plugin | 19:05 | |
*** ybunker has quit IRC | 19:15 | |
*** erus has quit IRC | 19:19 | |
lbragstad | canori01 it depends on if you have a custom policy file, some of the defaults in policy.v3cloudsample.json attempt to do that but we're actually fixing those issues directly in keystone | 19:20 |
lbragstad | fixes for those resources (projects being one of them) will be available in Stein | 19:21 |
canori01 | lbragstad: I don't curerntly have a policy.json. I was looking at policy.v3cloudsample.json. Is there somewhere I could see what the default rules are? | 19:24 |
canori01 | I'm running queens and just taking defaults I guess since my policy.json is blank | 19:25 |
lbragstad | correct | 19:25 |
lbragstad | you can generate the defaults from code using ``oslopolicy-sample-generator --namespace keystone`` | 19:25 |
canori01 | thank you | 19:26 |
*** awalende has joined #openstack-keystone | 19:28 | |
*** erus has joined #openstack-keystone | 19:34 | |
*** nkinder has joined #openstack-keystone | 19:57 | |
nkinder | cmurphy, let me know if my comment here addresses your question - https://review.openstack.org/#/c/649177/6/keystone/tests/unit/test_backend_ldap.py | 19:58 |
*** jamesmcarthur has quit IRC | 19:59 | |
cmurphy | nkinder: i think so, still need to stare at it for a bit | 20:03 |
cmurphy | also added gyee as reviewer | 20:03 |
*** starborn has quit IRC | 20:04 | |
*** sapd1 has quit IRC | 20:10 | |
*** sapd1 has joined #openstack-keystone | 20:16 | |
*** awalende has quit IRC | 20:19 | |
*** awalende has joined #openstack-keystone | 20:19 | |
*** awalende has quit IRC | 20:24 | |
*** sapd1 has quit IRC | 20:34 | |
*** phasespace has joined #openstack-keystone | 20:36 | |
*** sapd1 has joined #openstack-keystone | 20:37 | |
openstackgerrit | erus proposed openstack/keystone master: Add new attribute to the federation protocol API https://review.openstack.org/637305 | 20:46 |
*** mvkr has joined #openstack-keystone | 20:52 | |
openstackgerrit | erus proposed openstack/keystone master: Add new attribute to the federation protocol API https://review.openstack.org/637305 | 20:53 |
openstackgerrit | erus proposed openstack/keystone master: Add new attribute to the federation protocol API https://review.openstack.org/637305 | 20:59 |
*** raildo has quit IRC | 21:06 | |
*** sapd1 has quit IRC | 21:14 | |
*** sapd1 has joined #openstack-keystone | 21:19 | |
openstackgerrit | ayoung proposed openstack/keystone master: Predictable IDs for Roles https://review.openstack.org/651655 | 21:44 |
openstackgerrit | erus proposed openstack/keystone master: Add new attribute to the federation protocol API https://review.openstack.org/637305 | 21:46 |
*** sapd1 has quit IRC | 21:50 | |
*** sapd1 has joined #openstack-keystone | 21:54 | |
*** erus has quit IRC | 22:01 | |
*** erus has joined #openstack-keystone | 22:03 | |
*** rcernin has joined #openstack-keystone | 22:06 | |
*** sapd1 has quit IRC | 22:41 | |
*** sapd1 has joined #openstack-keystone | 22:42 | |
*** erus has quit IRC | 22:44 | |
*** whoami-rajat has quit IRC | 22:51 | |
*** tkajinam has joined #openstack-keystone | 22:53 | |
*** sapd1 has quit IRC | 22:55 | |
*** sapd1 has joined #openstack-keystone | 22:58 | |
cmurphy | lbragstad: http://git.openstack.org/cgit/openstack-infra/reviewstats/ | 23:03 |
lbragstad | oh - nice, it supports that? | 23:03 |
lbragstad | er - collecting all reviews for a given release? | 23:03 |
cmurphy | i couldn't get it to do per release but i used the number of days since stein opened | 23:04 |
cmurphy | so the number i ended up with is across all branches | 23:04 |
cmurphy | which i think is fine, it's still activity that happened since the last time we gave an update | 23:04 |
lbragstad | right - it's close enough | 23:10 |
*** sapd1 has quit IRC | 23:23 | |
*** sapd1 has joined #openstack-keystone | 23:24 | |
*** david-lyle has joined #openstack-keystone | 23:36 | |
*** dklyle has quit IRC | 23:36 | |
*** david-lyle has quit IRC | 23:46 | |
*** sapd1 has quit IRC | 23:48 | |
*** sapd1 has joined #openstack-keystone | 23:50 | |
*** rcernin has quit IRC | 23:52 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!