Friday, 2018-02-23

« Thursday, 2018-02-22 Index Saturday, 2018-02-24 »
*** itlinux has quit IRC00:03
*** daidv has quit IRC00:06
*** daidv has joined #openstack-keystone00:06
*** cmurphy has quit IRC00:17
*** cmurphy has joined #openstack-keystone00:17
*** AlexeyAbashkin has joined #openstack-keystone00:19
*** lbragstad has quit IRC00:23
*** AlexeyAbashkin has quit IRC00:23
*** oikiki has quit IRC00:26
*** oikiki has joined #openstack-keystone00:27
*** edmondsw has quit IRC00:31
*** oikiki has quit IRC00:39
*** r-daneel has quit IRC00:43
*** threestrands has joined #openstack-keystone00:44
openstackgerritTim Burke proposed openstack/keystonemiddleware master: Only include response body if there's a response  https://review.openstack.org/53810801:01
openstackgerritTim Burke proposed openstack/keystonemiddleware master: Properly zero out max_retries in test_http_error_not_cached_token  https://review.openstack.org/54722801:01
*** itlinux has joined #openstack-keystone01:46
*** edmondsw has joined #openstack-keystone02:17
*** edmondsw has quit IRC02:22
*** itlinux has quit IRC02:28
*** mattoliverau_ is now known as mattoliverau02:35
*** itlinux has joined #openstack-keystone02:38
*** annp has joined #openstack-keystone02:49
openstackgerritDinesh Bhor proposed openstack/python-keystoneclient master: Add return-request-id-to-caller function(v3/contrib)  https://review.openstack.org/26800302:56
openstackgerritDinesh Bhor proposed openstack/python-keystoneclient master: Add return-request-id-to-caller function(v3/contrib)  https://review.openstack.org/26800302:57
*** Faster-Fanboi_ has quit IRC03:10
*** oikiki has joined #openstack-keystone03:12
*** Faster-Fanboi has joined #openstack-keystone03:12
*** itlinux has quit IRC03:30
*** oikiki has quit IRC03:33
*** gyee has quit IRC03:56
*** edmondsw has joined #openstack-keystone04:06
*** edmondsw has quit IRC04:11
*** chergd has quit IRC04:11
*** AlexeyAbashkin has joined #openstack-keystone04:18
*** links has joined #openstack-keystone04:21
*** AlexeyAbashkin has quit IRC04:23
*** links has quit IRC04:33
*** itlinux has joined #openstack-keystone04:36
*** Faster-Fanboi has quit IRC04:46
*** links has joined #openstack-keystone04:47
*** Faster-Fanboi has joined #openstack-keystone04:52
*** Faster-Fanboi has quit IRC05:14
*** Faster-Fanboi has joined #openstack-keystone05:20
*** oikiki has joined #openstack-keystone05:26
openstackgerritSuramya proposed openstack/keystone master: Consolidate identity-domain-specific-config.rst  https://review.openstack.org/54727405:27
*** oikiki has quit IRC05:39
*** Suramya has joined #openstack-keystone05:46
openstackgerritSuramya proposed openstack/keystone master: Consolidate identity-token-binding.rst  https://review.openstack.org/54727705:50
*** edmondsw has joined #openstack-keystone05:55
*** edmondsw has quit IRC05:59
*** idlemind has quit IRC06:02
*** namnh has joined #openstack-keystone06:12
*** threestrands has quit IRC06:13
*** namnh has quit IRC06:16
*** annp has quit IRC06:16
*** namnh has joined #openstack-keystone06:16
*** annp has joined #openstack-keystone06:16
openstackgerritSuramya proposed openstack/keystone master: Consolidate endpoint-policy.rst  https://review.openstack.org/54728806:19
openstackgerritSuramya proposed openstack/keystone master: Consolidate endpoint-filtering.rst Consolidate from configuration.rst into endpoint-filtering.rst.  https://review.openstack.org/54729106:31
openstackgerritSuramya proposed openstack/keystone master: Consolidate oauth1.rst  https://review.openstack.org/54729306:47
*** pcaruana has joined #openstack-keystone06:56
openstackgerritSuramya proposed openstack/keystone master: Consolidate endpoint-filtering.rst  https://review.openstack.org/54729106:58
*** pcaruana has quit IRC07:02
*** pcaruana has joined #openstack-keystone07:02
openstackgerritSuramya proposed openstack/keystone master: Consolidate auth-totp.rst  https://review.openstack.org/54729807:09
*** josecastroleon has joined #openstack-keystone07:16
openstackgerritSuramya proposed openstack/keystone master: Consolidate service-catalog.rst  https://review.openstack.org/54730307:20
*** josecastroleon has quit IRC07:25
*** josecastroleon has joined #openstack-keystone07:25
*** cz2 has quit IRC07:30
*** eEbx- has quit IRC07:30
*** homeski has quit IRC07:30
*** itlinux has quit IRC07:31
*** nkinder has quit IRC07:31
*** panbalag has quit IRC07:31
*** sileht has quit IRC07:31
*** dgonzalez has quit IRC07:31
openstackgerritSuramya proposed openstack/keystone master: Consolidate health-check-middleware.rst  https://review.openstack.org/54730807:34
*** edmondsw has joined #openstack-keystone07:43
*** edmondsw has quit IRC07:48
*** josecastroleon has quit IRC07:51
*** itlinux has joined #openstack-keystone07:53
*** nkinder has joined #openstack-keystone07:53
*** panbalag has joined #openstack-keystone07:53
*** sileht has joined #openstack-keystone07:53
*** dgonzalez has joined #openstack-keystone07:53
*** josecastroleon has joined #openstack-keystone07:56
*** cz2 has joined #openstack-keystone07:57
*** eEbx- has joined #openstack-keystone07:57
*** homeski has joined #openstack-keystone07:57
*** cz2 has quit IRC08:11
*** eEbx- has quit IRC08:11
*** homeski has quit IRC08:11
*** itlinux has quit IRC08:11
*** nkinder has quit IRC08:11
*** panbalag has quit IRC08:11
*** sileht has quit IRC08:11
*** dgonzalez has quit IRC08:11
*** rcernin has quit IRC08:16
*** tesseract has joined #openstack-keystone08:19
*** dgonzalez has joined #openstack-keystone08:22
*** nkinder has joined #openstack-keystone08:22
*** panbalag has joined #openstack-keystone08:23
*** sileht has joined #openstack-keystone08:23
*** edmondsw has joined #openstack-keystone08:28
*** homeski has joined #openstack-keystone08:33
*** eEbx has joined #openstack-keystone08:33
*** cz2 has joined #openstack-keystone08:33
*** rcernin has joined #openstack-keystone09:09
*** slunkad has quit IRC09:13
*** slunkad has joined #openstack-keystone09:14
openstackgerritSuramya proposed openstack/keystone master: Consolidate event_notifications.rst  https://review.openstack.org/54733709:18
*** slunkad has quit IRC09:27
*** slunkad has joined #openstack-keystone09:28
*** slunkad has quit IRC09:38
*** belmoreira has joined #openstack-keystone09:42
*** slunkad has joined #openstack-keystone09:50
*** rcernin has quit IRC09:59
*** masber has quit IRC10:01
*** namnh has quit IRC10:44
*** daidv has quit IRC10:58
*** annp has quit IRC11:11
*** links has quit IRC11:42
*** d0ugal has quit IRC11:52
*** links has joined #openstack-keystone11:56
*** d0ugal has joined #openstack-keystone11:58
*** raildo has joined #openstack-keystone12:10
*** edmondsw has quit IRC12:18
*** edmondsw has joined #openstack-keystone12:18
*** edmondsw has quit IRC12:23
*** bhagyashris has quit IRC12:29
*** links has quit IRC12:30
*** masber has joined #openstack-keystone12:38
*** links has joined #openstack-keystone12:44
*** edmondsw has joined #openstack-keystone13:10
*** links has quit IRC13:56
*** dklyle has quit IRC14:04
*** idlemind has joined #openstack-keystone14:16
*** Guest60045 is now known as zeus14:18
*** zeus has joined #openstack-keystone14:18
*** nixi_girl has joined #openstack-keystone14:25
*** lbragstad has joined #openstack-keystone14:37
*** ChanServ sets mode: +o lbragstad14:37
openstackgerritLance Bragstad proposed openstack/keystone-specs master: Add specification for a capabilities API  https://review.openstack.org/54716214:45
*** r-daneel has joined #openstack-keystone14:54
*** spilla has joined #openstack-keystone14:56
*** edmondsw has quit IRC15:00
*** edmondsw has joined #openstack-keystone15:00
*** edmondsw has quit IRC15:05
*** masber has quit IRC15:05
lbragstadwould anyone be opposed to me moving the libraries session to the end of Wednesday to accommodate gmann_ here ? http://lists.openstack.org/pipermail/openstack-dev/2018-February/127670.html15:13
lbragstadso - after lunch, if the tempest discussion interests you, we could head there for the v2.0 removal bits and then pick up normal keystone stuff later, and wrap up the libraries discussion at the end of the day15:14
*** nixi_girl has quit IRC15:20
*** david-lyle has joined #openstack-keystone15:21
*** dave-mccowan has joined #openstack-keystone15:29
*** nixi_girl has joined #openstack-keystone15:37
knikollao/15:46
*** dklyle has joined #openstack-keystone15:50
gagehugoo/15:53
*** david-lyle has quit IRC15:53
gagehugono opposition from me15:54
lbragstadcool15:55
*** cwright has quit IRC15:59
*** nixi_girl has quit IRC16:00
*** jaosorior has quit IRC16:05
*** openstackgerrit has quit IRC16:33
*** oikiki has joined #openstack-keystone17:08
*** gyee has joined #openstack-keystone17:10
*** dave-mccowan has quit IRC17:15
*** dave-mccowan has joined #openstack-keystone17:16
*** dklyle has quit IRC17:17
*** edmondsw has joined #openstack-keystone17:18
*** Suramya has quit IRC17:23
*** belmoreira has quit IRC17:25
*** d0ugal has quit IRC17:46
*** d0ugal has joined #openstack-keystone17:54
*** rm_work has quit IRC18:06
*** dansmith has quit IRC18:07
*** zeus has quit IRC18:07
*** homeski has quit IRC18:07
*** tesseract has quit IRC18:19
*** openstackgerrit has joined #openstack-keystone18:20
openstackgerritOpenStack Release Bot proposed openstack/keystone master: Update reno for stable/newton  https://review.openstack.org/54759318:20
openstackgerritOpenStack Release Bot proposed openstack/keystoneauth master: Update reno for stable/newton  https://review.openstack.org/54759518:21
openstackgerritOpenStack Release Bot proposed openstack/keystonemiddleware master: Update reno for stable/newton  https://review.openstack.org/54759718:21
*** r-daneel has quit IRC18:24
*** lbragstad has quit IRC18:31
*** pcaruana has quit IRC18:33
*** dansmith has joined #openstack-keystone18:33
*** bigjools has joined #openstack-keystone18:33
*** dansmith is now known as Guest1702618:33
*** zeus has joined #openstack-keystone18:35
*** zeus is now known as Guest8913418:36
*** robcresswell_ has joined #openstack-keystone18:37
*** hogepodge_ has joined #openstack-keystone18:37
*** mwhahaha_ has joined #openstack-keystone18:37
*** gus_ has joined #openstack-keystone18:38
*** cmurphy_ has joined #openstack-keystone18:40
*** andreykurilin_ has joined #openstack-keystone18:43
*** jamielennox_ has joined #openstack-keystone18:43
*** johnthetubaguy_ has joined #openstack-keystone18:44
*** cmurphy has quit IRC18:44
*** robcresswell has quit IRC18:44
*** mwhahaha has quit IRC18:44
*** martinus__ has quit IRC18:44
*** andreykurilin has quit IRC18:44
*** hogepodge has quit IRC18:44
*** gus has quit IRC18:44
*** jamielennox has quit IRC18:44
*** johnthetubaguy has quit IRC18:44
*** mwhahaha_ is now known as mwhahaha18:44
*** jamielennox_ is now known as jamielennox18:44
*** robcresswell_ is now known as robcresswell18:44
*** hogepodge_ is now known as hogepodge18:44
*** martinus__ has joined #openstack-keystone18:51
*** rm_work has joined #openstack-keystone18:51
*** rm_work has quit IRC18:51
*** rm_work has joined #openstack-keystone18:51
*** lbragstad has joined #openstack-keystone18:57
*** ChanServ sets mode: +o lbragstad18:57
*** mchlumsky has quit IRC19:06
*** mchlumsky has joined #openstack-keystone19:07
*** itlinux has joined #openstack-keystone19:22
*** itlinux has quit IRC19:35
*** chergd has joined #openstack-keystone19:38
*** itlinux has joined #openstack-keystone19:44
*** knasim-wrs has joined #openstack-keystone19:46
knasim-wrshi, doing a skip-release (Newton->Pike) online migration, we sometimes see the DB contract fail19:47
knasim-wrsand its always in the 014_X DB contraction script19:48
knasim-wrsALTER TABLE local_user ADD CONSTRAINT local_user_user_id_fkey FOREIGN KEY(user_id, domain_id) REFERENCES "user" (id, domain_id) ON DELETE CASCADE ON UPDATE CASCADE19:48
knasim-wrsthe contraction is attempting to modify the local_user DB table to add domain_id as a foreign key and if you get unlucky then Keystone is doing an Identity backend call in the same DB which causes a deadlock19:49
knasim-wrsso the question is, do we need 014_X (the domain id foreign constraint) DB migration script or I can omit it19:49
knasim-wrsthe other option is to momentarily shut off Keystone before doing a Contract but that will be service disruptive19:50
*** itlinux has quit IRC19:50
knasim-wrs@lbragstad: We discussed this issue previously. Its quite reproducible on skip level Online migrations19:51
lbragstadthis is the migration in question - https://github.com/openstack/keystone/blob/1175a356fc71036b38c41025f5a90e16f91bbc98/keystone/common/sql/contract_repo/versions/014_contract_add_domain_id_to_user_table.py ?19:52
lbragstadit's failing here specifically - https://github.com/openstack/keystone/blob/1175a356fc71036b38c41025f5a90e16f91bbc98/keystone/common/sql/contract_repo/versions/014_contract_add_domain_id_to_user_table.py#L46-L49 ?19:53
lbragstadknasim-wrs another work around you could do - would be to roll out a custom policy that disables anyone from making an identity request (e.g. create_user, update_user, delete_user, etc...)19:58
lbragstadthat would prevent a partial service outage until you can run the migration19:58
lbragstadonce the migrations are complete you could revert the policy change and open up the identity API again19:59
lbragstadnot ideal - but it might be a better option that taking the service down19:59
knasim-wrsthanks Lance, but I think that one of my services (Nova) was simply doing an AUTH while the contraction happened. This is the Keystone DB operation that was happening at that time:20:01
knasim-wrsSELECT local_user.id AS local_user_id, local_user.user_id AS local_user_user_id, local_user.domain_id AS local_user_domain_id, local_user.name AS local_user_name, local_user.failed_auth_count AS local_user_failed_auth_count, local_user.failed_auth_at AS local_user_failed_auth_at, anon_1.user_id AS anon_1_user_id, anon_1.user_domain_id AS anon_1_use20:01
knasim-wrsr_domain_id20:01
knasim-wrsWHERE "user".id = '20bd4216910340bc8e6019f6d826f9d7') AS anon_1 JOIN local_user ON anon_1.user_id = local_user.user_id AND anon_1.user_domain_id = local_user.domain_id ORDER BY anon_1.user_id, anon_1.user_domain_id20:02
knasim-wrsyes lbragstad, the links you send indicate which DB migration is failing20:03
lbragstadi'm trying to figure out why the deadlock happened20:03
lbragstadknasim-wrs do you have a paste of the error?20:07
lbragstadwith a little more context?20:07
knasim-wrssure20:09
knasim-wrs1 sec20:09
knasim-wrshttps://thepasteb.in/p/0ghJWZQnm8Pf520:11
knasim-wrsso the problem is that DB contraction wants Exlusive Lock on the local_user DB table. Keystone wants Shared Lock on the local_user DB table hence the deadlock. The contraction could should play nice and either yield or take out a Shared lock20:13
*** oikiki has quit IRC20:14
knasim-wrsProcess 76955 (KEYSTONE) waits for AccessShareLock on relation 17776 of database 16401; blocked by process 76959 ( DB Contract).20:14
*** oikiki has joined #openstack-keystone20:14
lbragstadhmm20:14
lbragstadprocess 76955 is the read?20:15
lbragstadit looks like20:15
knasim-wrsProcess 76959(DB Contract) waits for AccessExclusiveLock on relation 17886 of database 16401; blocked by process 76955 (Keystone)20:15
lbragstadand process 76959 is the fk operation20:15
knasim-wrsthats right20:15
knasim-wrsfk operation wants exclusive lock20:15
lbragstadright20:15
knasim-wrsand keystone was doing an AUTH in the local_user DB at that time and wants a Shared lock on that table20:15
lbragstadhmm20:16
openstackgerritGage Hugo proposed openstack/keystone master: Add new setup commands for token keys  https://review.openstack.org/54765720:16
knasim-wrskeystone was just doing a read operation on the table, contract wants to do a write operation20:16
lbragstadright...20:16
lbragstadso - do you cache identity responses?20:16
*** itlinux has joined #openstack-keystone20:17
lbragstadhttps://github.com/openstack/keystone/blob/master/etc/keystone.conf.sample#L1055-L105720:17
lbragstadi was thinking that might help because calling the identity API to get a user by id would return a cached user reference instead of having to go pull it from the database20:19
lbragstadand hopefully eliminate the read operation20:20
knasim-wrscaching was not in this load. I can try with caching enabled so yes in that case it won't be hitting the DB20:22
*** david-lyle has joined #openstack-keystone20:25
lbragstadthat might be an easy workaround to get you past the migration20:31
*** pramodrj07 has joined #openstack-keystone20:34
*** itlinux has quit IRC20:36
*** Exhar has quit IRC20:41
*** itlinux has joined #openstack-keystone20:42
*** r-daneel has joined #openstack-keystone21:00
*** pramodrj07 has quit IRC21:02
*** rmascena has joined #openstack-keystone21:02
*** raildo has quit IRC21:05
*** rmascena has quit IRC21:06
*** oikiki has quit IRC21:07
*** oikiki has joined #openstack-keystone21:09
*** cmurphy_ is now known as cmurphy21:13
*** nkinder has quit IRC21:16
*** itlinux has quit IRC21:16
*** panbalag has quit IRC21:19
*** dave-mcc_ has joined #openstack-keystone21:35
*** dave-mccowan has quit IRC21:36
*** edmondsw has quit IRC21:40
*** Exhar has joined #openstack-keystone21:56
*** spilla has quit IRC22:17
*** masber has joined #openstack-keystone22:27
*** masber has quit IRC22:30
*** masber has joined #openstack-keystone22:30
*** dave-mcc_ has quit IRC22:34
*** dave-mccowan has joined #openstack-keystone22:46
gmann_lbragstad: thanks22:53
lbragstadgmann_ no problem, thanks for hosting the topic23:01
*** masuberu has joined #openstack-keystone23:07
*** masber has quit IRC23:09
*** dave-mccowan has quit IRC23:11
*** mchlumsky has quit IRC23:12
*** itlinux has joined #openstack-keystone23:22
*** r-daneel has quit IRC23:27
*** itlinux has quit IRC23:28
*** agrebennikov has joined #openstack-keystone23:57
« Thursday, 2018-02-22 Index Saturday, 2018-02-24 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!