Friday, 2018-11-30

*** Liang__ has joined #openstack-cinder00:03
*** dave-mccowan has quit IRC00:08
*** angela-s has quit IRC00:14
*** brinzhang has joined #openstack-cinder00:18
*** threestrands has joined #openstack-cinder00:18
brinzhangsmcginnis: Hi, could you please review this patch [1]. [1] https://review.openstack.org/#/c/615494/00:19
*** ganso has quit IRC00:38
*** lixiaoy1 has joined #openstack-cinder00:40
*** erlon has quit IRC01:38
*** chhagarw has joined #openstack-cinder01:44
*** chhagarw has quit IRC01:50
*** dave-mccowan has joined #openstack-cinder01:50
*** hedvig_01 has quit IRC01:57
*** dklyle has joined #openstack-cinder01:59
*** dklyle has quit IRC02:05
openstackgerritfutaotao proposed openstack/cinder master: Update FusionStorage Cinder Driver by using REST API  https://review.openstack.org/61995202:05
openstackgerritLeopard Ma proposed openstack/cinder master: Fix revert a volume from a snapshot  https://review.openstack.org/62105602:06
*** mriedem_afk has quit IRC02:19
*** Dinesh_Bhor has joined #openstack-cinder02:27
*** _hemna has quit IRC02:29
*** _hemna has joined #openstack-cinder02:35
*** psachin has joined #openstack-cinder02:39
*** mhen has quit IRC02:40
*** mhen has joined #openstack-cinder02:43
*** dave-mccowan has quit IRC02:46
*** dave-mccowan has joined #openstack-cinder02:55
*** _hemna has quit IRC02:56
*** _hemna has joined #openstack-cinder02:57
*** rosmaita has left #openstack-cinder03:01
*** Liang__ has quit IRC03:03
*** dklyle has joined #openstack-cinder03:11
*** whoami-rajat has joined #openstack-cinder03:12
*** _hemna has quit IRC03:13
*** dklyle has quit IRC03:17
openstackgerritMerged openstack/cinder master: Correct default policy file  https://review.openstack.org/62045003:25
*** _hemna has joined #openstack-cinder03:30
*** _hemna has quit IRC03:30
*** dave-mccowan has quit IRC03:33
*** dave-mccowan has joined #openstack-cinder03:34
openstackgerritBrian Rosmaita proposed openstack/cinder stable/rocky: Correct default policy file  https://review.openstack.org/62106403:39
*** psachin has quit IRC03:43
*** _hemna has joined #openstack-cinder03:46
*** _hemna has quit IRC03:50
*** _hemna has joined #openstack-cinder03:56
*** _hemna has quit IRC03:56
*** udesale has joined #openstack-cinder04:01
*** _hemna has joined #openstack-cinder04:11
*** _hemna has quit IRC04:11
*** _hemna has joined #openstack-cinder04:21
*** _hemna has quit IRC04:21
*** gnufied has quit IRC04:25
*** dave-mccowan has quit IRC04:30
*** markvoelker has quit IRC04:32
*** _hemna has joined #openstack-cinder04:40
*** _hemna has quit IRC04:40
*** _hemna has joined #openstack-cinder04:42
*** chhagarw has joined #openstack-cinder04:43
*** _hemna has quit IRC04:43
*** _hemna has joined #openstack-cinder04:45
*** _hemna has quit IRC04:45
*** _hemna has joined #openstack-cinder04:57
*** _hemna has quit IRC04:57
*** markvoelker has joined #openstack-cinder05:02
*** Nel1x has quit IRC05:14
*** _hemna has joined #openstack-cinder05:17
*** _hemna has quit IRC05:17
*** threestrands has quit IRC05:20
*** _hemna has joined #openstack-cinder05:31
*** _hemna has quit IRC05:32
*** pcaruana has quit IRC05:35
*** _hemna has joined #openstack-cinder05:38
*** _hemna has quit IRC05:39
*** imacdonn has quit IRC05:39
*** imacdonn has joined #openstack-cinder05:39
*** _hemna has joined #openstack-cinder05:45
*** _hemna has quit IRC05:45
*** _hemna has joined #openstack-cinder05:52
*** _hemna has quit IRC05:52
*** _hemna has joined #openstack-cinder05:59
*** _hemna has quit IRC05:59
*** _hemna has joined #openstack-cinder06:05
*** _hemna has quit IRC06:06
*** _hemna has joined #openstack-cinder06:13
*** _hemna has quit IRC06:16
*** _hemna has joined #openstack-cinder06:21
*** _hemna has quit IRC06:21
*** _hemna has joined #openstack-cinder06:29
*** _hemna has quit IRC06:29
*** hoangcx has quit IRC06:29
*** e0ne has joined #openstack-cinder06:32
*** _hemna has joined #openstack-cinder06:39
*** _hemna has quit IRC06:39
*** _hemna has joined #openstack-cinder06:47
*** _hemna has quit IRC06:48
*** noonedeadpunk[h] has joined #openstack-cinder06:48
*** noonedeadpunk has quit IRC06:49
*** _hemna has joined #openstack-cinder06:54
*** _hemna has quit IRC06:54
*** _hemna has joined #openstack-cinder07:01
*** _hemna has quit IRC07:02
*** FlorianFa has quit IRC07:06
*** _hemna has joined #openstack-cinder07:13
*** _hemna has quit IRC07:15
*** FlorianFa has joined #openstack-cinder07:20
*** _hemna has joined #openstack-cinder07:21
*** stakeda has joined #openstack-cinder07:22
*** pcaruana has joined #openstack-cinder07:22
*** e0ne has quit IRC07:31
*** slaweq has joined #openstack-cinder07:35
*** Luzi has joined #openstack-cinder07:38
*** rcernin has quit IRC08:06
*** dpawlik has joined #openstack-cinder08:07
openstackgerritwangwei proposed openstack/cinder master: Fix the problem of the scheduler status  https://review.openstack.org/61644008:39
*** dpawlik has quit IRC08:46
*** lixiaoy1 has quit IRC08:54
*** noonedeadpunk[h] is now known as noonedeadpunk09:14
*** dpawlik has joined #openstack-cinder09:20
*** dpawlik has quit IRC09:24
*** e0ne has joined #openstack-cinder09:27
*** stakeda has quit IRC09:33
*** s10 has joined #openstack-cinder09:40
openstackgerritIvan Kolodyazhny proposed openstack/cinder master: Cleanup code duplication in cinder.cmd.backup module  https://review.openstack.org/62087509:40
*** takamatsu has quit IRC09:41
openstackgerritIvan Kolodyazhny proposed openstack/cinder master: Publish backup capabilities to a scheduler  https://review.openstack.org/62088109:41
*** ganso has joined #openstack-cinder09:59
*** arne_wiebalck_ has joined #openstack-cinder10:02
*** luizbag has joined #openstack-cinder10:04
*** erlon has joined #openstack-cinder10:04
*** brinzhang has quit IRC10:05
*** Dinesh_Bhor has quit IRC10:09
*** tpsilva has joined #openstack-cinder10:31
*** slaweq has quit IRC10:32
*** mszwed has joined #openstack-cinder10:35
*** pcaruana has quit IRC10:44
*** pcaruana has joined #openstack-cinder10:50
*** arne_wiebalck_ has quit IRC10:51
*** udesale has quit IRC10:59
*** chhagarw has quit IRC11:14
*** Luzi has quit IRC12:00
*** pcaruana has quit IRC12:11
*** udesale has joined #openstack-cinder12:26
*** eharney has quit IRC12:34
*** Nel1x has joined #openstack-cinder12:46
*** e0ne has quit IRC12:54
*** erlon has quit IRC12:56
*** zul has joined #openstack-cinder13:06
*** erlon has joined #openstack-cinder13:11
*** abishop has joined #openstack-cinder13:13
*** dave-mccowan has joined #openstack-cinder13:19
*** arne_wiebalck_ has joined #openstack-cinder13:19
*** arne_wiebalck has quit IRC13:23
*** arne_wiebalck_ is now known as arne_wiebalck13:23
*** arne_wiebalck_ has joined #openstack-cinder13:26
*** jackivanov has quit IRC13:26
*** e0ne has joined #openstack-cinder13:37
*** arne_wiebalck_ has quit IRC13:40
*** mchlumsky has joined #openstack-cinder13:44
*** udesale has quit IRC13:48
*** whoami-rajat has quit IRC13:49
*** udesale has joined #openstack-cinder13:49
*** takamatsu has joined #openstack-cinder13:55
*** arne_wiebalck_ has joined #openstack-cinder14:13
*** etp has quit IRC14:18
*** mriedem has joined #openstack-cinder14:19
openstackgerritIvan Kolodyazhny proposed openstack/cinder master: Publish backup capabilities to a scheduler  https://review.openstack.org/62088114:20
*** etp has joined #openstack-cinder14:22
_hemna2mep14:35
*** lbragstad is now known as elbragstad14:36
smcginnis_hemna: Was wondering, do you know if your old house survived the CA fires?14:37
_hemnayah, my old pad was in Pilot Hill, CA14:38
smcginnisClose to there, wasn't it?14:38
*** whoami-rajat has joined #openstack-cinder14:38
_hemnafar enough away from the Paradise fires14:38
_hemnaabout an hour or so away14:38
smcginnisCool. Was thinking about that and Kurt. The HP office was in Paradise wasn't it?14:39
_hemnaIt was in Roseville14:39
_hemnaa close friend of mine's mom lived in Paradise14:40
_hemnashe barely got out alive, but lost everything14:40
smcginnisCrazy14:40
_hemnaI went to Chico state for college, about 20 minutes from Paradise14:40
smcginnisCan't even imagine what it was like out there.14:41
_hemnaI was the bay area during the camp fire14:42
_hemnathe air was like a dark thick fog14:42
*** eharney has joined #openstack-cinder14:42
smcginnisI head San Fran and the area was considered "most polluted air quality" cities at the time. Worse than LA even.14:43
smcginnisProbably still better than Beijing. :)14:44
*** rosmaita has joined #openstack-cinder14:45
_hemnahehe yah I think at the time the news said it was worse than Beijing14:45
*** arne_wiebalck has quit IRC14:45
*** arne_wiebalck_ is now known as arne_wiebalck14:45
smcginnisDang, that's bad then. :D14:45
*** takamatsu has quit IRC14:48
rosmaitasmcginnis: will we want to backport that policy file default change to queens also?14:51
smcginnisrosmaita: Hmm, yeah, I suppose so. The policy in code work was done in queens, so I guess we'll need it there too.14:52
rosmaitasmcginnis: cool, i should know this, but do i backport from master or rocky to queens?14:53
smcginnismaster to rocky, rocky to queens. ;)14:54
rosmaitamakes sense14:54
*** gnufied has joined #openstack-cinder15:01
jbernardsmcginnis: is avishay still around?15:15
smcginnisHe pops in every once and awhile, but not too often.15:16
jbernardsmcginnis: ok, i have some questions about his original migration code, specifically why new_type_id must be none for assisted migration to be considered15:17
jbernardsmcginnis: i don't follow that logical requirement15:17
smcginnisThat migration code and swapping around IDs has been a source of confusion and bugs for awhile.15:17
jbernardsmcginnis: and it prevents an otherwise possible assisted migration taking place without violating any volume type assertions15:17
smcginnisMy 2 second guess on that is if a backend can do the migration, then new_type_id might be set by its model update. But not sure.15:18
jbernardsmcginnis: the backend's function is guarded by this logic, so it's never asked if it can assist15:18
jbernardsmcginnis: and the path is decided in the manager15:18
jbernardsmcginnis: i may propose some changes there15:20
smcginnisI haven't seen any bug reports, but I'm sure that code could use another look over.15:23
*** esker has joined #openstack-cinder15:25
*** takamatsu has joined #openstack-cinder15:29
*** eharney has quit IRC15:36
*** dansmith is now known as SteelyDan15:43
*** mriedem has quit IRC15:50
*** eharney has joined #openstack-cinder15:51
*** takamatsu has quit IRC15:53
*** FlorianFa has quit IRC16:02
*** erlon has quit IRC16:15
*** s10 has quit IRC16:21
*** mriedem has joined #openstack-cinder16:21
*** mchlumsky has quit IRC16:21
jgriffithjbernard: IIRC that was related to the difference between migrate, retype and retype with migrate16:21
*** lemko has joined #openstack-cinder16:24
jgriffithjbernard: hmmm... https://github.com/openstack/cinder/blob/master/cinder/volume/rpcapi.py#L27016:25
*** pcaruana has joined #openstack-cinder16:26
jgriffiththose extra params don't appear to be used ever16:26
jgriffithOh, somebody threw in some trickery pasing the backend_p dict :)16:27
smcginnisThe overlap between the migrate, retype, and retype with migrate there I think has been a huge source of confusion. Both in the code implementation and the end user understanding of what's going on.16:31
jgriffithsmcginnis: yeah, it's been a tricky flow all along16:32
jgriffithsmcginnis: I remember all sorts of weirdness with the attachment work on that16:32
smcginnisAnd we've had multiple "fixes" for different parts of that that have broken other parts.16:33
*** _hemna has quit IRC16:47
*** whoami-rajat has quit IRC16:48
*** udesale has quit IRC16:49
*** abishop has quit IRC16:54
*** abishop has joined #openstack-cinder16:55
*** whoami-rajat has joined #openstack-cinder16:55
*** luizbag has quit IRC16:55
*** _hemna has joined #openstack-cinder16:57
kukaczjgriffith: hi John, I'm facing some strange behaviour during retype within solidfire backend in driver v2.0.7 (Ocata). have just submitted https://bugs.launchpad.net/cinder/+bug/1806084. not sure if that might be my misconfiguration or rather a bug16:58
openstackLaunchpad bug 1806084 in Cinder "Retyped SolidFire volume not being deleted from backend after fallback to generic mechanism" [Undecided,New]16:58
*** eharney has quit IRC16:59
kukaczjgriffith: I'll be happy if you kindly find some time to have a look17:00
*** pcaruana has quit IRC17:04
*** mriedem is now known as mriedem_lunch17:11
jgriffithkukacz: sure, not sure how much I can help but I'll have a look17:14
*** eharney has joined #openstack-cinder17:14
kukaczjgriffith: thanks! ahh, you are not working for solidfire/netapp anymore? sorry then, solidfire=john in my mind17:19
jgriffithgeesh, another retype/migration thing17:19
jgriffithkukacz: no worries, yeah... I've moved on to other things; but SolidFire will always be a pretty special deal to me17:20
smcginnisA special place in your heart. :D17:20
jgriffithsmcginnis: that's what that burning sensation is :)17:21
smcginnisHah!17:21
jgriffithkukacz: I might be able to help out, but it would likely be faster to get somebody like erlon involved who can actually duplicate things17:22
*** eharney has quit IRC17:22
jgriffithkukacz: the permission issues on the db update are a bit odd though17:22
*** esker has quit IRC17:23
*** erlon has joined #openstack-cinder17:23
jgriffithkukacz: this one may have already addressed the delete/cleanup part:  https://github.com/openstack/cinder/commit/d1b3adfbe471ad99ddf0dd2b483b77435c157aea#diff-edd3b1109653835d85834f80ae3c9b2c17:24
*** luizbag has joined #openstack-cinder17:29
kukaczjgriffith: great, thanks, I'll try to apply that and retest later today perhaps17:31
jgriffithkukacz: the other thing that's odd is the retype failing in the driver, but no messages indicating the failure.17:32
jgriffithkukacz: here https://github.com/openstack/cinder/blob/newton-eol/cinder/volume/drivers/solidfire.py#L178017:32
jgriffithor here: https://github.com/openstack/cinder/blob/newton-eol/cinder/volume/drivers/solidfire.py#L179117:33
jgriffithThe second one should show up in the logs if there's a problem, but if you have the ability you could look at adding some debug statements in those areas17:33
jgriffithSorry I don't have a cluster any more, I could build one but it's been so long since I've done it I probably wouldn't get it working before Christmas :)17:34
kukaczhmm, I'll try17:34
kukaczof course, I'm grateful for these hints17:34
kukaczI forgot to write that with admin role the retype works as it should17:35
jgriffithInteresting, there's something going on in front of the driver it sounds like (with the db access), but the driver seems to be silently ignoring whatever the problem is17:35
*** _hemna has quit IRC17:36
kukaczthere's an error "AdminRequired: User does not have admin privileges" logged17:36
jgriffithYeah, that's the one I was referring to17:37
kukaczit's really non-qualified guess, but could call like this be involved?: https://github.com/openstack/cinder/blob/newton-eol/cinder/volume/drivers/solidfire.py#L115817:38
*** _hemna has joined #openstack-cinder17:38
jgriffithcertainly could...17:39
kukacz... the get_admin_context() in qos lookup17:39
jgriffithI was poking around db/sql call... we'll see if that's the offending call or not17:40
jgriffithodd... 2018-11-30 15:04:36.200 1972 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 6808,17:42
jgriffithThere is no line 6808?17:43
*** e0ne has quit IRC17:45
jgriffithbut following that up in the trace a bit, I think you might be right... it looks like it's coming from a call to get_qos_specs in teh objects17:45
jgriffiththe17:45
jgriffithso we need to throw a new context in there and see if that fixes you up17:45
kukaczjgriffith: in my api.py, there is line 6808, it's the return statement of get_by_id method17:47
jgriffithoh, well that lines up; isn't this Newton?17:48
*** hedvig_01 has joined #openstack-cinder17:48
jgriffithOr did I make that up in my head :)17:48
jgriffithanyway, I think you're zoomed in on the right thing; that context doesn't seem to be working for some reason.17:49
jgriffithit would've been nice if whoever wrote that SF driver included some more debug logging :)17:50
kukacz:-)17:50
kukaczit's Ocata17:50
jgriffithbased on the logs though, I do think you're on to it.  The get_by_id that's failing is for the QOS specs object, so it all aligns17:51
kukaczI've just re-used the example from newton you've sent to highlight that context call17:51
*** openstackgerrit has quit IRC17:51
jgriffithGrrr.. I was off by 117:51
kukaczwhat can I do with the context call?17:52
kukacz(this should be the right place in ocata): https://github.com/openstack/cinder/blob/stable/ocata/cinder/volume/drivers/solidfire.py#L122017:53
jgriffithYeah, and I think I know how to fix that17:54
jgriffithHmm, I'll poke at it some more later.  Hopefully we can get erlon or somebody from NetApp to have a look.  Should be pretty easy to sort out once they have a system17:56
kukaczjgriffith: great. thank you a lot John! I really appreciate your help17:59
kukaczjgriffith: it's now about a year since you've helped me to another retype issue within a short IRC session. you created a bug fix in couple of hour then. I've been promoting that as the greatest community support example since then18:03
jgriffithkukacz: Thank you!!!!18:12
kukaczerlon: hi, may I kindly ask you to have a look at https://bugs.launchpad.net/cinder/+bug/1806084 when possible? jgriffith helped me to trace it in the driver code. there might be the admin context call involved perhaps: https://github.com/openstack/cinder/blob/stable/ocata/cinder/volume/drivers/solidfire.py#L122018:15
openstackLaunchpad bug 1806084 in Cinder "Retyped SolidFire volume not being deleted from backend after fallback to generic mechanism" [Undecided,New]18:15
*** luizbag has quit IRC18:40
*** lemko has quit IRC18:49
*** erlon has quit IRC19:02
*** mriedem_lunch is now known as mriedem19:13
*** noonedeadpunk has quit IRC19:47
*** eharney has joined #openstack-cinder20:07
*** whoami-rajat has quit IRC20:08
*** jistr has quit IRC20:11
*** jistr has joined #openstack-cinder20:13
*** itlinux has joined #openstack-cinder20:51
*** takamatsu has joined #openstack-cinder20:53
*** itlinux has quit IRC20:59
*** itlinux_ has joined #openstack-cinder20:59
*** ganso has quit IRC21:17
*** esker has joined #openstack-cinder21:36
*** openstackgerrit has joined #openstack-cinder21:42
openstackgerritBrian Rosmaita proposed openstack/cinder master: Make policy config docs consistent  https://review.openstack.org/62129621:42
spotzHey should cinder-rootwrap be running constantly? My guess is yes it's doing logging but wanted to verify21:44
*** abishop has quit IRC21:49
*** tpsilva has quit IRC21:51
*** esker has quit IRC22:11
*** itlinux_ has quit IRC22:13
openstackgerritMerged openstack/cinder stable/queens: Remove cinder-tox-compliance job  https://review.openstack.org/62066522:39
openstackgerritMerged openstack/cinder master: Stop cleaning images to be deleted in remotefs driver  https://review.openstack.org/62092622:39
*** mriedem has quit IRC22:54
openstackgerritMatt Smith (_alastor_) proposed openstack/cinder master: Adding support for fallback eval-based filter_func parser  https://review.openstack.org/61369522:54
*** Nel1x has quit IRC23:01
*** Miouge- has quit IRC23:05
*** itlinux has joined #openstack-cinder23:07
*** Miouge has joined #openstack-cinder23:08

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!