Tuesday, 2017-02-21

*** esberglu has quit IRC00:01
*** tjakobs has quit IRC00:01
*** esberglu has joined #openstack-powervm00:29
*** esberglu has quit IRC00:33
*** esberglu has joined #openstack-powervm00:38
*** svenkat has joined #openstack-powervm01:13
*** jwcroppe has joined #openstack-powervm01:26
*** thorst has joined #openstack-powervm01:29
*** mdrabe has quit IRC01:34
*** jwcroppe has quit IRC01:41
*** jwcroppe has joined #openstack-powervm01:42
*** jwcroppe has quit IRC01:46
*** esberglu has quit IRC01:49
*** svenkat has quit IRC02:00
*** svenkat has joined #openstack-powervm02:03
*** esberglu has joined #openstack-powervm02:04
*** esberglu has quit IRC02:04
*** edmondsw has joined #openstack-powervm02:20
*** edmondsw has quit IRC02:24
*** svenkat has quit IRC02:36
*** edmondsw has joined #openstack-powervm02:52
*** esberglu has joined #openstack-powervm02:58
*** esberglu has quit IRC03:03
*** thorst has quit IRC03:04
*** thorst has joined #openstack-powervm03:05
*** esberglu has joined #openstack-powervm03:12
*** esberglu has quit IRC03:12
*** thorst has quit IRC03:21
*** edmondsw has quit IRC05:33
*** edmondsw has joined #openstack-powervm05:33
*** edmondsw has quit IRC05:38
*** tlian has quit IRC05:41
*** tjakobs has joined #openstack-powervm06:53
*** tjakobs has quit IRC06:59
*** openstackgerrit has quit IRC07:32
*** k0da has joined #openstack-powervm08:18
*** edmondsw has joined #openstack-powervm09:34
*** edmondsw has quit IRC09:39
*** k0da has quit IRC10:58
*** smatzek has joined #openstack-powervm12:00
*** smatzek_ has joined #openstack-powervm12:02
*** smatzek has quit IRC12:02
*** edmondsw has joined #openstack-powervm12:03
*** seroyer has joined #openstack-powervm12:34
*** jpasqualetto has joined #openstack-powervm12:37
*** edmondsw has quit IRC12:59
*** xia has joined #openstack-powervm13:01
*** tblakes has joined #openstack-powervm13:02
*** seroyer has quit IRC13:07
*** svenkat has joined #openstack-powervm13:08
*** seroyer has joined #openstack-powervm13:17
*** thorst has joined #openstack-powervm13:24
*** edmondsw has joined #openstack-powervm13:30
*** thorst has quit IRC13:33
*** thorst has joined #openstack-powervm13:34
*** edmondsw has quit IRC13:35
*** thorst has quit IRC13:36
*** edmondsw has joined #openstack-powervm13:39
*** smatzek_ has quit IRC13:40
*** thorst has joined #openstack-powervm13:40
*** thorst_ has joined #openstack-powervm13:44
*** edmondsw has quit IRC13:44
*** thorst has quit IRC13:45
*** tlian has joined #openstack-powervm13:46
*** esberglu has joined #openstack-powervm13:47
*** dwayne has quit IRC13:48
*** smatzek has joined #openstack-powervm13:49
*** tlian has quit IRC13:49
efriedesberglu Looks like we're not properly applying the pypowervm local2remote patch in-tree?13:53
*** edmondsw has joined #openstack-powervm13:55
*** wangqwsh has joined #openstack-powervm13:59
esberglu#startmeeting powervm_driver_meeting14:02
openstackMeeting started Tue Feb 21 14:02:12 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.14:02
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:02
openstackThe meeting name has been set to 'powervm_driver_meeting'14:02
adrezneco/14:02
esbergluLooks like the CI is failing for in-tree and out-of-tree since last night14:03
xiao/14:03
efriedo/14:04
esbergluBut we can talk about it more in a topic14:04
esberglu#topic In Tree Driver14:04
efriedesberglu I don't have this meeting on my calendar.14:05
efriedCan you re-invite me?14:05
*** mdrabe has joined #openstack-powervm14:05
efriedSo I'm looking at the in-tree failure, and I believe it's because we're not applying the pypowervm patches correctly.14:06
efriedThe problem is here:14:06
efried    git fetch "$PYPOWERVM_REPO" $commit && git cherry-pick FETCH_HEAD14:06
efriedThe fetch is failing - though I can't tell why - so the cherry-pick isn't happening.14:06
efriedThis should really be split out, so that the whole script fails if the fetch fails.14:07
esbergluWhere are you seeing the fetch fail? I'm not seeing that14:07
efriedit's subtle.14:07
efriedhttp://184.172.12.213/80/427380/8/check/nova-in-tree-pvm/57d5ca6/console.html  Look around 20:30:3714:08
efriedYou see the git fetch, but you don't see the git cherry-pick14:09
efriedCompare and contrast to 20:30:53-54 where the cherry-pick succeeds, producing output, and then the cherry-pick happens.14:09
efriedThis is the danger of using that && syntax.14:10
efriedIt's really only for when you don't care if the LHS fails.14:10
efried...but the RHS is still dependent on the LHS.14:10
efriedI can repro the failure locally.14:11
efriedBut it still isn't saying anything.14:11
efriedNo idea why it's not working.  Will keep poking.14:11
esbergluI'm gonna turn off CI for now then. This is causing every run to fail14:12
*** kylek3h has joined #openstack-powervm14:13
efriedIt works when I use the corresponding refs/changes, but not when I use the commit hash.14:15
esbergluBut the same format is working for the other patches?14:16
*** thorst_ has quit IRC14:17
efriedSome of them yes, some of them no.14:17
esbergluI bet it14:17
esbergluis the ones that aren't merged yet14:17
*** thorst has joined #openstack-powervm14:17
esbergluSome of these patches are merged, some of them are not14:18
efriedWell, that makes zero sense, but I can check it.14:18
efriedNot that it matters.  I'll put up a fix.14:18
*** burgerk has joined #openstack-powervm14:22
esbergluIdk. But the 2 not merged patches are the patches that fail. Looks like the unmerged nova patch also isn't working14:23
esbergluThe nova patch fails with this message14:23
esbergluerror: no such remote ref db98f3c1f409faf3d2b5ca1f8951240f0a11414e14:23
esbergluOn the git fetch14:23
esbergluEither way. That's all I had for my side of things. Anyone else have discussion items?14:25
*** smatzek has quit IRC14:31
*** seroyer has quit IRC14:36
esbergluI will get the new invites out for this meeting and the thursday CI meeting. Thanks for joining14:38
esberglu#endmeeting14:39
openstackMeeting ended Tue Feb 21 14:39:02 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:39
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-02-21-14.02.html14:39
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-02-21-14.02.txt14:39
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-02-21-14.02.log.html14:39
efriedesberglu 489414:39
*** kylek3h has quit IRC14:42
*** kylek3h has joined #openstack-powervm14:42
*** kylek3h has quit IRC14:43
*** kylek3h has joined #openstack-powervm14:43
*** kylek3h has quit IRC14:43
*** kylek3h has joined #openstack-powervm14:44
*** kylek3h has quit IRC14:44
*** kylek3h has joined #openstack-powervm14:45
*** kylek3h has quit IRC14:45
*** k0da has joined #openstack-powervm14:49
*** smatzek has joined #openstack-powervm14:54
*** kylek3h has joined #openstack-powervm15:08
*** wangqwsh has quit IRC15:10
*** burgerk has quit IRC15:16
*** thorst has quit IRC15:18
*** edmondsw has quit IRC15:18
*** seroyer has joined #openstack-powervm15:20
*** edmondsw has joined #openstack-powervm15:23
*** edmondsw has quit IRC15:26
*** edmondsw has joined #openstack-powervm15:26
*** edmondsw_ has joined #openstack-powervm15:29
*** edmonds__ has joined #openstack-powervm15:30
*** edmondsw has quit IRC15:31
*** thorst has joined #openstack-powervm15:31
*** edmonds__ has quit IRC15:31
*** edmondsw has joined #openstack-powervm15:31
*** thorst has quit IRC15:32
*** edmondsw_ has quit IRC15:33
*** edmondsw_ has joined #openstack-powervm15:33
*** edmondsw has quit IRC15:36
*** edmondsw has joined #openstack-powervm15:38
*** edmondsw_ has quit IRC15:38
*** edmondsw has quit IRC15:39
*** tjakobs has joined #openstack-powervm15:39
*** edmondsw has joined #openstack-powervm15:39
*** thorst has joined #openstack-powervm15:44
*** thorst has quit IRC15:44
*** thorst has joined #openstack-powervm15:44
*** xia has quit IRC15:45
*** burgerk has joined #openstack-powervm15:55
*** apearson_ has joined #openstack-powervm15:55
*** k0da has quit IRC15:58
thorstefried: I was just pinged offline about https://review.openstack.org/#/c/435108/115:59
thorstIt looks fine to me.  I put a +215:59
thorstany concerns?15:59
*** burgerk has quit IRC16:00
*** mdrabe has quit IRC16:03
*** burgerk has joined #openstack-powervm16:07
*** dwayne has joined #openstack-powervm16:10
*** mdrabe has joined #openstack-powervm16:10
*** tlian has joined #openstack-powervm16:12
*** thorst_ has joined #openstack-powervm16:16
*** thorst has quit IRC16:17
*** k0da has joined #openstack-powervm16:20
mdrabeefried: I've got a migration failure happening in this call in nova-powervm16:20
mdrabehttps://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/driver.py#L153516:21
mdrabeProblem is, migration comes back as a success even though that call fails16:21
mdrabeMy question is, do you think it should fail out instead?16:21
efriedmdrabe I'm looking.  But we'll want thorst_ on that discussion.16:22
efriedmdrabe How does that call fail?16:24
efriedIf it raises, the live_migration will fail.16:24
mdrabeefried, thorst_ yea nevermind, I'm looking at this wrong16:25
mdrabeSomehow the migration is still coming back as succeeded16:25
efriedmdrabe How does that call fail?16:26
mdrabeefried: I'll pm you the stack trace16:26
efriedmdrabe So what makes you think the migration is succeeding?16:30
mdrabeWe get a message saying that it succeeded16:30
mdrabeAnd it actually shows as on the new host16:30
mdrabeIt's weird, I need to keep looking. I jumped the gun thinking it was nova-powervm related with that call16:31
*** kylek3h has quit IRC16:51
*** thorst_ has quit IRC17:00
*** thorst has joined #openstack-powervm17:05
*** edmondsw has quit IRC17:05
*** edmondsw has joined #openstack-powervm17:06
*** thorst has quit IRC17:07
*** thorst has joined #openstack-powervm17:07
*** edmondsw has quit IRC17:11
*** edmondsw has joined #openstack-powervm17:14
*** edmondsw_ has joined #openstack-powervm17:16
*** edmondsw has quit IRC17:18
*** edmondsw_ has quit IRC17:21
*** thorst has quit IRC17:23
*** thorst has joined #openstack-powervm17:28
*** thorst has quit IRC17:30
*** apearson_ has quit IRC17:32
*** thorst has joined #openstack-powervm17:32
*** kylek3h has joined #openstack-powervm17:38
*** nbante has joined #openstack-powervm17:38
*** kylek3h has quit IRC17:39
*** apearson has joined #openstack-powervm17:45
*** thorst has quit IRC17:46
*** thorst has joined #openstack-powervm17:51
*** edmondsw has joined #openstack-powervm17:52
*** thorst has quit IRC17:56
*** tjakobs has quit IRC17:56
*** thorst has joined #openstack-powervm17:56
*** edmondsw has quit IRC17:57
*** edmondsw has joined #openstack-powervm17:58
*** edmondsw_ has joined #openstack-powervm18:03
*** edmondsw has quit IRC18:03
*** edmondsw_ has quit IRC18:03
*** edmondsw has joined #openstack-powervm18:03
*** tjakobs has joined #openstack-powervm18:04
*** openstack has joined #openstack-powervm18:14
*** edmondsw has quit IRC18:15
*** nbante has quit IRC18:25
*** thorst_ has joined #openstack-powervm18:27
*** apearson has quit IRC18:28
*** thorst has quit IRC18:29
*** edmondsw has joined #openstack-powervm18:31
*** edmondsw has quit IRC18:41
*** edmondsw has joined #openstack-powervm18:42
*** tlian has quit IRC18:42
*** edmondsw has quit IRC18:43
*** edmondsw has joined #openstack-powervm18:43
*** edmondsw has quit IRC18:45
*** edmondsw has joined #openstack-powervm18:46
*** edmondsw has quit IRC18:48
*** edmondsw has joined #openstack-powervm18:48
*** smatzek has quit IRC18:50
*** smatzek has joined #openstack-powervm18:58
*** apearson has joined #openstack-powervm19:00
*** tlian has joined #openstack-powervm19:05
*** thorst_ has quit IRC19:05
*** thorst has joined #openstack-powervm19:09
*** jwcroppe has joined #openstack-powervm19:17
*** apearson has quit IRC19:17
*** apearson has joined #openstack-powervm19:20
*** tlian has quit IRC19:22
*** thorst has quit IRC19:32
*** jwcroppe has quit IRC19:35
*** jwcroppe has joined #openstack-powervm19:36
*** jwcroppe has quit IRC19:40
*** openstackgerrit has joined #openstack-powervm19:44
openstackgerritEric Fried proposed openstack/nova-powervm master: In-tree backports: Task UT, Delete args  https://review.openstack.org/43511419:44
*** thorst has joined #openstack-powervm19:50
*** edmondsw has quit IRC19:51
*** edmondsw has joined #openstack-powervm19:52
efriedesberglu Aborted?  What's going on there?19:55
esbergluTrying to figure it out. A few runs got stuck in a loop looking for an existing LU19:57
*** edmondsw has quit IRC19:57
*** edmondsw has joined #openstack-powervm19:58
*** edmondsw has quit IRC20:04
*** jwcroppe has joined #openstack-powervm20:04
*** edmondsw has joined #openstack-powervm20:04
*** edmondsw has quit IRC20:07
*** edmondsw has joined #openstack-powervm20:07
*** thorst_ has joined #openstack-powervm20:21
*** thorst has quit IRC20:23
efriedesberglu adreznec thorst_ Booyah, first two change sets are golden.  Just finished polishing the third (localdisk) for thorst_ review comments, ready for next review.20:34
*** tlian has joined #openstack-powervm20:36
*** tlian has quit IRC20:41
*** tlian has joined #openstack-powervm20:44
*** jwcroppe has quit IRC20:54
esbergluefried: We are going to need to update the in tree local.conf for localdisk to pass20:55
esbergluoslo_config.cfg.RequiredOptError: value required for option volume_group_name in group [powervm]20:55
efriedIndeeeed.20:55
efriedI just did that, too ;-)20:56
thorst_do we have any volume groups on these systems20:56
efriedI mean, added that as a required opt.20:56
*** thorst_ is now known as thorst_afk20:56
efriedThe SSP change set will make it not-required again.20:56
efriedesberglu Doesn't actually matter if we make the opt not-required, since we'd be trying to use the localdisk driver anyway.21:01
efriedSo I think we need to figure out a way to run CI with localdisk driver enabled.21:02
*** edmondsw has quit IRC21:03
*** edmondsw has joined #openstack-powervm21:04
thorst_afkefried: can we do the SSP one first?21:04
efriedthorst_afk Not easily ;-)21:05
efriedBut yeah, if we did that, we would never have to get localdisk working ;-)21:05
thorst_afkI think that may be easier than trying to get localdisk into the CI21:05
efriedAnyone here think it's a bad idea for us to run the CI at least once with localdisk?21:05
thorst_afkme.21:06
thorst_afkbecause we would have to fundamentally retool the CI for it21:06
thorst_afkI think it makes sense to run tempest tests against a custom setup21:07
thorst_afkbut not refactor the entire CI setup for it when we've tuned it fundamentally for SSPs21:07
efriedOkay.  I'll get on that.  Going to be a nontrivial reswizzle, though.21:08
efriedLike, more than just a rebase.21:08
efriedCause we geared localdisk as if it was going to stand alone, then SSP as if it was going on top.21:08
efriedSo gotta reverse that.21:08
*** edmondsw has quit IRC21:11
*** burgerk has quit IRC21:23
*** smatzek has quit IRC21:26
thorst_afkefried: yeah...sorry.  I think that's probably the better approach21:27
thorst_afkwe had originally done localdisk cause it made more sense initially...but...this is probably better21:27
efriedthorst_afk First slapping console on top of power on/off/reboot...21:44
efriedthorst_afk esberglu Done - that one should be able to pass CI now.21:46
*** tjakobs has quit IRC21:46
efriedhttps://review.openstack.org/#/c/409402/21:46
efriedThink I'm going to start a new SSP change set, rather than trying to rebase wangqwsh's21:47
*** tjakobs has joined #openstack-powervm21:49
*** thorst_afk has quit IRC21:51
*** tblakes has quit IRC21:58
*** svenkat has quit IRC22:02
*** thorst_afk has joined #openstack-powervm22:03
*** thorst_afk has quit IRC22:04
mdrabeefried: about that post_method call in the live_migration driver method. I'm not sure if this is a defect, but if that method fails when it's called nothing is done about it22:08
efriedAgain, define "fails".22:09
mdraberaises an exception22:09
efriedThen the live_migration method will raise an exception.22:09
mdrabeCorrect but the manager doesn't do anything about it22:09
efriedBut if the compute manager doesn't do anything about that, that's not under our control.22:09
efriedSo... is that a problem?22:09
mdrabeI was thinking the driver should do something about it, but wasn't sure22:09
efriedHave you found the code where the compute manager is running the live_migration method?22:10
efriedIs that guy handling specific exceptions specially?22:10
mdrabeSo that post_method, in my case, and maybe all cases is _post_live_migration22:10
mdrabeIt handles some exceptions22:10
mdrabeIt's _post_live_migration defined in the compute manager22:10
efriedCould we "fix" it by making our code raise an exception that's handled differently?22:10
mdrabeSo the problem that's specifically happening in my case is22:11
efriedWe call the post_method within our version of live_migration.  We could catch whatever exception you think ought to be fatal, and reraise something that the compute manager _does_ treat as fatal.22:11
efriedIf you think that's the right thing to do.22:11
mdrabe_post_live_migration fails in terminate_connection22:11
mdrabeThis causes the exception to bubble up and not be handled22:12
mdrabeSo there's no cleanup or post live migration handling on the destination node22:12
mdrabeThis leaves the instance in the driver's migrating dictionary on the destination22:12
mdrabeAnd causes some issues for subsequent migrations22:13
mdrabeThe thing is it's the driver that's supposed to handle the cleanup I think22:13
mdrabeThe manager isn't going to do anything22:13
efriedSo does that mean our live_migration method needs a 'finally' that cleans that stuff up regardless of whether the _post fails?22:15
*** tlian has quit IRC22:15
mdrabeI was thinking about maybe just invoking that _migration_exception_util in the event the post_method fails22:16
mdrabeThe _post_live_migration should really be handling some more exceptions22:18
esbergluefried: The console patch is passing CI. I'm setting up staging to see what additional tests we should add to the in-tree whitelist22:21
efriedesberglu Cool beans.  Remember we have to merge the first one before we update that.22:24
efriedesberglu How about the power on/off/reboot one?22:24
efriedDo that one first.22:24
*** tblakes has joined #openstack-powervm22:24
esbergluYep that was the plan.22:25
efriedmdrabe I'm not looking at the code, but I'm instinctively not liking the idea of using private methods.22:25
mdrabeefried: I'm boiling it down to a hole in nova22:26
efriedmdrabe Bug?22:26
efriedSounds like it22:26
mdrabeYea, it's a fairly rare case I think22:27
efriedThough it may be tough to push through, given it may break legacy.22:27
mdrabeTbh I don't even know what I'd want to happen22:27
mdrabeIt's not like we'd reverse migrate the VM back to the source at that point22:27
efriedWell, if it's "fail the migration" (and what, roll back?) then yeah, we have a bug.22:27
efriedIf it's "make sure we clean up anyway", then we can fix that in our driver with a `finally`.22:28
mdrabeThe cleanup that'd need to happen is more or less everything that happens in here: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5522-L563622:29
mdrabelibvirt doesn't do any driver side cleanup either22:32
efriedoho.22:33
efriedThe thing that's failing is calling into the consumer's cinder driver code.22:33
efriedSooooo... maybe that thing shouldn't "fail".22:34
mdrabegood point22:34
mdrabeours does22:34
efriedQuestion is, does consumer's terminate_connection get called from other places too?22:35
mdrabeI think so22:36
mdrabeProbably in the detach volume and delete flows22:36
efriedMight want to spy a look at other cinder drivers and see whether/how they avoid exceptions in their terminate_connection methods.22:37
mdrabeYea, I know who to talk to for that too22:38
efriedThink he's in Atlanta.22:39
*** tlian has joined #openstack-powervm22:40
mdrabeYea... got a few folks on the storage side though22:40
*** esberglu has quit IRC22:48
*** tlian has quit IRC22:49
*** tlian has joined #openstack-powervm22:59
*** dwayne has quit IRC23:02
*** seroyer has quit IRC23:10
*** k0da has quit IRC23:10
*** mdrabe_ has joined #openstack-powervm23:12
*** mdrabe has quit IRC23:16
*** mdrabe_ has quit IRC23:17
*** tlian has quit IRC23:18
*** esberglu has joined #openstack-powervm23:21
*** esberglu has quit IRC23:26
*** svenkat has joined #openstack-powervm23:29
*** apearson has quit IRC23:45
*** tjakobs has quit IRC23:51
*** tlian has joined #openstack-powervm23:59
*** tblakes has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!