| *** thorst_ has joined #openstack-powervm | 00:02 | |
| *** svenkat has joined #openstack-powervm | 00:06 | |
| *** thorst_ has quit IRC | 00:10 | |
| *** svenkat has quit IRC | 00:11 | |
| *** svenkat has joined #openstack-powervm | 00:21 | |
| *** thorst_ has joined #openstack-powervm | 00:43 | |
| *** thorst_ has quit IRC | 00:44 | |
| *** thorst_ has joined #openstack-powervm | 00:44 | |
| *** thorst_ has quit IRC | 00:52 | |
| *** Ashana has joined #openstack-powervm | 01:10 | |
| *** Ashana has quit IRC | 01:14 | |
| *** arnoldje has joined #openstack-powervm | 01:18 | |
| *** esberglu has joined #openstack-powervm | 01:24 | |
| *** thorst_ has joined #openstack-powervm | 01:25 | |
| *** thorst_ has quit IRC | 01:34 | |
| *** svenkat has quit IRC | 01:37 | |
| *** esberglu has quit IRC | 01:45 | |
| *** esberglu has joined #openstack-powervm | 01:55 | |
| *** esberglu has quit IRC | 02:02 | |
| *** thorst_ has joined #openstack-powervm | 02:04 | |
| *** thorst_ has quit IRC | 02:10 | |
| *** esberglu has joined #openstack-powervm | 02:13 | |
| *** esberglu has quit IRC | 02:24 | |
| *** thorst_ has joined #openstack-powervm | 02:40 | |
| *** svenkat has joined #openstack-powervm | 02:41 | |
| *** thorst_ has quit IRC | 02:41 | |
| *** svenkat has quit IRC | 02:45 | |
| *** esberglu has joined #openstack-powervm | 03:11 | |
| *** esberglu has quit IRC | 03:34 | |
| *** jwcroppe_ has joined #openstack-powervm | 04:28 | |
| *** tsjakobs has joined #openstack-powervm | 05:07 | |
| *** tsjakobs has quit IRC | 05:29 | |
| *** jwcroppe_ has quit IRC | 05:36 | |
| *** arnoldje has quit IRC | 05:38 | |
| *** kotra03 has joined #openstack-powervm | 05:39 | |
| *** madhaviy has joined #openstack-powervm | 05:46 | |
| *** madhaviy has quit IRC | 06:45 | |
| *** madhaviy_ has joined #openstack-powervm | 06:45 | |
| *** madhaviy_ is now known as madhaviy | 06:45 | |
| *** madhaviy_ has joined #openstack-powervm | 06:47 | |
| *** madhaviy has quit IRC | 06:47 | |
| *** madhaviy_ is now known as madhaviy | 06:48 | |
| *** madhaviy_ has joined #openstack-powervm | 07:14 | |
| *** madhaviy has quit IRC | 07:14 | |
| *** madhaviy_ is now known as madhaviy | 07:14 | |
| *** svenkat has joined #openstack-powervm | 07:24 | |
| *** svenkat has quit IRC | 07:28 | |
| *** k0da has joined #openstack-powervm | 08:11 | |
| *** svenkat has joined #openstack-powervm | 10:24 | |
| *** svenkat has quit IRC | 10:28 | |
| *** thorst has joined #openstack-powervm | 10:56 | |
| *** Ashana has joined #openstack-powervm | 11:43 | |
| *** svenkat has joined #openstack-powervm | 11:48 | |
| *** seroyer has joined #openstack-powervm | 12:26 | |
| *** edmondsw has joined #openstack-powervm | 13:07 | |
| *** edmondsw has quit IRC | 13:08 | |
| openstackgerrit | Drew Thorstensen proposed openstack/networking-powervm: Simplify host_uuid and gets https://review.openstack.org/344399 | 13:08 |
|---|---|---|
| *** mdrabe has joined #openstack-powervm | 13:09 | |
| *** arnoldje has joined #openstack-powervm | 13:09 | |
| *** tblakeslee has joined #openstack-powervm | 13:10 | |
| *** edmondsw has joined #openstack-powervm | 13:14 | |
| *** esberglu has joined #openstack-powervm | 13:35 | |
| thorst | esberglu: I think we need two CI changes for the networking-powervm bits | 13:39 |
| thorst | 1) Turn down the logging of the REST API bits (I had a 1.5 Gig log file because of that EEK) | 13:39 |
| thorst | 2) Set the 'heal_and_optimize_interval' to like 90000 | 13:40 |
| *** seroyer has quit IRC | 13:41 | |
| thorst | and I think https://review.openstack.org/#/c/344399/ will be our friend for speed ups | 13:43 |
| esberglu | esberglu: Okay. heal_and_optimize_interval is a config option for networking_powervm? | 13:47 |
| thorst | esberglu: yep. It won't solve the speed issue...but I think that leads to a few short delays on the longer runs. | 13:49 |
| *** apearson has joined #openstack-powervm | 13:49 | |
| thorst | and having multiple VMs on the same novalink hitting that at the same time...is too expensive | 13:49 |
| thorst | and the other review should fix an issue where we're making WAY too many LPAR feed calls | 13:50 |
| *** seroyer has joined #openstack-powervm | 13:56 | |
| esberglu | thorst: Awesome. What section of the config does it go in? | 13:58 |
| thorst | the agent section for powervm | 13:58 |
| thorst | not sure we have anything in there yet | 13:58 |
| esberglu | thorst: Doesn’t look like it. What’s the syntax for that? | 14:09 |
| esberglu | For adding that section I mean | 14:13 |
| thorst | let me find the file | 14:13 |
| *** efried has joined #openstack-powervm | 14:14 | |
| thorst | esberglu: See the roles / devstack-compute / templates / local.conf.j2 | 14:15 |
| thorst | the section is [AGENT] | 14:15 |
| thorst | the post config is... | 14:15 |
| thorst | [[post-config|$Q_PLUGIN_CONF_FILE]] | 14:16 |
| esberglu | I think it might need a slash. [[post-config|/$Q_PLUGIN_CONF_FILE]] | 14:17 |
| *** tsjakobs has joined #openstack-powervm | 14:21 | |
| thorst | yeah...it had that initially...I removed it because the nova one didn't have it | 14:22 |
| thorst | :-) | 14:22 |
| *** mdrabe has quit IRC | 14:27 | |
| *** efried has quit IRC | 14:29 | |
| *** efried has joined #openstack-powervm | 14:30 | |
| *** mdrabe has joined #openstack-powervm | 14:34 | |
| *** jwcroppe has joined #openstack-powervm | 14:40 | |
| *** jwcroppe has joined #openstack-powervm | 14:40 | |
| esberglu | thorst: How do I change the rest API logging? | 14:55 |
| *** tblakeslee has quit IRC | 14:55 | |
| thorst | where's the path to the local.conf.aio that we use? | 14:57 |
| thorst | let me look in there | 14:57 |
| esberglu | ci-management/templates/scripts/local.conf.aio | 14:57 |
| *** apearson has quit IRC | 14:59 | |
| *** apearson has joined #openstack-powervm | 15:01 | |
| thorst | efried: remember that one time I tried to switch pypowervm to pretty_tox and you said no? | 15:01 |
| efried | thorst, yeah, I remember. See commit message. | 15:02 |
| thorst | lol | 15:02 |
| thorst | ahh, vindicated | 15:02 |
| thorst | -1 | 15:02 |
| efried | balls. | 15:03 |
| efried | thorst, done. | 15:04 |
| efried | <sheepish> | 15:04 |
| efried | Course, the thing won't build until dclain gets jenkins in gear. | 15:04 |
| thorst | #devops | 15:05 |
| esberglu | thorst: We told stephen that we would increment by 8/2 if we we are going to go to 2.0.3 | 15:16 |
| thorst | adreznec: you may want to listen in here | 15:17 |
| thorst | efried: you too | 15:17 |
| efried | sup | 15:18 |
| *** openstackgerrit has quit IRC | 15:18 | |
| efried | who's Stephen? | 15:18 |
| thorst | so I don't think we need to increment ceilometer-powervm for Mitaka. I don't think anything went back there...so we can keep the existing version. Anyone disagree? (I in fact assert we must keep the existing version) | 15:18 |
| *** openstackgerrit has joined #openstack-powervm | 15:18 | |
| efried | thorst, looks like there's a couple change sets on top of mitaka. | 15:19 |
| thorst | efried: but how recent? Was it after we tagged last? | 15:19 |
| efried | | * 411bbb5 Fix package reference in version code | 15:19 |
| efried | * | 45809e9 Add ceilometer-powervm spec dir and template | 15:19 |
| efried | That second one is prolly fine - specs for newton/ocata | 15:20 |
| adreznec | Yeah | 15:20 |
| efried | But the first one, looking... | 15:20 |
| thorst | are you thinking mitaka? | 15:20 |
| thorst | I thought that was just master... | 15:20 |
| adreznec | Is setuptools locked at <20.2 for Mitaka? | 15:20 |
| efried | wait, are we diffing master with mitaka, or mitaka with liberty? | 15:20 |
| thorst | esberglu: when was the last tag? | 15:20 |
| adreznec | thorst: 5.19 | 15:21 |
| thorst | efried: we're looking to see if there was anything significant between last mitaka tag and now (in mitaka) | 15:21 |
| thorst | nothing for master... | 15:21 |
| adreznec | Translations were the last thing that went in | 15:21 |
| efried | Okay, that other change set fixes broken docs builds. | 15:21 |
| adreznec | I don't see anything critical | 15:21 |
| thorst | so last merge to ceilometer-powervm way may 17. We did our tag on may 19. So no version bump | 15:22 |
| thorst | agree? | 15:22 |
| thorst | (again - this is all mitaka) | 15:22 |
| adreznec | +21 | 15:22 |
| adreznec | *+1 | 15:22 |
| adreznec | Not quite that enthusiastic about it... | 15:23 |
| thorst | networking-powervm is the same way | 15:23 |
| thorst | last merge was May 17th...so I think we're fine there. | 15:23 |
| *** k0da has quit IRC | 15:24 | |
| adreznec | Is there anything that should go back that hasn't? | 15:24 |
| thorst | nova-powervm has had many since May 17th...so we definitely need a tag there. | 15:24 |
| adreznec | Do we care about the fix for https://bugs.launchpad.net/networking-powervm/+bug/1573180 going back into Mitaka? If not, then I agree on networking-powervm | 15:28 |
| openstack | Launchpad bug 1573180 in networking-powervm "Agent makes frequent get_devices_details_list RPC calls with an empty list" [Undecided,Fix released] - Assigned to Sridhar Venkat (svenkat) | 15:28 |
| adreznec | and yeah, nova-powervm definitely needs a tag | 15:28 |
| thorst | adreznec: I don't worry too much | 15:30 |
| thorst | it wasn't a huge impact | 15:30 |
| thorst | so...consensus. Increment the nova-powervm one? And if something urgent comes in...then Stephen will have to react to it :-( | 15:33 |
| adreznec | Yup | 15:34 |
| adreznec | Guess so | 15:34 |
| thorst | esberglu: Rip it | 15:35 |
| esberglu | thorst: Cool | 15:37 |
| adreznec | esberglu: While you're tagging that for Mitaka, can you tag our newton-2 milestones for Newton? | 15:44 |
| adreznec | I just realized we never did that | 15:44 |
| adreznec | Just tag the latest commit on master for each of the repos as 3.0.0.0b2 | 15:45 |
| thorst | +1 | 15:54 |
| esberglu | adreznec: Yeah. What about for networking-powervm? Since it’s a stadium project. | 15:54 |
| adreznec | Not anymore | 15:54 |
| adreznec | We have control back | 15:54 |
| esberglu | Oh yeah I remember you saying that | 15:54 |
| adreznec | We can push our own tags now | 15:54 |
| esberglu | I need to be added to the project release teams on gerrit I think | 15:55 |
| adreznec | Ah | 15:55 |
| adreznec | I think I can do that | 15:55 |
| adreznec | esberglu: Done | 15:56 |
| *** apearson has quit IRC | 16:03 | |
| *** apearson has joined #openstack-powervm | 16:04 | |
| esberglu | Alright everything is tagged | 16:09 |
| *** tblakeslee has joined #openstack-powervm | 16:19 | |
| thorst | alright alright alright | 16:21 |
| *** k0da has joined #openstack-powervm | 16:45 | |
| *** burgerk has joined #openstack-powervm | 16:47 | |
| thorst | burgerk: we've done MPIO with Ubuntu 16.04 LTS, right? | 16:55 |
| burgerk | yes, not sure how extensively it was tested ... i.e. shutting down paths, etc. | 16:59 |
| thorst | chmod6661rg is having trouble with it on his NL partition. I thought we had hit it though in our test. :-/ | 17:00 |
| thorst | might want to see if we can get the test team to take another look at it | 17:00 |
| burgerk | 16.04 as the GuestOS? or referring to the NovaLink partition? | 17:01 |
| thorst | well, he's doing it for the NL partition. But that's really just a specific type of Guest OS. So lets start with guest os | 17:02 |
| burgerk | ok | 17:02 |
| thorst | thx! | 17:03 |
| *** madhaviy has quit IRC | 17:07 | |
| esberglu | thorst: Still not finding any options to turn down the networking logging | 17:09 |
| thorst | https://github.com/openstack/nova-powervm/blob/master/devstack/local.conf.aio#L46-L49 | 17:17 |
| thorst | see in there where pypowervm is set to info | 17:17 |
| thorst | I think something like that is needed for pypowervm in the neutron conf | 17:17 |
| *** tblakeslee has quit IRC | 17:18 | |
| efried | thorst, care to weigh in on https://review.openstack.org/#/c/348014/ ? | 17:31 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm: Rebase on pypowervm.tasks.hdisk refactor https://review.openstack.org/347889 | 17:36 |
| thorst | efried: weighed in | 17:39 |
| efried | thx | 17:39 |
| *** k0da has quit IRC | 17:40 | |
| efried | thorst, mdrabe: can we take a moment to discuss? | 17:40 |
| thorst | I'm debugging something...can chat, but distracted | 17:41 |
| efried | thorst: looking at the code, I don't see any path where we don't destroy the disks. (Except where they didn't exist in the first place.) | 17:42 |
| efried | Do you? | 17:42 |
| thorst | yeah | 17:42 |
| thorst | let me get you link | 17:42 |
| thorst | its live migration | 17:42 |
| thorst | (also cold I think) | 17:43 |
| thorst | on shared storage, like a SSP | 17:43 |
| thorst | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L659-L660 | 17:43 |
| kotra03 | @thorst: This is regarding the bug 1602759 | 17:45 |
| openstack | bug 1602759 in nova-powervm "get_vm_qp in vm.py throws an exception if the instance no longer exists" [Undecided,In progress] https://launchpad.net/bugs/1602759 - Assigned to Ravi Kumar Kota (ravkota3) | 17:45 |
| kotra03 | @thorst: If we change 'get_vm_qp' method to not throw InstanceNotFound exception then that would break the existing code. | 17:45 |
| kotra03 | There are other methods such as 'instance_exists' which depend on the InstanceNotFound exception thrown by 'get_vm_qp' method. | 17:45 |
| kotra03 | All the methods that call get_vm_qp method are going to be affected if we don't throw that exception. | 17:45 |
| thorst | kotra03: didn't we just not want it to log or something? | 17:46 |
| efried | thorst, I see it now. | 17:46 |
| efried | It juuuust may be worth adding a flag to the scrubber to get around that. | 17:46 |
| thorst | efried: honestly...I kinda like reading the code the way it is | 17:47 |
| thorst | it makes sense. | 17:47 |
| thorst | it reads well... | 17:47 |
| efried | Cause you're used to it. | 17:48 |
| thorst | just saying 'blast all the things' leads me to question 'well is that getting all the disks' | 17:48 |
| thorst | how can it just assume that | 17:48 |
| efried | Because that's its job? | 17:48 |
| thorst | yeah, agree. Plus I did write chunks of it | 17:48 |
| thorst | kotra03: specifically this block barfs if we get an instance not found exception | 17:50 |
| thorst | https://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/driver.py#L1999-L2001 | 17:50 |
| thorst | and clogs up the logs. | 17:50 |
| thorst | one could argue 'well that makes sense, the instance isn't there.' | 17:50 |
| thorst | but it doesn't make sense that I'm getting logs from the destroy path for instance not found....because of course the REST API sends me events about a VM that got deleted...but then we can't process them because that VM is gone. | 17:51 |
| thorst | so we shouldn't chunk big log messages for that very normal scenario | 17:51 |
| efried | thorst, So there's the other option, which is to rework https://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/volume/vscsi.py#L371-L378 (again) so that it always removes the mappings, even if discover_hdisk fails. I believe we've established this is safe for the "no ITL" error mdrabe is seeing. | 17:51 |
| thorst | so scrub in there... | 17:52 |
| thorst | or a form of scrubbing. | 17:52 |
| *** tblakeslee has joined #openstack-powervm | 17:52 | |
| thorst | what if there are multiple volumes on the scsi bus...but only one wasn't found? | 17:52 |
| efried | That's the bug we're hitting (partially). | 17:53 |
| efried | One isn't found, but that makes the method bail and ignore the rest. | 17:53 |
| efried | Although even ignoring the first one is bad. | 17:53 |
| thorst | so it just returns False...you're saying the other volumes don't get processed? | 17:54 |
| efried | correct. | 17:54 |
| efried | mdrabe, keep me honest. | 17:54 |
| efried | oh, maybe I'm wrong. | 17:55 |
| efried | But the point is, I think we want to reach line 413 even if that discover_hdisk fails hard. | 17:56 |
| *** apearson has quit IRC | 17:58 | |
| *** apearson has joined #openstack-powervm | 18:00 | |
| mdrabe | efried, thorst: No all the volumes are processed | 18:07 |
| mdrabe | But each one returns False because none of the ITLs are there | 18:07 |
| efried | Right. | 18:07 |
| mdrabe | because it was ripped out by evacuation | 18:07 |
| efried | So we can try to patch up the above such that it does the disconnect (but not the delete) when it gets that particular error (or really any error). | 18:08 |
| efried | However, it would have to do the disconnect via a different mechanism, because the current mechanism (line 413) relies on having info about the disk, which we don't have. | 18:09 |
| efried | I reckon we can figure that out. | 18:09 |
| efried | My beef is that that chunk of code is already pretty convoluted. | 18:09 |
| efried | So | 18:09 |
| efried | thorst, mdrabe: Option 1: rework vscsi.py discon_vol_for_vio as described above. Option 2: hit the main destroy flow with a bigger hammer, adding a "destroy_disks=destroy_disks" option to the LPAR scrubber. | 18:10 |
| mdrabe | I honestly have no clue how the VIOS would respond to option 2 | 18:11 |
| efried | What do you mean? | 18:11 |
| efried | We use that scrub already. | 18:11 |
| efried | The VIOS responds just fine. | 18:11 |
| mdrabe | You can just remove all those mappings/elements no problem | 18:11 |
| mdrabe | ? | 18:12 |
| efried | Yeah. | 18:12 |
| *** kotra03 has quit IRC | 18:12 | |
| mdrabe | Where as there's designated tasks to remove them nicely, which seem to involve a lot of logic | 18:12 |
| mdrabe | That's my concern, but I guess if it works for everything currently... | 18:12 |
| efried | Correct. And which rely on the mappings/elements behaving nicely. | 18:12 |
| mdrabe | Wait though | 18:12 |
| efried | which they're not in this case. | 18:12 |
| mdrabe | That scrub is only being used in one place right? | 18:12 |
| efried | Today? | 18:13 |
| efried | I think two. | 18:13 |
| mdrabe | Yes | 18:13 |
| mdrabe | looking | 18:13 |
| efried | nova_powervm/virt/powervm/slot.py:123: pvm_tstor.add_lpar_storage_scrub_tasks([lpar_id], scrub_ftsk, | 18:13 |
| efried | nova_powervm/virt/powervm/tasks/vm.py:98: pvm_stg.add_lpar_storage_scrub_tasks([wrap.id], self.stg_ftsk, | 18:13 |
| mdrabe | So the example in slot.py is kind of not a good one | 18:13 |
| mdrabe | since that should be for stale mappings yes? Which doesn't happen _too_ often | 18:14 |
| efried | It's almost identical to this situation we're discussing. | 18:15 |
| mdrabe | I disagree because | 18:15 |
| mdrabe | https://review.openstack.org/#/c/348014/ would apply to all deletes | 18:15 |
| mdrabe | That's what you're talking about right? Doing the scrub there? Just wanna make sure we're on the same page | 18:16 |
| efried | Yes, applies to all deletes. | 18:17 |
| efried | Yes, doing the scrub during the main destroy flow. | 18:17 |
| efried | So okay, yeah, the slot.py example is doing the scrub during the create flow. So is the vm.py example. | 18:19 |
| efried | This would be the first attempt to use it during a "normal" destroy flow. | 18:19 |
| *** k0da has joined #openstack-powervm | 18:20 | |
| mdrabe | Alright efried, I'm gonna talk about vscsi since I know at least something about that | 18:21 |
| mdrabe | In the vscsi driver, the disconnect logic, the stuff that does the work is _add_remove_mapping and _add_remove_hdisk | 18:22 |
| mdrabe | Actually I start getting confused right here | 18:23 |
| mdrabe | _add_remove_mapping should remove the specific VSCSI mapping for volume X on VIOS A, which would include the PV as well no? So what does _add_remove_hdisk do? | 18:24 |
| efried | mdrabe, no. _add_remove_mapping removes the VSCSI mapping. add_remove_hdisk deletes the PV. | 18:25 |
| mdrabe | Got it ok. So the remove hdisk job essentially the opposite of LUA recovery? | 18:26 |
| efried | er, not really. | 18:26 |
| *** k0da has quit IRC | 18:26 | |
| efried | Think of LUA recovery as "run cfgmgr, then try to find the disk I care about". | 18:27 |
| efried | Remove hdisk is just rmdev | 18:27 |
| mdrabe | Got it got it | 18:27 |
| mdrabe | So does that scrub remove the hdisk? | 18:27 |
| efried | yup. | 18:27 |
| efried | uhhh. | 18:28 |
| efried | shoot, no. | 18:29 |
| mdrabe | K so that's one actually concrete conern | 18:29 |
| efried | Only vopt and vdisk (LV) | 18:29 |
| efried | Yeah. This won't work at all. | 18:29 |
| efried | Option 1 it is. | 18:29 |
| efried | Dang, it *almost* would have worked as is. | 18:31 |
| efried | If only [None] evaluated as False. | 18:31 |
| mdrabe | Ok so efried: so before the return False in the disconnect in the vscsi driver, there'll be some remove mapping logic | 18:33 |
| mdrabe | Which'll require some additional logic since we don't have the device name | 18:34 |
| *** catintheroof has joined #openstack-powervm | 18:34 | |
| efried | mdrabe, thorst: I'm actually thinking we remove *all* of the 'return False's; and then add logic around the _add_remove_*() calls such that, if device_name is None: | 18:37 |
| efried | => _add_remove_mappings converts [None] to None or [] (I think this will actually wind up "deleting" the mappings more than once - discussion to follow) | 18:37 |
| efried | => _add_remove_hdisk doesn't get run at all. | 18:37 |
| efried | So I believe that first thing will trigger deletion of all the mappings for the LPAR in question. Since this is in a generic disconnect flow, that may actually be a very bad thing. If we're doing disconnect on a running instance (rather than under the auspices of a full-VM op like destroy, LPM, cold migration, etc.), it'll disconnect disks we didn't want to tough. | 18:38 |
| efried | touch. | 18:38 |
| efried | So - is there some other way we can identify just the mapping that's busted? | 18:39 |
| mdrabe | We'll know there's busted mappings if we can't find the hdisk | 18:40 |
| efried | Is any part of the mapping object itself broken? | 18:40 |
| mdrabe | er well sorry, that doesn't really help | 18:40 |
| efried | If it's missing a client adapter, we're golden. | 18:40 |
| efried | If it's got a client adapter, but is missing the storage element, we *might* be able to use an existing scrubber - I would have to check whether we have a scrubber that identifies storage-less but client-having mappings. | 18:41 |
| mdrabe | I don't _think_ it is | 18:41 |
| efried | Else we could conceivably create such a scrubber. | 18:41 |
| efried | Are you able to reproduce this problem at this point? | 18:41 |
| efried | E.g. by pdbing and destroying the ITLs in an otherwise-normal flow? | 18:41 |
| mdrabe | It's reproducible, I can't reproduce it right now for lack of systems | 18:41 |
| efried | Oh, that doesn't help. We need to see whether the mappings are broken. | 18:41 |
| efried | brb | 18:41 |
| *** apearson has quit IRC | 18:46 | |
| efried | I don't have a scrubber for storage-less mappings. | 18:46 |
| efried | I'm not even sure if it's a valid thing to remove such mappings in general. | 18:47 |
| efried | May be an apearson question. | 18:47 |
| efried | mdrabe, can you do me this: | 18:47 |
| mdrabe | Yes, in this case | 18:47 |
| mdrabe | evacuated VM deletes have storage-less mappings | 18:47 |
| efried | Create a disk. Create a mapping to it. Rip the disk out (in the same way it's happening in this scenario). Then pull the mapping and see what it looks like. | 18:47 |
| efried | oh, unless you already have some XML like that. | 18:48 |
| efried | It's just... | 18:48 |
| efried | I wonder what it looks like if the ghost of the storage is there before discover_hdisk. | 18:48 |
| mdrabe | It shouldn't be just rip the disk out though | 18:48 |
| efried | Whether it shows up in the mapping, and if so, if that's distinguishable from when the disk is valid. | 18:49 |
| mdrabe | It should be rip the disk out (and I mean from the storage backend) and restart the VIOS OR run cfgmgr/cfgdev | 18:49 |
| mdrabe | Because if the system is down, which it should be for evacuation and the VIOS comes back up... | 18:50 |
| mdrabe | Then that's why I'm thinking the hdisk is no longer there | 18:50 |
| *** apearson has joined #openstack-powervm | 19:03 | |
| thorst | so..I was afk. Should I catch up on that 40 minutes of convo? | 19:04 |
| mdrabe | thorst: eh probably not | 19:07 |
| mdrabe | unless you want to | 19:07 |
| *** tblakeslee has quit IRC | 19:07 | |
| thorst | mdrabe: totally good not to | 19:08 |
| *** tblakeslee has joined #openstack-powervm | 19:30 | |
| efried | thorst, the upshot is my proposal isn't going to work. See my abandon comment. | 19:30 |
| *** apearson has quit IRC | 19:30 | |
| *** k0da has joined #openstack-powervm | 19:30 | |
| efried | Now we're investigating ways to make the mapping disappear in this error scenario. The problem is, since we have nothing with which to identify the backing storage, we need to figure out some other way to identify the mapping. Current speculation is that, in this situation, the Storage and/or TargetDev will actually be absent from the mapping. If so, we should be able to remove such mappings with impunity - though we'll | 19:31 |
| efried | It'll basically be a new kind of scrubber. | 19:32 |
| efried | Which we actually may as well institute in the general case (perhaps even as part of ComprehensiveScrub) | 19:32 |
| efried | ...assuming it's valid. | 19:32 |
| efried | We should consult with apearson about that. | 19:32 |
| thorst | yeah...this won't clear out the storage though? Just the mapping? | 19:33 |
| efried | thorst, Right. The storage is already gone. That's the point. | 19:35 |
| thorst | got it | 19:36 |
| *** apearson has joined #openstack-powervm | 19:54 | |
| *** dwayne has quit IRC | 20:08 | |
| efried | apearson: Advice on scrubbing storage-less mappings... | 20:09 |
| efried | The scenario is bringing a nova-powervm back up after crashing it and evacuating its instances to another host. | 20:11 |
| efried | Before the source host comes up, the PVs associated with those evacuated instances has been ripped out at the storage provider level (mdrabe, help me with the terminology here if I screwed that up) | 20:12 |
| efried | The nova compute comes up and runs a process that destroys evacuated instances. | 20:12 |
| efried | As part of that, we're trying to delete mappings and storage associated with such instances. | 20:12 |
| apearson | @efried - so in that case, you are trying to delete the lpar...and as part of that, absolutely you can clean up mappings. Or you could just delete the lpar and then let your mapping 'fix' code handle it as it would now have only a server adapter and no client (since the client was associated w/ the lpar) | 20:14 |
| mdrabe | apearson: If I delete the LPAR, does the client adapter for any associated mappings of that LPAR automatically get deleted? | 20:15 |
| efried | (mdrabe, no) | 20:15 |
| efried | apearson, okay, the quandary is that this code currently lives in the generic disconnect flow - meaning even if we're just trying to disconnect a (single) disk from a (live) VM that's running normally. | 20:15 |
| efried | For a normal living VM, is it possible to cause harm by removing mappings that have no Storage element? | 20:16 |
| apearson | no, not generally. The case where it causes harm is if somebody is manually changing stuff...removing a mapping and replacing it with something else (for example). | 20:17 |
| efried | Maybe the storage fabric is flaking and the disk is temporarily invisible so the mapping is missing its Storage? Normally the storage would come back to life at some point? In which case, if we deleted the mapping, we've actually disconnected a disk which was only temporarily sick? | 20:17 |
| apearson | VIOS should still be showing the disk...it's just in defined state perhaps. IE I believe you still see the mapping. I'm still confused why you even need to be doing this. | 20:25 |
| efried | apearson: In the case we're trying to fix, it's because we've failed to retrieve one or both of the disk name or UDID from discover_hdisk. | 20:26 |
| efried | aka lua_recovery | 20:30 |
| efried | ...to which we've provided the "volume_id", whatever that is (cause apparently it's not the UDID or the UUID) | 20:31 |
| openstackgerrit | Adam Reznechek proposed openstack/nova-powervm: Fix package setup configuration https://review.openstack.org/348544 | 20:34 |
| *** apearson has quit IRC | 20:35 | |
| *** apearson has joined #openstack-powervm | 20:36 | |
| thorst | adreznec: Curious if that passes devstack... | 20:38 |
| thorst | efried: you should probably take a peak at that ^^ | 20:38 |
| efried | I am | 20:38 |
| efried | I'm not sure why this change is needed. | 20:39 |
| thorst | OSA | 20:39 |
| thorst | eggs | 20:39 |
| thorst | venvs | 20:39 |
| adreznec | Not really OSA | 20:39 |
| adreznec | The way it was before wasn't actually causing the shim driver files to get properly installed into /nova/virt/powervm if you did a package (e.g. pip) install of the driver | 20:39 |
| adreznec | It would only install the files under nova_powervm, not those under nova/virt/powervm | 20:40 |
| adreznec | Meaning that you couldn't actually do an import of nova.virt.powervm.driver | 20:40 |
| adreznec | I'm doing a full OSA redeploy to ensure it works there as well, but that will take like an hour | 20:43 |
| efried | adreznec, okay, if you say so. | 20:43 |
| adreznec | Solved the problem in my smaller testing though | 20:44 |
| efried | thorst, I got the +1, nyah nyah. | 20:44 |
| adreznec | It's ok, thorst is an PEP420/python namespace extension pro | 20:45 |
| *** jwcroppe has quit IRC | 20:53 | |
| *** dwayne has joined #openstack-powervm | 20:54 | |
| efried | adreznec, really? Then why did I wind up stumbling through https://review.openstack.org/#/c/310610/ ?? | 20:55 |
| adreznec | efried: Uhhhhhhhhhhhhh...... learning experience? | 20:58 |
| efried | Ticket to Barcelona? | 20:59 |
| efried | I'll take it. | 20:59 |
| *** apearson has quit IRC | 21:04 | |
| thorst | adreznec efried: svenkat wins the day...he found a way to cut out a bunch of our networking-powervm code. | 21:10 |
| thorst | we should talk tomorrow | 21:10 |
| thorst | I've got to start biking home. | 21:10 |
| adreznec | Huh | 21:10 |
| adreznec | cool | 21:10 |
| efried | thorst, he figured out how to do it without monkey patching? | 21:10 |
| adreznec | That's always a nice thing | 21:10 |
| thorst | adreznec: yeah... | 21:10 |
| thorst | efried: not that yet. | 21:10 |
| thorst | this is our SEA code | 21:10 |
| svenkat | yes! no more monkey patching | 21:10 |
| thorst | he found a way to get the VLAN back to Nova-PowerVM | 21:10 |
| thorst | that is proper | 21:10 |
| efried | Cool, look forward to hearing about it. | 21:10 |
| adreznec | Yay | 21:10 |
| thorst | so we don't have to have our agent 'loop' and listen to events and do goofy stuff | 21:11 |
| thorst | should probably help with CI too. | 21:11 |
| adreznec | Less code to maintain! | 21:11 |
| thorst | yeah...too bad wpward isn't around to see it get deleted | 21:11 |
| thorst | I should tweet him so he knows | 21:11 |
| thorst | lol | 21:11 |
| efried | svenkat, is a review up yet? | 21:11 |
| svenkat | not yet. | 21:12 |
| *** svenkat has quit IRC | 21:13 | |
| *** apearson has joined #openstack-powervm | 21:29 | |
| *** apearson has quit IRC | 21:33 | |
| *** thorst has quit IRC | 21:33 | |
| *** thorst has joined #openstack-powervm | 21:34 | |
| *** thorst has quit IRC | 21:38 | |
| *** burgerk_ has joined #openstack-powervm | 21:39 | |
| *** burgerk has quit IRC | 21:42 | |
| *** burgerk_ has quit IRC | 21:43 | |
| *** edmondsw has quit IRC | 21:46 | |
| *** apearson has joined #openstack-powervm | 21:46 | |
| *** seroyer has quit IRC | 21:56 | |
| *** esberglu has quit IRC | 21:57 | |
| *** Ashana has quit IRC | 21:57 | |
| *** Ashana has joined #openstack-powervm | 21:58 | |
| apearson | @efried - but then aren't you trying to rip apart an existing mapping? Just because you don't understand the disk associated w/ the mapping, doesn't mean you should clean it up. | 21:59 |
| efried | apearson, exactly. | 21:59 |
| efried | It's the volume driver that has this volume_id thingy. | 21:59 |
| efried | It held onto that from the previous incarnation of the compute driver somehow. | 22:00 |
| efried | Point is, we now can't find the associated volume on the VIOS. | 22:00 |
| efried | Ergo we can't identify which of possibly several mappings associated with this LPAR actually held that volume. | 22:00 |
| *** tsjakobs has quit IRC | 22:01 | |
| efried | But the question is, will such a mapping show up without storage on it? And is there harm in removing such mappings even when we don't know for sure they're associated with the disk we're trying to disconnect? | 22:01 |
| *** mdrabe has quit IRC | 22:02 | |
| *** Ashana has quit IRC | 22:03 | |
| apearson | @efried - I don't believe so. But you can easily test this: 1) Create a pv-based mapping. 2) Remove the disk on the SAN, but make sure it still shows up in the vios as 'defined'. 3) See how that mapping looks. | 22:13 |
| efried | rgr, thx | 22:14 |
| *** jwcroppe has joined #openstack-powervm | 22:15 | |
| *** Ashana has joined #openstack-powervm | 22:16 | |
| *** Ashana has quit IRC | 22:20 | |
| *** Ashana has joined #openstack-powervm | 22:22 | |
| *** Ashana has quit IRC | 22:26 | |
| *** Ashana has joined #openstack-powervm | 22:28 | |
| *** Ashana has quit IRC | 22:32 | |
| *** k0da has quit IRC | 22:38 | |
| *** jwcroppe has quit IRC | 22:46 | |
| *** Ashana has joined #openstack-powervm | 22:57 | |
| *** tblakeslee has quit IRC | 22:58 | |
| *** Ashana has quit IRC | 23:01 | |
| *** Ashana has joined #openstack-powervm | 23:03 | |
| *** apearson has quit IRC | 23:04 | |
| *** Ashana has quit IRC | 23:07 | |
| *** Ashana has joined #openstack-powervm | 23:14 | |
| *** Ashana has quit IRC | 23:19 | |
| *** Ashana has joined #openstack-powervm | 23:20 | |
| *** Ashana has quit IRC | 23:25 | |
| *** Ashana has joined #openstack-powervm | 23:27 | |
| *** Ashana has quit IRC | 23:31 | |
| *** Ashana has joined #openstack-powervm | 23:33 | |
| *** Ashana has quit IRC | 23:37 | |
| *** Ashana has joined #openstack-powervm | 23:45 | |
| *** Ashana has quit IRC | 23:49 | |
| *** Ashana has joined #openstack-powervm | 23:50 | |
| *** Ashana has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!