| *** thorst_afk has quit IRC | 01:37 | |
| *** svenkat has quit IRC | 01:45 | |
| *** svenkat has joined #openstack-powervm | 01:55 | |
| *** svenkat has quit IRC | 02:09 | |
| *** svenkat has joined #openstack-powervm | 02:18 | |
| *** esberglu has quit IRC | 02:39 | |
| *** esberglu has joined #openstack-powervm | 02:39 | |
| *** svenkat has quit IRC | 02:46 | |
| *** esberglu has quit IRC | 02:47 | |
| *** esberglu has joined #openstack-powervm | 03:05 | |
| *** thorst_afk has joined #openstack-powervm | 03:38 | |
| *** thorst_afk has quit IRC | 03:52 | |
| *** esberglu has quit IRC | 05:23 | |
| *** thorst_afk has joined #openstack-powervm | 05:49 | |
| *** thorst_afk has quit IRC | 05:53 | |
| *** thorst_afk has joined #openstack-powervm | 07:22 | |
| *** thorst_afk has quit IRC | 07:27 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** edmondsw has joined #openstack-powervm | 09:22 | |
| *** thorst_afk has joined #openstack-powervm | 09:23 | |
| *** thorst_afk has quit IRC | 09:28 | |
| *** smatzek has joined #openstack-powervm | 11:37 | |
| *** thorst_afk has joined #openstack-powervm | 12:03 | |
| *** svenkat has joined #openstack-powervm | 12:05 | |
| *** svenkat has quit IRC | 12:10 | |
| *** svenkat has joined #openstack-powervm | 12:11 | |
| *** esberglu has joined #openstack-powervm | 12:40 | |
| *** esberglu has quit IRC | 12:45 | |
| *** kylek3h has joined #openstack-powervm | 12:52 | |
| *** apearson has joined #openstack-powervm | 12:59 | |
| *** esberglu has joined #openstack-powervm | 13:10 | |
| *** smatzek has quit IRC | 13:41 | |
| *** efried has joined #openstack-powervm | 13:58 | |
| *** smatzek has joined #openstack-powervm | 14:01 | |
| esberglu | efried: http://184.172.12.213/55/408955/72/check/nova-in-tree-pvm/852c09d/ | 14:03 |
|---|---|---|
| esberglu | Seeing this in n-cpu.log | 14:03 |
| esberglu | HttpError: HTTP error 500 for method DELETE on path /rest/api/uom/LogicalPartition/403003D2-5380-4F58-8DF1-87CB4A639D2A: Internal Server Error -- [PVME01050010-0056] This task is only allowed when the partition is powered off. | 14:03 |
| esberglu | Right before that is | 14:03 |
| esberglu | It took task 'pwr_off_vm' 0.00 seconds to finish | 14:03 |
| esberglu | So it looks like the power off either isn't working correctly or isn't actually complete when the destroy happens | 14:04 |
| efried | esberglu Have we confirmed what pypowervm version was running in that env? | 14:04 |
| *** smatzek has quit IRC | 14:05 | |
| esberglu | efried: Yeah | 14:05 |
| thorst_afk | mdrabe: just had another user hit https://bugs.launchpad.net/pypowervm/+bug/1708206 | 14:07 |
| openstack | Launchpad bug 1708206 in pypowervm "Orphan adapter cleanup removes valid *ANY* type adapters" [Undecided,New] - Assigned to Matt Rabe (mdrabe) | 14:07 |
| thorst_afk | a team out in poland :-) | 14:08 |
| efried | esberglu There are two instances with the same UUID. | 14:14 |
| *** smatzek has joined #openstack-powervm | 14:14 | |
| efried | esberglu Are we seeing this error frequently? | 14:14 |
| esberglu | efried: Decent amount | 14:15 |
| efried | Then I surmise it's because a rebuild actually creates the new instance with the same UUID before the old one is really gone. | 14:15 |
| mdrabe | thorst_afk That fix is awaiting your review :) | 14:15 |
| thorst_afk | mdrabe: boo. Why am I always the bottleneck :-) | 14:16 |
| *** chhavi has joined #openstack-powervm | 14:18 | |
| efried | thorst_afk Just point out a UT gap, buy yourself a couple hours. | 14:19 |
| thorst_afk | jerk | 14:21 |
| thorst_afk | mdrabe: I found another variation we need to consider... | 14:22 |
| thorst_afk | talking to the team doing it. Basically if there are mappings between the VIOS and the NovaLink - just don't scrub those. They're trying to create the mapping up front, and then add things later to it. The way our scrubber works is that we see the mapping doesn't have anything and we wipe it...which makes it hard for them to add later | 14:22 |
| esberglu | efried: Is there a good way to determine the difference between the 2? | 14:22 |
| thorst_afk | since it's a special VM...I think its OK for us to not clean the mappings. The rest we need to, but that one we should probably ignore | 14:23 |
| thorst_afk | I'll add to review. | 14:23 |
| efried | esberglu I don't know how it even begins to work. It shouldn't be possible for us to create both of them on the hypervisor at once. | 14:24 |
| efried | Someone needs to dig in and understand this rebuild flow. | 14:24 |
| mdrabe | thorst_afk: You're saying they create a server adapter without a client between VIOS and NL and add the client later? | 14:25 |
| thorst_afk | nah...they have the client/server adapter. They just don't have any storage in it yet | 14:28 |
| mdrabe | I don't think we scrub those | 14:28 |
| thorst_afk | no disks yet. Because the HMC they created it with can't connect a disk | 14:28 |
| thorst_afk | I think we do... efried - do we? | 14:28 |
| efried | thorst_afk Looking. | 14:29 |
| efried | which scrub is this? | 14:29 |
| efried | Comprehensive? | 14:29 |
| mdrabe | https://github.com/powervm/pypowervm/blob/master/pypowervm/tasks/storage.py#L943-L944 | 14:29 |
| mdrabe | Unless it's something else | 14:30 |
| thorst_afk | any scruber that runs on startup | 14:30 |
| thorst_afk | brb | 14:30 |
| *** thorst_afk has quit IRC | 14:30 | |
| mdrabe | Pretty sure that's the comprehensive from nova-powervm | 14:30 |
| efried | We'll remove any mapping whose LPAR ID doesn't correspond to a real LPAR. (And we'll remove any mapping that doesn't have a client adapter.) | 14:32 |
| efried | That first part may be where we're getting into trouble here. Is the nvl showing up as a VIOS or an LPAR in this env? | 14:32 |
| *** thorst_afk has joined #openstack-powervm | 14:32 | |
| efried | nah, never mind, we already take that into account. | 14:33 |
| efried | thorst_afk here's what you missed: | 14:34 |
| efried | We'll remove any mapping whose LPAR ID doesn't correspond to a real LPAR. (And we'll remove any mapping that doesn't have a client adapter.) | 14:34 |
| efried | That first part may be where we're getting into trouble here. Is the nvl showing up as a VIOS or an LPAR in this env? | 14:34 |
| *** thorst_afk has quit IRC | 14:37 | |
| *** thorst_afk has joined #openstack-powervm | 14:42 | |
| *** thorst_afk has quit IRC | 15:00 | |
| *** thorst_afk has joined #openstack-powervm | 15:01 | |
| *** apearson has quit IRC | 15:27 | |
| *** apearson has joined #openstack-powervm | 15:30 | |
| efried | thorst_afk Metrics can come in any order wrt datetime? | 15:33 |
| efried | Swhy we can't just pick the last two off the list? | 15:33 |
| esberglu | efried: edmondsw: Some of the runs are failing the discover_hosts command while stacking | 16:08 |
| *** thorst_afk has quit IRC | 16:08 | |
| esberglu | Well, not failing, but just failing to discover any hosts | 16:08 |
| efried | esberglu And we're bailing right away, or continuing on? | 16:09 |
| edmondsw | any idea why? | 16:10 |
| efried | TBH, the answer there might be the ill-fated patch that sdague is trying to land. | 16:10 |
| esberglu | efried: Well this is part of the actual stack script, we don't do discover_hosts on our own now | 16:10 |
| efried | It could be because the compute service won't register until it's up. | 16:10 |
| esberglu | So it just fails to discover any hosts and the just continues on with the script | 16:10 |
| esberglu | I'm thinking for now we can probably just toss another discover_hosts command post stack | 16:10 |
| edmondsw | esberglu efried see comment on 5641 before you merge that | 16:11 |
| efried | Good catch edmondsw | 16:13 |
| edmondsw | efried not sure I followed that about sdague's patch... that hasn't landed, so it can't be troubling us here unless it's in the dependency chain of the patch being run through the CI | 16:14 |
| efried | edmondsw I'm saying sdague's patch will *fix* this problem for us. | 16:15 |
| *** thorst_afk has joined #openstack-powervm | 16:15 | |
| edmondsw | hmm | 16:15 |
| efried | Because it waits for the compute service to come alive before proceeding with the stack. | 16:15 |
| efried | If the problem is discover_hosts failing because the compute service isn't alive yet... | 16:15 |
| edmondsw | efried why would we just now be seeing this, though? | 16:15 |
| efried | Because esberglu removed our discover_hosts loop. | 16:16 |
| efried | because we thought we could rely on the one in stack.sh itself. | 16:16 |
| efried | I could be misunderstanding. | 16:16 |
| edmondsw | oh... yeah | 16:16 |
| esberglu | Exactly. The discover_hosts in stack is sometimes finding the host, sometimes not | 16:19 |
| thorst_afk | efried: yes - metrics can come in any order. At least, I want to assume they can. | 16:21 |
| efried | thorst_afk Okay, see my latest -1 | 16:21 |
| thorst_afk | jerk | 16:21 |
| efried | thorst_afk You'll thank me for this one. | 16:24 |
| efried | LOC reduction, improved readability/maintainability, sonar appeasement... | 16:25 |
| efried | And the test cases shouldn't have to change, assuming they're written correctly ;-) | 16:25 |
| efried | (except for that unused var for the sonar issue) | 16:25 |
| *** apearson has quit IRC | 16:41 | |
| *** apearson has joined #openstack-powervm | 16:43 | |
| *** smatzek has quit IRC | 16:43 | |
| *** smatzek has joined #openstack-powervm | 17:01 | |
| *** efried has quit IRC | 17:01 | |
| *** chhavi has quit IRC | 17:29 | |
| *** chhavi has joined #openstack-powervm | 17:34 | |
| *** chhavi has quit IRC | 17:38 | |
| *** chhavi has joined #openstack-powervm | 18:07 | |
| *** chhavi has quit IRC | 18:12 | |
| *** chhavi has joined #openstack-powervm | 18:34 | |
| *** apearson has quit IRC | 19:32 | |
| *** apearson has joined #openstack-powervm | 19:34 | |
| *** esberglu has quit IRC | 19:38 | |
| *** efried has joined #openstack-powervm | 19:48 | |
| *** esberglu has joined #openstack-powervm | 19:53 | |
| *** apearson has quit IRC | 19:59 | |
| *** apearson has joined #openstack-powervm | 20:02 | |
| *** apearson has quit IRC | 20:14 | |
| *** apearson has joined #openstack-powervm | 20:14 | |
| *** chhavi has quit IRC | 20:20 | |
| *** smatzek has quit IRC | 20:31 | |
| *** thorst_afk has quit IRC | 21:10 | |
| *** thorst_afk has joined #openstack-powervm | 21:12 | |
| *** thorst_afk has quit IRC | 21:17 | |
| *** thorst_afk has joined #openstack-powervm | 21:30 | |
| *** thorst_afk has quit IRC | 21:34 | |
| *** apearson has quit IRC | 21:36 | |
| *** thorst_afk has joined #openstack-powervm | 21:50 | |
| *** thorst_afk has quit IRC | 21:55 | |
| *** openstackgerrit has joined #openstack-powervm | 21:56 | |
| openstackgerrit | Matt Rabe proposed openstack/nova-powervm master: WIP: Change NVRAM manager and Swift APIs to accept UUID https://review.openstack.org/471926 | 21:56 |
| *** dwayne has quit IRC | 21:57 | |
| *** esberglu has quit IRC | 22:06 | |
| *** svenkat has quit IRC | 22:06 | |
| *** esberglu has joined #openstack-powervm | 22:06 | |
| *** kylek3h has quit IRC | 22:08 | |
| *** esberglu has quit IRC | 22:10 | |
| *** edmondsw has quit IRC | 22:23 | |
| *** edmondsw has joined #openstack-powervm | 22:26 | |
| *** edmondsw has quit IRC | 22:31 | |
| *** svenkat has joined #openstack-powervm | 22:32 | |
| *** svenkat has quit IRC | 22:36 | |
| *** efried is now known as efried_zzz | 22:50 | |
| *** thorst_afk has joined #openstack-powervm | 23:26 | |
| *** thorst_afk has quit IRC | 23:32 | |
| *** edmondsw has joined #openstack-powervm | 23:54 | |
| *** edmondsw has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!