| *** kylek3h has joined #openstack-powervm | 00:23 | |
| *** apearson has joined #openstack-powervm | 00:55 | |
| *** esberglu has joined #openstack-powervm | 01:01 | |
| *** esberglu has quit IRC | 01:03 | |
| *** kylek3h has quit IRC | 01:19 | |
| *** jwcroppe has quit IRC | 01:27 | |
| *** jwcroppe has joined #openstack-powervm | 01:31 | |
| *** jwcroppe has quit IRC | 01:33 | |
| *** jwcroppe has joined #openstack-powervm | 01:35 | |
| *** jwcroppe has quit IRC | 01:39 | |
| *** svenkat has joined #openstack-powervm | 02:14 | |
| *** jwcroppe has joined #openstack-powervm | 02:36 | |
| *** jwcroppe has quit IRC | 02:53 | |
| *** svenkat has quit IRC | 03:54 | |
| *** svenkat has joined #openstack-powervm | 03:56 | |
| *** svenkat has quit IRC | 03:58 | |
| *** apearson has quit IRC | 04:17 | |
| *** jwcroppe has joined #openstack-powervm | 04:49 | |
| *** jwcroppe has quit IRC | 04:54 | |
| *** jwcroppe has joined #openstack-powervm | 05:51 | |
| *** jwcroppe has quit IRC | 05:56 | |
| *** madhaviy has joined #openstack-powervm | 06:59 | |
| *** jwcroppe has joined #openstack-powervm | 07:53 | |
| *** jwcroppe has quit IRC | 07:57 | |
| *** jwcroppe has joined #openstack-powervm | 08:54 | |
| *** jwcroppe has quit IRC | 08:58 | |
| *** madhaviy_ has joined #openstack-powervm | 09:10 | |
| *** madhaviy has quit IRC | 09:13 | |
| *** madhaviy_ is now known as madhaviy | 09:13 | |
| *** jwcroppe has joined #openstack-powervm | 10:55 | |
| *** jwcroppe has quit IRC | 11:00 | |
| *** madhaviy has quit IRC | 11:06 | |
| *** madhaviy has joined #openstack-powervm | 11:06 | |
| *** madhaviy has quit IRC | 11:39 | |
| *** svenkat has joined #openstack-powervm | 11:48 | |
| *** edmondsw has joined #openstack-powervm | 12:28 | |
| *** mdrabe has joined #openstack-powervm | 12:32 | |
| *** burgerk has joined #openstack-powervm | 12:43 | |
| *** esberglu has joined #openstack-powervm | 12:45 | |
| *** esberglu has quit IRC | 12:49 | |
| *** jwcroppe has joined #openstack-powervm | 12:57 | |
| *** jwcroppe has quit IRC | 13:01 | |
| *** jwcroppe has joined #openstack-powervm | 13:06 | |
| *** kylek3h has joined #openstack-powervm | 13:10 | |
| *** esberglu has joined #openstack-powervm | 13:19 | |
| *** seroyer has joined #openstack-powervm | 13:33 | |
| *** apearson has joined #openstack-powervm | 13:51 | |
| *** madhaviy has joined #openstack-powervm | 13:56 | |
| *** tblakes has joined #openstack-powervm | 14:02 | |
| *** seroyer has quit IRC | 14:11 | |
| *** thorst_ has joined #openstack-powervm | 14:12 | |
| *** kriskend has joined #openstack-powervm | 14:12 | |
| *** seroyer has joined #openstack-powervm | 14:13 | |
| *** burgerk has quit IRC | 14:13 | |
| esberglu | thorst_: Tried a few conf. variations last night, still no luck getting the compute to stack | 14:22 |
|---|---|---|
| thorst_ | yeah...I was playing with that. I think its just a legit bug? | 14:23 |
| *** seroyer has quit IRC | 14:23 | |
| thorst_ | I need to investigate more but don't have time at the moment (doing a scale test for someone). | 14:23 |
| thorst_ | adreznec: do you have chance to take a peak? | 14:23 |
| adreznec | thorst_: Is this still the stacking issue w/ Neutron? | 14:25 |
| esberglu | adreznec: yeah | 14:26 |
| kriskend | Pky93o | 14:27 |
| adreznec | Hmm ok | 14:28 |
| adreznec | I can try and take a look in a bit | 14:28 |
| adreznec | Have some other stuff to dig into first | 14:28 |
| esberglu | adreznec: Okay. I’ll keep looking | 14:30 |
| *** burgerk has joined #openstack-powervm | 14:38 | |
| *** tjakobs has joined #openstack-powervm | 14:40 | |
| *** apearson has quit IRC | 14:42 | |
| *** seroyer has joined #openstack-powervm | 14:43 | |
| *** seroyer has quit IRC | 14:45 | |
| *** seroyer has joined #openstack-powervm | 14:59 | |
| *** catintheroof has joined #openstack-powervm | 15:27 | |
| thorst_ | saving a 250 GB image is an awful idea... | 15:32 |
| *** seroyer has quit IRC | 15:50 | |
| *** seroyer has joined #openstack-powervm | 15:52 | |
| *** jwcroppe has quit IRC | 15:57 | |
| *** jwcroppe has joined #openstack-powervm | 15:58 | |
| openstackgerrit | Tyler Blakeslee proposed openstack/nova-powervm: Added revert for Resize in order to revert temp name change https://review.openstack.org/354179 | 15:58 |
| *** jwcroppe_ has joined #openstack-powervm | 15:59 | |
| *** jwcroppe has quit IRC | 16:03 | |
| *** seroyer has quit IRC | 16:13 | |
| *** seroyer has joined #openstack-powervm | 16:15 | |
| *** esberglu has quit IRC | 16:23 | |
| *** madhaviy has quit IRC | 16:41 | |
| openstackgerrit | Matt Rabe proposed openstack/nova-powervm: Move LPAR scrub tasks from build_map to Create task https://review.openstack.org/354217 | 16:45 |
| thorst_ | efried: That one above, you've been interlocked on? | 16:46 |
| tblakes | thorst: Just saw your comment on https://review.openstack.org/#/c/354179/ about using self.instance.name | 16:46 |
| tblakes | instance.name is used to revert the name back to the original, is that an issue? | 16:47 |
| tblakes | https://github.com/openstack/nova-powervm/blob/e90cc307ddd015d915fda7b555cc66d06274a416/nova_powervm/virt/powervm/driver.py#L1364 | 16:47 |
| thorst_ | yeah, because at that point in the code, you don't actually know that that was the original name | 16:47 |
| thorst_ | it fixes your bug...but what if you were renaming from a different original name | 16:47 |
| *** tblakes_ has joined #openstack-powervm | 16:47 | |
| efried | thorst_, yes, mdrabe and I have been working through that bug. Last we spoke (less than an hour ago) he was going to verify that there were no other gotchas related to initializing the rebuild map before the LPAR was created. Assume the above review has that taken into account. | 16:48 |
| thorst_ | sweet...just know you're smarter about that crap than I am | 16:49 |
| efried | Meanwhile, I will pay anyone a million dollars to get my freaking victim to stack AIO. | 16:49 |
| thorst_ | lol | 16:49 |
| thorst_ | talk to esberglu and adreznec | 16:49 |
| thorst_ | I had to overwrite parts of devstack while they're working that | 16:50 |
| seroyer | efried, get your checkbook ready first | 16:50 |
| *** tblakes has quit IRC | 16:51 | |
| *** tblakes_ is now known as tblakes | 16:52 | |
| efried | thorst_, where can I find your devstack hacks? | 16:56 |
| thorst_ | what error are you hitting. If AIO, I don't think you'd hit the same | 16:57 |
| thorst_ | just pastebin it? | 16:57 |
| *** tblakes has quit IRC | 17:00 | |
| efried | I don't know what a pastebin is. | 17:01 |
| efried | thorst_ http://pastebin.com/6izgjtw7 | 17:02 |
| thorst_ | efried: yeah, that's a different problem than what any of us are hitting. | 17:02 |
| efried | Well, I hacked the keystone version thinking that could help, but it didn't. | 17:03 |
| efried | lemme revert and do it again | 17:03 |
| thorst_ | ahh | 17:03 |
| efried | But the previous error was very similar. | 17:03 |
| *** tblakes has joined #openstack-powervm | 17:06 | |
| *** tblakes_ has joined #openstack-powervm | 17:10 | |
| efried | thorst_, http://pastebin.com/jP5X5Jib | 17:10 |
| thorst_ | efried: so I'm assuming you did the whole path of running that command on the system yourself | 17:11 |
| thorst_ | pre-unstack? | 17:11 |
| *** tblakes has quit IRC | 17:11 | |
| *** tblakes_ is now known as tblakes | 17:11 | |
| efried | The openstack project show command? Yeah. 500. | 17:11 |
| thorst_ | send my way the system...going to grab a bite to eat with the OSA team first | 17:14 |
| *** thorst_ has quit IRC | 17:14 | |
| adreznec | FYI efried I'm stacking a brand new AIO | 17:16 |
| adreznec | With the latest from upstream | 17:16 |
| adreznec | I'll let you know if I hit the same issue as you | 17:16 |
| efried | Thanks. | 17:16 |
| adreznec | I'm going to try and develop a new validated working aio local.conf | 17:17 |
| *** thorst_ has joined #openstack-powervm | 17:25 | |
| *** esberglu has joined #openstack-powervm | 17:27 | |
| *** thorst_ has quit IRC | 17:29 | |
| *** catintheroof has quit IRC | 17:30 | |
| *** thorst_ has joined #openstack-powervm | 17:33 | |
| *** esberglu has quit IRC | 18:01 | |
| efried | adreznec: pip uninstall & reinstall of PasteDeploy did the trick. | 18:02 |
| efried | for that error, anyway. | 18:02 |
| adreznec | Man | 18:03 |
| adreznec | That package | 18:03 |
| adreznec | I feel like I should just suggest that as the first fix for any problem at this point | 18:03 |
| *** esberglu has joined #openstack-powervm | 18:03 | |
| adreznec | Whether it seems related or not | 18:03 |
| efried | thorst_, still looking for you to address replies to PS12 comments on https://review.openstack.org/#/c/343423/ | 18:17 |
| thorst_ | efried: yeah, just juggling many patches here at the mid cycle | 18:17 |
| thorst_ | lots of doc stuff | 18:17 |
| efried | adreznec: now this: | 18:17 |
| efried | 2016-08-11 18:16:03.496 | +lib/neutron_plugins/services/l3:_neutron_configure_router_v6:362 sudo ip -6 addr replace 2001:db8::2/64 dev br-ex | 18:17 |
| efried | 2016-08-11 18:16:03.502 | RTNETLINK answers: Permission denied | 18:17 |
| efried | I thought I had turned off l3. | 18:19 |
| efried | or maybe that's the problem? | 18:19 |
| esberglu | efried: Did you see that you now have to disable q-l3 and neutron-l3? | 18:21 |
| efried | Explicitly put 'em in the disable list? | 18:22 |
| efried | no, will try. Thanks. | 18:22 |
| adreznec | Yeah... maybe that will work | 18:22 |
| adreznec | I'm restacking again now | 18:22 |
| adreznec | Turns out stacking all-in-one with 2gigs of memory doesn't work | 18:22 |
| adreznec | Who'd have guessed | 18:22 |
| *** catintheroof has joined #openstack-powervm | 18:34 | |
| *** esberglu has quit IRC | 18:35 | |
| *** esberglu has joined #openstack-powervm | 18:36 | |
| *** esberglu has quit IRC | 18:40 | |
| *** esberglu has joined #openstack-powervm | 18:43 | |
| *** apearson has joined #openstack-powervm | 18:50 | |
| *** tblakes has quit IRC | 18:54 | |
| efried | VIOLOG: Error log file was not opened when trying to attach shared memory | 19:09 |
| efried | VIOLOG: Error with attaching shared memory: errno=30 Read-only file system | 19:09 |
| efried | VIOLOG: Error handle is either invalid or corrupt | 19:09 |
| efried | VIOLOG: Error handle is either invalid or corrupt | 19:09 |
| efried | Does this mean someone yanked my VIOS disks out from under me? | 19:09 |
| efried | adreznec thorst_ | 19:10 |
| adreznec | efried: possibly... seroyer would probably know for sure | 19:11 |
| efried | Out of "possibly", "probably", and "for sure", I'm going to grasp at that last one like a dying man. seroyer? | 19:12 |
| seroyer | That’s the NovaLink partition? Sorry, I have no context. | 19:12 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm: WIP: VIF driver implementation for SR-IOV https://review.openstack.org/343419 | 19:13 |
| openstackgerrit | Eric Fried proposed openstack/networking-powervm: WIP: Mechanism driver & agent for powervm SR-IOV https://review.openstack.org/343423 | 19:14 |
| adreznec | seroyer: I thought it was efried's VIOS given the question | 19:15 |
| adreznec | Maybe I misunderstood | 19:15 |
| efried | seroyer adreznec: Attempting to run e.g. pvmctl vios list on a victim, which was working yesterday, now gives the above | 19:15 |
| efried | as part of 500 from REST. | 19:16 |
| seroyer | My IRC client is going nuts. Need to restart. Be back in a minute. | 19:16 |
| *** seroyer has left #openstack-powervm | 19:16 | |
| *** seroyer has joined #openstack-powervm | 19:17 | |
| seroyer | efried, try logging into your VIOS and check the error log. | 19:19 |
| efried | INIT: failed write of utmp entry: | 19:20 |
| efried | this is happening on both VIOSes. | 19:20 |
| efried | lots of messages to the console. | 19:20 |
| efried | Can't log in. | 19:20 |
| seroyer | On log in to the VIOS? That’s a pretty good indicator that writes aren’t working. Don’t prove that the disk was yanked (could be full). But yanked is the most likely cause. | 19:21 |
| seroyer | Doesn’t * | 19:21 |
| *** tblakes has joined #openstack-powervm | 19:25 | |
| efried | Especially if both VIOSes. | 19:27 |
| efried | svenkat: can you check the backing devs for the VIOSes off of .149? | 19:27 |
| svenkat | sure… one moment | 19:27 |
| efried | esberglu: Are you a wizard at setting up SSPs at this point? | 19:28 |
| *** catintheroof has quit IRC | 19:28 | |
| esberglu | efried: thorst has been creating them. I’ve just added and removed nodes | 19:29 |
| svenkat | efried: are you asking the physical ports of sriov on this host? i just did a pvmctl sriov list | 19:29 |
| efried | svenkat, no, the boot disks backing the VIOSes. I have reason to believe they've been yanked out from the SAN. | 19:29 |
| svenkat | oh | 19:29 |
| svenkat | i can ping vios but not able to ssh into it | 19:31 |
| adreznec | Hmm | 19:31 |
| adreznec | not a good sign | 19:31 |
| adreznec | Can you grab the console to it from the novalink? | 19:32 |
| svenkat | yes.. trying | 19:32 |
| svenkat | when i do pvmctl vios list, it complains about vios with id 3 | 19:33 |
| adreznec | Same error as efried's? | 19:33 |
| svenkat | Error occurred while querying for Vios Exchange Capability from VIOS with ID 3 in System xxxx | 19:33 |
| svenkat | let me see if vios with lpar id 2 is alive | 19:33 |
| *** apearson has quit IRC | 19:34 | |
| *** esberglu_ has joined #openstack-powervm | 19:34 | |
| svenkat | mkvterm —id 2 still working… | 19:34 |
| svenkat | now it says INIT: failed to write of utmp entry: | 19:34 |
| *** esberglu has quit IRC | 19:35 | |
| svenkat | let me as Doug about san backend for this system | 19:36 |
| *** apearson has joined #openstack-powervm | 19:37 | |
| *** esberglu_ has quit IRC | 19:38 | |
| *** esberglu has joined #openstack-powervm | 19:38 | |
| efried | thorst_, what mechanism are you using to create SSPs? pvmctl? Directly on the VIOSes? | 19:40 |
| thorst_ | efried: I have been creating ssp's on the VIOS | 19:42 |
| thorst_ | cluster -create commands | 19:42 |
| efried | k | 19:42 |
| thorst_ | svenkat: try rebooting the VIOS? | 19:42 |
| thorst_ | we hit that on a few other system | 19:42 |
| svenkat | ok.. | 19:42 |
| thorst_ | *systems | 19:42 |
| thorst_ | we just rebooted them. Something hiccup'd on the backing SAN | 19:43 |
| thorst_ | was a huge PITA...but of course this was a dev env | 19:43 |
| *** esberglu has quit IRC | 19:43 | |
| svenkat | restarting vios with id 2 | 19:44 |
| svenkat | got error messages | 19:45 |
| thorst_ | svenkat: keep in mind all the pain you'll feel | 19:45 |
| thorst_ | but I suspect you're already feeling it | 19:45 |
| thorst_ | (I/O backing system is going offline) | 19:45 |
| thorst_ | so any VMs - including novalink - will need to be rebooted then | 19:45 |
| adreznec | efried: This probably goes for you as well if you haven't already... | 19:45 |
| svenkat | Error occurred while querying for Vios Exchange Capability from VIOS with ID 3 in System | 19:46 |
| svenkat | and bunch of other errors | 19:46 |
| efried | fun times. | 19:46 |
| *** esberglu has joined #openstack-powervm | 19:46 | |
| *** apearson has quit IRC | 19:47 | |
| svenkat | pvmctl lpar list is showing ok but this does not contain vios list as expected | 19:48 |
| svenkat | shall I reboot novalink partition? | 19:52 |
| efried | fine by me | 19:52 |
| svenkat | i can kick it off in ssh session to it? | 19:52 |
| efried | I would expect so. | 19:53 |
| svenkat | pvmctl lpar restart --object-id id=1 | 19:53 |
| svenkat | is this fine? | 19:53 |
| efried | Or maybe | 19:54 |
| efried | sudo reboot now | 19:54 |
| efried | ? | 19:54 |
| *** apearson has joined #openstack-powervm | 19:55 | |
| *** apearson has quit IRC | 19:56 | |
| svenkat | which one, i can try sudo reboot now? | 19:56 |
| svenkat | Doug says there are no activities that he is aware of for this problem | 19:57 |
| *** apearson has joined #openstack-powervm | 19:59 | |
| *** catintheroof has joined #openstack-powervm | 20:00 | |
| svenkat | i will try sudo reboot now | 20:03 |
| svenkat | waiting for it to come up | 20:04 |
| svenkat | it is pinging now. let me ssh into it | 20:05 |
| svenkat | pvmctl vios list says : Failed to find any VirtualIOServer entries. | 20:06 |
| *** esberglu_ has joined #openstack-powervm | 20:18 | |
| *** esberglu has quit IRC | 20:19 | |
| openstackgerrit | Matt Rabe proposed openstack/nova-powervm: Move LPAR scrub tasks from build_map to Create task https://review.openstack.org/354217 | 20:19 |
| thorst_ | svenkat: what about now? RMC can take a while to boot... | 20:21 |
| *** esberglu_ has quit IRC | 20:22 | |
| svenkat | thorst_: checking | 20:24 |
| svenkat | pvmctl vios list still shows error.. waiting | 20:25 |
| svenkat | i can ping vios now | 20:26 |
| svenkat | both vioses are pingable, but pvmctl vios list shows error… i will wait | 20:29 |
| *** apearson has quit IRC | 20:30 | |
| *** apearson has joined #openstack-powervm | 20:30 | |
| svenkat | i am not able to ssh into vios and from novalink pvmctl vios list throws ‘Failed to find any VirtualIOServer entries.’ | 20:34 |
| svenkat | ssh to vios says : ssh_exchange_identification: Connection closed by remote host | 20:34 |
| thorst_ | svenkat: can you check with the vios team? Or debug with vterm or something? Not sure we can help in this forum | 20:45 |
| svenkat | ok… | 20:45 |
| *** esberglu has joined #openstack-powervm | 20:46 | |
| openstackgerrit | Matt Rabe proposed openstack/nova-powervm: Move LPAR scrub tasks from build_map to Create task https://review.openstack.org/354217 | 20:52 |
| *** esberglu_ has joined #openstack-powervm | 20:52 | |
| *** esberglu has quit IRC | 20:53 | |
| *** esberglu_ is now known as esberglu | 20:53 | |
| efried | svenkat thorst_: where does the network name 'default' come from? Is this an appropriate default for the community code? | 20:59 |
| *** esberglu has quit IRC | 20:59 | |
| svenkat | it is in ml2.conf file as a physical network name. | 20:59 |
| *** tblakes has quit IRC | 21:00 | |
| *** esberglu has joined #openstack-powervm | 21:00 | |
| svenkat | so when you create a neutron network, if you do not specify provider:physical_network, it defaults to default | 21:00 |
| thorst_ | efried: I'm not sure off hand...I just remember somewhere that if it wasn't set...that was the 'default' | 21:00 |
| efried | shrug, okay. | 21:00 |
| efried | Anyone know what a numa_node is? | 21:01 |
| *** esberglu has quit IRC | 21:07 | |
| seroyer | Non-uniform memory access. :) | 21:08 |
| *** esberglu has joined #openstack-powervm | 21:08 | |
| *** burgerk has quit IRC | 21:08 | |
| *** tblakes has joined #openstack-powervm | 21:11 | |
| *** esberglu_ has joined #openstack-powervm | 21:12 | |
| *** esberglu has quit IRC | 21:12 | |
| *** esberglu_ is now known as esberglu | 21:12 | |
| *** dwayne has quit IRC | 21:16 | |
| *** svenkat has quit IRC | 21:19 | |
| *** esberglu_ has joined #openstack-powervm | 21:30 | |
| *** esberglu has quit IRC | 21:31 | |
| *** esberglu_ is now known as esberglu | 21:31 | |
| *** esberglu_ has joined #openstack-powervm | 21:35 | |
| *** esberglu has quit IRC | 21:35 | |
| *** esberglu_ has quit IRC | 21:39 | |
| *** jwcroppe_ has quit IRC | 21:46 | |
| *** jwcroppe has joined #openstack-powervm | 21:47 | |
| *** esberglu has joined #openstack-powervm | 21:49 | |
| *** jwcroppe_ has joined #openstack-powervm | 21:49 | |
| *** jwcroppe has quit IRC | 21:51 | |
| *** kriskend has quit IRC | 21:52 | |
| *** jwcroppe_ has quit IRC | 21:54 | |
| *** apearson has quit IRC | 22:02 | |
| *** edmondsw has quit IRC | 22:03 | |
| *** svenkat has joined #openstack-powervm | 22:05 | |
| *** esberglu has quit IRC | 22:06 | |
| *** tblakes has quit IRC | 22:06 | |
| *** svenkat has quit IRC | 22:12 | |
| *** thorst_ has quit IRC | 22:17 | |
| *** esberglu has joined #openstack-powervm | 22:17 | |
| *** mdrabe has quit IRC | 22:19 | |
| *** thorst_ has joined #openstack-powervm | 22:20 | |
| *** thorst_ has quit IRC | 22:25 | |
| *** kylek3h has quit IRC | 22:33 | |
| *** catintheroof has quit IRC | 22:35 | |
| *** seroyer has quit IRC | 22:36 | |
| *** esberglu has quit IRC | 22:41 | |
| *** jwcroppe has joined #openstack-powervm | 22:50 | |
| *** tjakobs has quit IRC | 22:52 | |
| *** esberglu has joined #openstack-powervm | 22:52 | |
| *** jwcroppe has quit IRC | 22:59 | |
| *** tjakobs has joined #openstack-powervm | 23:19 | |
| *** tjakobs has quit IRC | 23:24 | |
| *** seroyer has joined #openstack-powervm | 23:34 | |
| *** esberglu has quit IRC | 23:43 | |
| *** jwcroppe has joined #openstack-powervm | 23:56 | |
| *** esberglu has joined #openstack-powervm | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!