| *** thorst has joined #openstack-powervm | 00:01 | |
| *** thorst has quit IRC | 00:02 | |
| *** edmondsw has joined #openstack-powervm | 00:37 | |
| *** edmondsw has quit IRC | 00:42 | |
| *** thorst has joined #openstack-powervm | 01:03 | |
| *** thorst has quit IRC | 01:08 | |
| *** esberglu has joined #openstack-powervm | 01:20 | |
| *** thorst has joined #openstack-powervm | 01:38 | |
| *** thorst has quit IRC | 01:39 | |
| *** thorst has joined #openstack-powervm | 01:42 | |
| *** esberglu has quit IRC | 01:52 | |
| *** svenkat has quit IRC | 02:03 | |
| *** thorst has quit IRC | 02:20 | |
| *** edmondsw has joined #openstack-powervm | 02:26 | |
| *** edmondsw has quit IRC | 02:30 | |
| *** thorst has joined #openstack-powervm | 03:20 | |
| *** thorst has quit IRC | 03:25 | |
| *** jwcroppe has joined #openstack-powervm | 03:51 | |
| *** edmondsw has joined #openstack-powervm | 04:14 | |
| *** jwcroppe has quit IRC | 04:17 | |
| *** edmondsw has quit IRC | 04:18 | |
| *** thorst has joined #openstack-powervm | 04:22 | |
| *** thorst has quit IRC | 04:26 | |
| *** openstack has quit IRC | 04:42 | |
| *** openstack has joined #openstack-powervm | 04:45 | |
| *** openstackstatus has joined #openstack-powervm | 04:45 | |
| *** ChanServ sets mode: +v openstackstatus | 04:46 | |
| *** kylek3h has joined #openstack-powervm | 04:46 | |
| *** zerick has joined #openstack-powervm | 04:49 | |
| *** adreznec has joined #openstack-powervm | 04:49 | |
| *** thorst has joined #openstack-powervm | 05:22 | |
| *** thorst has quit IRC | 05:27 | |
| *** YuYangWang has joined #openstack-powervm | 05:53 | |
| *** YuYangWang has quit IRC | 05:55 | |
| *** YuYangWang has joined #openstack-powervm | 05:56 | |
| *** edmondsw has joined #openstack-powervm | 06:02 | |
| *** edmondsw has quit IRC | 06:06 | |
| *** jwcroppe has joined #openstack-powervm | 06:18 | |
| *** jwcroppe has quit IRC | 06:23 | |
| *** thorst has joined #openstack-powervm | 06:23 | |
| *** thorst has quit IRC | 06:28 | |
| *** YuYangWang has quit IRC | 06:56 | |
| *** YuYangWang has joined #openstack-powervm | 06:56 | |
| *** YuYangWang has quit IRC | 07:14 | |
| *** YuYangWang has joined #openstack-powervm | 07:14 | |
| *** thorst has joined #openstack-powervm | 07:24 | |
| *** thorst has quit IRC | 07:29 | |
| *** edmondsw has joined #openstack-powervm | 07:50 | |
| *** edmondsw has quit IRC | 07:54 | |
| *** thorst has joined #openstack-powervm | 08:25 | |
| *** thorst has quit IRC | 08:30 | |
| *** thorst has joined #openstack-powervm | 09:26 | |
| *** thorst has quit IRC | 09:30 | |
| *** edmondsw has joined #openstack-powervm | 09:38 | |
| *** edmondsw has quit IRC | 09:42 | |
| *** YuYangWang has quit IRC | 10:23 | |
| *** thorst has joined #openstack-powervm | 10:27 | |
| *** thorst has quit IRC | 10:31 | |
| *** svenkat has joined #openstack-powervm | 11:11 | |
| *** svenkat has quit IRC | 11:13 | |
| *** mawi has joined #openstack-powervm | 11:21 | |
| *** edmondsw has joined #openstack-powervm | 11:26 | |
| *** thorst has joined #openstack-powervm | 11:27 | |
| *** edmondsw has quit IRC | 11:30 | |
| *** thorst has quit IRC | 11:32 | |
| *** thorst has joined #openstack-powervm | 11:45 | |
| *** svenkat has joined #openstack-powervm | 11:47 | |
| *** svenkat_ has joined #openstack-powervm | 11:51 | |
| *** svenkat has quit IRC | 11:52 | |
| *** svenkat_ is now known as svenkat | 11:52 | |
| mawi | hello. I can't instantiate vm on power box (with pvm_sea networking) due to the following error: ERROR nova.compute.manager [instance: 24dc2b4d-d861-4def-9ef9-f1573850c4a5] PortBindingFailed: Binding failed for port 2761eb50-d979-4e03-9902-c1879d82bd3d | 12:12 |
|---|---|---|
| mawi | I would appreciate any suggestions how to further investigate the root cause of it. | 12:12 |
| thorst | mawi - were we chatting earlier (sorry - mapping nic's from other means to IRC) | 12:13 |
| thorst | wondering if you were the one we wanted to get the nova.conf / neutron.conf's from our CI | 12:14 |
| mawi | yes :) | 12:14 |
| thorst | mind waiting ~1 hour for esberglu to pop online? | 12:14 |
| thorst | still early where he's at | 12:15 |
| mawi | not at all, I will wait. Thanks! | 12:15 |
| *** AndyWojo has joined #openstack-powervm | 12:20 | |
| *** edmondsw has joined #openstack-powervm | 12:25 | |
| *** esberglu has joined #openstack-powervm | 13:07 | |
| mawi | Hello esberglu. I am having problem with instantiating vm on power due to some networking configuration problems | 13:08 |
| mawi | I am getting ERROR nova.compute.manager [instance: 24dc2b4d-d861-4def-9ef9-f1573850c4a5] PortBindingFailed: Binding failed for port 2761eb50-d979-4e03-9902-c1879d82bd3d | 13:08 |
| mawi | would you suggest what to start with in order to find the root cause? | 13:09 |
| esberglu | mawi: Could you post the full trace on paste.openstack.org please? | 13:14 |
| mawi | will do | 13:14 |
| mawi | Paste #618681 | 13:19 |
| thorst | esberglu: I was hoping you could point mawi to our nova.conf and neutron.conf's from our CI runs | 13:19 |
| thorst | so that he can compare to his own | 13:19 |
| mawi | the stack trace is from nova-compute.log on NL system | 13:19 |
| mawi | I am not sure if in my configuration the networking-powervm agent is used at all (couldn't see that in log) | 13:20 |
| thorst | I suspect that even though the stack is in nova-compute, the error is on the neutron side | 13:20 |
| thorst | nova is just complaining the neutron didn't bind it properly | 13:20 |
| thorst | usually that means that the neutron server conf doesn't know about the pvm sea type | 13:20 |
| mawi | from the neutron-server.log I see following: | 13:22 |
| mawi | 2017-08-17 10:32:47.984 2788 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['linuxbridge', 'pvm_sea'] | 13:23 |
| mawi | 2017-08-17 10:32:47.988 2788 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['linuxbridge', 'pvm_sea'] | 13:23 |
| mawi | and | 13:23 |
| mawi | 2017-08-17 10:32:48.316 2788 INFO neutron.plugins.ml2.managers [req-edd87be2-3f2b-49ab-98a3-69e94fc40cd8 - - - - -] Initializing mechanism driver 'pvm_sea' | 13:23 |
| thorst | hmm...that is encouraging. | 13:23 |
| thorst | that means you must have installed the networking-powervm as well | 13:23 |
| thorst | the next thing I do is I set the neutron server log to debug | 13:24 |
| thorst | restart... | 13:24 |
| thorst | then try a VM deploy. | 13:24 |
| thorst | in the nova-compute log you'll see when we start to wait for the 'vif binding' | 13:24 |
| thorst | then go the the neutron server log and see what it's saying about a vif binding around that time | 13:24 |
| thorst | usually it has some esoteric error... | 13:24 |
| mawi | ok | 13:25 |
| thorst | mawi: really dumb question. You're using a VLAN network right? | 13:25 |
| thorst | not vxlan, flat or gre? | 13:25 |
| thorst | (networking-powervm only supports vlan) | 13:26 |
| mawi | excuse me this stupid question but are you talkin about following setting in the /etc/neutron/plugins/ml2/ml2_conf.ini file: | 13:29 |
| mawi | type_drivers = flat,vlan | 13:29 |
| mawi | ? | 13:29 |
| thorst | well, networking-powervm only supports vlan | 13:29 |
| thorst | so that may be one bit | 13:29 |
| esberglu | http://paste.openstack.org/show/618683/ | 13:29 |
| esberglu | ^ neutron and nova.conf. A lot of the nova.conf is probably irrelevant but I posted it in full | 13:30 |
| mawi | thank you esberglu, would you mind sending me also your ml2_conf.ini file? | 13:30 |
| esberglu | http://paste.openstack.org/show/618685/ | 13:32 |
| esberglu | mawi ^ | 13:32 |
| mawi | tanks a lot | 13:33 |
| esberglu | np | 13:33 |
| mawi | Am I assuming properly, that on the compute node (NL) PowerVM Shared Ethernet agent will be started | 13:33 |
| thorst | you need to start it there | 13:33 |
| mawi | automatically and I don't have to start it | 13:33 |
| thorst | you start the networking-powervm and nova-compute there. | 13:34 |
| thorst | and optionally ceilometer-powervm | 13:34 |
| mawi | networking-powervm as a service ? | 13:34 |
| thorst | yep | 13:34 |
| thorst | just like OVS neutron agent | 13:34 |
| thorst | they're independent / codependent services :-) | 13:34 |
| mawi | as I downloaded the code from github, I don't have such a service. What is the poper way to start it? | 13:35 |
| thorst | same way you start the nova compute? | 13:36 |
| mawi | but I installed nova-compute as ubuntu package | 13:36 |
| thorst | oooo...weird | 13:36 |
| thorst | surely not the powervm bit? | 13:36 |
| thorst | I don't know of anyone who's done that yet :-) | 13:37 |
| mawi | well hold on | 13:37 |
| mawi | what I did on NL was: | 13:37 |
| mawi | apt install software-properties-common add-apt-repository cloud-archive:ocata apt install nova-compute then I cloned git repo nova-powervm and linked directories from ocata branch to python dist-packages on NL | 13:40 |
| mawi | I alos cloned repo for networking-powervm and linked ocata branch to my python installation. What would be the proper way to install it? | 13:41 |
| mawi | I wasn't able to find ubuntu repositories with networking-powervm or nova-powervm | 13:42 |
| thorst | mawi: so the "proper" way is to use a distribution. Ubuntu hasn't brought in all the powervm packages into their repos for their distro. Though I suppose you could get the OVS one working (assuming its a new enough version) | 13:42 |
| thorst | we are integrated into the OSA distro though | 13:42 |
| thorst | I think that's just for OVS though, I don't think that has SEA yet | 13:43 |
| thorst | adreznec: can you confirm? | 13:43 |
| adreznec | thorst: Yeah, OSA is OVS/LBr only, no SEA | 13:43 |
| adreznec | mawi: thorst when you say you linked the ocata branch for networking-powervm to your python installation, what do you mean exactly? | 13:44 |
| adreznec | Did you pip install it? Something else? | 13:44 |
| mawi | from NL I git cloned the repository do local file system, changed branch to ocata, so I had these files from ocata branch in local filesystem. Finally I did | 13:46 |
| mawi | sudo ln -s /home/padmin/nova-powervm/nova/virt/powervm_ext /usr/lib/python2.7/dist-packages/nova/virt | 13:46 |
| mawi | sudo ln -s /home/padmin/nova-powervm/nova_powervm /usr/lib/python2.7/dist-packages | 13:47 |
| mawi | so the python could see the files among othe packages | 13:47 |
| mawi | what do you mean by OSA and OVS (open vswitch?)? | 13:48 |
| adreznec | OSA = OpenStack-Ansible, a deployment tool for openstack - https://github.com/openstack/openstack-ansible/ | 13:48 |
| adreznec | OVS is Open vSwitch, another network provider supported by PowerVM if you're using NovaLink with SDN | 13:48 |
| adreznec | Rather than using SEA | 13:49 |
| adreznec | Are you actually able to start the Nova compute service with those links in place? | 13:50 |
| mawi | ok. I am aware that there is SDN option too. However it looks lika I am having problems with the basic and simpler SEA option now.. | 13:50 |
| mawi | yes, I have nova-compute up and running and it load the nova-powervm driver | 13:51 |
| adreznec | Hmm interesting, ok. Wasn't sure if it'd work just linking over the nova/virt directory like that since nova usually has other files in there besides the virt drivers, but cool. | 13:52 |
| adreznec | So for networking there are a couple differences from how you did the compute stuff. In that case you're just re-using the service file from the nova-compute package you installed but telling it to load nova-powervm instead | 13:52 |
| adreznec | But for networking-powervm there doesn't seem to be a corresponding "generic" package you can install then point and tell it to use networking-powervm instead | 13:53 |
| mawi | ok. so I am still missing part how would I properly install networking-powervm on ubuntu NL lpar | 13:54 |
| adreznec | You should be able to get things up and running if you were to pip install networking-powervm, then configure your neutron options as per http://networking-powervm.readthedocs.io/en/latest/devref/usage.html#configuration-file-options | 13:55 |
| adreznec | thorst: It's been a while since I've done this, but I believe pip installing networking-powervm lays down a startable agent then | 13:56 |
| adreznec | networking-powervm-sharedethernet-agent or similar | 13:57 |
| mawi | well. I have read the doc under the link you sent many times :). This configuration should be done on the neutron controller system right? No configuration is needed on te NL (compute node) side right? | 13:57 |
| mawi | I also previously did: sudo pip install nova-powervm==5.0.0.0b4 ceilometer-powervm==5.0.0.0b3 networking-powervm==5.0.0.0b3 | 13:58 |
| mawi | and it installed correctly but I couldn't get nova-compute running. | 13:59 |
| adreznec | mawi: Some of that configuration has to be done on the compute/NL side as it could be unique for each node. Which mechanism agent (sea or sriov) or bridge-mappings, for example, | 13:59 |
| mawi | ok. (the doc didn't state this clearly). Do I need to install additional neutron packages? | 14:00 |
| adreznec | thorst or esberglu could probably help you with that configuration better than I could at this point | 14:00 |
| *** apearson has joined #openstack-powervm | 14:00 | |
| adreznec | I think pip installing networking-powervm should pull in the requisite dependencies on the compute node. | 14:01 |
| adreznec | Your control node would obviously be different depending on how you installed that (ubuntu I'm assuming) | 14:01 |
| mawi | one of the issues with pip I had was that without specifying version to be installed explicitly it didn't install the latest one and even after using the latest ones, it led to breaking some package dependecies on ubuntu | 14:02 |
| mawi | I can try that again if you think this is the way to go | 14:03 |
| mawi | yes. the control node is x86 ubuntu 16.04 with ubuntu openstack packages | 14:04 |
| adreznec | mawi: Sorry, had to change locations for a meeting quick. Trying pip would probably be the fastest option. The other option I can think of offhand would be to look at what the neutron-openvswitch-agent package for ubuntu openstack is doing and either replicate that but using the networking-powervm code, or install that and change the configuration to load networking-powervm instead | 14:06 |
| esberglu | efried: edmondsw: http://184.172.12.213/54/465954/23/check/nova-out-of-tree-pvm/e4c4371/powervm_os_ci.html | 14:07 |
| adreznec | Since there's no official package for networking-powervm | 14:07 |
| esberglu | The time.sleep call is timing out? | 14:07 |
| adreznec | For the Ubuntu distro | 14:07 |
| adreznec | Sorry, I have to handle this meeting now, but I'll check back in after. Hopefully if you have questions in the mean time thorst or esberglu can help out | 14:08 |
| mawi | thanks for you support | 14:08 |
| edmondsw | esberglu I assume that means that tempest timed out and it just so happened to be during the sleep | 14:11 |
| esberglu | edmondsw: If that were the case the timeout would be raised from here though? | 14:13 |
| esberglu | https://github.com/openstack/tempest/blob/master/tempest/common/waiters.py#L127 | 14:13 |
| edmondsw | esberglu yep... I assume it did, and the trace is just misleading | 14:14 |
| edmondsw | time.sleep can't raise a tempest-specific TimeoutException :) | 14:14 |
| edmondsw | esberglu wait... that's not the tempest TimeoutException, it's a fixtures one... | 14:17 |
| esberglu | edmondsw: Weird right? | 14:18 |
| esberglu | I've been looking at this failure wrong for a while because of that | 14:18 |
| *** mawi has quit IRC | 14:19 | |
| edmondsw | esberglu https://github.com/openstack/tempest/blob/master/tempest/test.py#L421 | 14:22 |
| edmondsw | I think that should get you going | 14:23 |
| esberglu | Ah so there is a separate test timeout | 14:24 |
| edmondsw | yup | 14:25 |
| edmondsw | esberglu what do we set for OS_TEST_TIMEOUT ? | 14:25 |
| edmondsw | looks like we could just disable that by setting it to 0 | 14:26 |
| edmondsw | not that we probably want to... | 14:26 |
| esberglu | edmondsw: I don't think we set it, so w/e the default is | 14:26 |
| esberglu | 500 | 14:26 |
| edmondsw | that explains it then, right? | 14:27 |
| esberglu | Yep | 14:27 |
| edmondsw | so I think we need to either bump that or play with TIMEOUT_SCALING_FACTOR for the specific tests where we're seeing timeouts | 14:27 |
| esberglu | edmondsw: I'm thinking we just double the OS_TEST_TIMEOUT for now. We can modify later as needed. Cool? | 14:44 |
| edmondsw | I think that's probably ok, but we'll need to keep an eye on things. These timeout settings are getting high... | 14:46 |
| edmondsw | efried agreed? | 14:46 |
| esberglu | edmondsw: Upping the timeout shouldn't affect anything except these long running tests though | 14:48 |
| esberglu | We can drop the build_timeout since it is currently longer than the OS_TEST_TIMEOUT | 14:49 |
| edmondsw | until/unless something goes wrong, yes | 14:49 |
| edmondsw | go ahead | 14:49 |
| efried | catching up... | 14:51 |
| efried | So we've identified why stuff was timing out quicker than we thought it shoulda? | 14:52 |
| esberglu | efried: Yeah | 14:53 |
| efried | Good. And yes, I'm on board with doubling that timeout and seeing if that helps. | 14:54 |
| esberglu | Is the best way to just set it in /etc/environment? I still don't fully understand what can see env. vars if using export. If you export, do all child processes have access? | 14:55 |
| edmondsw | esberglu I would have asked you that question :) | 14:57 |
| edmondsw | does it log what the vars are? If so, you could try it and see | 14:57 |
| esberglu | I *think* that we could export and it would pass to child processes. I know that /etc/environment will work | 14:58 |
| esberglu | edmondsw: Not that I know of | 14:59 |
| edmondsw | boo | 14:59 |
| esberglu | We use /etc/environment for other env. vars. might as well keep it consistent | 14:59 |
| edmondsw | yu | 14:59 |
| esberglu | Here's the next question | 15:01 |
| esberglu | Is a spawn, rebuild, and termination taking over 500s reasonabe? | 15:01 |
| esberglu | reasonable | 15:01 |
| *** tjakobs has joined #openstack-powervm | 15:01 | |
| efried | esberglu Sorry, missed the discussion. Yes, export gets it to child processes. That's what export is for. Do that instead of /etc/environment. But only if you are sure we can't just put it in tempest.conf, which seems like a much more reasonable place for it. | 15:32 |
| esberglu | There isn't a tempest.conf option for it | 15:42 |
| edmondsw | tempest seems to be a weird hodgepodge of using conf and using env vars | 15:46 |
| esberglu | Yep | 15:47 |
| esberglu | efried: If setting OS_TEST_TIMEOUT in the os_ci_tempest.conf files do I still need export? Or does sourcing make that unnecessary? | 15:48 |
| efried | esberglu I believe you need to export it. | 15:49 |
| efried | esberglu Which you can do from within the .conf btw. | 15:50 |
| efried | Sourcing is like the code was in the same file. | 15:50 |
| *** k0da has joined #openstack-powervm | 16:20 | |
| efried | https://bugs.launchpad.net/nova-powervm/+bug/1711410 <== anyone feel like tackling this beast? | 16:38 |
| openstack | Launchpad bug 1711410 in nova-powervm "get_available_resource/get_inventory should account for resources used by the NovaLink, VIOSes, and non-OS partitions" [Undecided,New] | 16:38 |
| efried | esberglu Are there tempest tests for PCI passthrough, that we currently have disabled? | 16:45 |
| *** gianpero has joined #openstack-powervm | 16:46 | |
| *** gianpero has quit IRC | 16:47 | |
| esberglu | efried: Not seeing any | 17:17 |
| esberglu | edmondsw: Did you see efried's latest comment on the OS_TEST_TIMEOUT change? Thoughts? | 17:20 |
| *** k0da has quit IRC | 17:40 | |
| edmondsw | esberglu replied | 17:57 |
| *** esberglu has quit IRC | 18:13 | |
| *** esberglu has joined #openstack-powervm | 18:14 | |
| *** esberglu has quit IRC | 18:18 | |
| *** esberglu has joined #openstack-powervm | 18:24 | |
| *** k0da has joined #openstack-powervm | 18:50 | |
| *** efried has quit IRC | 19:18 | |
| *** efried has joined #openstack-powervm | 19:24 | |
| *** apearson has quit IRC | 19:31 | |
| *** apearson has joined #openstack-powervm | 19:36 | |
| *** tjakobs has quit IRC | 20:22 | |
| *** tonyb has joined #openstack-powervm | 21:05 | |
| *** svenkat has quit IRC | 21:43 | |
| *** esberglu has quit IRC | 21:47 | |
| *** esberglu has joined #openstack-powervm | 21:54 | |
| *** thorst has quit IRC | 22:00 | |
| *** k0da has quit IRC | 22:09 | |
| *** thorst has joined #openstack-powervm | 22:11 | |
| *** thorst has quit IRC | 22:23 | |
| *** thorst has joined #openstack-powervm | 22:28 | |
| *** apearson has quit IRC | 22:31 | |
| *** edmondsw has quit IRC | 22:36 | |
| *** thorst has quit IRC | 22:51 | |
| *** openstackgerrit has joined #openstack-powervm | 23:00 | |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: Spec: PowerVM Support for PCI Passthrough https://review.openstack.org/494733 | 23:00 |
| *** svenkat has joined #openstack-powervm | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!