| *** thorst has joined #openstack-powervm | 01:59 | |
| *** thorst has quit IRC | 02:00 | |
| *** thorst has joined #openstack-powervm | 02:31 | |
| *** thorst has quit IRC | 02:48 | |
| *** thorst has joined #openstack-powervm | 03:45 | |
| *** thorst has quit IRC | 04:04 | |
| *** thorst has joined #openstack-powervm | 06:02 | |
| *** thorst has quit IRC | 06:21 | |
| *** thorst has joined #openstack-powervm | 07:18 | |
| *** thorst has quit IRC | 07:23 | |
| *** thorst has joined #openstack-powervm | 09:20 | |
| *** thorst has quit IRC | 09:24 | |
| *** thorst has joined #openstack-powervm | 10:53 | |
| *** thorst has quit IRC | 10:57 | |
| *** smatzek has joined #openstack-powervm | 11:09 | |
| *** thorst has joined #openstack-powervm | 11:55 | |
| *** kylek3h has quit IRC | 11:56 | |
| *** edmondsw has joined #openstack-powervm | 12:14 | |
| *** kylek3h has joined #openstack-powervm | 12:30 | |
| openstackgerrit | Merged openstack/nova-powervm master: Check for presence of volume data before doing cleanup https://review.openstack.org/456206 | 12:32 |
|---|---|---|
| *** mdrabe has joined #openstack-powervm | 12:35 | |
| efried | thorst https://review.openstack.org/#/c/457231/ Was this really worth a bug and a backport? | 12:50 |
| efried | Did I misunderstand the fix? Just avoiding a log message if there's no volume data? | 12:51 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: Get flavor from instance https://review.openstack.org/456741 | 12:53 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: Get flavor from instance https://review.openstack.org/456741 | 12:54 |
| *** jpasqualetto has joined #openstack-powervm | 12:55 | |
| thorst | efried: I think it was something Manas got a defect on...I didn't really see a reason not to port back | 13:02 |
| thorst | I'd rather he not carry patches | 13:02 |
| efried | shrug | 13:02 |
| *** jpasqualetto has quit IRC | 13:25 | |
| *** jpasqualetto has joined #openstack-powervm | 13:33 | |
| *** smatzek has quit IRC | 13:36 | |
| *** smatzek has joined #openstack-powervm | 13:57 | |
| *** esberglu has joined #openstack-powervm | 14:02 | |
| *** apearson has joined #openstack-powervm | 14:17 | |
| *** tjakobs has joined #openstack-powervm | 14:24 | |
| *** burgerk has joined #openstack-powervm | 14:41 | |
| *** dwayne has quit IRC | 14:53 | |
| openstackgerrit | Eric Larese proposed openstack/networking-powervm master: [WIP] Exclude non-SEA vifs from list_vifs https://review.openstack.org/456966 | 14:54 |
| efried | esberglu Rebased up to console in-tree. | 15:08 |
| efried | edmondsw Added you to reviews of bottom half dozen or so in-tree change sets. | 15:09 |
| efried | It's about as gentle an introduction to the community driver as yer gonna get ;-) | 15:09 |
| edmondsw | efried :) tx | 15:09 |
| efried | Easiest to review 'em in order. If you load up any one of 'em, look on the RHS of the gerrit window and go bottom up. | 15:10 |
| esberglu | efried: Okay. I'm gonna let that last one run silently just as a sanity check. As soon as it looks okay, prod will go live | 15:11 |
| esberglu | And then we can go through and recheck the in-tree patches | 15:12 |
| efried | esberglu Yup, sounds like the right plan. | 15:12 |
| *** apearson has quit IRC | 15:28 | |
| *** apearson has joined #openstack-powervm | 15:30 | |
| esberglu | efried: thorst: adreznec: CI is publishing | 15:49 |
| efried | esberglu Good deal. | 15:49 |
| thorst | nice! | 15:49 |
| esberglu | efried: Want me to kick off rechecks for in-tree patches? | 15:50 |
| adreznec | neat | 15:50 |
| efried | esberglu Sure. | 15:50 |
| *** dwayne has joined #openstack-powervm | 16:18 | |
| esberglu | efried: Want to talk CI session config quick? | 16:19 |
| *** apearson has quit IRC | 16:33 | |
| *** apearson has joined #openstack-powervm | 16:35 | |
| *** k0da has joined #openstack-powervm | 17:10 | |
| openstackgerrit | Eric Larese proposed openstack/networking-powervm master: [WIP] Exclude non-SEA vifs from list_vifs https://review.openstack.org/456966 | 17:16 |
| openstackgerrit | Eric Larese proposed openstack/networking-powervm master: Exclude non-SEA vifs from list_vifs https://review.openstack.org/456966 | 18:05 |
| *** kjw3 has joined #openstack-powervm | 18:18 | |
| *** kjw3 has quit IRC | 19:08 | |
| efried | esberglu Sorry, missed that alert. CI session config. Yeah. | 19:11 |
| esberglu | efried: I ended up just putting another patch up if you haven't seen that yet | 19:11 |
| efried | stand by | 19:11 |
| efried | Yeah, my view doesn't go bold because I introduced the change set. And I check my email like twice a day ;-) | 19:12 |
| esberglu | Essentially the same thing as before, just doing the version comparison within prep_devstack and getting rid of the python package comparison | 19:13 |
| efried | esberglu Does dpkg version compare work if pip installed? | 19:13 |
| esberglu | Yep | 19:13 |
| efried | cool beans. | 19:14 |
| esberglu | I haven't tested any of this on CI yet. About to deploy staging with if you don't have further comments | 19:15 |
| efried | esberglu Where was the $pypowervm_branch variable getting set before?? | 19:16 |
| efried | That line - `git checkout $pypowervm_branch` - was a no-op before :) | 19:16 |
| efried | oh, I see it now, dah. | 19:17 |
| efried | esberglu You don't still want to use --keep-redundant-commits ? | 19:19 |
| efried | If you don't, then we may end up having to maintain separate patch lists for each release. | 19:19 |
| esberglu | efried: I meant to add that. Will update | 19:20 |
| efried | cool | 19:20 |
| efried | esberglu The CI is looking really good. | 19:20 |
| efried | It seems to be completing really fast - 33 minutes or so. Is that... normal? | 19:21 |
| esberglu | We doubled the memory for the instances | 19:21 |
| *** kjw3 has joined #openstack-powervm | 19:22 | |
| efried | Wow, okay. I'm a little surprised that'd do it. | 19:22 |
| efried | But cool. | 19:22 |
| efried | freakin xenserver is still awful. | 19:23 |
| efried | But at least it takes 2h to run. | 19:23 |
| esberglu | Yeah I'm not sure how much of a performance increase we would expect from upping mem. But there's nothing else that has happened recently that should impact run time | 19:25 |
| efried | I dig it. | 19:25 |
| efried | Gift horse | 19:25 |
| efried | mouth | 19:25 |
| efried | That whole thing. | 19:25 |
| efried | thorst adreznec esberglu On jay's system, seeing a lot of this: | 19:38 |
| efried | @mock.patch('nova_powervm.virt.powervm.disk.driver.IterableToFileAdapter') | 19:38 |
| efried | whoops, not that. | 19:39 |
| efried | 2017-04-17 15:38:40.539 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: timed out from (pid=126938) _raise_timeout /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:1054 | 19:39 |
| efried | 2017-04-17 15:38:40.539 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" from (pid=126938) _raise_timeout /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabb | 19:39 |
| efried | That. | 19:39 |
| efried | Every second or two. | 19:39 |
| efried | One assumes that'll cause all kinds of problems, including inability to communicate between n-cpu and our pvm-sea agent. | 19:39 |
| esberglu | Is rabbitmq running? | 19:39 |
| efried | according to service status, yeah. | 19:40 |
| esberglu | Hmm. I think the only time I have seen that before when rabbitmq was down | 19:40 |
| efried | mebbe I try restarting it. | 19:41 |
| esberglu | Worth a try | 19:41 |
| efried | It's also possible I'm not used to amqp logging at debug level. | 19:42 |
| efried | Like, perhaps that timeout is normal? | 19:42 |
| efried | I wouldn't have thought so, but... | 19:42 |
| esberglu | Is jay having other issues related to this? Or you just happened to notice the logging | 19:44 |
| efried | Well, he's having issues where vif plug is failing. | 19:44 |
| efried | He's got debug = True in his nova.conf, which is blowing up the logs. I'm going to turn that off and try a deploy and see if I notice anything... unusual. | 19:45 |
| efried | thorst adreznec I think jay's issue is that the pvm-sea agent plug isn't getting the message back to nova anymore. | 19:56 |
| efried | n-cpu just sits there waiting for it. And eventually times out. | 19:56 |
| efried | I'm seeing the provision happen on the agent. | 19:57 |
| thorst | efried: so I didn't see the provision actually go through on Jay's system. | 19:58 |
| thorst | I saw the request come into networking-powervm | 19:58 |
| thorst | but not an actual up being sent back | 19:58 |
| thorst | (if I remember right) | 19:58 |
| efried | thorst https://pastebin.com/KHrtPwCT | 19:58 |
| thorst | efried: I stand corrected | 19:59 |
| thorst | so if you look in the neutron server... | 19:59 |
| efried | Well, it's doing it in this case. It's a new spawn. | 19:59 |
| thorst | neutron.notifiers.nova | 19:59 |
| thorst | that is the thing that is supposed to take that active message | 20:00 |
| thorst | and then determine whether or not to send to nova | 20:00 |
| thorst | the logging is done in neutron server though | 20:00 |
| efried | okay, looking... | 20:00 |
| efried | The timestamp would be that last one, where it says sending to neutron, right? | 20:01 |
| thorst | around then - yeah | 20:02 |
| efried | thorst https://pastebin.com/RbKkantM | 20:02 |
| efried | That looks nice to me. | 20:02 |
| efried | But what do I know? | 20:02 |
| efried | There is an error in the q-svc log about 13s earlier | 20:02 |
| efried | q-svc is also claiming to have three agents running. Is that right? | 20:03 |
| thorst | that seems right. | 20:03 |
| efried | our SEA & SR-IOV agents, and... what else? | 20:03 |
| thorst | maybe L3? | 20:03 |
| thorst | oo....maybe the third agent is what is blocking it | 20:03 |
| efried | Would I see that in screen? | 20:04 |
| thorst | the message you'd want to look for is.... "Sending events:" | 20:04 |
| thorst | probably, what are all the agents running there? | 20:04 |
| efried | 0$ shell 1$(L) dstat 2$(L) key 3$(L) key-access 4$(L) g-reg 5$(L) g-api 6$(L) n-api 7$(L) q-svc* 8$(L) q-dhcp 9$(L) n-cond 10$(L) n-crt 11$(L) n-sch 12$(L) n-novnc 13$(L) n-cauth 14-$(L) n-cpu 15$(L) placement-api 16$(L) horizon 17$(L) pvm-q-sea-agt 18$(L) pvm-q-sriov-agt | 20:04 |
| thorst | q-dhcp is the third | 20:05 |
| thorst | I don't think that one cares... | 20:05 |
| thorst | because that gets used when the VM boots | 20:05 |
| efried | okay. | 20:05 |
| thorst | (and isn't using config drive) | 20:05 |
| efried | In q-svc init, I only see two mech drivers listed. | 20:05 |
| thorst | yep | 20:05 |
| efried | and the 'qos' extension driver | 20:05 |
| thorst | all seems correct | 20:05 |
| efried | and 'port_security' | 20:05 |
| thorst | welcome to neutron | 20:05 |
| thorst | :-) | 20:05 |
| efried | thorst Okay, so next debug step? | 20:06 |
| thorst | In that second paste you sent me, I don't see a "Sending events" debug message | 20:06 |
| thorst | so we probably need to pdb around the neutron.notifiers.nova code | 20:07 |
| thorst | maybe pdb in there a bit... | 20:07 |
| efried | I don't see "Sending events" anywhere in q-svc log. | 20:07 |
| thorst | yep...probably time to root around with pdb then | 20:12 |
| thorst | that's the file I'd start with | 20:12 |
| efried | thorst Remind me - import pdb; pdb.set_trace() ? | 20:23 |
| thorst | yes sir | 20:23 |
| *** smatzek has quit IRC | 20:28 | |
| *** thorst has quit IRC | 21:02 | |
| *** smatzek has joined #openstack-powervm | 21:06 | |
| *** edmondsw has quit IRC | 21:10 | |
| *** dwayne has quit IRC | 21:19 | |
| *** burgerk has quit IRC | 21:30 | |
| *** smatzek has quit IRC | 21:31 | |
| *** esberglu has quit IRC | 21:34 | |
| *** esberglu has joined #openstack-powervm | 21:35 | |
| *** esberglu has quit IRC | 21:39 | |
| *** kylek3h has quit IRC | 21:45 | |
| *** esberglu has joined #openstack-powervm | 21:48 | |
| *** esberglu has quit IRC | 21:52 | |
| *** mdrabe has quit IRC | 21:56 | |
| *** apearson has quit IRC | 22:18 | |
| *** kjw3 has quit IRC | 22:30 | |
| *** tjakobs has quit IRC | 22:41 | |
| *** jpasqualetto has quit IRC | 22:42 | |
| *** kjw3 has joined #openstack-powervm | 22:46 | |
| *** edmondsw has joined #openstack-powervm | 22:49 | |
| *** edmondsw has quit IRC | 22:53 | |
| *** kjw3 has quit IRC | 22:55 | |
| *** kjw3 has joined #openstack-powervm | 23:03 | |
| *** k0da has quit IRC | 23:10 | |
| *** kjw3 has quit IRC | 23:17 | |
| *** thorst has joined #openstack-powervm | 23:37 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!