| *** esberglu has joined #openstack-powervm | 00:04 | |
| *** esberglu has quit IRC | 00:10 | |
| *** esberglu has joined #openstack-powervm | 00:26 | |
| *** esberglu has quit IRC | 00:27 | |
| *** thorst has joined #openstack-powervm | 00:28 | |
| *** esberglu has joined #openstack-powervm | 00:37 | |
| *** esberglu has quit IRC | 00:49 | |
| *** esberglu has joined #openstack-powervm | 00:55 | |
| *** esberglu has quit IRC | 01:30 | |
| *** esberglu has joined #openstack-powervm | 01:34 | |
| *** esberglu has quit IRC | 01:39 | |
| *** esberglu has joined #openstack-powervm | 01:48 | |
| *** esberglu has quit IRC | 01:49 | |
| *** thorst has quit IRC | 01:49 | |
| *** thorst has joined #openstack-powervm | 01:50 | |
| *** esberglu has joined #openstack-powervm | 01:51 | |
| *** seroyer has quit IRC | 01:53 | |
| *** esberglu has quit IRC | 01:55 | |
| *** thorst has quit IRC | 01:58 | |
| *** tblakes has joined #openstack-powervm | 02:03 | |
| *** tblakes has quit IRC | 02:04 | |
| *** esberglu has joined #openstack-powervm | 02:06 | |
| *** esberglu has quit IRC | 02:06 | |
| *** esberglu has joined #openstack-powervm | 02:27 | |
| *** esberglu has quit IRC | 02:28 | |
| *** esberglu has joined #openstack-powervm | 02:34 | |
| *** esberglu has quit IRC | 02:37 | |
| *** esberglu has joined #openstack-powervm | 02:38 | |
| *** thorst has joined #openstack-powervm | 02:40 | |
| *** thorst has quit IRC | 02:42 | |
| *** thorst has joined #openstack-powervm | 02:42 | |
| *** apearson has joined #openstack-powervm | 02:47 | |
| *** thorst has quit IRC | 02:51 | |
| *** esberglu has quit IRC | 02:54 | |
| *** esberglu has joined #openstack-powervm | 03:16 | |
| *** esberglu has quit IRC | 03:19 | |
| *** thorst has joined #openstack-powervm | 03:49 | |
| *** thorst has quit IRC | 03:56 | |
| *** tjakobs has joined #openstack-powervm | 04:09 | |
| *** tjakobs has quit IRC | 04:28 | |
| *** esberglu has joined #openstack-powervm | 04:44 | |
| *** esberglu has quit IRC | 04:48 | |
| *** thorst has joined #openstack-powervm | 04:54 | |
| *** thorst has quit IRC | 05:01 | |
| *** thorst has joined #openstack-powervm | 06:00 | |
| *** thorst has quit IRC | 06:06 | |
| *** thorst has joined #openstack-powervm | 07:04 | |
| *** thorst has quit IRC | 07:11 | |
| *** k0da has joined #openstack-powervm | 07:17 | |
| *** esberglu has joined #openstack-powervm | 07:49 | |
| *** esberglu has quit IRC | 07:53 | |
| *** thorst has joined #openstack-powervm | 08:11 | |
| *** thorst has quit IRC | 08:16 | |
| *** esberglu has joined #openstack-powervm | 08:26 | |
| *** esberglu has quit IRC | 08:31 | |
| *** thorst has joined #openstack-powervm | 09:14 | |
| *** ChanServ has quit IRC | 09:14 | |
| *** adreznec has quit IRC | 09:17 | |
| *** apearson has quit IRC | 09:17 | |
| *** adi___ has quit IRC | 09:17 | |
| *** miltonm has quit IRC | 09:17 | |
| *** dwayne has quit IRC | 09:17 | |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** toan has quit IRC | 09:17 | |
| *** AlexeyAbashkin has quit IRC | 09:17 | |
| *** efried has quit IRC | 09:17 | |
| *** AndyWojo has quit IRC | 09:17 | |
| *** antonym has quit IRC | 09:17 | |
| *** tonyb has quit IRC | 09:17 | |
| *** k0da has quit IRC | 09:20 | |
| *** k0da has joined #openstack-powervm | 09:20 | |
| *** thorst has quit IRC | 09:20 | |
| *** k0da has quit IRC | 09:39 | |
| *** apearson has joined #openstack-powervm | 10:06 | |
| *** antonym has joined #openstack-powervm | 10:07 | |
| *** tonyb has joined #openstack-powervm | 10:07 | |
| *** efried has joined #openstack-powervm | 10:09 | |
| *** AndyWojo has joined #openstack-powervm | 10:09 | |
| *** adreznec has joined #openstack-powervm | 10:10 | |
| *** adreznec has quit IRC | 10:10 | |
| *** efried has quit IRC | 10:10 | |
| *** AndyWojo has quit IRC | 10:10 | |
| *** antonym has quit IRC | 10:10 | |
| *** tonyb has quit IRC | 10:10 | |
| *** apearson has quit IRC | 10:10 | |
| *** esberglu has joined #openstack-powervm | 10:12 | |
| *** esberglu has quit IRC | 10:17 | |
| *** antonym has joined #openstack-powervm | 10:26 | |
| *** tonyb has joined #openstack-powervm | 10:26 | |
| *** adreznec has joined #openstack-powervm | 10:27 | |
| *** apearson has joined #openstack-powervm | 10:27 | |
| *** efried has joined #openstack-powervm | 10:27 | |
| *** AndyWojo has joined #openstack-powervm | 10:27 | |
| *** adi___ has joined #openstack-powervm | 10:28 | |
| *** miltonm has joined #openstack-powervm | 10:28 | |
| *** dwayne has joined #openstack-powervm | 10:28 | |
| *** openstackgerrit has joined #openstack-powervm | 10:28 | |
| *** toan has joined #openstack-powervm | 10:28 | |
| *** AlexeyAbashkin has joined #openstack-powervm | 10:28 | |
| *** ChanServ has joined #openstack-powervm | 10:30 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 10:30 | |
| *** smatzek has joined #openstack-powervm | 11:02 | |
| *** thorst has joined #openstack-powervm | 11:22 | |
| *** seroyer has joined #openstack-powervm | 11:31 | |
| *** thorst has quit IRC | 11:31 | |
| *** thorst has joined #openstack-powervm | 11:52 | |
| *** smatzek_ has joined #openstack-powervm | 11:54 | |
| *** smatzek has quit IRC | 11:57 | |
| *** kylek3h has quit IRC | 12:01 | |
| *** esberglu has joined #openstack-powervm | 12:01 | |
| *** esberglu has quit IRC | 12:05 | |
| *** apearson has quit IRC | 12:12 | |
| *** edmondsw has joined #openstack-powervm | 12:13 | |
| *** svenkat has joined #openstack-powervm | 12:15 | |
| *** esberglu has joined #openstack-powervm | 12:26 | |
| *** seroyer has quit IRC | 12:28 | |
| *** kylek3h has joined #openstack-powervm | 12:35 | |
| *** seroyer has joined #openstack-powervm | 12:40 | |
| *** kylek3h has quit IRC | 12:45 | |
| efried | thorst, you cherry-pickin' this morning? | 12:49 |
|---|---|---|
| *** tblakes has joined #openstack-powervm | 12:56 | |
| thorst | efried: I will | 12:57 |
| efried | I'll be waiting with my axe, George Washington style. | 12:58 |
| *** seroyer has quit IRC | 12:58 | |
| *** apearson has joined #openstack-powervm | 12:59 | |
| *** burgerk has joined #openstack-powervm | 13:00 | |
| *** seroyer has joined #openstack-powervm | 13:07 | |
| *** mdrabe has joined #openstack-powervm | 13:12 | |
| *** smatzek_ has quit IRC | 13:33 | |
| *** k0da has joined #openstack-powervm | 13:33 | |
| *** burgerk has quit IRC | 13:49 | |
| thorst | efried: https://review.openstack.org/#/c/385995/ | 13:51 |
| *** smatzek_ has joined #openstack-powervm | 13:57 | |
| *** tblakes has quit IRC | 14:03 | |
| *** viclarson has joined #openstack-powervm | 14:04 | |
| viclarson | hi | 14:04 |
| viclarson | it works | 14:04 |
| viclarson | joke | 14:04 |
| viclarson | we have some problem | 14:04 |
| thorst | heh - what's up | 14:04 |
| viclarson | aix vm provisioned by openstack | 14:05 |
| thorst | wooo! | 14:05 |
| viclarson | cant see attached volumes | 14:05 |
| viclarson | until reboot | 14:05 |
| viclarson | cfgmgr doesn't help | 14:05 |
| viclarson | until first reboot | 14:05 |
| thorst | viclarson: are you using vSCSI I assume? | 14:05 |
| viclarson | y | 14:05 |
| viclarson | after first reboot | 14:06 |
| thorst | and the volumes were attached post spawn or on original boot? | 14:06 |
| viclarson | doesn't matter | 14:06 |
| thorst | k | 14:06 |
| thorst | SSP for nova ephemeral tho? | 14:06 |
| viclarson | hm | 14:06 |
| viclarson | ssp | 14:07 |
| viclarson | post spawn | 14:07 |
| viclarson | sry | 14:07 |
| viclarson | after start vm | 14:07 |
| thorst | can you try on VM boot? | 14:07 |
| viclarson | doesn't matter after or before cloudinit | 14:07 |
| viclarson | on vm boot? | 14:08 |
| viclarson | tried | 14:08 |
| thorst | k | 14:08 |
| viclarson | 1) after rmc connected | 14:08 |
| viclarson | 2) after cloud-init finished | 14:08 |
| viclarson | before rmc connected were troubles but workarounded | 14:09 |
| thorst | yeah...well both of those would need to be done. | 14:09 |
| viclarson | i like my english | 14:09 |
| thorst | RMC should connect via the MGMTSWITCH, should be seamless. | 14:09 |
| viclarson | if rmc dont connected volumes cant be attached | 14:09 |
| viclarson | but they are attached | 14:10 |
| thorst | right, I'm curious why RMC wasn't seamless for you...but that's a different discussion | 14:10 |
| thorst | I think we'd have to ask someone from the AIX team to look at that. | 14:10 |
| thorst | apearson: do you know anyone that could help? | 14:10 |
| *** kriskend has joined #openstack-powervm | 14:12 | |
| viclarson | i saw wrong ct_node_id can affect volume connection | 14:13 |
| viclarson | but it seems all attached correctly | 14:13 |
| viclarson | and simple reboot repairs it | 14:13 |
| viclarson | mb reinit of rmc d be done before network configuration? | 14:14 |
| thorst | viclarson: rmc is just a communication/mgmt channel...it shouldn't really affect boot process | 14:16 |
| thorst | I think we need an AIX guru here...just trying to locate one | 14:16 |
| *** openstackgerrit has quit IRC | 14:18 | |
| viclarson | i see this postponed command /usr/sbin/chdev -l hdisk0 -a hcheck_interval=60 -P but i think this cant affect scsi rescan too | 14:18 |
| *** openstackgerrit has joined #openstack-powervm | 14:18 | |
| *** tjakobs has joined #openstack-powervm | 14:24 | |
| *** tblakes has joined #openstack-powervm | 14:25 | |
| *** tblakes_ has joined #openstack-powervm | 14:26 | |
| *** tblakes has quit IRC | 14:29 | |
| *** tblakes_ is now known as tblakes | 14:29 | |
| *** burgerk has joined #openstack-powervm | 14:31 | |
| *** seroyer has quit IRC | 14:34 | |
| efried | thorst, did you find an AIX person yet? | 14:36 |
| efried | Start with Veena or Uma. | 14:37 |
| efried | viclarson, is the chdev command hanging? If it's holding an ODM lock, it could certainly hold up cfgmgr. But then cfgmgr oughtta hang. | 14:37 |
| viclarson | no | 14:39 |
| viclarson | efried, no | 14:40 |
| viclarson | efried, no hangs | 14:40 |
| efried | viclarson, okay. So let me make sure I'm understanding: you've got a volume that's attached to your AIX VM, but it's not showing up inside the VM? | 14:41 |
| viclarson | yes | 14:42 |
| thorst | unless you reboot | 14:42 |
| thorst | indicating its wired correctly. | 14:42 |
| efried | Oh, it appears on reboot? Yeah, that's odd. Even without RMC, cfgmgr oughtta discover it. | 14:42 |
| efried | thorst, viclarson: what kind of volume is it? | 14:44 |
| thorst | vscsi FC | 14:44 |
| efried | viclarson, what's the oslevel of the AIX LPAR? | 14:47 |
| viclarson | efried, 7.2.0.0 | 14:48 |
| efried | thanks | 14:48 |
| *** esberglu has quit IRC | 14:49 | |
| *** esberglu has joined #openstack-powervm | 14:49 | |
| *** seroyer has joined #openstack-powervm | 14:49 | |
| thorst | adreznec: do you know if the Xen CI team posts their nova.conf? | 14:51 |
| *** esberglu has quit IRC | 14:51 | |
| *** esberglu has joined #openstack-powervm | 14:51 | |
| adreznec | thorst: http://logs.openstack.xenproject.org/42/384742/2/check/dsvm-tempest-xen/4098d64/logs/etc/nova/nova.conf.txt.gz | 14:52 |
| thorst | cool | 14:52 |
| thorst | looks like they don't use glance v1....which is good | 14:53 |
| adreznec | Also http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/42/384742/2/check/dsvm-tempest-neutron-network/f57b2c9/logs/etc/nova/nova.conf.txt.gz | 14:53 |
| adreznec | For the other Xen CI | 14:53 |
| adreznec | First one was libvirt Xen, second is XenAPI Xen | 14:54 |
| *** esberglu has quit IRC | 14:56 | |
| adreznec | thorst: esberglu I actually just realized that we don't publish our conf files (nova.conf, neutron.conf, etc) for our CI runs | 14:56 |
| adreznec | We should probably consider doing that as a backlog item | 14:56 |
| thorst | adreznec: yeah...maybe something qingwu could add? | 14:56 |
| *** kylek3h has joined #openstack-powervm | 14:56 | |
| adreznec | Mhm | 14:58 |
| efried | thorst, viclarson: It sounds like sans RMC, the attach could succeed, but if the DLPAR stuff doesn't happen, the reboot would be required. | 14:59 |
| efried | I didn't quite follow the discussion above - was RMC alive when the attach was performed? | 14:59 |
| viclarson | yes | 14:59 |
| viclarson | but rmc was reconfiged since vm boot | 15:00 |
| efried | Before the attach? | 15:01 |
| viclarson | in one case before | 15:01 |
| viclarson | in second after | 15:01 |
| viclarson | the same result | 15:01 |
| efried | okay. | 15:01 |
| *** esberglu has joined #openstack-powervm | 15:03 | |
| viclarson | efried, DLPAR stuff? what d happen? | 15:03 |
| efried | I'm really not very well versed on that aspect. apearson, yt? | 15:05 |
| apearson | viclarson / efried - Why RMC isn't coming up is a mystery. After you do the deploy, can you run 'ctsnap' on the AIX lpar and the same command on the NovaLink LPAR and drop the logs somewhere where I can take a look? | 15:06 |
| efried | Sounded to me like RMC was okay | 15:07 |
| viclarson | rmc comed up | 15:07 |
| *** mdrabe has quit IRC | 15:08 | |
| efried | apearson ^^ So if RMC was okay, DLPAR stuff should've happened, right? | 15:08 |
| viclarson | i just see volume attached to lpar from vios side, but dont see from lpar | 15:09 |
| viclarson | until reboot | 15:09 |
| efried | If RMC glitched in the middle, but the LPAR was still active, would we fail the attach, or would we proceed and just skip the DLPARy stuff? | 15:09 |
| apearson | viclarson - if the volume attach was happening AFTER the VM was booted, a cfgmgr will be needed on the VM to re-discover it. | 15:09 |
| *** smatzek_ is now known as smatzek | 15:09 | |
| efried | apearson - right - cfgmgr didn't discover it - only reboot did. | 15:10 |
| efried | That's the mystery. | 15:10 |
| viclarson | yes, but cfg cant re-discover until reboot | 15:10 |
| apearson | but looks like you said cfgmgr didn't do anything...you absolutely positive? | 15:10 |
| efried | Sounds like this is reproducible too. viclarson, happens every time, or intermittent? | 15:10 |
| viclarson | every time | 15:11 |
| efried | viclarson, have you tried the same thing on other types of LPARs? | 15:11 |
| viclarson | other types? | 15:12 |
| viclarson | other oses? | 15:12 |
| apearson | efried - checking with Pafumi to see if they've ever seen that sort of behavior before on the vscsi | 15:12 |
| viclarson | tried with ubuntu | 15:12 |
| viclarson | i think it worked | 15:12 |
| efried | apearson, cool. I talked to Uma, and am chatting with Ruzek as well. | 15:13 |
| efried | viclarson: If you can reproduce, then immediately run | 15:13 |
| efried | alog -otcfg > /some/file | 15:13 |
| efried | then send us the file. | 15:13 |
| apearson | viclarson - waht kind of disk is the volume? And did you apply 'vios rules' when you installed the VIOS? | 15:13 |
| efried | (that command runs on the LPAR) | 15:13 |
| apearson | 'rules -o apply' I believe... | 15:13 |
| viclarson | i did't install vios | 15:14 |
| viclarson | it came to me already installed | 15:14 |
| apearson | And what kind of disk is the volume? | 15:15 |
| viclarson | vscsi FC | 15:15 |
| viclarson | is it answer? | 15:15 |
| *** mdrabe has joined #openstack-powervm | 15:16 | |
| apearson | Yup - ok, so the issue is likely with max transfer sizes conflicting on the disk that you add. So that means as the VIOS discoveres a new disk it may get a different transfer size than the disk you're already have assigned to the lpar...and if they differ, it gets masked off. | 15:17 |
| apearson | So - apply the VIOS rules...this will make it so that all new disks will get the same transfer size, and this problem should go away. | 15:18 |
| viclarson | -f rulesFile | 15:21 |
| viclarson | what rules file? | 15:21 |
| viclarson | https://dl.dropboxusercontent.com/u/13321033/1.txt | 15:26 |
| viclarson | logs | 15:26 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm: Change devstack pypowervm branch to develop https://review.openstack.org/386044 | 15:26 |
| openstackgerrit | Eric Fried proposed openstack/networking-powervm: Change devstack pypowervm branch to develop https://review.openstack.org/386047 | 15:27 |
| openstackgerrit | Eric Fried proposed openstack/ceilometer-powervm: Change devstack pypowervm branch to develop https://review.openstack.org/386048 | 15:28 |
| efried | apearson - see viclarson question above | 15:31 |
| thorst | efried: lol...what a miss. | 15:33 |
| *** k0da has quit IRC | 15:37 | |
| apearson | viclarson - skip the -f...use the default one. | 15:39 |
| apearson | so just 'rules -o deploy' | 15:39 |
| viclarson | A manual post-operation is required for the changes to take effect, please reboot the system. | 15:40 |
| viclarson | are you sure? | 15:40 |
| openstackgerrit | Merged openstack/nova-powervm: Change devstack pypowervm branch to develop https://review.openstack.org/386044 | 15:40 |
| openstackgerrit | Merged openstack/ceilometer-powervm: Change devstack pypowervm branch to develop https://review.openstack.org/386048 | 15:41 |
| viclarson | https://dl.dropboxusercontent.com/u/13321033/4.txt | 15:42 |
| viclarson | mb this is logs | 15:42 |
| viclarson | how to safety reboot it? | 15:42 |
| viclarson | is it possible? | 15:42 |
| viclarson | man who can recover it on vacation now | 15:43 |
| openstackgerrit | Merged openstack/networking-powervm: Change devstack pypowervm branch to develop https://review.openstack.org/386047 | 15:45 |
| *** mdrabe has quit IRC | 15:45 | |
| *** mdrabe has joined #openstack-powervm | 15:45 | |
| apearson | so is this a single VIOS? If so, power off all your other lpars first...if it's dual, you should be able to just shutdown -r. Make sure you have console access... | 15:46 |
| apearson | viclarson ^^ | 15:46 |
| viclarson | apearson, two vioses | 15:54 |
| apearson | ok - in that case rebooting one of them should be fine as long as I/O is all redundant... | 15:55 |
| viclarson | apearson, after reboot of vms all attached devices was detected. vios dont mask devices if vm rebooted? | 15:57 |
| viclarson | is it that problem? | 15:57 |
| thorst | efried: mind re-reviewing 4046? | 16:21 |
| thorst | we've got a team waiting on it | 16:21 |
| efried | thorst, ack. | 16:30 |
| efried | thorst, looks like you need to fix sonar, tho | 16:31 |
| *** esberglu has quit IRC | 16:33 | |
| thorst | bleh | 16:33 |
| *** esberglu has joined #openstack-powervm | 16:33 | |
| thorst | sonar: "I don't like the color of this character" | 16:33 |
| thorst | me: what? | 16:33 |
| efried | thorst, you can ignore the unused args - that rule is nonfatal. | 16:34 |
| efried | But you need to get rid of the unused class. | 16:34 |
| thorst | yeah, it found something this one time | 16:34 |
| thorst | amazing | 16:34 |
| thorst | next rev is up | 16:35 |
| efried | +2 | 16:35 |
| *** esberglu has quit IRC | 16:37 | |
| thorst | efried: thx | 16:41 |
| *** esberglu has joined #openstack-powervm | 17:05 | |
| *** esberglu has quit IRC | 17:08 | |
| *** esberglu has joined #openstack-powervm | 17:08 | |
| *** esberglu has quit IRC | 17:13 | |
| *** esberglu has joined #openstack-powervm | 17:21 | |
| apearson | viclarson - on reboot, the client is able to re-negotiate the transfer size to match the lowest listed... | 17:25 |
| *** viclarson has quit IRC | 17:29 | |
| *** k0da has joined #openstack-powervm | 17:51 | |
| openstackgerrit | Drew Thorstensen (thorst) proposed openstack/nova-powervm: Separate out glance read logic https://review.openstack.org/386113 | 17:53 |
| thorst | kriskend: ^^ | 17:54 |
| thorst | I would ask you take a look at that... | 17:54 |
| thorst | I think it'll make debug of the streaming issue easier. | 17:55 |
| *** tjakobs has quit IRC | 18:06 | |
| *** tjakobs has joined #openstack-powervm | 18:11 | |
| kriskend | going to load the patch now | 18:12 |
| kriskend | waiting for Steve to finish setting up l2pop and then I will try again | 18:21 |
| *** tblakes has quit IRC | 18:38 | |
| *** tblakes has joined #openstack-powervm | 18:44 | |
| kriskend | thorst: So much output... | 18:55 |
| thorst | really? | 18:55 |
| thorst | o, you have debug on | 18:55 |
| thorst | aren't the chunks like 16 meg tho? | 18:55 |
| kriskend | we restacked... shouldn't that have blown away my glance debug setting in nova.conf | 18:56 |
| kriskend | watching the screen might give me a seizure | 18:57 |
| thorst | ooo, I want to see | 18:57 |
| thorst | PM me the compute node | 18:57 |
| *** apearson has quit IRC | 19:00 | |
| thorst | well.... kriskend the chunks are significantly smaller than I expected... | 19:02 |
| thorst | I thought 16M, not 65K | 19:02 |
| kriskend | lots of chunks | 19:02 |
| thorst | funky chunks | 19:03 |
| kriskend | it is spewing chunks | 19:03 |
| *** apearson has joined #openstack-powervm | 19:03 | |
| thorst | :-) | 19:04 |
| kriskend | so sad that irc does not have giphy | 19:05 |
| thorst | or a blessing | 19:05 |
| kriskend | yeah..probably best | 19:05 |
| kriskend | so started the spawns 15 minutes ago... none are to the point of booting yet | 19:07 |
| thorst | well, that much debug is bound to slow it way down | 19:07 |
| thorst | but it looked to me like there were 6 reads going on in the logs? | 19:07 |
| thorst | some that started at 13:51, and another round of three at 14:01? | 19:08 |
| kriskend | Error: Failed to perform requested operation on instance "KAx-4", the instance has an error status: Please try again later [Error: ('Connection broken: IncompleteRead(7395 bytes read)', IncompleteRead(7395 bytes read))]. | 19:11 |
| thorst | yeah...that's new.... | 19:12 |
| thorst | all at the same time | 19:12 |
| kriskend | the 3rd batch of 3 all started and failed around the same time | 19:12 |
| kriskend | and the very last one never starts | 19:12 |
| thorst | yeah...but aren't you glad we have all those chunks in there? | 19:13 |
| thorst | now we know where it failed | 19:13 |
| kriskend | that is true | 19:13 |
| kriskend | i am just happy to be making any progress on this issue | 19:13 |
| seroyer | Was watching iostat at the time. All three made it to 1600 bytes written. | 19:13 |
| seroyer | Which is different from before, when there were no bytes written. | 19:14 |
| seroyer | thorst: The ones that failed? It actually detected the failures this time and cleaned up the VMs. The socat processes are gone, but the LVs are still there. There is a socat for the last one, but no writes to it. | 19:18 |
| kriskend | so my vote is that this is better than before :-) | 19:22 |
| seroyer | I see that it did a couple of reads: Read chunk 1 from glance image. Saw up to chunk 3. Nothing after that. | 19:24 |
| thorst | I saw up to chunk 24 on one | 19:26 |
| thorst | seroyer: can we do another deploy? do they get hung? | 19:27 |
| kriskend | without cleaning up the existing? | 19:28 |
| thorst | yeah? | 19:28 |
| thorst | cause we want to see if it hangs...right? | 19:28 |
| thorst | see if we're past the deadlock? | 19:28 |
| kriskend | kicked one off | 19:29 |
| thorst | seems like we're past the deadlock? | 19:29 |
| kriskend | looks like it is going.. | 19:29 |
| thorst | huzzah? | 19:29 |
| kriskend | so it is something in just continuing on with the existing set of deploys...weird | 19:30 |
| thorst | not sure. I kinda wonder if the two closes I had there screwed things up | 19:30 |
| thorst | so I'm going to push 4046 through | 19:31 |
| thorst | k? | 19:31 |
| thorst | then work on this GlanceReader side of things | 19:31 |
| seroyer | FYI: I saw 4 IncompleteRead error stacks in n-cpu.log. Need to correlate them to VMs… | 19:32 |
| seroyer | I think that the last read from glance for the 10th VM happened at almost the exact same time as 3 of the IncompleteRead errors: Wonder if n-cpu lost contact with glance? Network hiccup? | 19:34 |
| thorst | seroyer: yeah...that's exactly what I was thinking | 19:36 |
| thorst | these all happen right at the same time | 19:36 |
| thorst | but you're on a private network.... | 19:36 |
| kriskend | is there any extra debug we can turn on on the controller? | 19:37 |
| thorst | if your switch was something modern....I could hop on and look | 19:37 |
| kriskend | haha | 19:37 |
| seroyer | 2016-10-13 14:10:37.752 fail /opt/pvm-rest/data/fileupload/673bd467-1296-4194-9180-97e356783c5d | 19:37 |
| seroyer | 2016-10-13 14:10:38.927 read chunk 3 for VM 10 | 19:37 |
| adreznec | Still waiting for that replacement switch thorst | 19:37 |
| seroyer | 2016-10-13 14:10:38.929 fail /opt/pvm-rest/data/fileupload/b0d3cea3-5c83-4454-bf99-fc71cbf0d588 | 19:37 |
| adreznec | :P | 19:37 |
| seroyer | 2016-10-13 14:10:38.933 fail /opt/pvm-rest/data/fileupload/2fa70990-ddee-4eb9-a353-b32170934eee | 19:37 |
| seroyer | 2016-10-13 14:10:38.936 fail /opt/pvm-rest/data/fileupload/18d04270-6cd4-4517-aeb8-7d7d22ac3e2d | 19:37 |
| thorst | I installed iftop and ifstat on your compute nodes | 19:37 |
| thorst | you had some odd fluctuations of traffic during the downloads | 19:38 |
| kriskend | ok so what is the next step? | 19:54 |
| kriskend | thorst: did you find anything in the the controller logs? | 19:55 |
| thorst | kriskend: not yet | 19:55 |
| kriskend | Ok trying the new code out on another system that multiple deploys normally work fine on | 19:56 |
| *** burgerk has quit IRC | 19:58 | |
| thorst | efried: had a q on 4212 | 20:11 |
| efried | thorst, yeah, motivation was actually so clbush could use it in pvmctl. | 20:11 |
| efried | Hence public. | 20:11 |
| thorst | ahh | 20:11 |
| thorst | looked kinda like a silly method | 20:11 |
| efried | Yeah, it was frustrating us in pvmctl because clbush was doing a bunch of logic that .marshal was already doing - but not *all* of it. | 20:12 |
| thorst | that method seems really weird | 20:12 |
| thorst | very dependent on what 'val' is into it | 20:12 |
| efried | So he either had to keep his duplication, or use .marshal and then reverse engineer the result to get it *back* to the input list if it had been sanitized, kind of thing. | 20:13 |
| thorst | and in some cases it checks upper cased versus other checks doesn't look into casing | 20:13 |
| efried | Yeah, that's the whole point of the method - if 'val' is one of the consts (or a list containing just one of those), it sanitizes the const and returns it. Otherwise, returns just the original list. | 20:13 |
| efried | In other words, "take something the user input as a list or string, and turn it into a value you can stuff into the .bld method blindly" | 20:14 |
| thorst | so if they type in 'all' you'll convert it to ALL | 20:14 |
| efried | Oh, actually, no, you're right | 20:15 |
| thorst | #confused | 20:15 |
| efried | If they send in 'all', we fail. But if they use ['all'] we return ALL. That's bad. Will fix. | 20:15 |
| thorst | but if the pass in ['all', 'none'], you return ['all', 'none'] | 20:16 |
| thorst | which is weird as well. | 20:16 |
| thorst | maybe doc it as well a little more | 20:16 |
| efried | Yeah, that'll get sanitized later by .bld in this case, but I was tempted to run the sanitizer in this method. I think that's appropriate. | 20:16 |
| efried | Although 'all' will still sanitize as a valid MAC address. We don't check *too* thoroughly. | 20:16 |
| thorst | you and your weird methods...now back to writing my funky_chunky_iter | 20:17 |
| efried | ooo, looking forward to reviewing that one. | 20:18 |
| *** openstackgerrit has quit IRC | 20:19 | |
| *** openstackgerrit has joined #openstack-powervm | 20:19 | |
| kriskend | thorst: what log file would you expect to see interesting info for glance in? | 20:29 |
| kriskend | g-api? | 20:29 |
| thorst | kriskend: yeah, that's where I was looking, but there wasn't anything interesting in there. | 20:30 |
| efried | thorst, can I validate a MAC address as always being ten hex digits, upper or lower, optionally separated by colons every two digits? | 20:30 |
| *** smatzek has quit IRC | 20:30 | |
| kriskend | Is there a way to turn on more interesting glance logs on the controller? | 20:31 |
| efried | kriskend: in the conf (whichever one the glance CLI is using), you can set debug = True. That'll get you DEBUG for the whole world. Or you can use default_log_levels (see conf template comment for examples) | 20:33 |
| kriskend | hmm glance api conf already has debug = True | 20:38 |
| kriskend | i guess glance just doesn't log very interesting things | 20:41 |
| openstackgerrit | Drew Thorstensen (thorst) proposed openstack/nova-powervm: Separate out glance read logic https://review.openstack.org/386113 | 20:41 |
| thorst | kriskend efried: ^^ | 20:41 |
| thorst | though I'm not sure I want that debug statement in that tight loop... | 20:41 |
| kriskend | So with your patch, a VM hung in spawning on the system that multiple deploys use to always work on ... | 20:50 |
| thorst | I don't comprehend. | 20:51 |
| thorst | efried: FYI - nova meeting in ten | 20:51 |
| efried | thorst, okay, remind me again then. | 20:52 |
| efried | In #openstack-meeting? | 20:52 |
| thorst | yep | 20:52 |
| *** smatzek has joined #openstack-powervm | 20:55 | |
| *** edmondsw has quit IRC | 20:58 | |
| thorst | and we're off. adreznec FYI | 21:00 |
| * adreznec hides under table | 21:01 | |
| *** kriskend has quit IRC | 21:14 | |
| openstackgerrit | Taylor Jakobson proposed openstack/nova-powervm: Create snapshot of VM https://review.openstack.org/368157 | 21:32 |
| *** seroyer has quit IRC | 21:34 | |
| *** dwayne has quit IRC | 21:35 | |
| *** esberglu has quit IRC | 21:39 | |
| *** esberglu has joined #openstack-powervm | 21:40 | |
| *** esberglu has quit IRC | 21:41 | |
| *** k0da has quit IRC | 21:43 | |
| *** svenkat has quit IRC | 21:45 | |
| *** mdrabe has quit IRC | 21:45 | |
| *** mdrabe has joined #openstack-powervm | 21:45 | |
| efried | thorst, aw shoot, I missed it. | 21:46 |
| efried | thorst, looks like it went relatively well. Even Still wasn't in total veto mode. | 21:50 |
| thorst | efried: best we could hope for | 21:50 |
| thorst | :-) | 21:50 |
| efried | thorst, made improvements to 4212. I like it better now. | 21:51 |
| efried | aaaand I'm out. Later y'all. | 21:52 |
| *** tjakobs has quit IRC | 22:01 | |
| *** mdrabe has quit IRC | 22:07 | |
| *** thorst has quit IRC | 22:09 | |
| *** mdrabe has joined #openstack-powervm | 22:11 | |
| *** smatzek has quit IRC | 22:19 | |
| *** mdrabe has quit IRC | 22:20 | |
| *** tblakes has quit IRC | 22:21 | |
| *** kylek3h has quit IRC | 22:27 | |
| *** apearson has quit IRC | 22:40 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!