| *** dmsimard_away is now known as dmsimard | 00:40 | |
| *** dmsimard is now known as dmsimard_away | 00:55 | |
| *** dmsimard_away is now known as dmsimard | 01:19 | |
| *** emagana has joined #openstack-operators | 01:21 | |
| *** emagana has quit IRC | 01:25 | |
| *** emagana has joined #openstack-operators | 01:26 | |
| *** dmsimard is now known as dmsimard_away | 01:28 | |
| *** emagana has quit IRC | 01:30 | |
| *** jaypipes has joined #openstack-operators | 03:25 | |
| *** simon-AS559 has joined #openstack-operators | 05:06 | |
| *** simon-AS5591 has joined #openstack-operators | 05:08 | |
| *** simon-AS559 has quit IRC | 05:11 | |
| *** simon-AS5591 has quit IRC | 05:22 | |
| *** simon-AS559 has joined #openstack-operators | 05:49 | |
| *** simon-AS5591 has joined #openstack-operators | 05:51 | |
| *** simon-AS559 has quit IRC | 05:54 | |
| *** simon-AS5591 has quit IRC | 06:01 | |
| *** simon-AS559 has joined #openstack-operators | 06:49 | |
| *** judf has joined #openstack-operators | 07:24 | |
| *** simon-AS559 has quit IRC | 09:17 | |
| *** simon-AS559 has joined #openstack-operators | 09:29 | |
| *** simon-AS5591 has joined #openstack-operators | 09:31 | |
| *** simon-AS559 has quit IRC | 09:34 | |
| *** simon-AS559 has joined #openstack-operators | 09:49 | |
| *** simon-AS5591 has quit IRC | 09:51 | |
| *** simon-AS5592 has joined #openstack-operators | 09:51 | |
| *** simon-AS559 has quit IRC | 09:53 | |
| *** dmsimard_away is now known as dmsimard | 10:41 | |
| *** simon-AS5592 has quit IRC | 10:42 | |
| *** simon-AS559 has joined #openstack-operators | 10:44 | |
| *** simon-AS5591 has joined #openstack-operators | 10:46 | |
| *** simon-AS559 has quit IRC | 10:49 | |
| *** pboros has joined #openstack-operators | 12:46 | |
| *** pboros has quit IRC | 12:46 | |
| *** pboros has joined #openstack-operators | 12:48 | |
| *** radez_g0n3 is now known as radez | 13:19 | |
| *** mattgriffin has joined #openstack-operators | 14:39 | |
| *** emagana has joined #openstack-operators | 14:42 | |
| *** klindgren has joined #openstack-operators | 14:52 | |
| *** emagana has quit IRC | 15:00 | |
| *** emagana has joined #openstack-operators | 15:00 | |
| *** emagana has quit IRC | 15:03 | |
| *** emagana has joined #openstack-operators | 15:03 | |
| *** cjellick has joined #openstack-operators | 15:21 | |
| *** mdorman has joined #openstack-operators | 15:51 | |
| *** cjellick has quit IRC | 15:55 | |
| *** cjellick has joined #openstack-operators | 15:55 | |
| *** emagana has quit IRC | 16:01 | |
| *** emagana has joined #openstack-operators | 16:02 | |
| *** emagana has quit IRC | 16:06 | |
| *** emagana has joined #openstack-operators | 16:11 | |
| *** reed has joined #openstack-operators | 16:23 | |
| *** emagana has quit IRC | 16:42 | |
| *** emagana has joined #openstack-operators | 16:43 | |
| *** simon-AS5591 has quit IRC | 17:01 | |
| *** harlowja_away is now known as harlowja | 17:21 | |
| *** SimonChung has joined #openstack-operators | 17:27 | |
| *** cjellick has quit IRC | 17:30 | |
| *** cjellick has joined #openstack-operators | 17:38 | |
| *** salv-orlando has joined #openstack-operators | 18:02 | |
| *** SimonChung1 has joined #openstack-operators | 18:03 | |
| *** SimonChung has quit IRC | 18:03 | |
| *** harlowja has quit IRC | 18:05 | |
| *** harlowja_ has joined #openstack-operators | 18:05 | |
| *** SimonChung has joined #openstack-operators | 18:06 | |
| *** SimonChung1 has quit IRC | 18:06 | |
| *** cjellick has quit IRC | 18:30 | |
| *** emagana has quit IRC | 18:40 | |
| *** emagana has joined #openstack-operators | 18:41 | |
| *** simon-AS559 has joined #openstack-operators | 18:41 | |
| *** simon-AS5591 has joined #openstack-operators | 18:42 | |
| *** emagana has quit IRC | 18:42 | |
| *** emagana has joined #openstack-operators | 18:43 | |
| *** emagana has quit IRC | 18:44 | |
| *** sdcbeeb has joined #openstack-operators | 18:44 | |
| *** simon-AS559 has quit IRC | 18:46 | |
| *** simon-AS5591 has quit IRC | 18:47 | |
| *** openstack has joined #openstack-operators | 18:55 | |
| -sendak.freenode.net- [freenode-info] if you're at a conference and other people are having trouble connecting, please mention it to staff: http://freenode.net/faq.shtml#gettinghelp | 18:55 | |
| *** mdorman has quit IRC | 19:00 | |
| xavpaice | sdcbeeb: sounds like you might have different versions of kvm on the hosts - did you run an update or some such? | 19:32 |
|---|---|---|
| *** mdorman has joined #openstack-operators | 19:40 | |
| sdcbeeb | No version updates. What makes it look like kvm version differences? | 19:53 |
| xavpaice | I saw a very similar message when we updated the compute nodes a while ago, that's all. The boxes had the 'kvm' package installed, but it wasn't needed, more the qemu-kvm packages which replaced it (Precise, Havana). | 19:55 |
| xavpaice | it looks like it's just a warning though, and shouldn't be an issue for the monitor connection. Certainly it didn't cause breakages for us. | 19:56 |
| xavpaice | perhaps run with debug on and get a few more logs? | 19:56 |
| *** simon-AS559 has joined #openstack-operators | 19:57 | |
| sdcbeeb | I see. I just checked the md5sum's on the /usr/bin/qemu-system-x86_64 files on all nodes and they match. That is another oddity of that message saying to "please use qemu-system-x86_64" since that is what is being used. I don't know why it thinks that kvm is being used. | 19:58 |
| xavpaice | oh, you're using qemu not kvm? | 19:59 |
| *** simon-AS5591 has joined #openstack-operators | 19:59 | |
| xavpaice | I guess you've put libvirt_type in nova.con? | 19:59 |
| xavpaice | s/con/conf/ | 20:00 |
| sdcbeeb | xavpaice: the binary is called qemu-system-x86-64, however that is kvm right? | 20:00 |
| sdcbeeb | yes: libvirt_type=kvm | 20:01 |
| sdcbeeb | Should that be libvirt_type=qemu-system-x86_64 instead? | 20:01 |
| *** simon-AS559 has quit IRC | 20:02 | |
| xavpaice | if you want to use kvm for the hypervisor, then kvm is what you want. We set qemu for dev only, it's not hardware virt so is really slow. | 20:02 |
| xavpaice | what distro is this? | 20:03 |
| sdcbeeb | This is Ubuntu 12.04.3 LTS | 20:04 |
| xavpaice | do you have the qemu-kvm package on all the compute hosts? | 20:04 |
| sdcbeeb | I didn't set this system up. I'm trying to get my head around it after our cloud admin left. | 20:04 |
| * xavpaice sympathy | 20:05 | |
| sdcbeeb | thanks. I'm checking on the qemu-kvm package. I'm not a big Ubuntu user either so it takes a while. | 20:06 |
| sdcbeeb | yes qemu-kvm is installed | 20:07 |
| sdcbeeb | So you think emulation is being used instead of hardware virtualization? | 20:08 |
| xavpaice | no, I think you need a bunch more logs. You can set debug=true in the nova.conf and restart nova-compute - that will enable more logs | 20:10 |
| sdcbeeb | Yeah, | 20:10 |
| xavpaice | also check network comms etc between the nodes | 20:10 |
| xavpaice | maybe look at what might have changed since it was working, verify that nothing has interfered | 20:10 |
| sdcbeeb | Actually I turned debug off first thing because it was so noisy. I'll turn it back on to check on this. | 20:11 |
| xavpaice | possibly look at netstat for a listening port from libvirtd also - could be something realy simple (hopes) | 20:11 |
| sdcbeeb | It (live-migration) has been working and then it just stopped. I thought maybe it thought the node was full or something but it hasn't reached the over-subscription level yet (8:1) and that matches the memory of the machines. | 20:13 |
| sdcbeeb | That is, we have 8GB per physical core. | 20:13 |
| xavpaice | it's not always such a clear cut calculation tbh - but usually there's something in the logs about it if that's the issue | 20:14 |
| * xavpaice has to run, hope debug gives you some more clues :) | 20:17 | |
| sdcbeeb | I'll check networking. The first thing I see is dropped RX packets and overruns on the admin nodes. Also on some of the tap interfaces have dropped packets. | 20:18 |
| sdcbeeb | Thanks a lot for your help xavpaice | 20:18 |
| sdcbeeb | I don't see much more with debugging on, other than all of the DEBUG listings of the rabbitmq messages for the live-migration (and other) events. The one thing that does show though is the AUDIT messages. The node that I want to migrate to shows: VCPUS: -30. It doesn't seem to be calculating the cpu_allocation_ratio=8.0 in /etc/nova/nova.conf. | 21:11 |
| *** radez is now known as radez_g0n3 | 21:14 | |
| sdcbeeb | These nodes have 16 cores and with 8:1 ratio we should be able to get to 128 VCPU's. "nova-manage service describe_resource test3" shows that 46 VCPU's are used on node test3 but it also says (total) 16 and 16 - 46 = -30. It seems that (total) should be 128 instead of 16. Anyone know why this is like this? | 21:15 |
| *** mattgriffin has quit IRC | 21:40 | |
| *** alop has joined #openstack-operators | 21:45 | |
| *** sdcbeeb has quit IRC | 21:47 | |
| *** harlowja_ has quit IRC | 21:49 | |
| *** harlowja has joined #openstack-operators | 21:49 | |
| *** simon-AS5591 has quit IRC | 21:59 | |
| *** mdorman has quit IRC | 22:15 | |
| *** jaypipes has quit IRC | 22:41 | |
| *** dmsimard is now known as dmsimard_away | 22:55 | |
| *** reed has quit IRC | 22:56 | |
| *** reed has joined #openstack-operators | 22:57 | |
| *** dmsimard_away is now known as dmsimard | 22:58 | |
| *** dmsimard is now known as dmsimard_away | 22:59 | |
| notmyname | anyone around who runs Swift on not-Linux? | 23:40 |
| *** cjellick has joined #openstack-operators | 23:47 | |
| *** cjellick has quit IRC | 23:47 | |
| *** cjellick has joined #openstack-operators | 23:48 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!