| *** macz_ has joined #openstack-nova | 00:00 | |
| openstackgerrit | Merged openstack/nova master: libvirt: Add guest generation for vDPA https://review.opendev.org/c/openstack/nova/+/770532 | 00:03 |
|---|---|---|
| *** macz_ has quit IRC | 00:04 | |
| openstackgerrit | Vlad Gusev proposed openstack/nova stable/stein: Use subqueryload() instead of joinedload() for (system_)metadata https://review.opendev.org/c/openstack/nova/+/761812 | 00:13 |
| *** tosky has quit IRC | 00:35 | |
| brinzhang | bauzas, gibi: hope we can continue to discuss the question with add accel_uuids in periodic task (_poll_shelved_instances) while you are all online, thanks | 00:47 |
| *** ociuhandu has joined #openstack-nova | 00:47 | |
| *** ociuhandu has quit IRC | 01:00 | |
| *** brinzhang_ has joined #openstack-nova | 01:03 | |
| *** dpawlik6 has joined #openstack-nova | 01:04 | |
| *** lemko5 has joined #openstack-nova | 01:04 | |
| *** sapd1 has quit IRC | 01:05 | |
| *** lemko has quit IRC | 01:12 | |
| *** brinzhang has quit IRC | 01:12 | |
| *** dpawlik has quit IRC | 01:12 | |
| *** lemko5 is now known as lemko | 01:13 | |
| *** dpawlik6 is now known as dpawlik | 01:13 | |
| *** jamesdenton has quit IRC | 01:22 | |
| *** jamesdenton has joined #openstack-nova | 01:22 | |
| *** hamalq has quit IRC | 01:23 | |
| *** mlavalle has quit IRC | 01:43 | |
| *** mkrai has joined #openstack-nova | 01:59 | |
| *** rcernin has quit IRC | 02:37 | |
| *** sean-k-mooney has quit IRC | 02:48 | |
| *** sean-k-mooney has joined #openstack-nova | 02:50 | |
| *** kd has joined #openstack-nova | 03:03 | |
| *** k-s-dean has quit IRC | 03:04 | |
| *** k-s-dean has joined #openstack-nova | 03:06 | |
| *** kd has quit IRC | 03:08 | |
| *** k-s-dean has quit IRC | 03:18 | |
| *** rcernin has joined #openstack-nova | 03:19 | |
| *** whoami-rajat_ has joined #openstack-nova | 03:24 | |
| *** rcernin has quit IRC | 03:26 | |
| *** rcernin has joined #openstack-nova | 03:31 | |
| *** psachin has joined #openstack-nova | 03:38 | |
| *** mkrai has quit IRC | 03:43 | |
| *** DinaBelova has quit IRC | 03:49 | |
| *** DinaBelova has joined #openstack-nova | 03:53 | |
| *** mkrai has joined #openstack-nova | 03:55 | |
| *** zzzeek has quit IRC | 04:18 | |
| *** sean-k-mooney has quit IRC | 04:21 | |
| *** zzzeek has joined #openstack-nova | 04:22 | |
| *** ociuhandu has joined #openstack-nova | 04:24 | |
| *** ociuhandu has quit IRC | 04:30 | |
| *** ratailor has joined #openstack-nova | 04:33 | |
| *** vishalmanchanda has joined #openstack-nova | 04:33 | |
| *** dviroel has quit IRC | 05:02 | |
| *** links has joined #openstack-nova | 05:15 | |
| *** jamesdenton has quit IRC | 05:21 | |
| *** jamesden_ has joined #openstack-nova | 05:22 | |
| *** khomesh24 has joined #openstack-nova | 06:01 | |
| *** k_mouza has joined #openstack-nova | 06:06 | |
| *** k_mouza has quit IRC | 06:10 | |
| *** ociuhandu has joined #openstack-nova | 06:13 | |
| *** ociuhandu has quit IRC | 06:17 | |
| *** ociuhandu has joined #openstack-nova | 06:18 | |
| *** ociuhandu has quit IRC | 06:28 | |
| *** ociuhandu has joined #openstack-nova | 06:29 | |
| openstackgerrit | Wenping Song proposed openstack/nova-specs master: Trival change: correct some nits https://review.opendev.org/c/openstack/nova-specs/+/780443 | 06:31 |
| *** ociuhandu has quit IRC | 06:35 | |
| *** LinPeiWen25 has joined #openstack-nova | 06:47 | |
| *** slaweq has joined #openstack-nova | 06:50 | |
| *** hemanth_n has joined #openstack-nova | 06:59 | |
| *** ociuhandu has joined #openstack-nova | 07:00 | |
| *** ociuhandu has quit IRC | 07:00 | |
| *** ociuhandu has joined #openstack-nova | 07:00 | |
| *** whoami-rajat_ is now known as whoami-rajat | 07:09 | |
| *** ignaziocassano has joined #openstack-nova | 07:18 | |
| ignaziocassano | hello All, please any help on live migration on queens ? when I migrate the vm crash on destination node | 07:20 |
| ignaziocassano | Ignazio Cassano <ignaziocassano@gmail.com> | 07:20 |
| ignaziocassano | lun 15 mar, 18:59 (13 ore fa) | 07:20 |
| ignaziocassano | a openstack-discuss | 07:20 |
| ignaziocassano | Hello, | 07:20 |
| ignaziocassano | looking at destination kvm host I got the following in instance log under /var/log/libvirt/qemu: | 07:20 |
| ignaziocassano | 2021-03-15 11:48:31.996+0000: starting up libvirt version: 4.5.0, package: 36.el7_9.3 (CentOS BuildSystem <http://bugs.centos.org>, 2020-11-16-16:25:20, x86-01.bsys.centos.org), qemu version: 2.12.0qemu-kvm-ev-2.12.0-44.1.el7_8.1, kernel: 3.10.0-1160.15.2.el7.x86_64, hostname: podto2-kvmae | 07:20 |
| ignaziocassano | LC_ALL=C \ | 07:20 |
| ignaziocassano | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \ | 07:20 |
| ignaziocassano | QEMU_AUDIO_DRV=none \ | 07:20 |
| ignaziocassano | -name guest=instance-00002a52,debug-threads=on \ | 07:20 |
| ignaziocassano | -S \ | 07:20 |
| ignaziocassano | -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-73-instance-00002a52/master-key.aes \ | 07:20 |
| ignaziocassano | -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \ | 07:20 |
| ignaziocassano | -cpu Broadwell-IBRS,vme=on,f16c=on,rdrand=on,hypervisor=on,arat=on,xsaveopt=on,abm=on \ | 07:20 |
| ignaziocassano | -m 4096 \ | 07:20 |
| ignaziocassano | -realtime mlock=off \ | 07:20 |
| ignaziocassano | -smp 2,sockets=2,cores=1,threads=1 \ | 07:20 |
| ignaziocassano | -uuid c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6 \ | 07:20 |
| ignaziocassano | -smbios 'type=1,manufacturer=RDO,product=OpenStack Compute,version=17.0.11-1.el7,serial=3dec30fe-a31f-4ea6-971f-6f993589ef04,uuid=c6ea7ed2-e7ce-4df6-a767-6bb95ae8fdc6,family=Virtual Machine' \ | 07:20 |
| ignaziocassano | -no-user-config \ | 07:20 |
| ignaziocassano | -vnc 0.0.0.0:55 \ | 07:21 |
| ignaziocassano | -k en-us \ | 07:21 |
| ignaziocassano | -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \ | 07:21 |
| ignaziocassano | -incoming defer \ | 07:21 |
| ignaziocassano | -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \ | 07:21 |
| ignaziocassano | -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ | 07:21 |
| ignaziocassano | -msg timestamp=on | 07:21 |
| ignaziocassano | 2021-03-15 11:48:31.996+0000: Domain id=73 is tainted: high-privileges | 07:21 |
| ignaziocassano | 2021-03-15T11:48:32.163025Z qemu-kvm: -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on: char device redirected to /dev/pts/57 (label charserial0) | 07:21 |
| ignaziocassano | 2021-03-15T11:48:32.167206Z qemu-kvm: -drive file=/var/lib/nova/mnt/7eb4b0178ee3ec9ad7cbbc20c62b1912/volume-d5c812c5-2c27-4e82-a38d-83fc79ab848e,format=raw,if=none,id=drive-virtio-disk0,serial=d5c812c5-2c27-4e82-a38d-83fc79ab848e,cache=none,aio=native: 'serial' is deprecated, please use the corresponding option of '-device' instead | 07:21 |
| ignaziocassano | 2021-03-15T11:48:37.779611Z qemu-kvm: Failed to load virtio_pci/modern_queue_state:desc | 07:21 |
| ignaziocassano | 2021-03-15T11:48:37.780020Z qemu-kvm: Failed to load virtio_pci/modern_state:vqs | 07:21 |
| ignaziocassano | 2021-03-15T11:48:37.780042Z qemu-kvm: Failed to load virtio/extra_state:extra_state | 07:21 |
| ignaziocassano | 2021-03-15T11:48:37.780062Z qemu-kvm: Failed to load virtio-balloon:virtio | 07:21 |
| ignaziocassano | 2021-03-15T11:48:37.780082Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:06.0/virtio-balloon' | 07:21 |
| ignaziocassano | 2021-03-15T11:48:37.781465Z qemu-kvm: load of migration failed: Input/output error | 07:21 |
| ignaziocassano | 2021-03-15 11:48:38.231+0000: shutting down, reason=crashed | 07:21 |
| ignaziocassano | "instance-00002a52.log" 102L, 7122C | 07:21 |
| *** rcernin has quit IRC | 07:24 | |
| *** jamesden_ has quit IRC | 07:49 | |
| *** jamesdenton has joined #openstack-nova | 07:50 | |
| gibi | brinzhang_: sorry, I was mostly off yesterday | 07:52 |
| brinzhang_ | gibi: np^ | 07:52 |
| brinzhang_ | please review firstly, Ihave a meeting, after end this meeting I will back, thanks | 07:53 |
| *** rcernin has joined #openstack-nova | 07:54 | |
| *** dklyle has quit IRC | 07:56 | |
| *** rpittau|afk is now known as rpittau | 07:58 | |
| *** tesseract has joined #openstack-nova | 08:03 | |
| *** rcernin has quit IRC | 08:06 | |
| *** andrewbonney has joined #openstack-nova | 08:10 | |
| *** ociuhandu has quit IRC | 08:19 | |
| *** k-s-dean has joined #openstack-nova | 08:28 | |
| *** zigo has joined #openstack-nova | 08:30 | |
| gibi | brinzhang_, bauzas: replyied in https://review.opendev.org/c/openstack/nova/+/778440 | 08:33 |
| gibi | I will have to go offline again, sorry, I will be available during the day | 08:33 |
| bauzas | gibi: brinzhang_: sorry, a bit not paying attention to the IRC chan as I'm trying to update the RPC API | 08:34 |
| bauzas | gibi: np | 08:34 |
| brinzhang_ | gibi, bauzas: np, I will review your reply comments, thanks | 08:39 |
| *** tosky has joined #openstack-nova | 09:00 | |
| *** khomesh24 has quit IRC | 09:01 | |
| *** xarlos has joined #openstack-nova | 09:01 | |
| *** lpetrut has joined #openstack-nova | 09:02 | |
| *** k-s-dean has quit IRC | 09:03 | |
| *** k-s-dean has joined #openstack-nova | 09:03 | |
| *** derekh has joined #openstack-nova | 09:06 | |
| *** lucasagomes has joined #openstack-nova | 09:08 | |
| *** brinzhang0 has joined #openstack-nova | 09:12 | |
| *** ignaziocassano has quit IRC | 09:14 | |
| *** brinzhang_ has quit IRC | 09:16 | |
| *** lee2 has joined #openstack-nova | 09:24 | |
| *** lee2 is now known as lyarwood | 09:24 | |
| *** ratailor has quit IRC | 09:50 | |
| *** ratailor has joined #openstack-nova | 09:51 | |
| *** dtantsur|afk is now known as dtantsur | 09:56 | |
| *** k_mouza has joined #openstack-nova | 09:59 | |
| hemanth_n | sean-k-mooney stephenfin: can i get some reviews on old backport patch https://review.opendev.org/c/openstack/nova/+/761824 when you have time, thanks | 10:07 |
| *** ratailor_ has joined #openstack-nova | 10:15 | |
| *** ratailor_ has quit IRC | 10:16 | |
| *** ratailor_ has joined #openstack-nova | 10:16 | |
| *** ratailor_ has quit IRC | 10:17 | |
| *** ratailor_ has joined #openstack-nova | 10:18 | |
| *** ratailor has quit IRC | 10:18 | |
| *** martinkennelly has joined #openstack-nova | 10:21 | |
| *** ratailor__ has joined #openstack-nova | 10:23 | |
| *** ratailor_ has quit IRC | 10:27 | |
| *** jangutter_ has quit IRC | 10:42 | |
| *** supamatt has quit IRC | 10:43 | |
| *** jangutter has joined #openstack-nova | 10:43 | |
| *** ociuhandu has joined #openstack-nova | 10:44 | |
| lyarwood | bauzas / melwitt ; https://review.opendev.org/q/b9333125790682f9d60bc74fdbb12a098565e7c2 - really simple backports if you have time this week | 10:48 |
| *** smcginnis has joined #openstack-nova | 10:48 | |
| *** dviroel has joined #openstack-nova | 10:49 | |
| lyarwood | ah nvm I forgot to include another change so that's borked after victoria | 10:52 |
| * lyarwood fixes | 10:52 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: Use absolute path during qemu img rebase https://review.opendev.org/c/openstack/nova/+/757083 | 10:54 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780780 | 10:54 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: Use absolute path during qemu img rebase https://review.opendev.org/c/openstack/nova/+/757084 | 11:00 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780782 | 11:00 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: Use absolute path during qemu img rebase https://review.opendev.org/c/openstack/nova/+/757085 | 11:01 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780784 | 11:01 |
| *** jangutter_ has joined #openstack-nova | 11:08 | |
| *** jangutter has quit IRC | 11:11 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780787 | 11:12 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: Use absolute path during qemu img rebase https://review.opendev.org/c/openstack/nova/+/780788 | 11:12 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780789 | 11:17 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Use absolute path during qemu img rebase https://review.opendev.org/c/openstack/nova/+/780790 | 11:17 |
| brinzhang0 | bauzas: if you are free from the rpc version patch, pls review gibi's comments, thanks^ | 11:21 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/pike: Update resources once in update_available_resource https://review.opendev.org/c/openstack/nova/+/612295 | 11:31 |
| *** ratailor__ has quit IRC | 11:46 | |
| openstackgerrit | Merged openstack/nova master: libvirt: Wire up 'os_secure_boot' property https://review.opendev.org/c/openstack/nova/+/776681 | 11:47 |
| openstackgerrit | Merged openstack/nova master: libvirt: Report secure boot support to scheduler https://review.opendev.org/c/openstack/nova/+/775690 | 11:48 |
| openstackgerrit | Merged openstack/nova master: tests: Add functional tests for UEFI, secure boot https://review.opendev.org/c/openstack/nova/+/776682 | 11:48 |
| openstackgerrit | Merged openstack/nova master: tests: Remove duplicated 'start_compute' helper https://review.opendev.org/c/openstack/nova/+/776683 | 11:49 |
| *** dosaboy has quit IRC | 11:49 | |
| *** hkominos has joined #openstack-nova | 11:49 | |
| openstackgerrit | Merged openstack/nova master: docs: Document UEFI secure boot feature https://review.opendev.org/c/openstack/nova/+/776684 | 11:50 |
| *** smcginnis has quit IRC | 11:50 | |
| *** dosaboy has joined #openstack-nova | 11:50 | |
| hkominos | Hi guys. Can i please ask a quick question regarding an exception that I am facing in my deployment? I believe the input from a developer would be more helpful for me than asking in the openstack channel | 11:51 |
| *** ratailor has joined #openstack-nova | 11:51 | |
| *** stand has quit IRC | 11:52 | |
| k-s-dean | hkominos, ask away. someone might be see it later. If i can answer you I will. | 11:52 |
| hkominos | k-s-dean Hi!. What do you make of this : https://paste.centos.org/view/741688bb | 11:52 |
| k-s-dean | hkominos, are you running cyborg ? | 11:54 |
| hkominos | no | 11:54 |
| hkominos | This appeared after a host was rebooted with some Vms on it | 11:54 |
| *** ratailor_ has joined #openstack-nova | 11:54 | |
| hkominos | which now refuse to come up | 11:54 |
| k-s-dean | whats the underlying hardware | 11:55 |
| *** Luzi has joined #openstack-nova | 11:57 | |
| *** ratailor has quit IRC | 11:57 | |
| k-s-dean | to me that sounds like a hardware issue. | 11:57 |
| k-s-dean | hkominos, have you checked the compute logs on the host | 11:58 |
| k-s-dean | \? | 11:58 |
| hkominos | yes. https://paste.centos.org/view/73064e38. But before I start looking for hardware vendors I want to understand the problem. I think the issues probably more into nova placement but Idk. that is why I asked here. | 12:01 |
| k-s-dean | hkominos, has this machine got a graphics card in it ? | 12:04 |
| hkominos | yes. | 12:04 |
| k-s-dean | Has the graphics card failed ? | 12:04 |
| hkominos | lets double check | 12:05 |
| *** smcginnis has joined #openstack-nova | 12:07 | |
| hkominos | does not look like it. | 12:08 |
| k-s-dean | ok. | 12:09 |
| k-s-dean | any reason why you have 15 instances in the placement database and 10 instances on the hypervisor. | 12:10 |
| k-s-dean | can you run virsh list --all on the host ? | 12:10 |
| *** ratailor_ has quit IRC | 12:15 | |
| *** ociuhandu has quit IRC | 12:18 | |
| *** hemanth_n has quit IRC | 12:25 | |
| gibi | kashyap: I guess https://review.opendev.org/c/openstack/nova/+/682627 is not a mandatory part of the secure boot feature for W | 12:28 |
| *** tbachman has quit IRC | 12:28 | |
| gibi | am I correct? | 12:28 |
| * kashyap clicks | 12:28 | |
| gibi | the rest of the secure boot series landed | 12:28 |
| kashyap | gibi: Cool; just see it in the scrollback. So that auto-detect can come later -- unfortunately, a bug in libvirt blocking that :-( | 12:29 |
| gibi | kashyap: ack, then marking the bp implemented for W | 12:29 |
| gibi | thanks | 12:29 |
| *** tbachman has joined #openstack-nova | 12:29 | |
| kashyap | gibi: Thank you. And stephenfin, particularly | 12:30 |
| gibi | stephenfin: will you resolve the merge conflict in the vdpa series? | 12:33 |
| stephenfin | working on it atm | 12:33 |
| gibi | cool | 12:33 |
| gibi | thanks | 12:33 |
| *** ociuhandu has joined #openstack-nova | 12:38 | |
| *** ociuhandu has quit IRC | 12:39 | |
| *** tbachman_ has joined #openstack-nova | 12:39 | |
| *** ociuhandu has joined #openstack-nova | 12:40 | |
| *** smcginnis has quit IRC | 12:40 | |
| *** tbachman has quit IRC | 12:42 | |
| *** tbachman_ is now known as tbachman | 12:42 | |
| *** macz_ has joined #openstack-nova | 12:43 | |
| *** rcernin has joined #openstack-nova | 12:47 | |
| *** macz_ has quit IRC | 12:48 | |
| *** smcginnis has joined #openstack-nova | 12:50 | |
| *** READ10 has joined #openstack-nova | 12:50 | |
| *** ociuhandu has quit IRC | 12:51 | |
| *** jangutter_ has quit IRC | 12:53 | |
| *** rcernin has quit IRC | 12:56 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780782 | 12:57 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: Use absolute path during qemu img rebase https://review.opendev.org/c/openstack/nova/+/757084 | 12:57 |
| *** jangutter has joined #openstack-nova | 12:59 | |
| *** ociuhandu has joined #openstack-nova | 13:02 | |
| *** hemanth_n has joined #openstack-nova | 13:12 | |
| *** smcginnis has quit IRC | 13:12 | |
| *** zul has joined #openstack-nova | 13:14 | |
| *** hemanth_n has quit IRC | 13:16 | |
| *** smcginnis has joined #openstack-nova | 13:18 | |
| *** artom has quit IRC | 13:21 | |
| *** artom has joined #openstack-nova | 13:22 | |
| *** smcginnis has quit IRC | 13:24 | |
| *** supamatt has joined #openstack-nova | 13:29 | |
| gibi | is there anything that I should review now? | 13:38 |
| *** smcginnis has joined #openstack-nova | 13:41 | |
| hkominos | k-s-dean virsh list all shows (or did show 10) VMs | 13:49 |
| hkominos | now why placement thought 15 Is because it had some garbage VMs that did not spawn on this node. | 13:49 |
| k-s-dean | Those should have been cleaned up. | 13:50 |
| hkominos | Will do | 13:51 |
| *** sapd1 has joined #openstack-nova | 13:51 | |
| *** yoctozepto has quit IRC | 13:53 | |
| openstackgerrit | Elod Illes proposed openstack/nova stable/pike: [stable-only] gate: Pin CEPH_RELEASE to nautilus in LM hook https://review.opendev.org/c/openstack/nova/+/780852 | 13:58 |
| *** mlavalle has joined #openstack-nova | 14:00 | |
| *** tbachman has quit IRC | 14:03 | |
| *** tbachman_ has joined #openstack-nova | 14:03 | |
| stephenfin | gibi: I'm reworking Sean's "block unsupported ops with vDPA interface" patch to return HTTP 409 (Conflict) instead of HTTP (Forbidden) since that seems more sensible. It's not a permissions or access issue, IMO. Do you agree? | 14:10 |
| *** jobewan has joined #openstack-nova | 14:23 | |
| *** sapd1 has quit IRC | 14:24 | |
| *** spatel has joined #openstack-nova | 14:25 | |
| *** macz_ has joined #openstack-nova | 14:27 | |
| Luzi | hey, is someone around who knows the qemu native tls config options? I think i might have found a bug in the guide: https://docs.openstack.org/nova/latest/admin/secure-live-migration-with-qemu-native-tls.html | 14:31 |
| *** macz_ has quit IRC | 14:33 | |
| kashyap | Luzi: Hi, I think I wrote that doc :) | 14:34 |
| kashyap | Luzi: What's the bug? Feel free to amend / send a pull-req :) | 14:35 |
| kashyap | s/pull-req/patch/ | 14:35 |
| kashyap | At the bottom right, there is also "found an error? report a bug" | 14:36 |
| Luzi | i followed that guide but got to a point when openstack did not use the tls path, but kept using the tcp path | 14:36 |
| Luzi | i onlny found out because i used tcpdump on both ports | 14:36 |
| kashyap | Luzi: Hmm, if you followed that guide to the word, then your setup should definitely be using the QEMU-native TLS | 14:37 |
| Luzi | i was looking through code and found out, that the uri used for live migration is different from the connection uri and uses a config option not mentioned in the guide | 14:38 |
| kashyap | Luzi: Hmm, possible the guide got slightly outdated ... as migration-related code got reworked | 14:39 |
| Luzi | only after i set live_migration_scheme = tls | 14:39 |
| Luzi | the traffic was encrypted | 14:39 |
| Luzi | and using the correct port :) | 14:39 |
| kashyap | Luzi: I see. That's correct _scheme is favoured | 14:41 |
| kashyap | And _uri parameter is deprecated in favor of the above | 14:41 |
| Luzi | but when you don't set it, as i had not done it, the code uses the hardcoded tcp parameter | 14:41 |
| fungi | looks like that config option was added by https://review.openstack.org/410817 which merged in early 2017 | 14:41 |
| kashyap | fungi: Heya; yep. That timeframe is right ... /me clicks | 14:42 |
| kashyap | Yep; patch is also correct. | 14:42 |
| Luzi | live_migration_scheme = tls | 14:42 |
| fungi | first appeared in nova 15.0.0, so ocata and later | 14:42 |
| kashyap | Luzi: Also you don't tell what version of OpenStack you were using? | 14:42 |
| kashyap | Yeah, what fungi says. | 14:43 |
| Luzi | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1224 | 14:43 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: pci: Add vDPA vnic to PCI request mapping and filtering https://review.opendev.org/c/openstack/nova/+/778350 | 14:43 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Block unsupported actions with vDPA https://review.opendev.org/c/openstack/nova/+/780333 | 14:43 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | 14:43 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: tests: Make mdev stubs work like vDPA https://review.opendev.org/c/openstack/nova/+/780234 | 14:43 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add release note for vDPA https://review.opendev.org/c/openstack/nova/+/780866 | 14:43 |
| stephenfin | gibi: lyarwood: ^ | 14:43 |
| Luzi | i am working on train and looking to upgrade soon | 14:43 |
| Luzi | i posted the line of code which seems to be still on master | 14:44 |
| fungi | same change deprecated live_migration_uri | 14:45 |
| kashyap | (Yep) | 14:45 |
| *** sapd1 has joined #openstack-nova | 14:45 | |
| Luzi | in the guide the only config option which is mentioned is live_migration_with_native_tls = true | 14:46 |
| Luzi | which is definitely not enough to enable native tls | 14:46 |
| kashyap | Luzi: So, indeed - it checks if the config option is set, then uses the _scheme, if not defaults to TLS | 14:46 |
| fungi | so alternatively, the document could warn that setting the _scheme will prevent a natural fallback to tls | 14:47 |
| Luzi | well i thought it worked, until i started tcpdump | 14:47 |
| Luzi | tcp | 14:47 |
| Luzi | the default in code is tcp | 14:47 |
| kashyap | Luzi: Err, defaults to TCP, I mean, I'm sorry. | 14:48 |
| Luzi | the guid should at least say it is needed to set the _scheme to tls | 14:48 |
| fungi | ayup, i agree it defaults to 'tcp' there | 14:48 |
| kashyap | Luzi: Right; I'll check the installer code to double-confirm; and then can update the doc | 14:49 |
| fungi | okay, so the doc just needs updating to mention setting live_migration_scheme="tls"? | 14:49 |
| Luzi | i'm just worried, that this may also concern some users which did not check their deployment with a tcpdump :D | 14:49 |
| fungi | it got mentioned in the release notes (under features) but yeah that's fairly hidden | 14:49 |
| Luzi | it should be enough, to update the guide, i think | 14:50 |
| fungi | and even the release note doesn't come out and say it's needed for turning on tls | 14:50 |
| Luzi | fungi, yeah thats the point | 14:50 |
| Luzi | i needed a whole day to find out :D | 14:51 |
| kashyap | fungi: Yep - on the doc needs updating. Luzi: Good catch! | 14:51 |
| fungi | Luzi: if you're worried that we need to do some outreach, the typical process for that is an openstack security note: https://wiki.openstack.org/wiki/Security/Security_Note_Process | 14:51 |
| Luzi | i need to go off now, thank you for replying so quickly kashyap and fungi :) | 14:52 |
| kashyap | Luzi: So you need _both_: | 14:52 |
| kashyap | - live_migration_scheme | 14:52 |
| fungi | once published, we can send a copy to the openstack-announce and openstack-discuss mailing lists | 14:52 |
| kashyap | - live_migration_with_native_tls | 14:52 |
| Luzi | fungi, i will do that when i'm at home :) | 14:52 |
| Luzi | kashyap, yes | 14:52 |
| *** Luzi has quit IRC | 14:53 | |
| *** macz_ has joined #openstack-nova | 14:54 | |
| kashyap | (Ah, Luzi is gone before I wanted to mention a way to actually verify how to check native TLS is in effect for disks during migration w/ low-level logs.) | 14:54 |
| lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/780333 - quick commit message nit in there but otherwise LGTM | 15:03 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Block unsupported actions with vDPA https://review.opendev.org/c/openstack/nova/+/780333 | 15:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | 15:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add release note for vDPA https://review.opendev.org/c/openstack/nova/+/780866 | 15:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: tests: Make mdev stubs work like vDPA https://review.opendev.org/c/openstack/nova/+/780234 | 15:07 |
| lyarwood | ta | 15:08 |
| lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/780866 - same in the releasenote btw | 15:17 |
| stephenfin | gdi :) | 15:17 |
| stephenfin | will fix | 15:17 |
| lyarwood | np my review ratio thanks you ;) | 15:18 |
| lyarwood | joys of being a stable core | 15:18 |
| gibi | stephenfin: ack, I have no problem with 409, and I will review the series soon | 15:21 |
| *** Techy2493 has joined #openstack-nova | 15:21 | |
| *** mkrai has quit IRC | 15:22 | |
| *** mkrai_ has joined #openstack-nova | 15:22 | |
| *** dklyle has joined #openstack-nova | 15:23 | |
| *** hemanth_n has joined #openstack-nova | 15:29 | |
| *** sapd1 has quit IRC | 15:33 | |
| *** psachin has quit IRC | 15:36 | |
| *** gyee has joined #openstack-nova | 15:43 | |
| *** sapd1 has joined #openstack-nova | 15:49 | |
| *** hemanth_n has quit IRC | 15:51 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Deprecate `live_migration_tunnelled` https://review.opendev.org/c/openstack/nova/+/780908 | 15:52 |
| kashyap | gibi: stephenfin: Would be nice to get it going in Wallaby --^. Will reduce some future "live migration config fatigue" | 15:54 |
| gibi | kashyap: ack, I will check after the vdpa series | 15:55 |
| kashyap | Sure; no prob. It's just a deprecation meta-work; that's all. | 15:55 |
| kashyap | I'll add a code-reference in the change comment | 15:55 |
| lyarwood | kashyap: random question, do we have coverage of live_migration_with_native_tls in the gate somewhere? | 15:58 |
| lyarwood | kashyap: I can't find anything but that smells like something we should enable in nova-next at least | 15:59 |
| kashyap | lyarwood: Good question, I do not know; and near as I know, no. It requires a full TLS env to be setup -- which is what TripleO does | 15:59 |
| kashyap | See my prerequisite here: https://docs.openstack.org/nova/latest/admin/secure-live-migration-with-qemu-native-tls.html#prerequisites | 15:59 |
| *** LinPeiWen25 has quit IRC | 16:00 | |
| kashyap | lyarwood: I swear there is some automated test of this upstream (perhaps the upstream whitebox from OSP QE folks), can't find a link yet | 16:00 |
| lyarwood | kashyap: yeah it's likely on the TripleO side and sorry I thought the virt tools would automate the CA creation etc, if not ignore me as it's going to be too much work to enable in devstack for nova-next etc | 16:01 |
| kashyap | lyarwood: Yeah, it requires good old PKI setup. It _can_ be automated, using GnuTLS | 16:01 |
| kashyap | lyarwood: If you don't mind holding your nose, something like ... https://kashyapc.fedorapeople.org/Auto-Setup-GnuTLS/Setup-GnuTLS-CA-and-ServerCert.bash | 16:02 |
| lyarwood | that said we do have some novnc tls stuff in devstack so... | 16:02 |
| kashyap | I see | 16:03 |
| *** vishalmanchanda has quit IRC | 16:03 | |
| kashyap | But pretty sure I recall chatting w/ TripleO folks to have an automated test. Once I find a URL, I'll link it in the change or post here | 16:03 |
| *** adrianc has quit IRC | 16:07 | |
| *** adrianc has joined #openstack-nova | 16:07 | |
| *** spatel has quit IRC | 16:07 | |
| *** Techy2493 has quit IRC | 16:17 | |
| *** Luzi has joined #openstack-nova | 16:21 | |
| *** ociuhandu has quit IRC | 16:27 | |
| *** ociuhandu has joined #openstack-nova | 16:27 | |
| bauzas | gibi: others, folks, i'm taking an urgent PTO tomorrow, my wife is turning 40 as well | 16:30 |
| *** manuvakery1 has joined #openstack-nova | 16:32 | |
| gibi | bauzas: ack, have a nice celebration! | 16:32 |
| gibi | bauzas: will you push a new rev from the rpc bump patch before you leave? | 16:33 |
| bauzas | gibi: not sure :( | 16:34 |
| bauzas | still fixing to not support 5.0 | 16:34 |
| gibi | bauzas: no worries just preparing my review queue | 16:36 |
| Luzi | kashyap, are you around? | 16:39 |
| kashyap | Luzi: Hi, yes | 16:39 |
| kashyap | Luzi: So, I wanted to mention one more to thing to you, before you left | 16:39 |
| kashyap | Luzi: If you want to see native TLS is *actually* in effect, you can verify it by a slightly tedious method in logs | 16:40 |
| kashyap | Luzi: Is your env. production? Or do you have a staging setup? | 16:40 |
| Luzi | testing setup | 16:40 |
| kashyap | Luzi: Perfect. So here we go: | 16:41 |
| kashyap | Luzi: What OS do you have? Ubuntu or Fedora? Doesn't matter: install the "libvirt-admin" tool | 16:41 |
| kashyap | s/tool/package/ | 16:41 |
| kashyap | To get the 'virt-admin' tool. Note: Ubuntu might name it differently. So "grep" your package repository for the tool. In Fedora, I can do it like this: `dnf whatprovides *virt-admin` | 16:43 |
| kashyap | Luzi: Then follow this to enable the dynamic libvirt log filters on your compute nodes: https://kashyapc.fedorapeople.org/virt/virt-admin.txt | 16:43 |
| kashyap | Once you have that setup; you can migrate a guest, and then you can look for: `grep tls-creds-x509` on your source and destination libvirtd.log. | 16:45 |
| kashyap | Luzi: Did you hav a different question for me? I began talking right after you pinged me :) | 16:45 |
| *** Luzi_ has joined #openstack-nova | 16:46 | |
| *** Luzi has quit IRC | 16:46 | |
| Luzi_ | here again, needed to change to vpn | 16:46 |
| Luzi_ | i missed the last 6 minutes or so - did you wrote something kashyap ? | 16:47 |
| kashyap | Luzi_: Yes, I did post something. I was talking to myself. Let me post a pastebin | 16:47 |
| kashyap | Luzi_: http://paste.openstack.org/show/803621/ | 16:48 |
| *** hamalq has joined #openstack-nova | 16:48 | |
| kashyap | Luzi_: For comparison, you should see commands like these in your _destination_ libvirt log file: https://kashyapc.fedorapeople.org/Native-TLS/Test-Evidence/DEST-QMP-commands-TLS-over-NBD-guestHyp2.log | 16:50 |
| kashyap | Luzi_: I need to head out shortly. If you have a question, ask now :-) (Assuming you're not disconnected again.) | 16:54 |
| Luzi_ | okay, it seems i adjusted the log level, I will now migrate with and without the config option set | 16:54 |
| *** ociuhandu_ has joined #openstack-nova | 16:59 | |
| *** lucasagomes has quit IRC | 17:02 | |
| Luzi_ | kashyap, it seems the logs collected by the environment are having another log level, may be because of the deployment setting | 17:02 |
| gibi | stephenfin: I'm +2 on the vdpa series, thanks for picking it up. The reno needs a respin as spotted by lyarwood. | 17:02 |
| *** ociuhandu has quit IRC | 17:02 | |
| Luzi_ | i would still open a bug and display the commands i used to catch the traffic with - for both ways | 17:02 |
| *** ociuhandu_ has quit IRC | 17:03 | |
| *** tesseract has quit IRC | 17:04 | |
| gibi | kashyap: Im +2 on the deprecation of the tunneled live migration patch | 17:05 |
| kashyap | Luzi_: Strange, do you already have the config settings manually done in /etc/libvirt/libvirtd.conf? | 17:06 |
| Luzi_ | maybe | 17:07 |
| kashyap | Luzi_: Check if you have these two configs set in your /etc/libvirt/libvirtd.conf: | 17:07 |
| kashyap | - log_filters | 17:07 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: Bump the Compute RPC API to version 6.0 https://review.opendev.org/c/openstack/nova/+/761452 | 17:07 |
| kashyap | - log_outputs | 17:07 |
| kashyap | gibi: Thank you! | 17:07 |
| bauzas | gibi: I eventually gave up given the time, and I just fixed dansmith's nits | 17:08 |
| bauzas | dansmith: gibiI'll try to still work on no longer supporting 5.0 on Thursday | 17:08 |
| kashyap | Luzi_: I need to head out for a walk, but feel free to gather your thoughts in a file. | 17:08 |
| kashyap | (And email here, or post them in a non-expiring pastebin somewhere. I'll come back and check.) | 17:10 |
| kashyap | Luzi_: In short, use either manual approach or the dynamic filters. See the "Gathering libvirt logs..." section here: https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt | 17:13 |
| Luzi_ | I need to head out too, so I just complete the bug report and maybe add some more tomorrow | 17:15 |
| gibi | bauzas: ack, I will review what you pushed | 17:20 |
| gibi | bauzas: have a nice PTO tomorrow | 17:20 |
| bauzas | thanks | 17:20 |
| *** rpittau is now known as rpittau|afk | 17:29 | |
| *** ociuhandu has joined #openstack-nova | 17:30 | |
| *** Luzi_ has quit IRC | 17:31 | |
| openstackgerrit | Merged openstack/nova master: Remove VFSLocalFS https://review.opendev.org/c/openstack/nova/+/778506 | 17:33 |
| *** ociuhandu has quit IRC | 17:34 | |
| *** terdei has joined #openstack-nova | 17:43 | |
| *** dtantsur is now known as dtantsur|afk | 17:47 | |
| *** vishalmanchanda has joined #openstack-nova | 17:48 | |
| *** READ10 has quit IRC | 17:59 | |
| *** lpetrut has quit IRC | 18:00 | |
| *** xek has quit IRC | 18:02 | |
| *** derekh has quit IRC | 18:03 | |
| *** lbragstad has quit IRC | 18:07 | |
| *** k_mouza has quit IRC | 18:09 | |
| *** xek has joined #openstack-nova | 18:11 | |
| *** xek has quit IRC | 18:11 | |
| *** lbragstad has joined #openstack-nova | 18:27 | |
| *** hkominos has quit IRC | 18:46 | |
| *** ralonsoh has quit IRC | 18:51 | |
| *** andrewbonney has quit IRC | 19:09 | |
| *** efried1 has joined #openstack-nova | 19:13 | |
| *** efried has quit IRC | 19:14 | |
| *** efried1 is now known as efried | 19:14 | |
| *** whoami-rajat has quit IRC | 19:26 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Delegate OVS plug to os-vif https://review.opendev.org/c/openstack/nova/+/602432 | 19:41 |
| melwitt | gmann: hey, wondering if you have seen this tempest-slow-py3 gate failure before ""tempest.scenario.test_network_v6.TestGettingAddress Bad router request: Cidr 2001:db8::/64 of subnet f3908f8d-a960-444f-9708-78ae906fbd63 overlaps with cidr 2001:db8::/64 of subnet d8fe1d15-5cdd-40e1-96e5-d1ac105253c3" I only see it on stable/stein in nova | 19:55 |
| melwitt | https://zuul.opendev.org/t/openstack/build/0d21e41fc0294f1d891c484dea84adb6/logs | 19:55 |
| *** mkrai_ has quit IRC | 19:59 | |
| *** tosky has quit IRC | 20:05 | |
| *** tosky has joined #openstack-nova | 20:05 | |
| *** vishalmanchanda has quit IRC | 20:14 | |
| *** yoctozepto has joined #openstack-nova | 20:14 | |
| *** links has quit IRC | 20:15 | |
| frickler | melwitt: this looks related https://review.opendev.org/c/openstack/neutron/+/777389 | 20:19 |
| melwitt | frickler: that does look like it, thank you! | 20:20 |
| *** Techy2493 has joined #openstack-nova | 20:22 | |
| *** slaweq has quit IRC | 20:26 | |
| *** hemna has quit IRC | 20:28 | |
| *** slaweq has joined #openstack-nova | 20:29 | |
| *** hemna has joined #openstack-nova | 20:29 | |
| *** manuvakery1 has quit IRC | 20:30 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: pci: Add vDPA vnic to PCI request mapping and filtering https://review.opendev.org/c/openstack/nova/+/778350 | 20:39 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Block unsupported actions with vDPA https://review.opendev.org/c/openstack/nova/+/780333 | 20:39 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | 20:39 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add release note for vDPA https://review.opendev.org/c/openstack/nova/+/780866 | 20:39 |
| *** Techy2493 has quit IRC | 21:10 | |
| openstackgerrit | Merged openstack/nova master: libvirt: Deprecate `live_migration_tunnelled` https://review.opendev.org/c/openstack/nova/+/780908 | 21:13 |
| *** mlavalle has quit IRC | 21:13 | |
| *** slaweq has quit IRC | 21:24 | |
| *** Techy2493 has joined #openstack-nova | 21:43 | |
| *** k-s-dean has quit IRC | 22:17 | |
| spotz | Is there a maximum number of security groups you cane have? Not talking about quota restricted | 22:21 |
| *** rcernin has joined #openstack-nova | 22:24 | |
| *** brinzhang_ has joined #openstack-nova | 22:26 | |
| *** lbragstad_ has joined #openstack-nova | 22:29 | |
| *** trozet has joined #openstack-nova | 22:29 | |
| *** dosaboy_ has joined #openstack-nova | 22:30 | |
| *** Techy2493 has quit IRC | 22:32 | |
| *** lbragstad has quit IRC | 22:35 | |
| *** dosaboy has quit IRC | 22:35 | |
| *** supamatt has quit IRC | 22:35 | |
| *** brinzhang0 has quit IRC | 22:35 | |
| melwitt | spotz: security groups will be a question for the neutron team, nova hasn't done them since the olden days of nova-network | 22:37 |
| spotz | haha thanks melwitt! | 22:37 |
| melwitt | yw | 22:37 |
| *** gyee has quit IRC | 22:40 | |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: [stable-only] Specify IPv6 CIDR in tempest-slow-py3 https://review.opendev.org/c/openstack/nova/+/780991 | 22:47 |
| *** mlavalle has joined #openstack-nova | 22:50 | |
| *** k-s-dean has joined #openstack-nova | 23:18 | |
| *** rcernin has quit IRC | 23:38 | |
| *** rcernin has joined #openstack-nova | 23:39 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!