| imacdonn | tonyb: around? | 00:10 |
|---|---|---|
| tonyb | imacdonn: Kinda ... in meetings (yes plural) ;P | 00:11 |
| *** rcernin has joined #openstack-nova | 00:11 | |
| imacdonn | tonyb: heh, ok .. whenever you get a chance, take a look at this backport .... https://review.openstack.org/611701 | 00:11 |
| tonyb | imacdonn: Will do. I'm thinking I might ask on the operators list before we merge | 00:12 |
| tonyb | It | 00:12 |
| imacdonn | tonyb: fair enough - thanks! | 00:12 |
| *** rcernin_ has quit IRC | 00:12 | |
| *** rcernin has quit IRC | 00:13 | |
| *** rcernin has joined #openstack-nova | 00:14 | |
| *** mlavalle has quit IRC | 00:32 | |
| *** takashin has joined #openstack-nova | 00:33 | |
| openstackgerrit | Merged openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093 | 00:43 |
| *** tetsuro has joined #openstack-nova | 00:45 | |
| *** gouthamr has joined #openstack-nova | 00:47 | |
| *** brinzhang has joined #openstack-nova | 00:50 | |
| *** tetsuro has quit IRC | 01:06 | |
| *** tetsuro_ has joined #openstack-nova | 01:06 | |
| *** mrsoul has quit IRC | 01:21 | |
| *** spatel has joined #openstack-nova | 01:22 | |
| *** imacdonn has quit IRC | 01:23 | |
| *** imacdonn has joined #openstack-nova | 01:23 | |
| *** markvoelker has joined #openstack-nova | 01:25 | |
| *** hongbin has joined #openstack-nova | 01:27 | |
| *** tiendc has joined #openstack-nova | 01:29 | |
| *** hongbin has quit IRC | 01:36 | |
| openstackgerrit | Merged openstack/nova master: Zuul: Update barbican experimental job https://review.openstack.org/610141 | 01:37 |
| *** hongbin has joined #openstack-nova | 01:37 | |
| *** hongbin_ has joined #openstack-nova | 01:41 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:42 | |
| *** hongbin has quit IRC | 01:43 | |
| *** trungnv has joined #openstack-nova | 01:44 | |
| *** mhen has quit IRC | 01:44 | |
| Dinesh_Bhor | Hi All, Is there a recommended/official API rate limiter for Nova? | 01:46 |
| *** mhen has joined #openstack-nova | 01:47 | |
| *** TuanDA has joined #openstack-nova | 01:49 | |
| *** idlemind has joined #openstack-nova | 01:53 | |
| *** Dinesh_Bhor has quit IRC | 01:59 | |
| *** hongbin has joined #openstack-nova | 02:05 | |
| *** hongbin_ has quit IRC | 02:07 | |
| openstackgerrit | Merged openstack/nova master: Remove the extensions framework from wsgi.py https://review.openstack.org/607092 | 02:12 |
| *** lei-zh has joined #openstack-nova | 02:22 | |
| * alex_xu begin the spec review day | 02:22 | |
| openstackgerrit | Merged openstack/nova master: Remove duplicate legacy-tempest-dsvm-multinode-full job https://review.openstack.org/610931 | 02:24 |
| *** lei-zh has quit IRC | 02:30 | |
| *** lei-zh1 has joined #openstack-nova | 02:30 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:35 | |
| *** trungnv has quit IRC | 02:36 | |
| *** TuanDA has quit IRC | 02:36 | |
| *** trungnv has joined #openstack-nova | 02:36 | |
| *** TuanDA has joined #openstack-nova | 02:36 | |
| openstackgerrit | Fan Zhang proposed openstack/nova master: Retry after hitting libvirt error VIR_ERR_OPERATION_INVALID in live migration. https://review.openstack.org/612272 | 02:48 |
| *** psachin has joined #openstack-nova | 02:57 | |
| *** dave-mccowan has quit IRC | 02:57 | |
| openstackgerrit | Seyeong Kim proposed openstack/nova master: Enable connection_info refresh for new-style attachments https://review.openstack.org/579004 | 02:59 |
| *** mikeoschen has joined #openstack-nova | 03:00 | |
| *** munimeha1 has quit IRC | 03:01 | |
| openstackgerrit | Merged openstack/nova stable/rocky: Use nova-consoleauth only if workaround enabled https://review.openstack.org/610673 | 03:03 |
| openstackgerrit | Merged openstack/nova master: conductor: Recreate volume attachments during a reschedule https://review.openstack.org/587071 | 03:10 |
| *** Dinesh_Bhor has quit IRC | 03:37 | |
| *** lei-zh1 has quit IRC | 03:39 | |
| *** hongbin has quit IRC | 03:52 | |
| *** udesale has joined #openstack-nova | 03:58 | |
| openstackgerrit | Sam Morrison proposed openstack/nova stable/rocky: Fix up compute rpcapi version for pike release https://review.openstack.org/612561 | 04:02 |
| openstackgerrit | Sam Morrison proposed openstack/nova stable/queens: Fix up compute rpcapi version for pike release https://review.openstack.org/612562 | 04:03 |
| *** Dinesh_Bhor has joined #openstack-nova | 04:42 | |
| openstackgerrit | Seyeong Kim proposed openstack/nova master: Enable connection_info refresh for new-style attachments https://review.openstack.org/579004 | 04:51 |
| *** lei-zh1 has joined #openstack-nova | 04:54 | |
| *** janki has joined #openstack-nova | 04:59 | |
| *** spatel has quit IRC | 05:02 | |
| *** ratailor has joined #openstack-nova | 05:18 | |
| *** jiaopengju has quit IRC | 05:20 | |
| *** jiaopengju has joined #openstack-nova | 05:23 | |
| *** mikeoschen has quit IRC | 05:23 | |
| *** tbachman has quit IRC | 05:33 | |
| *** tbachman has joined #openstack-nova | 05:37 | |
| *** ralonsoh has joined #openstack-nova | 05:40 | |
| *** ttsiouts has quit IRC | 05:42 | |
| *** ttsiouts has joined #openstack-nova | 05:43 | |
| *** Luzi has joined #openstack-nova | 05:44 | |
| *** ttsiouts has quit IRC | 05:47 | |
| *** takashin has left #openstack-nova | 05:48 | |
| *** spsurya has joined #openstack-nova | 05:48 | |
| *** tetsuro_ has quit IRC | 05:50 | |
| *** jangutter has joined #openstack-nova | 06:03 | |
| *** hamdyk has joined #openstack-nova | 06:06 | |
| *** maciejjozefczyk has quit IRC | 06:09 | |
| *** adrianc has joined #openstack-nova | 06:26 | |
| *** adrianc_ has joined #openstack-nova | 06:26 | |
| *** ccamacho has joined #openstack-nova | 06:31 | |
| *** slaweq has joined #openstack-nova | 06:43 | |
| *** moshele has joined #openstack-nova | 06:50 | |
| *** Dinesh_Bhor has quit IRC | 06:52 | |
| *** pcaruana has joined #openstack-nova | 06:56 | |
| aperevalov | Hello, community! Do you have functional test for Direct ports. Does this test requires HW with SR-IOV support? I can't find something similar in the neutron-tempest-test. | 07:01 |
| bauzas | good morning Nova | 07:02 |
| openstackgerrit | Seyeong Kim proposed openstack/nova master: Enable connection_info refresh for new-style attachments https://review.openstack.org/579004 | 07:02 |
| *** jaosorior has quit IRC | 07:02 | |
| *** jaosorior has joined #openstack-nova | 07:05 | |
| *** maciejjozefczyk has joined #openstack-nova | 07:05 | |
| *** rcernin has quit IRC | 07:06 | |
| *** rpittau has quit IRC | 07:07 | |
| *** rpittau has joined #openstack-nova | 07:07 | |
| *** alexchadin has joined #openstack-nova | 07:10 | |
| *** sahid has joined #openstack-nova | 07:16 | |
| *** Dinesh_Bhor has joined #openstack-nova | 07:16 | |
| *** helenafm has joined #openstack-nova | 07:20 | |
| *** lei-zh1 has quit IRC | 07:24 | |
| *** lei-zh1 has joined #openstack-nova | 07:24 | |
| openstackgerrit | Jan Gutter proposed openstack/nova-specs master: Spec to implement generic HW offloads for os-vif https://review.openstack.org/607610 | 07:40 |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: WIP - SRIOV live migration https://review.openstack.org/612620 | 07:44 |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: WIP - SRIOV live migration https://review.openstack.org/612620 | 08:02 |
| *** bhagyashris has joined #openstack-nova | 08:07 | |
| openstackgerrit | Pooja Jadhav proposed openstack/nova master: Ignore root_gb for BFV in simple tenant usage API https://review.openstack.org/612626 | 08:15 |
| *** k_mouza has joined #openstack-nova | 08:15 | |
| *** pvradu has joined #openstack-nova | 08:16 | |
| *** ttsiouts has joined #openstack-nova | 08:17 | |
| *** dtantsur|afk is now known as dtantsur | 08:22 | |
| *** mvkr has quit IRC | 08:26 | |
| *** derekh has joined #openstack-nova | 08:29 | |
| openstackgerrit | Merged openstack/nova master: Convert legacy-tempest-dsvm-neutron-src-oslo.versionedobjects job https://review.openstack.org/610271 | 08:29 |
| *** moshele has quit IRC | 08:30 | |
| *** gouthamr has quit IRC | 08:32 | |
| openstackgerrit | Elod Illes proposed openstack/nova master: Transform compute_task notifications https://review.openstack.org/482629 | 08:37 |
| *** k_mouza has quit IRC | 08:41 | |
| openstackgerrit | Elod Illes proposed openstack/nova master: Transform compute_task notifications https://review.openstack.org/482629 | 08:43 |
| *** k_mouza has joined #openstack-nova | 08:43 | |
| *** priteau has joined #openstack-nova | 08:45 | |
| *** moshele has joined #openstack-nova | 08:50 | |
| *** cdent has joined #openstack-nova | 08:51 | |
| *** mvkr has joined #openstack-nova | 08:51 | |
| *** Dinesh_Bhor has quit IRC | 08:54 | |
| *** moshele has quit IRC | 09:00 | |
| *** k_mouza has quit IRC | 09:06 | |
| *** k_mouza has joined #openstack-nova | 09:07 | |
| *** gouthamr has joined #openstack-nova | 09:11 | |
| *** tetsuro has joined #openstack-nova | 09:14 | |
| *** bzhao__ has quit IRC | 09:20 | |
| *** jpena|off is now known as jpena | 09:22 | |
| *** alex_xu has quit IRC | 09:23 | |
| *** lei-zh1 has quit IRC | 09:28 | |
| *** k_mouza has quit IRC | 09:28 | |
| *** alex_xu has joined #openstack-nova | 09:29 | |
| bauzas | stephenfin: good morning | 09:33 |
| bauzas | stephenfin: do we have some documentation about PCI NUMA policies in https://docs.openstack.org/nova/queens/admin/adv-config.html ? | 09:33 |
| bauzas | stephenfin: or shall I refer to the implemented spec https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/share-pci-between-numa-nodes.html ? | 09:33 |
| bauzas | stephenfin: context is me trying to update https://review.openstack.org/#/c/552924/10/specs/stein/approved/numa-topology-with-rps.rst | 09:33 |
| *** k_mouza has joined #openstack-nova | 09:35 | |
| *** mvkr has quit IRC | 09:36 | |
| *** bhagyashris has quit IRC | 09:39 | |
| *** mvkr has joined #openstack-nova | 09:50 | |
| stephenfin | bauzas: We should have. Sec | 09:51 |
| *** udesale has quit IRC | 09:51 | |
| *** udesale has joined #openstack-nova | 09:52 | |
| *** abhi89 has joined #openstack-nova | 09:55 | |
| stephenfin | bauzas: This is all we have https://docs.openstack.org/nova/rocky/configuration/config.html#pci | 09:55 |
| stephenfin | We should probably expand that out. As such though, the spec looks like the best option | 09:56 |
| abhi89 | Hi.. whenever nova service runs a command using sudo, does this make a call to unix_chkpwd program which is part of pam_unix module? | 09:58 |
| *** pvradu has quit IRC | 10:00 | |
| *** pvradu has joined #openstack-nova | 10:00 | |
| openstackgerrit | Elod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | 10:01 |
| *** TuanDA has quit IRC | 10:04 | |
| *** pvradu_ has joined #openstack-nova | 10:05 | |
| *** Dinesh_Bhor has joined #openstack-nova | 10:07 | |
| *** sahid has quit IRC | 10:08 | |
| *** adrianc has quit IRC | 10:08 | |
| *** pvradu has quit IRC | 10:09 | |
| *** adrianc_ has quit IRC | 10:09 | |
| *** pvc_ has joined #openstack-nova | 10:15 | |
| pvc_ | hi bauzas can you help me regarding vgpu? | 10:15 |
| *** Dinesh_Bhor has quit IRC | 10:15 | |
| pvc_ | http://paste.openstack.org/show/732763/ | 10:15 |
| bauzas | stephenfin: ack, thanks | 10:16 |
| bauzas | stephenfin: anway, it's just for explainint that it won't be modified by my spec :) | 10:16 |
| *** cdent has quit IRC | 10:16 | |
| pvc_ | bauzas hi | 10:16 |
| *** brinzh has joined #openstack-nova | 10:17 | |
| *** brinzhang has quit IRC | 10:20 | |
| pvc_ | can help me regarding vgpu bauzas? | 10:22 |
| bauzas | pvc_: I'm pretty busy today with specs reviews and writing, can we discuss other days ? | 10:22 |
| bauzas | pvc_: anyway, looking at your paste, doesn't seem related to nova at all | 10:23 |
| pvc_ | noted on this i cant install the driver | 10:23 |
| pvc_ | thank you | 10:23 |
| *** fanzhang has quit IRC | 10:26 | |
| stephenfin | Oh, today is spec review day. Oops | 10:32 |
| pvc_ | can i ask if i need the vfio-pci or not? | 10:36 |
| pvc_ | this one NVIDIA-Linux-x86_64-390.72-vgpu-kvm.run? | 10:38 |
| *** cdent has joined #openstack-nova | 10:43 | |
| *** mrch has joined #openstack-nova | 10:43 | |
| *** moshele has joined #openstack-nova | 10:45 | |
| *** adrianc_ has joined #openstack-nova | 10:45 | |
| *** adrianc has joined #openstack-nova | 10:45 | |
| *** tbachman has quit IRC | 10:47 | |
| pvc_ | how can i add now a vgpu to use by my instance? | 10:50 |
| sean-k-mooney | pvc_: you need to add a resouce class request to your instance flavor | 10:54 |
| sean-k-mooney | if you do then the libvirt dirver will automatically add it to your vm | 10:54 |
| sean-k-mooney | that said that assumes your host has an nvida vgpus capable graphics card and you have the correct drivers on the host to enable it | 10:55 |
| sean-k-mooney | pvc_: e.g. you card needs to be on this list https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html | 10:56 |
| sean-k-mooney | pvc_: if your card is not on that list you cannot use the vgpu support and can only use pci passthough instead to dedicate the entire gpu to the guest | 10:57 |
| pvc_ | i have this mdev | 10:57 |
| *** erlon has joined #openstack-nova | 10:58 | |
| pvc_ | sean-k-mooney http://paste.openstack.org/show/732798/ | 10:58 |
| sean-k-mooney | those are the mdev typs not mdevs but you need to choose one to enable in teh nova config | 10:59 |
| pvc_ | i already have this wait | 11:00 |
| pvc_ | enabled_vgpu_types = nvidia-160 | 11:01 |
| pvc_ | i add it on overcloud-compute | 11:01 |
| pvc_ | Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4c6de80f-cc4d-47c9-a63d-f72a5584ecb6. | 11:02 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs https://review.openstack.org/552924 | 11:03 |
| pvc_ | do i need to do something sean-k-mooney? | 11:03 |
| sean-k-mooney | pvc_: overcloud compute are you doing a triple-o deployment? if so im assumeing the compute node is an ironic baremental node | 11:03 |
| pvc_ | yes it is a tripleo-deployment | 11:03 |
| *** ttsiouts has quit IRC | 11:04 | |
| sean-k-mooney | looking at the docs https://docs.openstack.org/nova/queens/admin/virtual-gpu.html i dont see anything specifically | 11:05 |
| pvc_ | i already set my flavor http://paste.openstack.org/show/732799/ | 11:06 |
| openstackgerrit | Merged openstack/nova master: Rename tempest-nova job to follow conventions https://review.openstack.org/612230 | 11:08 |
| *** yikun has quit IRC | 11:09 | |
| pvc_ | libvirtError: internal error: qemu unexpectedly closed the monitor: 2018-10-23T11:07:33.700541Z qemu-kvm: -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/9d0df47b-1fc5-4868-bd39-3cf301918e78,bus=pci.0,addr=0x5: vfio error: 9d0df47b-1fc5-4868-bd39-3cf301918e78: error getting device from group 58: Input/output error | 11:09 |
| pvc_ | sean-k-mooney http://paste.openstack.org/show/732800/ | 11:10 |
| sean-k-mooney | pvc_: that look like you have having issue with your iommu config | 11:10 |
| openstackgerrit | Merged openstack/nova master: Use assertRegex instead of assertRegexpMatches https://review.openstack.org/611608 | 11:11 |
| sean-k-mooney | just to check a few things you are useing kvm as the hyptervior correct | 11:11 |
| sean-k-mooney | can you show me your kernel commandline | 11:11 |
| sean-k-mooney | you should have intel_iommu=on and iommu=pt set | 11:11 |
| sean-k-mooney | similarly you should have vt-d enabled in the bios asd you would for sriov | 11:12 |
| pvc_ | http://paste.openstack.org/show/732802/ sean-k-mooney | 11:12 |
| *** tetsuro has quit IRC | 11:12 | |
| sean-k-mooney | but my guess is that the slot the gpu is in is shareing an iommu group with another device | 11:13 |
| sean-k-mooney | pvc_: add iommu=pt to the host as a starting point. | 11:13 |
| pvc_ | wait | 11:13 |
| pvc_ | i add it then reboot | 11:13 |
| sean-k-mooney | after you regenerate your grub file yes | 11:14 |
| pvc_ | i dont need the vfio_pci right? | 11:14 |
| *** udesale has quit IRC | 11:14 | |
| sean-k-mooney | you do | 11:14 |
| pvc_ | vfio-pci module? | 11:14 |
| sean-k-mooney | mdev stands for vfio mediated device | 11:14 |
| pvc_ | i disable it since i cannot install the NVIDIA-kvm driver | 11:15 |
| sean-k-mooney | if you disable it then kvm cannot pass through any device that depend on the kernel vfio framework which included mdevs | 11:16 |
| pvc_ | okay i'll set it back | 11:16 |
| sean-k-mooney | did you follow instruction that said you should disable it from nvidia? | 11:16 |
| pvc_ | but my compute node already have the nvidia driver | 11:16 |
| pvc_ | because earlier i cant install the nvidia-driver | 11:16 |
| pvc_ | after disabled the vfio_pci, i successfully installed it | 11:17 |
| pvc_ | i will get it back | 11:17 |
| sean-k-mooney | so looking at https://images.nvidia.com/content/grid/pdf/GRID-vGPU-User-Guide.pdf in section 3.1 it does not say anything about disableing vfio so i would guess that is the issue | 11:18 |
| pvc_ | i will get it back sean-k-mooney wait | 11:18 |
| pvc_ | so install the driver then enable it. I enable it first after installing the driver | 11:19 |
| pvc_ | sean-k-mooney can i add all the available vgpu on the nova.conf? | 11:22 |
| pvc_ | then create a flavor that have 1 VGPU, 2 VGPU, 3 VGPU? | 11:23 |
| sean-k-mooney | pvc_: you can only have 1 vgpu per vm currently and you can only enable 1 vgpu mdev type per phyical host | 11:25 |
| sean-k-mooney | part of the limitation comes form libvirt and part form nvidia. multiple mdevs can be attached to the same instance but its a rather new addtion to the kernel and things have not really mautred yet in libvirt/qemu | 11:26 |
| pvc_ | so i cannot use the 24gpus on my physical host? | 11:26 |
| jangutter | sean-k-mooney: minor correction, intel_iommu=on is required, and "iommu=pt" is discouraged. | 11:27 |
| *** panda is now known as panda|lunch | 11:27 | |
| jangutter | sean-k-mooney: iommu=pt used to be required when DPDK hadn't set vfio-pci as it's default transport yet, and people used uio. | 11:28 |
| *** ratailor has quit IRC | 11:29 | |
| sean-k-mooney | jangutter: yes but i still recommend iommu=pt as the iommu is picky somethimes and i find you hit less corner cases if you limit its scope to pasthrough devices | 11:29 |
| pvc_ | ERROR nova.compute.manager [instance: 014fcc5f-660b-40df-92ed-9f4587993fa7] Verify all devices in group 58 are bound to vfio-<bus> or pci-stub and not already in use | 11:30 |
| sean-k-mooney | pvc_: yes so as i seaid previously i think your error is the gpus is sharingin an iommu group with anothe rdevice | 11:31 |
| sean-k-mooney | pvc_: all devices in a iommu group must use the same kernel driver | 11:31 |
| jangutter | sean-k-mooney: heh, iommu=pt is one of the worst-named options. The "passthrough" mapping it enables is a way to 'bypass' the IOMMU by creating a 1:1 memory map for the PCI space. | 11:31 |
| pvc_ | how can i check that sean-k-mooney? | 11:31 |
| sean-k-mooney | jangutter: yes and that 1:1 mapping fixes so manny things :P | 11:31 |
| jangutter | pvc_: yeah, you have to hand off _all_ the devices in an iommu group, and some platforms can't divide between them. | 11:32 |
| sean-k-mooney | pvc_: am you can find this in sysfs | 11:32 |
| pvc_ | sysfs? | 11:32 |
| sean-k-mooney | let me see if i can remember | 11:32 |
| pvc_ | thank you | 11:32 |
| pvc_ | jangutter i cannot use the 12vgpus of my 1 tesla? | 11:32 |
| pvc_ | only 1 vgpus? | 11:32 |
| *** jpena is now known as jpena|lunch | 11:33 | |
| jangutter | pvc_: it depends, the chipset may allow you to pass the entire card at once, but not a portion of it. | 11:33 |
| sean-k-mooney | pvc_: you can but if your tesla share an iommu group with a nic then they both need to be bound to vfio-pci or pci-stub | 11:34 |
| jangutter | pvc_: the kernel documentation (low level warning) is at: https://www.kernel.org/doc/Documentation/vfio.txt | 11:34 |
| pvc_ | thankyou jangutter, how can i do that sean-k-mooney? | 11:34 |
| pvc_ | sorry this is my first time using vgpu, im using just the pci-passthroigh | 11:34 |
| sean-k-mooney | first we need to see what is in iommu group 58 in /sys/class/iommu/ | 11:35 |
| jangutter | pvc_: you can do something like: ls -l /sys/bus/pci/devices/0000:06:0d.0/iommu_group/devices | 11:35 |
| jangutter | pvc_: where the pci address is obviously yours. | 11:35 |
| jangutter | pvc_: that's a list of PCI devices in one group | 11:35 |
| jangutter | pvc: they _all_ have to be passed through together, or it will fail. | 11:36 |
| pvc_ | lrwxrwxrwx. 1 root root 0 Oct 23 11:36 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:02.0/0000:06:00.0 | 11:36 |
| jangutter | Also check /sys/kernel/iommu_groups/58 ? | 11:37 |
| openstackgerrit | Elod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | 11:37 |
| pvc_ | 979d010e-17a3-4ac9-987a-565f9ba4b4a6 | 11:38 |
| pvc_ | [root@overcloud-novacompute-0 iommu]# ls /sys/kernel/iommu_groups/58/devices/ 979d010e-17a3-4ac9-987a-565f9ba4b4a6 | 11:38 |
| jangutter | pvc_: interesting... I haven't seen a UUID there yet. can you ls -l it? | 11:39 |
| sean-k-mooney | jangutter: my guess is the uuid is a mdev uuid | 11:39 |
| pvc_ | http://paste.openstack.org/show/732803/ | 11:39 |
| pvc_ | mdev bus types plus driver http://paste.openstack.org/show/732805/ | 11:41 |
| *** janki has quit IRC | 11:41 | |
| *** claudiub has joined #openstack-nova | 11:41 | |
| claudiub | heyo. since it's spec review day, could you take a look again at the live-resize one? https://review.openstack.org/#/c/141219/ | 11:42 |
| jangutter | pvc_, sean-k-mooney: should the PCI passthrough libxml element set "managed=true"? | 11:43 |
| jangutter | sean-k-mooney: managed=true generally means that it will auto-bind vfio-pci to the device before attempting passthrough? | 11:43 |
| bauzas | jangutter, sean-k-mooney: context ? | 11:43 |
| sean-k-mooney | jangutter: sorry where was the libvirt xml i missed that | 11:43 |
| *** ttsiouts has joined #openstack-nova | 11:43 | |
| sean-k-mooney | bauzas: trying to help pvc_ with there vgpu issue | 11:43 |
| bauzas | and? | 11:44 |
| sean-k-mooney | bauzas: pvc_ is seeing a weird iommu error | 11:44 |
| jangutter | hang on, looking up the libxml doc. | 11:44 |
| bauzas | we don't manage the iommu group | 11:44 |
| *** cdent has quit IRC | 11:45 | |
| jangutter | (rofl) s/libxml/libvirt/ | 11:45 |
| sean-k-mooney | bauzas: yes that is managed by the uefi and kernel | 11:45 |
| *** eharney has joined #openstack-nova | 11:45 | |
| *** markvoelker has quit IRC | 11:45 | |
| bauzas | oh wait | 11:46 |
| jangutter | sean-k-mooney: when doing https://libvirt.org/formatdomain.html#elementsHostDevSubsys <--- there's a 'managed=yes' element in the xml. if that's set it will do the vfio-pci binding for you. | 11:46 |
| bauzas | is pvc_ doing PXI | 11:46 |
| sean-k-mooney | bauzas: pvc_ is getting Verify all devices in group 58 are bound to vfio-<bus> or pci-stub and not already in use | 11:46 |
| bauzas | pci passthrough? | 11:46 |
| pvc_ | im using vgpu bauzas | 11:47 |
| sean-k-mooney | bauzas: no pvc_ is trying to do vgpu passthrouhg not pci | 11:47 |
| sean-k-mooney | bauzas: this is there flavor http://paste.openstack.org/show/732799/ | 11:47 |
| bauzas | sec, GPU passthrough? | 11:47 |
| sean-k-mooney | and pvc_ has set the gpu type in the config to nvidia-160 | 11:47 |
| bauzas | I'm confused | 11:48 |
| jangutter | pvc_: what does "readlink /sys/bus/pci/devices/0000:06:00.0" say? | 11:48 |
| bauzas | virtual GPU or GPU passthrough? | 11:48 |
| *** ttsiouts has quit IRC | 11:48 | |
| sean-k-mooney | bauzas: pvc_ has a tesla gp100 and is trying to ues the mdev based virtual gpu | 11:48 |
| bauzas | then don't do vfio bus | 11:49 |
| bauzas | or pci stub | 11:49 |
| bauzas | just use the nvidia kernel module | 11:50 |
| jangutter | aaah, the penny drops. | 11:50 |
| sean-k-mooney | pvc_: can you provide the libvirt xml that nova generated so we can see what its doing | 11:50 |
| pvc_ | i add an option of options vfio-pci ids=10de:15f8 should i disable this? | 11:50 |
| pvc_ | then new error is occured | 11:50 |
| bauzas | sean-k-mooney : I think pvc_ is mixing two different things | 11:50 |
| pvc_ | 2018-10-23 11:49:46.754 7 WARNING nova.virt.libvirt.driver [req-a3570604-eed3-4f8f-a244-202ac2b92b7d - - - - -] Error from libvirt while getting description of instance-00000001: [Error Code 42] Domain not found: no domain with matching uuid '2ac6c395-5f92-4e9e-a52a-cf90b9d551c5' (instance-00000001): libvirtError: Domain not found: no domain with matching uuid '2ac6c395-5f92-4e9e-a52a-cf90b9d551c5' (instance-00000001) | 11:50 |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709 | 11:50 |
| sean-k-mooney | pvc_: you should not be disableing or foceing vfio-pci | 11:51 |
| sean-k-mooney | you should allow it to be loaded if need but oterwise do not set any options for the vfio kernel module at all | 11:52 |
| pvc_ | http://paste.openstack.org/show/732806/ | 11:53 |
| bauzas | yup this | 11:54 |
| sean-k-mooney | pvc_: the phyical gpu needs to be bound to the nvidia grid driver and the mdevs will be created via the vfio framwork in the kenel but you should not force the pgpu to use vfio-pci | 11:54 |
| pvc_ | i need to remove that? | 11:54 |
| bauzas | yup it's only for GPU passthrough | 11:54 |
| pvc_ | okay wait | 11:54 |
| bauzas | hence my confusion | 11:54 |
| pvc_ | i'll remove | 11:54 |
| sean-k-mooney | bauzas: im guessing the driver in use should be nvidia_vgpu_vfio or nvida correct | 11:54 |
| sean-k-mooney | praobly nvidia_vgpu_vfio | 11:55 |
| pvc_ | i reboot again the hypervisor wait | 11:55 |
| bauzas | sean-k-mooney, I don't remember the module name but yeah something like that | 11:56 |
| pvc_ | bauzas is it possible to use the 12vgpus of my gpu? | 11:57 |
| bauzas | FWIW, I'll have connection issues this afternoon due to some planned outage in my street | 11:57 |
| sean-k-mooney | well of the 3 nouveau, nvidia_vgpu_vfio, nvidia. nouveau is the opensouce driver for the pgpu, nvidia is the binary driver from nvida for the same so that just leaves nvidia_vgpu_vfio | 11:58 |
| sean-k-mooney | pvc_: that depends on the mdev type you selected. but you should be able to however you can only request 1 vgpu per guest currently | 11:59 |
| pvc_ | 06:00.0 3D controller [0302]: NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB] [10de:15f8] (rev a1) Subsystem: NVIDIA Corporation Device [10de:118f] Kernel driver in use: nvidia | 12:00 |
| pvc_ | it's already nvidia | 12:00 |
| pvc_ | http://paste.openstack.org/show/732807/ | 12:00 |
| sean-k-mooney | pvc_: yes but it may need to be nvidia_vgpu_vfio. nvidia is gust the normal binary driver for using the gpu on the host | 12:01 |
| sean-k-mooney | best thing to do is try and boot a vm and see what happens | 12:01 |
| pvc_ | same error :( http://paste.openstack.org/show/732808/ | 12:02 |
| *** ttsiouts has joined #openstack-nova | 12:02 | |
| pvc_ | http://paste.openstack.org/show/732808/ bauzas and sean-k-mooney | 12:02 |
| *** spatel has joined #openstack-nova | 12:04 | |
| sean-k-mooney | pvc_: in this case try unbinding the card from nvdia driver and bind it to nvidia_vgpu_vfio | 12:04 |
| pvc_ | noted on this | 12:04 |
| pvc_ | i add this on module | 12:04 |
| pvc_ | options nvidia_vgpu_vfio ids=10de:15f8 | 12:05 |
| sean-k-mooney | can yo bind it by hand instad of via the moduel file to test it | 12:05 |
| pvc_ | how can i bind it? i'm sorry im not done it before | 12:05 |
| bauzas | that's super weird | 12:05 |
| pvc_ | this is my conf | 12:06 |
| pvc_ | http://paste.openstack.org/show/732809/ | 12:06 |
| *** dave-mccowan has joined #openstack-nova | 12:07 | |
| bauzas | pvc_ did you remove the nouveau driver ? | 12:07 |
| sean-k-mooney | echo 06:00.0 | sudo tee /sys/bus/pci/drivers/nvidia/unbind | 12:07 |
| sean-k-mooney | echo 06:00.0 | sudo tee /sys/bus/pci/drivers/nvidia_vgpu_vfio/bind | 12:07 |
| pvc_ | [root@overcloud-novacompute-0 nova]# lsmod | grep nou [root@overcloud-novacompute-0 nova]# | 12:07 |
| pvc_ | yes bauzas | 12:07 |
| pvc_ | tee: /sys/bus/pci/drivers/nvidia/unbind: No such device | 12:08 |
| *** spatel has quit IRC | 12:08 | |
| pvc_ | I install this driver | 12:09 |
| pvc_ | NVIDIA-Linux-x86_64-390.72-vgpu-kvm.run | 12:09 |
| bauzas | I need to drop, planned outage here | 12:09 |
| sean-k-mooney | pvc_: from http://paste.openstack.org/show/732807/ that should have been the driver in use | 12:10 |
| pvc_ | i use this 0000:06:00.0 | 12:10 |
| pvc_ | tee: /sys/bus/pci/drivers/nvidia_vgpu_vfio/bind: No such file or directory | 12:10 |
| pvc_ | it is already unbind | 12:10 |
| pvc_ | no nvidia_vgpu_vfio on drivers | 12:11 |
| pvc_ | just nvidia | 12:11 |
| *** janki has joined #openstack-nova | 12:14 | |
| sean-k-mooney | pvc_: this is the latest verion of the nvdia vgpu user guide https://docs.nvidia.com/grid/5.0/pdf/grid-vgpu-user-guide.pdf i think you need to back through it and section 4.2 specifically | 12:14 |
| pvc_ | sean-k-mooney im using a ubuntu image with img_hide_hypervisor_id='true' | 12:14 |
| sean-k-mooney | pvc_: i asked thi earliar but is the compute node a phyical server or a vm | 12:15 |
| pvc_ | compute node is a physical server | 12:15 |
| pvc_ | on docs it said the grid driver | 12:15 |
| pvc_ | but this is the driver i installed | 12:15 |
| pvc_ | installed NVIDIA-Linux-x86_64-390.72-vgpu-kvm.run | 12:16 |
| pvc_ | i have this grid driver NVIDIA-Linux-x86_64-390.75-grid.run | 12:16 |
| sean-k-mooney | pvc_: oh ok i think i understand the issue then | 12:17 |
| sean-k-mooney | you install the guest driver on the host | 12:17 |
| pvc_ | yes sean-k-mooney | 12:17 |
| pvc_ | to enable this | 12:17 |
| pvc_ | /sys/class/mdev_bus/*/mdev_supported_types | 12:17 |
| pvc_ | i install the driver on my compute node ( baremetal server ) im using a tripleo-deployment | 12:18 |
| *** tbachman has joined #openstack-nova | 12:20 | |
| aperevalov | hello, do nova or neutron has functional test for direct (SR-IOV) port (something like tempest test)? | 12:22 |
| pvc_ | sean-k-mooney what will i do then? | 12:22 |
| sean-k-mooney | aperevalov: i dont belive so in upstream tempest. neutron may have fullstack test but ingerneral our sriov testing is limited | 12:23 |
| *** abhi89 has quit IRC | 12:23 | |
| *** tbachman_ has joined #openstack-nova | 12:24 | |
| *** cdent has joined #openstack-nova | 12:24 | |
| sean-k-mooney | pvc_: this seams to be a driver issue not a nova one. there is little more advice i can give other then they to follow the nvida docs form start to finish exactly | 12:24 |
| aperevalov | sean-k-mooney: I assume, if such functional test exists it uses real HW with SR-IOV, but not simulators or emulators. | 12:25 |
| *** tbachman has quit IRC | 12:25 | |
| *** tbachman_ is now known as tbachman | 12:25 | |
| pvc_ | so you think i installed the grid one? | 12:25 |
| *** janki has quit IRC | 12:25 | |
| pvc_ | my baremetal is centos 7 | 12:25 |
| sean-k-mooney | aperevalov: so the fact we can use simulator/emulators for sriov is the main reason we have such limited testing | 12:25 |
| *** janki has joined #openstack-nova | 12:25 | |
| sean-k-mooney | pvc_: yes you need to install the grid one | 12:25 |
| sean-k-mooney | at least i think so. | 12:26 |
| pvc_ | okay wait | 12:26 |
| sean-k-mooney | aperevalov: i recently looked at using the netdevsim kernel module for sriov testing but it only emulates the kernel netdevs it does not emulate the pci devices or virtual function so we cant use it for testing | 12:27 |
| *** janki has quit IRC | 12:27 | |
| aperevalov | sean-k-mooney: I checked netdevsim on the latest kernel too, there are no device to put it into docker container. Docker container it's because I'm doing such research for kuryr-kubernetes. | 12:28 |
| aperevalov | sean-k-mooney: yes netdevsim is based on its own bus, but not pci. I also found attemp to submit to QEMU's pci emulation the SR-IOV support. | 12:29 |
| sean-k-mooney | aperevalov: yes so there is a netdev but no pci device on the virtual pci device. so you can use ip link and allocate vfs via sysfs but they dont show up on the virtual pci bus | 12:30 |
| sean-k-mooney | aperevalov: yes i have see that in the past but that wont help us with testing as we would need the hosting vms provided by the could providers to emulate pci device that support sriov | 12:31 |
| aperevalov | sean-k-mooney: but it was postponed due to lack of existing working qemu drivers, initial author suggested copied e1000, but it wasn't in working condition. | 12:31 |
| *** udesale has joined #openstack-nova | 12:31 | |
| sean-k-mooney | aperevalov: so first qemu would have to be extended then libvirt and then nova. once that is done we would need cloud providers to proved devices with virt sriov capable nics then we could start using it in the upstream gate | 12:32 |
| sean-k-mooney | *provided vms with ... | 12:32 |
| sean-k-mooney | aperevalov: effectivly we rely on thirdpart cis to test sriov. either intels or melonox's ci | 12:33 |
| sean-k-mooney | intels ci was intented to have signifcatily more sriov testing then it currently has but that never happened | 12:33 |
| *** janki has joined #openstack-nova | 12:34 | |
| aperevalov | sean-k-mooney, I see it's a long way, and seems netdevsim (just kernel module) looks like easiest (if kernel community will be agreed to bind it with pci bus). | 12:34 |
| sean-k-mooney | aperevalov: most people dont know that sr-iov is a specificaiton from the PCI-SIG i dont think the current netdevsim moudle is technically a confroming sriov implentaiton without the pci emulation | 12:35 |
| pvc_ | sean-k-mooney ls: cannot access /sys/class/mdev_bus/*/mdev_supported_types: No such file or directory | 12:36 |
| sean-k-mooney | aperevalov: that said its a netdev simulator not a sriov simulator so they skipped the bits they did not need for there own testing | 12:36 |
| *** markvoelker has joined #openstack-nova | 12:36 | |
| pvc_ | i reboot again to reflect the new driver | 12:36 |
| *** markvoelker has quit IRC | 12:37 | |
| *** jchhatbar has joined #openstack-nova | 12:38 | |
| aperevalov | sean-k-mooney: ok, I'll talk with authors of netdevsim. Thank you for information. BTW is intel or mellanox ci is publicly available. Or that work is going through their teems involved into openstack community? | 12:38 |
| *** jamesdenton has joined #openstack-nova | 12:39 | |
| *** janki has quit IRC | 12:40 | |
| *** jpena|lunch is now known as jpena | 12:40 | |
| aperevalov | sean-k-mooney: I think moshele knows it. | 12:40 |
| *** brinzh has quit IRC | 12:40 | |
| pvc_ | hi sean-k-mooney | 12:41 |
| moshele | aperevalov: I know what? | 12:41 |
| pvc_ | after installign the grid driver the mdev is gone | 12:41 |
| sean-k-mooney | so i used to work at intel and one of my roles there was product owner of the intel nfv ci. it used to fall to the upstream teams to identify which features needed ci testing and either add it or request that the team maintaining it add it to there backlog | 12:41 |
| pvc_ | ls: cannot access /sys/class/mdev_bus/*/mdev_supported_types: No such file or directory | 12:41 |
| pvc_ | sean-k-mooney any ideas? ls: cannot access /sys/class/mdev_bus/*/mdev_supported_types: No such file or directory | 12:41 |
| pvc_ | after installing the grid driver only not the vgpu | 12:42 |
| sean-k-mooney | im not sure how the intel nfv ci currently works but if you reach out to the new maintianer im sure they will respond | 12:42 |
| moshele | aperevalov: The Mellanox CI is public but not part of the openstack community. and I think intel CI is the same, because it depend on nic vendor | 12:45 |
| sean-k-mooney | pvc_: sorry not really. as i said this seams to be an nvidia driver issue. i dod not have acess to the hardware to test it my self so beyond reading there docs there is not much more advice i can give | 12:45 |
| pvc_ | but sean-k-mooney when i install the kvm-vgpu i can list the mdev | 12:46 |
| aperevalov | moshele: does it use in gerrit integration tests by zuul? | 12:47 |
| moshele | aperevalov: we run tempest scenarios and configure the tempest to use vnic_type=direct | 12:48 |
| moshele | aperevalov: see https://github.com/openstack/tempest/blob/master/tempest/config.py#L628 | 12:49 |
| sean-k-mooney | moshele: oh when was that option added? | 12:50 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/rocky: fixtures: Track volume attachments within CinderFixtureNewAttachFlow https://review.openstack.org/612485 | 12:50 |
| sean-k-mooney | that is useful to know about | 12:50 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/rocky: Add regression test for bug#1784353 https://review.openstack.org/612486 | 12:50 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/rocky: conductor: Recreate volume attachments during a reschedule https://review.openstack.org/612487 | 12:50 |
| moshele | aperevalov: long time ago | 12:50 |
| moshele | sean-k-mooney: long time ago | 12:50 |
| *** jchhatbar is now known as janki | 12:50 | |
| pvc_ | vfio_iommu_type1.allow_unsafe_interrupts=1 | 12:51 |
| pvc_ | sean-k-mooney vfio_iommu_type1.allow_unsafe_interrupts=1? | 12:51 |
| sean-k-mooney | moshele: cool aperevalov the intel ci does somthing similar | 12:51 |
| sean-k-mooney | aperevalov: the intel ci uses the standard senario test but addes extraflaovr extraspecs for cpu pinning hugepages numa toplogy exctra | 12:52 |
| aperevalov | sean-k-mooney: If I trully understood, kuryr-kubernetes (tempest test) can also be running there? | 12:52 |
| sean-k-mooney | pvc_: are you getting a message in dmesg? vfio_iommu_type1.allow_unsafe_interrupts=1 is specificaly for working around old buggy hardware | 12:52 |
| sean-k-mooney | aperevalov: the intel nfv ci does not load the kuryr-kuberntese tempest module or deploy tempetst | 12:53 |
| sean-k-mooney | at least it didnt when i was invovled with it | 12:53 |
| pvc_ | Oct 23 12:53:40 overcloud-novacompute-0 journal: 2018-10-23 12:53:40.708+0000: 3128: warning : virDomainAuditHostdev:424 : Unexpected hostdev type while en | 12:53 |
| pvc_ | Oct 23 12:53:40 overcloud-novacompute-0 journal: libvirt: QEMU Driver error : Requested operation is not valid: domain is not running | 12:53 |
| sean-k-mooney | * or deply kuryr-kubernetes | 12:53 |
| sean-k-mooney | pvc_: i dont really have time to contiue debugging sorry. i need to update some review and catch up on spec review today | 12:54 |
| pvc_ | 2018-10-23 12:41:10.331+0000: 3216: error : virPCIDeviceNew:1787 : Device 0003:01:05.1 not found: could not access /sys/bus/pci/devices/0003:01:05.1/config | 12:56 |
| pvc_ | virPidFileAcquirePath:422 : Failed to acquire pid file '/var/run/libvirtd.pid': Resource temporarily unavailable | 12:57 |
| pvc_ | sean-k-mooney there is an issue on my nova_libvirt | 13:02 |
| sean-k-mooney | pvc_: ok but that is not an nova issue. its a either a libvirt or a docker/triplo issue assuming you can acess /sys/bus/pci/devices/0003:01:05.1/config from the host. | 13:05 |
| * jroll glances at channel topic | 13:06 | |
| pvc_ | there is no 0003:01:01.1 sean | 13:06 |
| *** bnemec has joined #openstack-nova | 13:08 | |
| *** psachin has quit IRC | 13:10 | |
| *** cdent has quit IRC | 13:12 | |
| *** cdent has joined #openstack-nova | 13:14 | |
| pvc_ | sean-k-mooney is libvirtd not running is not an issue? | 13:15 |
| efried | bauzas: https://review.openstack.org/#/c/612497/ <== provider config yaml file, split out from the device passthrough spec (with some of jaypipes' Rocky provider config file mixed in) | 13:17 |
| bauzas | efried: ack | 13:19 |
| bauzas | I have some planned outage this EU afternoon hence me being a bit afk but will look later tonight | 13:19 |
| pvc_ | bauzas Failed to acquire pid file '/var/run/libvirtd.pid': Resource temporarily unavailable :( | 13:20 |
| pvc_ | bauzas fio error: cad68f60-930c-4d9b-b954-3e0cd855651e: error getting device from group 58: Input/output error | 13:20 |
| pvc_ | anyone can help? | 13:33 |
| *** mriedem has joined #openstack-nova | 13:34 | |
| *** adrianc_ has quit IRC | 13:35 | |
| *** adrianc has quit IRC | 13:35 | |
| *** boden has joined #openstack-nova | 13:42 | |
| *** awaugama has joined #openstack-nova | 13:42 | |
| *** boden has left #openstack-nova | 13:42 | |
| *** liuyulong has joined #openstack-nova | 13:48 | |
| *** mlavalle has joined #openstack-nova | 13:52 | |
| pvc_ | hi sean-k-moone do i need to hide the hypervisor of the image? | 13:53 |
| pvc_ | hi sean-k-mooney do i need to hide the hypervisor of the image? | 13:53 |
| alex_xu | jaypipes: for https://review.openstack.org/#/c/555081, are you saying that the user must specify guest numa topology when using resources:PCPU=1 or resources:VCPU=1 | 13:53 |
| sean-k-mooney | alex_xu: im not suer if cpu pinning auto creates a numa toplogy today but it is does not its one of the few numa specifc things that does not | 13:54 |
| pvc_ | sean-k-mooney i have an error on my XML | 13:56 |
| pvc_ | 2bf12bf5 - default default] Error launching a defined domain with XML: <domain type='kvm'> | 13:56 |
| alex_xu | sean-k-mooney: yes, I also think that. If the flavor doesn't include any guest numa topo, then we will get a None value for the InstanceTopologyObj. But jaypipes still want to use InstnaceTopology to store the cpu pinning. that is my confuse. | 13:57 |
| sean-k-mooney | alex_xu: well cpus have numa affintiy so i would be fine with saying if your request pinning you now have a numa toploy of 1 numa node for the vm unless you set a numa toploygy explcitly | 13:58 |
| sean-k-mooney | alex_xu: we do this for hugepages | 13:58 |
| sean-k-mooney | personly i have normally argued against that but we have too much presdent to change it at this point | 13:59 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make CellDatabases fixture reentrant https://review.openstack.org/611665 | 13:59 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663 | 13:59 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785 | 13:59 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934 | 13:59 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | 13:59 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | 13:59 |
| pvc_ | is that related sean-k-mooney you think? | 13:59 |
| sean-k-mooney | pvc_: i dont know but im busy with 3 other things. i do not have time to help futher sorry. | 14:00 |
| *** edmondsw has joined #openstack-nova | 14:00 | |
| alex_xu | sean-k-mooney: yea, that should be ok, that is just a clarify I ask on the spec, since it isn't clear about that | 14:01 |
| *** cdent has quit IRC | 14:01 | |
| sean-k-mooney | alex_xu: for what its worth the free cpus are already tracked in the numa toployg blob in the nova db so i dont hink jay was proposing changing that | 14:02 |
| alex_xu | jaypipes: ^ probably that is what I'm asking, are you plan to change the guest without numa topo to single numa cell topo | 14:02 |
| jaypipes | alex_xu: *currently* there is no way for a user to get pinned CPUs without the instance_extra.numa fields containing a serialized blob of InstanceNUMATopology object. | 14:03 |
| jaypipes | alex_xu: because, as you know, we couple the CPU pinning, memory page and NUMA topology stuff all together in the InstanceNUMATopology object :( | 14:03 |
| sean-k-mooney | jaypipes: do we currently invent a singel numa node topology today. its been to long since i looked at the details of that code to rember that off the top of my head | 14:03 |
| jaypipes | alex_xu: the cpu-resource-tracking spec proposes absolutely no changes to any of that. | 14:04 |
| *** panda|lunch is now known as panda | 14:04 | |
| sean-k-mooney | jaypipes: i have you spec on my list to review but i assumed we would still contiue to do whatever we do today on that front | 14:04 |
| jaypipes | sean-k-mooney: mriedem has basically shot down the possibility of cpu-resource-tracking happening in stein anyway, so I haven't been spending much time on it. :( | 14:05 |
| alex_xu | jaypipes: yes, but your spec didn't say I must specify HW:NUMA_xxx stuff when I use resources:PCPU=1 | 14:05 |
| mriedem | once again i am the killer of all hopes and dreams and kittens | 14:05 |
| jroll | always and forever | 14:06 |
| mriedem | if others think we can pull that change off in stein and are planning on devoting review time to it, then go nuts | 14:06 |
| jaypipes | alex_xu: I was asked by bauzas to take all NUMA stuff out of the cpu-resource-tracking spec so he could address it in his numa spec. | 14:06 |
| *** Luzi has quit IRC | 14:06 | |
| alex_xu | jaypipes: and what does mean for CONF.shared_cpu_set, it is for the VCPU will pinning to the CPU set of CONF.shared_cpu_set, and then I must specify HW:NUMA_xxx with resources:VCPU=1? | 14:06 |
| jaypipes | alex_xu: I don't understand your question. could you rephrase? | 14:08 |
| alex_xu | let me try :) | 14:08 |
| jaypipes | alex_xu: CONF.cpu_shared_set already exists, btw | 14:08 |
| sean-k-mooney | alex_xu: jaypipes so jsut looking at https://github.com/openstack/nova/blob/297de7fb9fbabe74b5305ef0aa82e196d5f48d5e/nova/virt/hardware.py#L1543-L1554 we create a singel node numa toplogy for the guest today if using pinning unless you say otherwise | 14:09 |
| jaypipes | sean-k-mooney: right, because we've coupled CPU pinning and NUMA together into the InstanceNUMAToplogy object. | 14:09 |
| sean-k-mooney | alex_xu: so if you just set resources:PCPU=1 then i would assume we would create a singel numa toplogy | 14:09 |
| *** moshele has quit IRC | 14:10 | |
| sean-k-mooney | jaypipes: ya. if you want to decouple them and fix it im happy with that idea too | 14:10 |
| jaypipes | sean-k-mooney: mriedem would never approve such a gigantic change. :P | 14:10 |
| alex_xu | sean-k-mooney: no, we return early at https://github.com/openstack/nova/blob/297de7fb9fbabe74b5305ef0aa82e196d5f48d5e/nova/virt/hardware.py#L1538, actualy it is NOne | 14:10 |
| sean-k-mooney | alex_xu: that for shared cpus | 14:11 |
| jaypipes | alex_xu: resources=VCPU:1 does not equal cpu_policy:shared | 14:11 |
| sean-k-mooney | pinned cpus have cpu_policy==dedicated | 14:11 |
| jaypipes | alex_xu: just another example of terrible coupling in this interface :( | 14:12 |
| jaypipes | if cpu_policy == fields.CPUAllocationPolicy.SHARED: | 14:12 |
| jaypipes | ^^ that is not the same as resources=VCPU:1 | 14:12 |
| alex_xu | jaypipes: your spec is about decouple cpu pinning and numa. so conf.cpu_shared_set defined the pcpus which the shared VCPU is running. if the conf.cpu_shared_set=7-15, dose it means nova-compute will pin the guest vcpus to the physical cpu 7 to 15? | 14:12 |
| sean-k-mooney | alex_xu: it will float them over that rage | 14:13 |
| sean-k-mooney | *range | 14:13 |
| sean-k-mooney | but it wont 1:1 pin the shared cpus | 14:13 |
| jaypipes | alex_xu: no, my spec is not about decoupling CPU pinning and NUMA... my spec is about handling the allocation of dedicated CPUs in a deliberate way. My spec does not touch assignment of host processor to guest vCPU thread, which is what you are referring to. | 14:13 |
| sean-k-mooney | that will be left to the kernel | 14:13 |
| alex_xu | sean-k-mooney: jaypipes so for the request resources:VCPU=1, this VCPU sitll can running on the pcpu which defined in conf.cpu_dedicated_set... | 14:15 |
| sean-k-mooney | alex_xu: with jays spec no. if i rememebre correctly jay was proposeing depercating the hw:cpu_policy extra spec and vcpu would courrespond to the shared set and PCPU reousfce woudl be from dedicated set | 14:16 |
| jaypipes | alex_xu: if the virt driver isn't changed to assign one of the dedicated host CPUs, yep. But from placement (and resource tracking) perspective, we don't care about that. All we care about is that some amount of dedicated (or shared) CPU resources are being deducted from the appropriate inventory of that class of resource (either VCPU or PCPU) | 14:17 |
| sean-k-mooney | alex_xu: jays spec is basicaly discribing how we will keep a tally count of PCPU and VCPU in placement | 14:18 |
| sean-k-mooney | the asiginment of vms to host dedicated or shared cpu sets will be handeled by the virt driver not placment using the exisitng numa toplogy blob in the nova db as we do today | 14:19 |
| sean-k-mooney | placement will just make sure we have enough cpus to fulltile the request without tracking which ones are free | 14:19 |
| sean-k-mooney | thats the virt driver/ resouce trakers jobs | 14:19 |
| jaypipes | sean-k-mooney: and yes, you're right that my spec proposes deprecating the cpu_policy extra spec. | 14:19 |
| sean-k-mooney | i think i left a comment about may using it to translate the flavor VCPU filed into resouces:VCPU=X or resources:PCPU=x to ease transition but long term it would nolonger be needed | 14:21 |
| alex_xu | ah....I probably I see...give me more seconds... | 14:21 |
| *** janki has quit IRC | 14:21 | |
| mriedem | is tpatil intel? | 14:24 |
| mriedem | oh NTT | 14:24 |
| *** tiendc has quit IRC | 14:24 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova stable/rocky: Move live_migration.pre.start to the start of the method https://review.openstack.org/612714 | 14:25 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova stable/rocky: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612715 | 14:25 |
| artom | mriedem, ^^ it has begun *dun dun dun* | 14:25 |
| pvc_ | hi anyone | 14:26 |
| pvc_ | how can i remove a pci devices? | 14:26 |
| pvc_ | nova_libvirt searching for it but it is not existing | 14:26 |
| *** cdent has joined #openstack-nova | 14:27 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Fix test bug when host doesn't have /etc/machine-id https://review.openstack.org/612717 | 14:28 |
| *** alexchadin has quit IRC | 14:30 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733 | 14:32 |
| openstack | bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9) | 14:32 |
| alex_xu | sean-k-mooney: jaypipes with that spec, the request with resources:PCPU=1 and without any HW:NUMA_.. stuff, that vcpu is also floating on all the pcpus? | 14:32 |
| alex_xu | since that spec is only about the counting pcpu and vcpu... | 14:33 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846 | 14:33 |
| *** medberry has joined #openstack-nova | 14:34 | |
| mriedem | pvc_: just fyi, today is a spec review sprint in nova so most people are busy with that. you could try asking your questions in the #openstack or #openstack-operators channels. for pci passthrough questions i'd normally direct you to sahid or cfriesen or moshele but none of them are online right now. | 14:35 |
| mriedem | i'd also think that excluding the pci devices you don't want to expose from https://docs.openstack.org/nova/latest/configuration/config.html#pci.passthrough_whitelist would work, but i don't know a lot about that code | 14:35 |
| mriedem | pvc_: you could also post a question to the openstack-dev mailing list | 14:35 |
| mriedem | if this is a common problem and we don't have documentation for it, then we should have a docs bug | 14:36 |
| pvc_ | thank you so much | 14:36 |
| pvc_ | i will do that | 14:36 |
| mriedem | yw | 14:36 |
| jaypipes | alex_xu: yes | 14:38 |
| jaypipes | alex_xu: since the hw:numa_xxx tags are currently the only way to trigger any pinning behaviour. | 14:39 |
| sean-k-mooney | well not quite you can use hw:cpu_policy | 14:39 |
| alex_xu | jaypipes: ok, i see now, then in the future, we want that case work correctly, right? | 14:39 |
| jaypipes | alex_xu: and since my spec doesn't propose any changes to that, then the existing behaviour if an instance does not have the hw:numa_xxx specs means its vCPU threads float over whatever host processors are in CONF.cpu_shared_set. | 14:39 |
| alex_xu | sean-k-mooney: yea, without hw_cpu_polciy also | 14:40 |
| sean-k-mooney | alex_xu: you should assume that if you have resouce:PCPU=X the virt drive will pin those cores but how it does that is not really realted to jays spec | 14:40 |
| jaypipes | sean-k-mooney: you should NOT assume that. | 14:40 |
| sean-k-mooney | jaypipes: why that was the prerequeit for deprecating hw:cpu_policy | 14:41 |
| jaypipes | sean-k-mooney: the only thing that guarantees assignment to a particular host CPU is the presence of hw:numa_xxx specs | 14:41 |
| sean-k-mooney | the hw:numa_xxx specs today do not do that | 14:41 |
| jaypipes | sean-k-mooney: that is a change that the virt driver will need to make, yes. but that change isn't part of my spec... | 14:41 |
| *** hamdyk has quit IRC | 14:43 | |
| alex_xu | so...probably we need to doc at somewhere for the user, resources:PCPU doesn't means you get a dedicated cpu for your guest... | 14:43 |
| sean-k-mooney | alex_xu: if we deprecated hw:cpu_policy we dont need to because it will. if we dont then yes | 14:44 |
| sean-k-mooney | i guess we should document it in either case but the point being that if the only way to remove hw:cpu_policy is to either make resources:PCPU mean the virt driver will pin you or add a trait for pinned cpus but that seams dumb | 14:47 |
| sean-k-mooney | from a placement point of view it does not care care if you are pinned or not | 14:48 |
| sean-k-mooney | its jsut a resouce class the fact that we are giving it special semantic is a nova thing not placement | 14:48 |
| alex_xu | sean-k-mooney: I see now | 14:49 |
| *** ccamacho has quit IRC | 14:57 | |
| *** hamzy has quit IRC | 15:04 | |
| melwitt | 15:14 | |
| *** ccamacho has joined #openstack-nova | 15:17 | |
| *** ccamacho has quit IRC | 15:17 | |
| *** k_mouza has quit IRC | 15:18 | |
| *** k_mouza has joined #openstack-nova | 15:18 | |
| alex_xu | sean-k-mooney: jaypipes thanks for helping me understand correctly, I see now. I leave my to 0 now, still not sure we let resources:PCPU works as that, that confuses for the end user. maybe we should set cpu policy to dedicated in numa implement when only have resources:PCPU, but we deprecate and remove the cpu_policy extra spec. so not sure | 15:24 |
| mriedem | Kevin_Zheng: the detach/attach root volume on stopped instance spec might have applications in the rescue a volume-backed instance spec https://review.openstack.org/#/c/532410/ | 15:24 |
| mriedem | just FYI | 15:24 |
| * alex_xu ends his spec review day... | 15:24 | |
| mriedem | alex_xu: o/ | 15:24 |
| alex_xu | mriedem: enjoy~ | 15:25 |
| mriedem | oh you know i will :) | 15:27 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Migrate "reboot an instance" user guide docs https://review.openstack.org/612730 | 15:28 |
| *** dtantsur is now known as dtantsur|brb | 15:32 | |
| *** hamzy has joined #openstack-nova | 15:38 | |
| *** mvkr has quit IRC | 15:39 | |
| *** liuyulong is now known as liuyulong|away | 15:41 | |
| pvc_ | hi guys can i ask? So if ever i want to use vgpu i can just use one vgpu on enabled_vgpu_types = nvidia-35, then how about its performance if i launch 5 instance on that flavor? | 15:41 |
| *** ivve has joined #openstack-nova | 15:42 | |
| mriedem | bauzas: ^ | 15:44 |
| melwitt | dansmith: could you take a look at this bug fix for a wrong pike compute rpc api alias causing problems with rolling upgrade pike => queens? https://review.openstack.org/612231 needs a +W | 15:44 |
| pvc_ | is there a way that i can use all the vgpus of my gpu considering this on docs As of the Queens release, Nova only supports a single type. If more than one vGPU type is specified (as a comma-separated list), only the first one will be used. | 15:49 |
| dansmith | melwitt: yep done, looks legit | 15:53 |
| melwitt | dansmith: ty | 15:54 |
| *** helenafm has quit IRC | 15:56 | |
| melwitt | pvc_: I can't comment on the performance but the limitation on vGPU type means having multiple enabled_vgpu_types on one compute host. at present, you can't have more than one type on the same compute host | 15:56 |
| melwitt | we are working on adding support for multiple types this cycle | 15:57 |
| pvc_ | thank you melwitt, i can launch many instance on flavor with gpu resources but im worried on the performance since we will install an gpu application on it. | 15:58 |
| sean-k-mooney | pvc_: be aware that supporting vGPU types on the same host will not enable multipel gpus to be consumed by a singel vm | 16:02 |
| pvc_ | i login to instance then query the nvidia, may i know if this is the driver that is need to be show? 0:05.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:15f8] (rev a1) | 16:03 |
| *** tbachman has quit IRC | 16:04 | |
| openstackgerrit | Jan Gutter proposed openstack/os-vif master: Extend port profiles with datapath offload type https://review.openstack.org/572081 | 16:05 |
| *** udesale has quit IRC | 16:06 | |
| *** tbachman has joined #openstack-nova | 16:06 | |
| *** mrch has quit IRC | 16:09 | |
| *** ttsiouts has quit IRC | 16:16 | |
| *** gyee has joined #openstack-nova | 16:16 | |
| *** ttsiouts has joined #openstack-nova | 16:17 | |
| *** ttsiouts has quit IRC | 16:21 | |
| *** cdent has quit IRC | 16:33 | |
| melwitt | dansmith: I've been meaning to ask you if you could review our cycle priorities doc where I've written down themes https://review.openstack.org/609807 | 16:36 |
| pvc_ | melwitt but it is possible to launch many instance on that flavor even if it just only one right? | 16:37 |
| *** pvradu_ has quit IRC | 16:37 | |
| melwitt | pvc_: yes, it is. it's just that all the instances on the same compute host will have to use the same gpu type, for example nvidia-35 | 16:38 |
| *** cdent has joined #openstack-nova | 16:38 | |
| *** moshele has joined #openstack-nova | 16:38 | |
| dansmith | mriedem: especially given it's spec day, could you circle back to this at some point? https://review.openstack.org/#/c/609709/3 | 16:38 |
| mriedem | do i have to? | 16:39 |
| pvc_ | so that is an issue on performance, so if ever i have 24 gpus on my compute node i can only use 1 right? | 16:39 |
| dansmith | melwitt: ack | 16:40 |
| *** k_mouza has quit IRC | 16:40 | |
| dansmith | mriedem: no, but you were the last to -1 it, I asked a question which you never answered, and jroll has updated it | 16:40 |
| mriedem | ok i hadn't seen, nor was looking | 16:41 |
| mriedem | i didn't pay attention to any of this at the PTG, so if there are more details from the ptg on alternatives and such, those should be in the spec, including caveats about what happens if the conductor group for a node is updated (as i had several questions about that) | 16:42 |
| mriedem | it sounds like, well that might work automagically, or it might not, shrug | 16:42 |
| mriedem | and i guess we don't want anything in the hypervisors API for this because it would be too ironic specific... | 16:44 |
| melwitt | pvc_: hm, looks like you're right, it says only one vGPU per instance. I don't know why that is limited https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#configure-a-flavor-controller | 16:44 |
| melwitt | bauzas ^ | 16:44 |
| sean-k-mooney | melwitt: its limited because libvirt cannot support multiple vgpus on a singel instance currently | 16:45 |
| melwitt | sean-k-mooney: thanks | 16:45 |
| pvc_ | yes but the problem is we launch instance it will use the same vgpu enabled on the nova.conf, is there no way to use the another vgpu? | 16:45 |
| sean-k-mooney | pvc_: that is not how that works | 16:46 |
| pvc_ | it's just for sharing sean-k-mooney? | 16:46 |
| sean-k-mooney | what you are enabling in the nova.conf is the type of vgpu | 16:46 |
| dansmith | mriedem: there are probably sequencing recommendations for when you change the mappings, but I think those are docs and don't need to be in the spec in great detail, IMHO. I don't think the behaviors are really any different than just spinning up a new or stopping a hashring-balanced compute today | 16:46 |
| pvc_ | so if i use nvida-211 | 16:46 |
| sean-k-mooney | your are not enabvling a specific mdev instnce jsut the type of mdev that will be allocated when a vm requests a vgpu | 16:47 |
| pvc_ | so in terms of performance it is okay sean-k-mooney? | 16:47 |
| dansmith | mriedem: if there's more that needs to be here, let's tell them what that is and let them move on | 16:47 |
| sean-k-mooney | pvc_: mdevs are not shared between instances. so the perfromance of the vgpus will be determined by the mdev type you select | 16:48 |
| pvc_ | so if ever i have 3 mdev types ( nvidia-1, nvidia-2, nvidia-3), i use the nvidia-1 on nova.conf. What will happen to the 2 mdev types? | 16:48 |
| sean-k-mooney | higher perfroamce mdev types consume more physical resouce and allow less vgpus to be createed so its a tradeoff | 16:49 |
| sean-k-mooney | if you set nvidia-1 the nova will only create mdevs of the nvida-1 type | 16:49 |
| sean-k-mooney | there are stict rules about if and how mdev types can be mixed | 16:50 |
| sean-k-mooney | they are vendor specific so initally we chose not to allow mixing them | 16:50 |
| pvc_ | is that okay to be use for many instances? | 16:51 |
| sean-k-mooney | yes we will limit you to the number of mdevs that the enabled type reporst it can allocate | 16:51 |
| sean-k-mooney | there no over subsiptoin or sharing of vgpus | 16:52 |
| pvc_ | so if ever i have 24 mdev types i can launch 24 instance, correct me if im wrong. | 16:52 |
| sean-k-mooney | pvc_: no | 16:53 |
| sean-k-mooney | an mdev type is like a nove flavor. its corresponed to a set of phyical resouces form the gpu. in the nvida case its a set of max reolutions, cuda cores, and vRAM | 16:54 |
| sean-k-mooney | this is all detailed in the nvidia documentation | 16:54 |
| pvc_ | thank you sean-k-mooney | 16:55 |
| pvc_ | so if i have a chance to launch an instance is it okay and nova will tell me if ever i cannot allocate omre | 16:55 |
| sean-k-mooney | yes you will get a no valid host error if all vpus have been consumed | 16:55 |
| sean-k-mooney | nova will read the number of allication that can be made form the mdev-type specified by the nova.conf and report that to placement | 16:56 |
| pvc_ | i understand now, it is okay if you only specifiy one enabled_vgpu_types on nova.conf. But by default it will consume the mdev types. Thank you :) | 16:57 |
| pvc_ | But by default it will consume the all mdev types* | 16:57 |
| pvc_ | am i right? | 16:57 |
| sean-k-mooney | pvc_: no if you dont specyu an enabled vgpu type the no vgpus will be consumable | 16:58 |
| pvc_ | i already specify one enabled ( example nvidia-160) | 16:58 |
| *** derekh has quit IRC | 17:00 | |
| pvc_ | thank you sean-k-mooney. :) | 17:00 |
| sean-k-mooney | pvc_: you should look at https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#vgpu-types-tesla-p100 to determin how your tesla P100 can be subdevied | 17:01 |
| pvc_ | noted on this. thanks for your help. :) | 17:03 |
| *** pvradu has joined #openstack-nova | 17:04 | |
| openstackgerrit | Merged openstack/nova-specs master: Document Stein review priorities https://review.openstack.org/609807 | 17:07 |
| *** auggy has joined #openstack-nova | 17:08 | |
| *** pvradu has quit IRC | 17:09 | |
| *** hamzy has quit IRC | 17:12 | |
| *** hamzy has joined #openstack-nova | 17:12 | |
| *** dtantsur|brb is now known as dtantsur | 17:13 | |
| melwitt | dtantsur: I've been meaning to ask you a question about this ironic bug we fixed around RC time https://bugs.launchpad.net/nova/+bug/1787910 in comment #2 you mentioned the regression broke the ironic-inspector CI upstream. do you happen to know why the ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa job we run in nova did not break the same way? | 17:17 |
| openstack | Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem) | 17:17 |
| dtantsur | melwitt: looking | 17:17 |
| *** lbragstad is now known as lbragstad_f00d | 17:17 | |
| melwitt | thanks. I don't know the differences between the ironic-inspector job and our job | 17:18 |
| dtantsur | melwitt: okay, so tripleo got broken because it still used disk/ram filters with ironic | 17:18 |
| melwitt | trying to learn if there's a test coverage gap we can close to catch more issues | 17:18 |
| dtantsur | as to inspector, I don't remember why exactly I mentioned that. but there is at least one difference | 17:18 |
| dtantsur | ironic CI does not set vcpus/memory_mb on nodes, which leads to them not exposed to nova for some time | 17:19 |
| dtantsur | ironic-inspector, as part of its functioning, discovers these properties and sets them | 17:19 |
| sean-k-mooney | dtantsur: i was under the impression we ended up keeping the disk/ram filter for that reason in rocky and it was going to be fixed in ironic this cycle | 17:19 |
| dtantsur | sean-k-mooney: I don't think it was for this reason, but I may be missing something | 17:20 |
| dtantsur | for ironic these properties (and filters) are optional since.. mmm.. pike? maybe queens | 17:20 |
| melwitt | dtantsur: thanks for the pointer | 17:21 |
| dtantsur | np | 17:21 |
| sean-k-mooney | dtantsur: we defintly had a bug in the RC period related to them during rocky release. mriedem do you remeber what the bug related to ironc and the disk/ram filter was? | 17:21 |
| dtantsur | sean-k-mooney: the link melwitt posted above? yeah, essentially disk/ram filter stopped working for ironic. since nobody cares about this case any more, we just ended up fixing tripleo to not enable these filters. | 17:22 |
| dtantsur | no further investigation was done IIUC | 17:22 |
| sean-k-mooney | dtantsur: maybe not sure | 17:23 |
| melwitt | yeah, looks like it, use of the filters resulted in NoValidHost because of the bug | 17:24 |
| melwitt | so I probably the failure to update_available_resource was showing up in our ironic job, but was hidden because things otherwise worked (without ram/disk filters). which seems like it would be unexpected | 17:25 |
| sean-k-mooney | i think the only reason to use the disk/ram filter today would be if you were using the caching scheduler since it does not use palcement | 17:25 |
| *** pvc_ has quit IRC | 17:26 | |
| dtantsur | yep | 17:27 |
| *** dtantsur is now known as dtantsur|afk | 17:29 | |
| openstackgerrit | sean mooney proposed openstack/nova-specs master: Add spec for sriov live migration https://review.openstack.org/605116 | 17:31 |
| melwitt | looks like there were two different bugs in this bug. one was the update_available_resource fail (which wasn't caught by any CI) and then the core/ram/disk filter problem which was "unfixable", that is, only way out was to stop configuring deployments to use the filters | 17:31 |
| melwitt | it just so happened that because tripleo CI was failing because it was using the core/ram/disk filters, they also noticed the update_available_resource failure in the logs | 17:33 |
| sean-k-mooney | brb | 17:33 |
| *** jpena is now known as jpena|off | 17:34 | |
| openstackgerrit | Merged openstack/nova-specs master: Detach and attach boot volumes - Stein https://review.openstack.org/600628 | 17:34 |
| *** pvradu has joined #openstack-nova | 17:42 | |
| *** lbragstad_f00d is now known as lbragstad | 17:45 | |
| mriedem | efried: do you/anyone care about this anymore? https://review.openstack.org/#/c/560174/ it was mostly just historical documentation right? | 17:52 |
| *** pvradu has quit IRC | 17:53 | |
| *** ralonsoh has quit IRC | 17:57 | |
| *** hamzy has quit IRC | 17:59 | |
| *** hamzy has joined #openstack-nova | 18:00 | |
| *** moshele has quit IRC | 18:12 | |
| *** medberry has quit IRC | 18:18 | |
| *** hamzy has quit IRC | 18:23 | |
| *** hamzy has joined #openstack-nova | 18:24 | |
| efried | mriedem: I'm not sure. The information is probably still useful. I guess the last sentence would need to be updated to reflect how we actually solved it. Swhy I hadn't abandoned it yet. | 18:27 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.openstack.org/603955 | 18:27 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova stable/queens: Move live_migration.pre.start to the start of the method https://review.openstack.org/612773 | 18:33 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova stable/queens: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612774 | 18:33 |
| efried | mriedem: abandoned | 18:35 |
| melwitt | huh, looks no longer possible to filter logstash by n-cpu log type only. I guess not enough people used it | 18:35 |
| *** irclogbot_4 has joined #openstack-nova | 18:35 | |
| artom | Huh, stable/pike is going to be very problematic for ^^ | 18:37 |
| mriedem | melwitt: tags:"screen-n-cpu.txt" | 18:38 |
| melwitt | mriedem: filters out my result when I do | 18:38 |
| artom | Oh wait, our downstream bug is OSP13/queens, so we're good | 18:39 |
| melwitt | oh wait | 18:39 |
| melwitt | mriedem: user error, my bad | 18:39 |
| edleafe | cdent: ^^ Got all the tests passing locally \o/ | 18:40 |
| edleafe | doh! fat fingers ^^ | 18:40 |
| melwitt | I want to create an e-r query and I'm rusty | 18:41 |
| mriedem | well, there are specs to be reviewed if you wanted to do that instead :) | 18:41 |
| melwitt | I'm doing that too | 18:41 |
| melwitt | I thought this would be quicker than it's being | 18:42 |
| *** panda has quit IRC | 18:45 | |
| *** panda has joined #openstack-nova | 18:45 | |
| melwitt | bah, there's already a query for this. just e-r hasn't posted anything on it | 18:49 |
| mriedem | jroll: you might want to take a gander at this ironic volume-backed resize/cold migrate spec https://review.openstack.org/#/c/449155/ | 18:51 |
| mriedem | i haven't been through it in quite awhile | 18:51 |
| mriedem | but i'm also not very ironically inclined | 18:51 |
| *** cfriesen has joined #openstack-nova | 18:52 | |
| mriedem | seems that tuba guy would also care about this | 18:52 |
| *** pvc has joined #openstack-nova | 18:53 | |
| pvc | sean-k-mooney suddenly root@test-vgpu:/home/ubuntu# nvidia-smi No devices were found | 18:53 |
| pvc | so sad | 18:53 |
| cfriesen | since it's spec review day, I'd appreciate some eyes on https://review.openstack.org/#/c/571111/ | 19:01 |
| cfriesen | (the emulated TPM spec) | 19:01 |
| pvc | nervermind me | 19:02 |
| cfriesen | I think the open questions are whether we want to support CRB at this point (and if so how to ask for it), and whether we need to explicitly call out what happens for non-x86 architectures or leave that for the implementation. | 19:04 |
| *** edleafe_ has joined #openstack-nova | 19:15 | |
| *** edleafe_ has quit IRC | 19:16 | |
| *** pcaruana has quit IRC | 19:23 | |
| *** david-lyle has joined #openstack-nova | 19:27 | |
| *** dklyle has quit IRC | 19:28 | |
| *** lbragstad has quit IRC | 19:50 | |
| *** lbragstad has joined #openstack-nova | 19:53 | |
| *** david-lyle is now known as dklyle | 19:53 | |
| mriedem | jaypipes: are there any plans for this? https://review.openstack.org/#/c/529135/ | 19:58 |
| sean-k-mooney | mriedem: by the way sorry to be so negitive on https://review.openstack.org/#/c/612500 i understand why huawei and zte wants this but i really dont think its a viable option | 20:02 |
| cdent | any particularly exciting specs to look at? | 20:03 |
| sean-k-mooney | i dont know is https://review.openstack.org/#/c/600016/ worth reading ? | 20:04 |
| cdent | sean-k-mooney: jay likes it, eric's less sure. I think if I update it to include the use case that eric's describing, it's a useful feature, but not likely to happen in stein | 20:05 |
| jaypipes | mriedem: that's another one I tired of fighting about. | 20:05 |
| artom | Wouldn't that be more or a placement spec at this point? | 20:05 |
| jaypipes | mriedem: would be nice to get a real solution for it. apparently Oath has a whole code series for that backported against Ocata that I am supposed to figure out what is wrong with. :( | 20:06 |
| *** openstackgerrit has quit IRC | 20:06 | |
| sean-k-mooney | cdent: i have not read it yet but my first reaction to "GET /resource_providers?having=VCPU" was thats not already a thing? | 20:06 |
| sean-k-mooney | e.g. i just assumed there was an effiect way to say give me all there resouce provierd with X resouce class | 20:07 |
| cdent | nope, you can't easily list _any_ resource provider that has a particular class of resource, as it will leave out providers that are full | 20:07 |
| sean-k-mooney | ah so its the and are not full bit that is missing right | 20:07 |
| sean-k-mooney | well and may or maynot be full | 20:08 |
| mriedem | sean-k-mooney: it's not my spec | 20:08 |
| sean-k-mooney | e.g. just list all the RPs that have X regarless of there fullness | 20:08 |
| cdent | exactly | 20:08 |
| cdent | artom: placement specs is not a thing yet | 20:09 |
| sean-k-mooney | mriedem: ya i kindof assumed you jsut repoposed after the spec repo rebase | 20:09 |
| sean-k-mooney | mriedem: but did you not say huawei wanted this back in dublin | 20:09 |
| mriedem | huawei wants sriov bond yes | 20:09 |
| mriedem | *huawei's customers want sriov bond | 20:09 |
| sean-k-mooney | mriedem: ya thats want i ment rather then that spec specifcally | 20:10 |
| mriedem | but i can't get the network product guys internally to clearly describe the requirements they have against nova | 20:11 |
| artom | cdent, oh. Um, should it be, with the split and everything? | 20:11 |
| mriedem | so i can't really compare what they want, or their proposed solutions, to the zte spec | 20:11 |
| *** mvkr has joined #openstack-nova | 20:12 | |
| mriedem | except they all say "something has to do the bond within the guest" hand wave hand wave | 20:12 |
| sean-k-mooney | mriedem: ya i know that feeling. the zte solution is very non cloudy | 20:12 |
| cdent | artom: the idea is that it will likely happen once the split is deemed complete. We have a list of things to complete before we declare that | 20:12 |
| *** spatel has joined #openstack-nova | 20:12 | |
| mriedem | jaypipes: you need to get your build request user metadata scheduler filter thing done first | 20:13 |
| sean-k-mooney | mriedem: what is often missed is sure you can do the bond in the guest but unless you tell neutron about it so it can confiugre the top of rack switch you screwed if you want to use lacp | 20:13 |
| artom | cdent, fair enough, thanks :) And reading the commit message, you just wanted to stash https://review.openstack.org/#/c/600016/ somewhere | 20:13 |
| *** tbachman has quit IRC | 20:13 | |
| jaypipes | mriedem: yep. | 20:14 |
| cdent | artom: yup | 20:14 |
| jaypipes | mriedem: right after I shoot myself in the head from looking at Chef recipe bullshit. | 20:14 |
| melwitt | mriedem: just so I'm clear, does the initial allocation ratios spec cover the ability to set per aggregate allocation ratios? https://review.openstack.org/552105 I mean, I know it's not called out in the spec, but does the spec allow it to work as a side effect? | 20:17 |
| *** medberry has joined #openstack-nova | 20:17 | |
| melwitt | or is there more work or another spec we would need to restore per aggregate allocation ratio abilities? | 20:18 |
| sean-k-mooney | melwitt: i think that would be a different spec. but if the desire for aggragate allocation ratios was just to be able to set it via an api then plament does that | 20:19 |
| melwitt | yeah, that's what I mean, if it's already possible to set allocation ratio per aggregate in placement, then I guess after the spec I linked is implemented, users will have what they need to do it | 20:20 |
| sean-k-mooney | melwitt: well no thats not possible but you can prgramtclly set an allocation via the api. you would have to loop over the RPs in the placement aggreate to set them via the placemetn api to get the same effect as nova old aggreate allocation ratios | 20:21 |
| mriedem | melwitt: no | 20:22 |
| mriedem | melwitt: you're looking for https://review.openstack.org/544683 | 20:22 |
| mriedem | which doesn't have an owner | 20:22 |
| mriedem | and is contentious in implementation | 20:22 |
| melwitt | urgh | 20:22 |
| sean-k-mooney | cdent: misusing your spec to help with ^ | 20:23 |
| sean-k-mooney | any opion on /resource_providers?having=VCPU&member-of=Y | 20:23 |
| sean-k-mooney | so that in osc-placement we could say somthing like " set allocation ration for all resocue class X in aggreate Y" | 20:24 |
| mriedem | melwitt: i think the summary on https://review.openstack.org/#/c/544683/ is that it's all possible outside of nova; adding it within nova was desirable as a proxy since we have a host aggregates API already with rbac, which placement didn't have until rocky. in dublin we said a simple meta CLI could be added to osc-placement to do this for people, but that didn't happen. but since we mirror the compute host aggregates to plac | 20:24 |
| mriedem | t aggregates, this would build on that for mirroring the allocation ratios as well if *_allocation_ratio was set on a compute host aggregate in nova. | 20:24 |
| melwitt | ok, so that's what the "proxy" talk was about. so the proxy would still have to set a ratio per RP, there's no concept of an aggregate spanning ratio | 20:27 |
| sean-k-mooney | mriedem: the meta cli being somting like "openstack placement allocation ratio set 1.0 --class VCPU --mem-of <my aggreate>" | 20:27 |
| mriedem | melwitt: correct | 20:27 |
| *** openstackgerrit has joined #openstack-nova | 20:27 | |
| openstackgerrit | Merged openstack/nova-specs master: Add .idea folder to .gitignore https://review.openstack.org/581611 | 20:27 |
| mriedem | resource provider aggregates in placement are just a group of linked providers, there is no metadata about the aggregate | 20:27 |
| mriedem | so the upside to nova doing the proxy is you get back to what we had before we broke this, and you only have to set aggregate allocation ratios in one place, rather than both nova and placement separately. | 20:28 |
| melwitt | hm. trying to think how the old filters worked, how did operators express the per aggregate ratio /me looks for it | 20:28 |
| mriedem | the downside is it's more proxy stuff in the comptue API | 20:28 |
| mriedem | they do it with metadata on the host aggregate | 20:28 |
| melwitt | ok, I understand now | 20:29 |
| mriedem | https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregateramfilter | 20:29 |
| mriedem | "If the host is in more than one aggregate and thus more than one value is found, the minimum value will be used." | 20:29 |
| mriedem | i don't know if jay's spec deals with that | 20:29 |
| melwitt | thinking... | 20:32 |
| melwitt | if you're only interested in per aggregate ratios, then you'd only need to set them in placement and not nova, I think | 20:32 |
| melwitt | so maybe the scenario of "set them in both nova and placement separately" would be a rare one | 20:33 |
| jaypipes | melwitt: as long as nova doesn't overwrite them over and over again.. | 20:34 |
| melwitt | you're either going to be setting them in nova, or in placement, depending on whether you want to do it per compute (or via conf) or per aggregate (or via API) | 20:34 |
| mriedem | right, so if you manage this via the api you don't set the config overrides (iweb case) | 20:34 |
| mriedem | if you're cern, you only use config and not the api | 20:34 |
| mriedem | but we don't support both | 20:34 |
| mriedem | if set, config takes precedence | 20:34 |
| melwitt | right. and if you're oath, you'd use only the api and not the conf | 20:34 |
| jaypipes | mriedem, melwitt: my light at the end of this tunnel is that jules is making kielbasa and cabbage for dinner, which is one of my favorites. yum. | 20:35 |
| mriedem | i can smell your house from here | 20:35 |
| melwitt | ok, I see. based on this, I think maybe we shouldn't try to proxy. it doesn't seem like it would buy much even for users | 20:35 |
| jaypipes | mriedem: you'll be able to smell it in the morning too. | 20:35 |
| mriedem | hi-o | 20:35 |
| jaypipes | lol | 20:35 |
| mriedem | melwitt: well you probably want to run that by mgagne | 20:35 |
| melwitt | and penick | 20:36 |
| melwitt | yeah, I'm just speculating here. will definitely want to talk to them and see if they would be ok with a placement CLI that can fan out to aggregate RPs | 20:36 |
| melwitt | or if being able to set the aggregate metadata is really that valuable to them | 20:37 |
| jaypipes | mriedem: how will melwitt get ahold of penick, though? | 20:37 |
| jaypipes | mriedem: he is a tough cookie to track down. | 20:37 |
| mriedem | paper airplane? | 20:37 |
| jaypipes | hehe | 20:37 |
| melwitt | yeah, that part will be tough. I don't know him that well | 20:37 |
| jaypipes | melwitt: you can set five upon him. | 20:38 |
| jaypipes | for some reason I always think of He-Man's Attack Cat when I picture Five... | 20:38 |
| mriedem | melwitt: it is important to mgagne because they allow non-admin users access to set host aggregate allocation ratios via the API | 20:38 |
| melwitt | jaypipes: lol battlecat, that's awesome | 20:38 |
| mriedem | that's why the RBAC need was big in placement for mgagne | 20:38 |
| jaypipes | melwitt: battlecat, right :) | 20:39 |
| melwitt | I had a battlecat, so cool | 20:39 |
| melwitt | jaypipes: have you watched that "toys that made us" show on netflix? they have one that has the he-man toy line. I found it really interesting | 20:40 |
| *** hamzy has quit IRC | 20:40 | |
| *** hamzy has joined #openstack-nova | 20:40 | |
| melwitt | mriedem: ahhh ok | 20:40 |
| jaypipes | melwitt: hmm, no... sounds like it would be good to watch, though! | 20:42 |
| melwitt | there's only like 4 episodes, I watched em all | 20:42 |
| *** ivve has quit IRC | 20:42 | |
| melwitt | the more I think about it, I'm like proxying isn't so terrible either is it? being that we're already mirroring aggregates anyway. setting the ratios would be a small extra step, it seems. but if placement has RBAC, then would mgagne be happy... hrm. | 20:44 |
| jaypipes | all that hard work I did this morning from 3:30am - 8am in reducing my browser tab count by 63 tabs has been slowly destroyed again. | 20:49 |
| melwitt | you were working at 3:30am?? | 20:50 |
| mriedem | cdent: if you have people internally that still care about these https://review.openstack.org/#/c/552190/ https://review.openstack.org/#/c/549067/ you might want to kick them to update them for stein | 20:51 |
| cdent | mriedem: i've kicked them several times to no avail | 20:52 |
| mriedem | shall i drop the abandon hammer then? | 20:52 |
| mriedem | that way you're not the bad guy | 20:52 |
| *** spatel has quit IRC | 20:52 | |
| cdent | mriedem: meh, I'd let them ride a bit longer, might still be a chance | 20:52 |
| jaypipes | melwitt: yeah, couldn't sleep. | 20:53 |
| mriedem | overcommitting a dedicated pcpu... https://review.openstack.org/#/c/599957/ seems...odd | 20:53 |
| jaypipes | BTW, I have COMPLETELY failed in my sean-k-mooney spellchecker powers today. | 20:54 |
| mriedem | isn't that an oxymoron? | 20:54 |
| mriedem | oxymoron = overcommitted dedicated pcpu | 20:54 |
| melwitt | thinking about that hurts my brain | 20:54 |
| jaypipes | mriedem: yeah, it is, and I chatted with tpatil about that in Denver | 20:54 |
| melwitt | definitely sounds contradictory | 20:55 |
| jaypipes | mriedem: mostly they just need the whole "allow a single compute host to have dedicated stuff and non-dedicated stuff on the same box" thing | 20:55 |
| jaypipes | mriedem: I doubt Tushar will follow up on that spec. | 20:55 |
| mriedem | and then PCPU inventory has an allocation_ratio>1.0? | 20:55 |
| *** ttsiouts has joined #openstack-nova | 20:56 | |
| mriedem | b/c this spec https://review.openstack.org/#/c/599957/ is all about many new config options | 20:56 |
| mriedem | which kinda sucks | 20:56 |
| melwitt | they're saying two of the options (cpu_dedicated_set, cpu_shared_set) come from a different spec though, `Standardize CPU resource tracking` | 21:00 |
| *** mchlumsky has quit IRC | 21:01 | |
| melwitt | so only the cpu_pinning_allocation_ratio would be new. which really I guess is meaning pcpu_allocation_ratio, whereas the existing cpu_allocation_ratio technically means vcpu_allocation_ratio? | 21:02 |
| *** erlon has quit IRC | 21:03 | |
| melwitt | so that they separate handling of vcpu vs pcpu | 21:03 |
| *** cdent has quit IRC | 21:03 | |
| melwitt | also interesting, it says based on the prereq spec https://review.openstack.org/#/c/555081 that PCPU inventory will be created with hardcoded allocation ratio of 1.0 and they want to be able to change it/overcommit it. so wouldn't that just be a call to the placement API? | 21:05 |
| melwitt | but I guess they want to be able to set it the same way as is possible for VCPU | 21:06 |
| melwitt | via nova.conf | 21:06 |
| *** priteau has quit IRC | 21:09 | |
| *** dklyle has quit IRC | 21:19 | |
| *** dklyle has joined #openstack-nova | 21:20 | |
| mriedem | edleafe: all sorts of API gross for you to munch on here https://review.openstack.org/#/c/580336/4 | 21:36 |
| *** spsurya has quit IRC | 21:38 | |
| *** slaweq has quit IRC | 21:39 | |
| *** awaugama has quit IRC | 21:39 | |
| *** rtjure has quit IRC | 21:39 | |
| cfriesen | I think my question about the "overcommit PCPUs" idea is what does it buy you that a low CPU overcommit ratio (with non-dedicated cpus) wouldn't? | 21:41 |
| sean-k-mooney | jaypipes: haha was there a partical pharse that exausted the spellchecking ablity today :) | 21:46 |
| sean-k-mooney | cfriesen: over commiting pinned cpus i think would be less objectionable but the fact we choose hw:cpu_policy=dedicated|shared and not hw:cpu_policy=pinned|floating makes me dislike the idea | 21:48 |
| edleafe | mriedem: gee thanks! | 21:49 |
| cfriesen | sean-k-mooney: once you have more than one instance using a cpu, it's shared. I don't see what this would buy us compared to just using a shared CPU | 21:52 |
| sean-k-mooney | cfriesen: well i very limited cases it may improve you performance as pinning will result in numa affinity for leagacy reasons but i would personally prefer hw:cpu_policy=share + hw:numa_nodes=1 | 21:54 |
| cfriesen | sean-k-mooney: agreed. And if you were using hugepages you'd get the numa affinity anyways. | 21:55 |
| sean-k-mooney | cfriesen: also since we are going to be using the dedictate_cpu_set for allocating realtime cpus i think this is likely to break that also | 21:56 |
| cfriesen | wait...how is a realtime cpu different from a regular dedicated cpu? | 21:57 |
| sean-k-mooney | cfriesen: realtime we set the "nice" value or whatever the priority values is on linux to realtime so it does not get premented if you have a realtime kernel | 21:58 |
| sean-k-mooney | cfriesen: we dont set the tread prioity for dedicated cpus | 21:58 |
| *** claudiub has quit IRC | 22:02 | |
| cfriesen | sean-k-mooney: that's what I thought, just making sure. if you've already got a "dedicated" cpu with nothing else running on it, what's the benefit of making the vcpu task realtime? | 22:02 |
| cfriesen | (unless the host hasn't properly moved all kernel work off that cpu) | 22:02 |
| sean-k-mooney | basiclly ^ | 22:03 |
| sean-k-mooney | or there is some backgound process that is on the host that is not confied properly | 22:03 |
| sean-k-mooney | but very little | 22:03 |
| cfriesen | but if the host hasn't moved kernel work off that cpu, and you run the qemu task as realtime, don't you risk priority inversion anyways if the guest is doing a busy-loop and never letting other tasks run? | 22:05 |
| *** slaweq has joined #openstack-nova | 22:05 | |
| jaypipes | sean-k-mooney: no, just distracted generally :) | 22:05 |
| sean-k-mooney | cfriesen: one if its actully a kernel thread then the kernel will run it if it needs too | 22:06 |
| *** tbachman has joined #openstack-nova | 22:06 | |
| *** ttsiouts has quit IRC | 22:06 | |
| sean-k-mooney | cfriesen: second if its a user tread the kernel can rescudle it i think if the vm is in a busy loop | 22:06 |
| mriedem | melwitt: can we abandon https://review.openstack.org/#/c/509042/ or are there plans to update that? | 22:06 |
| *** ttsiouts has joined #openstack-nova | 22:06 | |
| sean-k-mooney | cfriesen: i think we tell people dont use this unless you have set up your host properly for realtime workloads and properly isolated the cores | 22:07 |
| melwitt | mriedem: I'd like to update it but we need allocation ownership concepts in placement else it's moot | 22:07 |
| mriedem | so is anyone driving that dependency? | 22:07 |
| sean-k-mooney | cfriesen: at least i tell people do ues the realtime feature unless you have set up the host properly | 22:07 |
| cfriesen | sean-k-mooney: on hosts with the RT kernel a bunch of kernel things get run in schedulable threads | 22:08 |
| melwitt | mriedem: no, just saying, that's why it's stuck | 22:08 |
| melwitt | and maybe, I guess I could ask jaypipes because I thought I saw mention of the idea of an owner attribute in some other spec | 22:09 |
| mriedem | ok, so....if it's stuck, and no one is working on unstucking it, and it's not high enough priority to are, should we just abandon | 22:09 |
| sean-k-mooney | cfriesen: on https://review.openstack.org/#/c/599957 i suggested just adding a hw:cpu_policy=pinned which would pin the vm to one of the shared cpu set cores instead. does that sound better then over commiting dedicated cpus to you? | 22:09 |
| mriedem | *care | 22:09 |
| openstackgerrit | Merged openstack/nova-specs master: Dynamically find releases for move-implemented-specs https://review.openstack.org/592628 | 22:09 |
| sean-k-mooney | cfriesen: oh ya but you have to use isolcpus too if you useing the realtime core extra specs in nova correctly | 22:10 |
| cfriesen | sean-k-mooney: I think you'd need to pin each vCPU in the VM to one of the shared cpu set cores. | 22:10 |
| sean-k-mooney | cfriesen: yes that is what i was suggesting | 22:10 |
| melwitt | mriedem: for the record, I care about it a lot but I can't argue that allocation ownership in placement is a priority given everything else that's going on. I can abandon it on that basis | 22:10 |
| cfriesen | sean-k-mooney: isolcpus means no scheduling, so only works with explicit pinning and one vcpu per pcpu. | 22:10 |
| *** ttsiouts has quit IRC | 22:11 | |
| sean-k-mooney | cfriesen: yep | 22:11 |
| cfriesen | sean-k-mooney: yeah, so pinning to shared cpu set cores makes more sense to me | 22:11 |
| *** eharney has quit IRC | 22:13 | |
| sean-k-mooney | cfriesen: ok that comment is in there spec but while i kind of get this usescase i also think its not a great one. | 22:18 |
| cfriesen | it's purely a small performance optimization compared to just using "shared" | 22:20 |
| cfriesen | which is valid, but it makes the resource tracking really messy | 22:21 |
| sean-k-mooney | cfriesen: well in tushars case they want to have more over subsction with the same perfromce i think | 22:21 |
| cfriesen | sean-k-mooney: their performance document is comparing against "shared" | 22:21 |
| sean-k-mooney | yes but they are not comparing against shared with hw:numa_nodes=1 are they | 22:22 |
| sean-k-mooney | also where is the doc? | 22:22 |
| cfriesen | nope. | 22:22 |
| cfriesen | linked at the bottom of the spec | 22:22 |
| *** rcernin has joined #openstack-nova | 22:24 | |
| openstackgerrit | Merged openstack/nova-specs master: fix tox python3 overrides https://review.openstack.org/579793 | 22:24 |
| sean-k-mooney | not to kill this entirlaly but the other thing that makes me uncomfortable about this is by doing this you increase the risk of specter or l1tf | 22:24 |
| sean-k-mooney | in a public cloud enve you are now allowing two instace that could be from different host to be pinned to the same core where they will context switch | 22:25 |
| *** bnemec has quit IRC | 22:25 | |
| *** mriedem has quit IRC | 22:26 | |
| cfriesen | sean-k-mooney: agreed, that seems sketchy. ideally you'd want to ensure only instances from the same tenant were permanently pinned together | 22:27 |
| sean-k-mooney | ya which is a pain | 22:27 |
| *** slaweq has quit IRC | 22:38 | |
| *** hamzy has quit IRC | 22:59 | |
| *** threestrands has joined #openstack-nova | 23:02 | |
| *** slaweq has joined #openstack-nova | 23:11 | |
| *** idlemind has quit IRC | 23:38 | |
| *** slaweq has quit IRC | 23:45 | |
| *** gyee has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!