| *** hamalq_ has quit IRC | 00:06 | |
| *** mlavalle has quit IRC | 00:23 | |
| *** hamalq has joined #openstack-nova | 00:28 | |
| *** dlbewley has quit IRC | 00:30 | |
| *** dlbewley has joined #openstack-nova | 00:31 | |
| *** hamalq_ has joined #openstack-nova | 00:42 | |
| *** hamalq has quit IRC | 00:46 | |
| *** dlbewley has quit IRC | 01:00 | |
| *** xiaolin has joined #openstack-nova | 01:01 | |
| *** dlbewley has joined #openstack-nova | 01:01 | |
| *** brinzhang has joined #openstack-nova | 01:02 | |
| *** xiaolin has quit IRC | 01:14 | |
| *** Liang__ has joined #openstack-nova | 01:21 | |
| *** Liang__ is now known as LiangFang | 01:21 | |
| *** hamalq_ has quit IRC | 01:36 | |
| *** hamalq has joined #openstack-nova | 01:47 | |
| *** brinzhang_ has joined #openstack-nova | 01:59 | |
| *** markmcclain has quit IRC | 02:01 | |
| *** brinzhang has quit IRC | 02:02 | |
| *** markmcclain has joined #openstack-nova | 02:02 | |
| *** dlbewley has quit IRC | 02:02 | |
| *** dlbewley has joined #openstack-nova | 02:03 | |
| *** tinwood has quit IRC | 02:08 | |
| *** tinwood has joined #openstack-nova | 02:10 | |
| *** spatel has joined #openstack-nova | 02:20 | |
| *** lbragstad has quit IRC | 02:21 | |
| *** brinzhang0 has joined #openstack-nova | 02:26 | |
| *** brinzhang_ has quit IRC | 02:29 | |
| *** boxiang_ has joined #openstack-nova | 02:30 | |
| *** boxiang has quit IRC | 02:33 | |
| *** xinranwang_ has joined #openstack-nova | 02:43 | |
| *** brinzhang_ has joined #openstack-nova | 02:47 | |
| *** rcernin has quit IRC | 02:48 | |
| *** brinzhang0 has quit IRC | 02:50 | |
| *** hamalq has quit IRC | 02:50 | |
| *** hoonetorg has quit IRC | 02:51 | |
| *** spatel has quit IRC | 02:55 | |
| *** hamalq has joined #openstack-nova | 02:57 | |
| *** rcernin has joined #openstack-nova | 02:59 | |
| *** hamalq has quit IRC | 02:59 | |
| *** hamalq has joined #openstack-nova | 02:59 | |
| *** brinzhang_ is now known as brinzhang | 03:01 | |
| *** rcernin has quit IRC | 03:05 | |
| *** spatel has joined #openstack-nova | 03:16 | |
| *** mkrai has joined #openstack-nova | 03:17 | |
| *** rcernin has joined #openstack-nova | 03:20 | |
| *** rcernin has quit IRC | 03:22 | |
| *** rcernin has joined #openstack-nova | 03:22 | |
| *** hamalq has quit IRC | 03:29 | |
| *** ravsingh has joined #openstack-nova | 03:30 | |
| openstackgerrit | Jinsheng Zhang proposed openstack/nova-specs master: Add nova-support-multiple-boot-volume-with-boot-order-selection spec https://review.opendev.org/736422 | 03:37 |
|---|---|---|
| *** psachin has joined #openstack-nova | 03:37 | |
| *** hamalq has joined #openstack-nova | 03:52 | |
| openstackgerrit | Jinsheng Zhang proposed openstack/nova-specs master: fix "Line limited" error https://review.opendev.org/736425 | 04:05 |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-nova | 04:33 | |
| *** hoonetorg has joined #openstack-nova | 04:35 | |
| *** hamalq has quit IRC | 04:40 | |
| *** hamalq has joined #openstack-nova | 04:45 | |
| *** hamalq has quit IRC | 04:50 | |
| *** hamalq has joined #openstack-nova | 04:52 | |
| *** spatel has quit IRC | 04:53 | |
| *** hamalq has quit IRC | 04:57 | |
| openstackgerrit | Jinsheng Zhang proposed openstack/nova-specs master: Add nova-support-multiple-boot-volume-with-boot-order-selection spec https://review.opendev.org/736422 | 05:07 |
| *** ratailor has joined #openstack-nova | 05:07 | |
| *** ociuhandu has joined #openstack-nova | 05:21 | |
| *** ociuhandu has quit IRC | 05:26 | |
| *** udesale has joined #openstack-nova | 05:42 | |
| openstackgerrit | Jinsheng Zhang proposed openstack/nova-specs master: Add nova-support-multiple-boot-volume-with-boot-order-selection spec fix document error https://review.opendev.org/736422 | 05:43 |
| *** hamalq has joined #openstack-nova | 05:44 | |
| *** links has joined #openstack-nova | 05:46 | |
| *** alex_xu has joined #openstack-nova | 05:47 | |
| *** jsuchome has joined #openstack-nova | 05:50 | |
| *** hamalq has quit IRC | 05:50 | |
| *** markvoelker has joined #openstack-nova | 05:51 | |
| *** markvoelker has quit IRC | 05:55 | |
| *** mkrai has quit IRC | 05:59 | |
| *** mkrai has joined #openstack-nova | 06:04 | |
| *** links has quit IRC | 06:07 | |
| *** links has joined #openstack-nova | 06:12 | |
| *** dlbewley has quit IRC | 06:16 | |
| *** dlbewley has joined #openstack-nova | 06:16 | |
| *** rchurch has quit IRC | 06:19 | |
| *** rchurch has joined #openstack-nova | 06:21 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:22 | |
| *** dlbewley has quit IRC | 06:26 | |
| *** dlbewley has joined #openstack-nova | 06:27 | |
| *** links has quit IRC | 06:32 | |
| *** rpittau|afk is now known as rpittau | 06:43 | |
| *** dklyle has quit IRC | 06:44 | |
| *** xek has joined #openstack-nova | 07:00 | |
| *** links has joined #openstack-nova | 07:01 | |
| *** ttsiouts has joined #openstack-nova | 07:06 | |
| *** slaweq has joined #openstack-nova | 07:07 | |
| *** breizhkoala has joined #openstack-nova | 07:11 | |
| *** tesseract has joined #openstack-nova | 07:14 | |
| *** hamalq has joined #openstack-nova | 07:15 | |
| *** ttsiouts has quit IRC | 07:19 | |
| *** ttsiouts has joined #openstack-nova | 07:19 | |
| *** maciejjozefczyk has quit IRC | 07:19 | |
| *** ralonsoh has joined #openstack-nova | 07:21 | |
| *** tosky has joined #openstack-nova | 07:25 | |
| *** elod has quit IRC | 07:26 | |
| *** hamalq has quit IRC | 07:26 | |
| *** LiangFang has quit IRC | 07:31 | |
| *** mkrai has quit IRC | 07:32 | |
| *** elod has joined #openstack-nova | 07:33 | |
| *** Liang__ has joined #openstack-nova | 07:34 | |
| *** maciejjozefczyk has joined #openstack-nova | 07:37 | |
| *** hamalq has joined #openstack-nova | 07:42 | |
| *** Liang__ has quit IRC | 07:44 | |
| *** Liang__ has joined #openstack-nova | 07:46 | |
| *** hamalq has quit IRC | 07:47 | |
| gibi | sean-k-mooney, melwitt: re: resize without changing the flavor, the context on the PTG was that changing a resource request of a port might also needs a move operation without changing the flavor | 07:50 |
| *** dlbewley has quit IRC | 07:50 | |
| *** dlbewley has joined #openstack-nova | 07:51 | |
| gibi | sean-k-mooney, melwitt: I'll try to propose a spec for discussion during this cycle so your input about machine_type change and video mode change being similar use case helps. | 07:51 |
| gibi | sean-k-mooney, melwitt: however I think this spec will be targeted to W not V due to time constraints and discussion needs | 07:52 |
| *** rcernin has quit IRC | 07:54 | |
| *** rcernin_ has joined #openstack-nova | 07:54 | |
| *** ttsiouts has quit IRC | 07:54 | |
| *** ttsiouts has joined #openstack-nova | 07:58 | |
| *** links has quit IRC | 08:00 | |
| *** dlbewley has quit IRC | 08:00 | |
| sean-k-mooney | gibi: ah yes i could not recall what it was in relation too | 08:00 |
| *** dlbewley has joined #openstack-nova | 08:01 | |
| sean-k-mooney | gibi: i think there are other usecases too such as recreatign the vm with the same image and flavor but just updating the embeded copy to pick up change to extra_specs and image metadata without actually imaging the disk | 08:01 |
| *** songwenping_ has joined #openstack-nova | 08:02 | |
| *** songwenping__ has joined #openstack-nova | 08:03 | |
| gibi | but still cold migrating the VM if the current host is not good for the updted extra_spec / image metadata? | 08:03 |
| sean-k-mooney | gibi: yes | 08:03 |
| sean-k-mooney | i would guess in generall it would be a move opertation | 08:03 |
| gibi | OK, that seems like the same operation that I would need for the qos update | 08:03 |
| sean-k-mooney | it might be vaild to use the same host but likely the schduler would not select the same host | 08:04 |
| sean-k-mooney | it might but basically the same as same host resize | 08:04 |
| sean-k-mooney | i guess we could always add a weigher to make it prefer the same host | 08:05 |
| sean-k-mooney | but thats not really important | 08:05 |
| *** martinkennelly has joined #openstack-nova | 08:05 | |
| *** songwenping_ has quit IRC | 08:06 | |
| *** hamalq has joined #openstack-nova | 08:08 | |
| gibi | yeah, I can imagine that the end user needs an to update these data if it does not cause downtime (cold migration) of the VM | 08:08 |
| gibi | * needs an option to update | 08:09 |
| gibi | but same host resize is still downtime today | 08:09 |
| sean-k-mooney | gibi: currently you need to do a db edit and a hard reboot | 08:09 |
| *** links has joined #openstack-nova | 08:10 | |
| sean-k-mooney | the downstream issue that melwitt was looking at is related to the fact that in rhel 8.2 the default we use in nova "cirrus" was deprecated | 08:10 |
| sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=1651994 | 08:10 |
| openstack | sean-k-mooney: Error: Error getting bugzilla.redhat.com bug #1651994: NotPermitted | 08:10 |
| sean-k-mooney | oh thats private well that is what it was tracking | 08:11 |
| sean-k-mooney | anyway 16.0 which is based on train was released on 8.1 16.1 will be on 8.2 | 08:11 |
| sean-k-mooney | so customer are now in the situation where the default model that nova will select is deprecaed and there is no way via the api to change this on existing instance other than rebuild | 08:12 |
| *** ttsiouts has quit IRC | 08:12 | |
| *** hamalq has quit IRC | 08:12 | |
| sean-k-mooney | we wont be able to back port this new command or whatever it will be but it might help in the future | 08:13 |
| *** ttsiouts has joined #openstack-nova | 08:13 | |
| *** ttsiouts_ has joined #openstack-nova | 08:16 | |
| *** ttsiouts has quit IRC | 08:17 | |
| *** udesale_ has joined #openstack-nova | 08:19 | |
| *** elod has quit IRC | 08:19 | |
| *** ravsingh has quit IRC | 08:20 | |
| *** rcernin_ has quit IRC | 08:20 | |
| *** udesale has quit IRC | 08:21 | |
| *** elod has joined #openstack-nova | 08:26 | |
| *** dtantsur|afk is now known as dtantsur | 08:29 | |
| *** ravsingh has joined #openstack-nova | 08:33 | |
| *** ttsiouts has joined #openstack-nova | 08:36 | |
| *** ttsiout__ has joined #openstack-nova | 08:37 | |
| openstackgerrit | Wenping Song proposed openstack/nova master: delete sub resource provider when delete resource provider https://review.opendev.org/719163 | 08:39 |
| *** ttsiouts_ has quit IRC | 08:40 | |
| *** ttsiouts has quit IRC | 08:41 | |
| *** ociuhandu has joined #openstack-nova | 08:42 | |
| *** salmankhan has joined #openstack-nova | 08:46 | |
| *** elod has quit IRC | 09:00 | |
| *** jawad_axd has joined #openstack-nova | 09:09 | |
| *** ccamacho has quit IRC | 09:16 | |
| *** vishalmanchanda has joined #openstack-nova | 09:17 | |
| *** elod has joined #openstack-nova | 09:17 | |
| *** elod has quit IRC | 09:19 | |
| *** yaawang_ has quit IRC | 09:24 | |
| *** elod has joined #openstack-nova | 09:25 | |
| *** links has quit IRC | 09:27 | |
| *** links has joined #openstack-nova | 09:28 | |
| *** mkrai has joined #openstack-nova | 09:28 | |
| *** dlbewley has quit IRC | 09:30 | |
| *** dlbewley has joined #openstack-nova | 09:31 | |
| *** martinkennelly has quit IRC | 09:31 | |
| *** martinkennelly has joined #openstack-nova | 09:32 | |
| *** ttsiouts has joined #openstack-nova | 09:32 | |
| *** ttsiout__ has quit IRC | 09:32 | |
| *** martinkennelly has quit IRC | 09:35 | |
| *** martinkennelly has joined #openstack-nova | 09:36 | |
| *** martinkennelly has quit IRC | 09:38 | |
| *** martinkennelly has joined #openstack-nova | 09:39 | |
| *** martinkennelly has quit IRC | 09:41 | |
| *** yaawang_ has joined #openstack-nova | 09:41 | |
| *** martinkennelly has joined #openstack-nova | 09:43 | |
| *** martinkennelly has quit IRC | 09:49 | |
| *** avolkov has joined #openstack-nova | 09:51 | |
| *** ociuhandu has quit IRC | 09:52 | |
| openstackgerrit | Wenping Song proposed openstack/nova master: delete sub resource provider when delete resource provider https://review.opendev.org/719163 | 09:54 |
| *** ociuhandu has joined #openstack-nova | 09:59 | |
| *** tkajinam has quit IRC | 10:05 | |
| *** dlbewley has quit IRC | 10:05 | |
| *** rpittau is now known as rpittau|bbl | 10:06 | |
| *** dlbewley has joined #openstack-nova | 10:06 | |
| *** Liang__ has quit IRC | 10:18 | |
| *** ravsingh has quit IRC | 10:32 | |
| *** xek has quit IRC | 10:34 | |
| *** xek has joined #openstack-nova | 10:34 | |
| *** dlbewley has quit IRC | 10:35 | |
| *** dlbewley has joined #openstack-nova | 10:35 | |
| arne_wiebalck | TheJulia: dansmith sean-k-mooney Sorry, I missed the discussion yesterday. | 10:38 |
| arne_wiebalck | Let me add a little background info: physical instances at CERN are all done via Nova and Ironic. The main user is OpenStack itself and we rely on Ironic's software RAID and standard cloud images | 10:38 |
| arne_wiebalck | to deploy. However, other users (like the Ceph team) want to partition/RAID their physical instances in a specific way as needed for their service. There is no convenient way to express this is via Nova/Ironic at the moment. So, they reinstall these instances after initial deployment through Nova/Ironic once more with kickstart. This double installation is what we'd like to get rid of, and this triggered | 10:38 |
| arne_wiebalck | the whole discussion. | 10:38 |
| arne_wiebalck | One of the main points for user adoption probably is that whenever sth needs to change, this should be feasible without having the Ironic admin to re-clean nodes or the Nova admin to add new flavors (with 5000 nodes in Ironic we have around 150 resource classes and hence 150 flavors). This is why we came up with the idea of a "kickstart" driver where Ironic uses a kickstart/preseed file to do the | 10:38 |
| arne_wiebalck | deployment (and skip the usual image deployment) as it would provide the user with the same flexibility as there is now. But since we would like to keep Nova in the mix (for various reasons), we may need some tooling help on the Nova side. | 10:38 |
| *** ratailor has quit IRC | 10:40 | |
| *** ratailor has joined #openstack-nova | 10:40 | |
| *** dlbewley has quit IRC | 10:45 | |
| *** dlbewley has joined #openstack-nova | 10:45 | |
| *** rcernin_ has joined #openstack-nova | 10:48 | |
| *** lvdombrkr has joined #openstack-nova | 11:07 | |
| *** udesale_ has quit IRC | 11:12 | |
| *** ccamacho has joined #openstack-nova | 11:22 | |
| *** markvoelker has joined #openstack-nova | 11:24 | |
| *** lvdombrkr has quit IRC | 11:25 | |
| *** dlbewley has quit IRC | 11:26 | |
| *** dlbewley has joined #openstack-nova | 11:27 | |
| *** derekh has joined #openstack-nova | 11:27 | |
| *** markvoelker has quit IRC | 11:29 | |
| *** lvdombrkr has joined #openstack-nova | 11:29 | |
| *** belmoreira has joined #openstack-nova | 11:33 | |
| *** mkrai has quit IRC | 11:33 | |
| *** xiaolin has joined #openstack-nova | 11:33 | |
| *** hamalq has joined #openstack-nova | 11:39 | |
| *** mgariepy has quit IRC | 11:43 | |
| *** hamalq has quit IRC | 11:45 | |
| *** nweinber has joined #openstack-nova | 11:48 | |
| *** ociuhandu has quit IRC | 11:50 | |
| *** raildo has joined #openstack-nova | 11:50 | |
| *** ttsiouts has quit IRC | 12:06 | |
| *** ttsiouts has joined #openstack-nova | 12:07 | |
| *** ttsiouts has quit IRC | 12:11 | |
| *** mgariepy has joined #openstack-nova | 12:19 | |
| *** rpittau|bbl is now known as rpittau | 12:20 | |
| *** ttsiouts has joined #openstack-nova | 12:28 | |
| *** derekh has quit IRC | 12:33 | |
| aarents | Hi nova, | 12:36 |
| aarents | sean-k-mooney: dansmith Let me know if it needs further amend on https://review.opendev.org/#/c/736169/ https://review.opendev.org/#/c/734776/ thanks! | 12:38 |
| openstackgerrit | Alexandre Arents proposed openstack/nova master: libvirt: ensure disk_over_commit is not negative https://review.opendev.org/719008 | 12:39 |
| sean-k-mooney | ^ that should be done by the config option | 12:39 |
| sean-k-mooney | we set min 0 i think | 12:39 |
| *** spatel has joined #openstack-nova | 12:40 | |
| *** dlbewley has quit IRC | 12:40 | |
| sean-k-mooney | oh that is not the ratio | 12:40 |
| *** derekh has joined #openstack-nova | 12:40 | |
| sean-k-mooney | for that to be negitive the size on disk would have to be larger then the virtual size | 12:41 |
| *** dlbewley has joined #openstack-nova | 12:41 | |
| aarents | sean-k-mooney: yes | 12:41 |
| sean-k-mooney | does it being negitive break something | 12:41 |
| sean-k-mooney | ah i see | 12:43 |
| aarents | It just mislead calcuation of available_disk_least on which rely disk_filter | 12:43 |
| *** ttsiouts has quit IRC | 12:47 | |
| *** jawad_axd has quit IRC | 12:47 | |
| sean-k-mooney | ya clamping the value shoudl be fine. | 12:47 |
| *** ttsiouts has joined #openstack-nova | 12:47 | |
| *** bbowen_ has quit IRC | 12:49 | |
| *** bbowen_ has joined #openstack-nova | 12:49 | |
| *** ttsiouts has quit IRC | 12:52 | |
| *** ttsiouts has joined #openstack-nova | 12:54 | |
| *** ratailor has quit IRC | 12:54 | |
| *** dlbewley has quit IRC | 12:55 | |
| *** dlbewley has joined #openstack-nova | 12:56 | |
| *** rmart04 has joined #openstack-nova | 13:08 | |
| rmart04 | Hey all, I'm wondering if anyone could help me. Not strictly dev specific, but I'm having trouble with NUMA information being passed through to my virtual machines. /sys/bus/pci/devices*/numa_node always = -1. I'm running Rocky on C7, with numa_nodes=2 and cpu_sockets=2. | 13:12 |
| rmart04 | and cpu pinning policy set to dedicated | 13:13 |
| sean-k-mooney | is the -1 on the host or in the guest | 13:14 |
| sean-k-mooney | if its in the guest that is expected | 13:14 |
| rmart04 | In the guest | 13:14 |
| sean-k-mooney | so we do not currently create a pcie root complex per numa node | 13:14 |
| sean-k-mooney | since there is only one pci root all devices are childerne of that root | 13:15 |
| sean-k-mooney | wehn we do passthough we dont affiniteis the pci device to the virutal numa node of the guest | 13:15 |
| sean-k-mooney | so it is reported as -1 meaning no numa affinity in the guest | 13:15 |
| *** lbragstad has joined #openstack-nova | 13:16 | |
| sean-k-mooney | to change that we would likely have to use the q35 machine type and create a pcie root complete per numa node then add the passthough deivice to the correct pci root | 13:16 |
| sean-k-mooney | that has other implciation mainliy that on move operation either we have to allow the toploty to change or we have to limit the host we can select to maintain the current toplogy | 13:17 |
| sean-k-mooney | if we allow the toplogy to change the virtual pci address of the devices in the guest would also change | 13:18 |
| *** ttsiouts has quit IRC | 13:18 | |
| sean-k-mooney | rmart04: but yes if you have a multi numa node guest this can result in cross numa traffic worse case twice because you dont know the numa affinity of the device | 13:19 |
| *** ttsiouts has joined #openstack-nova | 13:19 | |
| openstackgerrit | Merged openstack/nova master: libvirt: Mark e1000e VIF as supported https://review.opendev.org/734777 | 13:20 |
| *** psachin has quit IRC | 13:20 | |
| rmart04 | OK, appreciate all the info SeanKMooney. I guess another way around this is to split the host into two guests, one on each numa node with their associated pci-passthrough devices (GPUs). Currently I appear to be blocked on this by my older kernel. 3.10. I bump into an issue allocating memory from the second NUMA node for the second machine. I believe this is fixed in 4.14. | 13:22 |
| rmart04 | qq, you mention the q35 machine type, what type do we use by default? | 13:23 |
| *** ttsiouts has quit IRC | 13:23 | |
| sean-k-mooney | rmart04: if the guest has a numa toploty we do not allow its memory to come form a remote numa node by design | 13:25 |
| sean-k-mooney | rmart04: we use pc | 13:26 |
| sean-k-mooney | or pc-i440fx | 13:26 |
| sean-k-mooney | something like that | 13:26 |
| sean-k-mooney | rmart04: what version of openstack are you using | 13:26 |
| *** ttsiouts has joined #openstack-nova | 13:26 | |
| rmart04 | Rocky (Stein upgrade this weekend) | 13:27 |
| sean-k-mooney | do you have gpus on all host numa nodes | 13:27 |
| sean-k-mooney | or just numa 0 | 13:27 |
| rmart04 | Yep, 2 sockets, 8 GPUs | 13:27 |
| rmart04 | 4 each | 13:27 |
| sean-k-mooney | ok what iw was going to say is you might need to use nuam_policy=preferred in the alias | 13:28 |
| sean-k-mooney | if you did not have them split | 13:28 |
| sean-k-mooney | if you do then yes 2 vms with 1 numa each and the default legacy polciy which enforce numa affintiy between cpu/memory and the pci device is what you will want | 13:28 |
| sean-k-mooney | you can create a dual numa guest but the limitation is you will know know what numa node in the guest maps to the actull location of the device on the host | 13:29 |
| sean-k-mooney | rmart04: are your vms using 1 gpu earch or multiple | 13:29 |
| sean-k-mooney | it wont affect the answer just wondering | 13:30 |
| rmart04 | Initial approach was 1VM 8 GPUs, second approach is 2VM's 4 each | 13:30 |
| sean-k-mooney | cool if you can horizontally scale then yes 2 vm with a singel numa node each shoudl give better performance | 13:31 |
| *** dlbewley has quit IRC | 13:31 | |
| sean-k-mooney | since there will be no corss numa trafic fo the vm cpu memory and gpus | 13:31 |
| *** dlbewley has joined #openstack-nova | 13:31 | |
| rmart04 | Thats the plan, but previously I tried this and got a cannot allocate memory issue, which seemed to be related to no dma32 on node1 in /proc/zoneinfo. Which I believe may be due to the older kernel | 13:32 |
| *** boxiang_ has quit IRC | 13:32 | |
| *** boxiang_ has joined #openstack-nova | 13:32 | |
| sean-k-mooney | rmart04: oh you hit that | 13:33 |
| sean-k-mooney | so that is not really a kernel issue so much as a kernl/bios/firmware issue that we worked around with a kvm change | 13:33 |
| sean-k-mooney | rmart04: really there should have been a dma32 region allcoated per numa node | 13:34 |
| sean-k-mooney | the kvm fix was not to require numa affinity for the dma32 region | 13:34 |
| rmart04 | Oh right, interesting. Could you point me at the info for the kvm change? | 13:34 |
| rmart04 | ah Ok, is that strict=false or similar | 13:34 |
| sean-k-mooney | kind of but that would have done it for all the vms memory | 13:35 |
| sean-k-mooney | that was the alternitive workaround | 13:35 |
| rmart04 | Please tell me its fixed in Stein? :D | 13:35 |
| sean-k-mooney | https://lkml.org/lkml/2018/7/24/843 | 13:36 |
| sean-k-mooney | this is not an openstack bug | 13:36 |
| sean-k-mooney | so we did not modify nova | 13:36 |
| sean-k-mooney | what distro are you using | 13:37 |
| rmart04 | ah OK, Yes this is what I was looking at, I thought it was a Kernel patch | 13:37 |
| rmart04 | Centos7 | 13:37 |
| rmart04 | 3.10 kernel | 13:37 |
| sean-k-mooney | it is for the kvm kernel module | 13:37 |
| sean-k-mooney | there might have been another patch too | 13:38 |
| sean-k-mooney | ok i know we backported this in rhel 7 | 13:38 |
| sean-k-mooney | may in 7.6 | 13:38 |
| sean-k-mooney | so hopefully you have that in the lates centos 7 too | 13:39 |
| sean-k-mooney | let me see if i have the bz for it in my history | 13:39 |
| rmart04 | ah that would be amazing | 13:39 |
| sean-k-mooney | so this is the nova patch we decied not to go with https://review.opendev.org/#/c/684375/ partly because we could not test it | 13:43 |
| sean-k-mooney | rmart04: the commit meassage has the links to the relevent bugs and converations | 13:43 |
| sean-k-mooney | hum it look like https://bugzilla.redhat.com/show_bug.cgi?id=1010885#c2 might also be a workaround but i dont think it is | 13:45 |
| openstack | bugzilla.redhat.com bug 1010885 in libvirt "kvm_init_vcpu failed: Cannot allocate memory in NUMA" [Medium,Closed: errata] - Assigned to mkletzan | 13:45 |
| *** dlbewley has quit IRC | 13:50 | |
| *** dlbewley has joined #openstack-nova | 13:51 | |
| rmart04 | Remove cpuset from cgroup controllers? | 13:52 |
| rmart04 | Is that what also makes the pinning work? | 13:53 |
| sean-k-mooney | rmart04: yes but i dont know if that fully disables pinning | 13:53 |
| sean-k-mooney | so the issue is that the wya libvirt appliees the cgrpus it also confines the allcoations of kernel memory | 13:53 |
| sean-k-mooney | one of the fixes that was only a partial fix was to move that later so that the dma region could be allocate before the cpus are pinned | 13:54 |
| sean-k-mooney | that was done in https://libvirt.org/git/?p=libvirt.git;a=commit;h=7e72ac7878 | 13:54 |
| sean-k-mooney | but that was backin 2014 so it obviouslyu was not a full fix or it was broken angain later | 13:55 |
| rmart04 | OK :/ | 13:56 |
| sean-k-mooney | rmart04: this was the final kernel fix i belive https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ee6268ba3a68 | 13:59 |
| sean-k-mooney | that was in 4.19 | 14:00 |
| rmart04 | ah ok, my bad I said 4.14 earliar | 14:00 |
| *** dklyle has joined #openstack-nova | 14:00 | |
| rmart04 | How easy is it to find out whether it was backported in C7? | 14:01 |
| sean-k-mooney | its in the rhel kernel-3.10.0-957.26.1.el7 pacakge and needs libivrt ibvirt-4.5.0-13.el7 or higher | 14:03 |
| rmart04 | OK thats amazing thank you. I think i'll see how far we are away from those package versions and expidite moving to them | 14:06 |
| *** rcernin_ has quit IRC | 14:14 | |
| *** dlbewley has quit IRC | 14:20 | |
| *** dlbewley has joined #openstack-nova | 14:21 | |
| *** ttsiouts has quit IRC | 14:30 | |
| *** mlavalle has joined #openstack-nova | 14:30 | |
| *** ttsiouts has joined #openstack-nova | 14:31 | |
| *** ttsiouts has quit IRC | 14:36 | |
| *** boxiang_ has quit IRC | 14:44 | |
| *** boxiang_ has joined #openstack-nova | 14:45 | |
| mordred | sean-k-mooney: you remember every conversation we've had about things from the past, right? | 14:46 |
| *** ttsiouts has joined #openstack-nova | 14:46 | |
| sean-k-mooney | mordred: ocourse i was right and you were .... | 14:46 |
| sean-k-mooney | mordred: is there a converstation in partcalar? | 14:46 |
| mordred | sean-k-mooney: what's the story with the api_servers config option for glance - we've asked about deprecating/removing it and getting rid of the idea of being a poor-mans-client-side-load-balancer | 14:46 |
| mordred | but I can't remember where the discussion got to on that | 14:47 |
| sean-k-mooney | right we wanted to not do the crappy round robin thing we do in nova anymore | 14:47 |
| mordred | (I'm trying to help cyborg with their glance support, but since it's copied from nova I wnat to make sure anything I do would eventually be transferrable back to nova) | 14:47 |
| sean-k-mooney | i belive dansmith was onboard with that | 14:47 |
| sean-k-mooney | i dont know if we have done anything to change it in nova | 14:47 |
| mordred | yeah. I guess I should look to see if we marked the option as deprecated yet | 14:47 |
| sean-k-mooney | mordred: efried did bring it up but then he had to move on | 14:48 |
| dansmith | I was not on board with removing it, | 14:48 |
| dansmith | but I think I was the only nova person in that position | 14:48 |
| mordred | Support for image service configuration via standard keystoneauth1 Adapter | 14:49 |
| mordred | options was added in the 17.0.0 Queens release. The api_servers option was | 14:49 |
| mordred | retained temporarily to allow consumers time to cut over to a real load | 14:49 |
| mordred | balancing solution. | 14:49 |
| mordred | there's the help text- so it implies that post-queens its existence is temporary | 14:49 |
| mordred | but with no official deprecation story :) | 14:49 |
| mordred | basically - I would like to either kill this or if we can never kill it support it in ksa so that we can stop it with copying the round-robin code everywhere | 14:50 |
| sean-k-mooney | dansmith: oh i just rememebered you had an opion and i generally rememeber when you dont like something so i assume you were ok with it | 14:50 |
| *** dlbewley has quit IRC | 14:50 | |
| *** ociuhandu has joined #openstack-nova | 14:50 | |
| efried | I would have deprecated it if I had been allowed to. I may be misremembering, but I think we put out a RFC on the ML and someone put up their hand and said they were still using it. Might even have been dansmith :P | 14:51 |
| *** dlbewley has joined #openstack-nova | 14:51 | |
| mordred | efried: :) | 14:51 |
| dansmith | no, wasn't me, | 14:51 |
| dansmith | but there are people in redhat, tripleo and edge-related IIRC, that definitely don't want to lose it | 14:52 |
| sean-k-mooney | because its used for rabbit mq? | 14:52 |
| efried | Also, I think I commented on the cyborg stuff when it went in, saying they really shouldn't be carrying all this warty stuff over from nova -- that is, they *never* should have supported [glance]api_servers. But I think they wound up just merging it for expediency. | 14:52 |
| mordred | I thought all those people thought k8s was super sexy - why is a lb hard? | 14:52 |
| dansmith | the history is a little dim for me without digging that back up, but basically for a very small number of remote edge machines, a "real load balancer" is not an option and having nova be able to try multiple glance servers is a major win | 14:53 |
| mordred | nod | 14:53 |
| efried | If ^ is not an issue for cyborg, I say they kill it, with prejudice. | 14:53 |
| mordred | ok - that's fair enough I suppose | 14:53 |
| sean-k-mooney | dansmith: really? ha proxy is pretty light weight | 14:53 |
| efried | esp if it means they're no longer anchored to ksa and can cut over to sdk | 14:53 |
| sean-k-mooney | im not sure i by that given it will be handleing very little traffic | 14:54 |
| dansmith | sean-k-mooney: it's not just the haproxy of course, it's the config, the need for shared L2 failover VIP, pacemaker to manage it, etc | 14:54 |
| dansmith | it's all the complexity that comes with deploying that architecture | 14:54 |
| mordred | sure. although I'll again point my fingers at all of the people rushing to install k8s in those contexts so they should have an easier time of it | 14:55 |
| dansmith | I haven't heard any consistent plans about how to do that in a non-toy arrrangement, fwiw.. lots of chest-puffing and hand waving | 14:55 |
| *** ociuhandu has quit IRC | 14:55 | |
| dansmith | not that it doesn't exist, but just I haven't heard a clear plan | 14:56 |
| dansmith | and all I'm doing is communicating the request | 14:56 |
| mordred | dansmith: fair | 14:56 |
| efried | If there really is a need for api_servers, then I guess we should support it in sdk so that's not something that blocks $consumer from cutting over. | 14:56 |
| mordred | I guess from my end I actually don't care other than wanting to mock people who think they can deploy a cloud but can't deploy a load balancer ... | 14:56 |
| mordred | efried: exactly | 14:56 |
| dansmith | personally, I do not understand how relying on a centralized (even if HA'd) load balancer is better than the clients knowing the options and being able to help themselves out of a failure to talk to one | 14:56 |
| mordred | if this is a thing that has to stay around | 14:56 |
| efried | Easy for me to say, I won't be the one doing it. | 14:56 |
| sean-k-mooney | dansmith: ok that more hevy wait then kolla does so i guess that is different in ooo | 14:57 |
| mordred | then I want to just say screw it and support it in the client layer | 14:57 |
| efried | dansmith: ftr, api_servers isn't really a load-balancer. I think it's a deafdumbandblind round robin. | 14:57 |
| mordred | it's way too fundamental of a piece of config to stich in in the way it's being done | 14:57 |
| mordred | a very dumb one | 14:57 |
| mordred | it doesn't do retries | 14:57 |
| efried | or failovers, or anything. | 14:57 |
| mordred | yah | 14:57 |
| efried | You simply get the next one on each subsequent api req | 14:57 |
| mordred | it's probalby _less_ HA | 14:58 |
| efried | heh | 14:58 |
| efried | indeed | 14:58 |
| mordred | maybe it's time to one-more-time go to the mailing list | 14:58 |
| dansmith | I understand, but it still gets you 66% success instead of zero if 1/3 is down, and it could be made to do the right thing | 14:58 |
| mordred | maybe life has changed for peopel since queens | 14:58 |
| dansmith | mordred: this was not queens when we had this discussion, fwiw | 14:58 |
| sean-k-mooney | mordred: for the better? | 14:58 |
| mordred | dansmith: no? that's the release mentioned in the nova config docs | 14:59 |
| dansmith | mordred: anyway, don't use my name in the email, I'm not prepared to argue for it anymore | 14:59 |
| mordred | dansmith: I won't | 14:59 |
| dansmith | mordred: that's when it was deprecated I think, but the concern about removing it was within the last year | 14:59 |
| mordred | nod | 14:59 |
| dansmith | but also, we were told that it was going to go away regardless so those people are assuming it's going or gone I think | 14:59 |
| mordred | oh - well that's something perhaps | 15:00 |
| artom | stephenfin, did the flavor extra spec validation spec ever go anywhere? | 15:00 |
| stephenfin | artom: it landed last cycle | 15:00 |
| stephenfin | API microversion 2.84, I think? | 15:00 |
| *** dlbewley has quit IRC | 15:00 | |
| *** dlbewley has joined #openstack-nova | 15:01 | |
| *** Luzi has joined #openstack-nova | 15:01 | |
| artom | stephenfin, 2.86, but yeah, thanks for the pointer | 15:01 |
| artom | So we could use that for the cpu_dedicated_mask, right? | 15:01 |
| efried | Deprecate: https://review.opendev.org/#/c/692227/ (merged) | 15:02 |
| efried | I don't see a patch (at least owned by me) proposing removing it. | 15:02 |
| artom | Context is my latest comment on https://review.opendev.org/#/c/468203/15 | 15:02 |
| efried | Comments in above patch contain ML links mordred | 15:03 |
| efried | and also refers to some people who wanted to keep it. | 15:03 |
| stephenfin | artom: fair comment. In the middle of something at the moment but I'll get a reply to it | 15:04 |
| stephenfin | we can't rely on the validation though, mind you, since it's an API change | 15:04 |
| artom | stephenfin, no rush. Today's my "upstream day", trying to do some reviews where I can :) | 15:04 |
| stephenfin | *in an API microversion | 15:04 |
| *** mlavalle has quit IRC | 15:05 | |
| *** ttsiouts has quit IRC | 15:07 | |
| *** ttsiouts has joined #openstack-nova | 15:07 | |
| mordred | efried: ooh! so it is deprecated! | 15:07 |
| *** breizhkoala has quit IRC | 15:08 | |
| mordred | efried: how many releases do we have to wait to remove a deprecated thing? can we remove that now in victoria? or do we have to wait until w? | 15:08 |
| *** mgariepy has quit IRC | 15:08 | |
| *** ttsiouts has quit IRC | 15:09 | |
| dansmith | mordred: should probably ask the ptl | 15:09 |
| *** mlavalle has joined #openstack-nova | 15:09 | |
| *** ttsiouts has joined #openstack-nova | 15:09 | |
| stephenfin | mordred: it's six months or one release, whatever is greater | 15:10 |
| stephenfin | though that's obviously tempered by impact | 15:10 |
| dansmith | I thought it was two releases for config options | 15:10 |
| dansmith | you can't remove it in the N+1 release, but you can in the N+2, | 15:10 |
| dansmith | else we'd break N->N+1 config compatibility | 15:10 |
| mordred | gotcha. so we can remove in W | 15:11 |
| mordred | but not today | 15:11 |
| stephenfin | you deprecate in N, and remove in N+1 | 15:12 |
| stephenfin | if you removed in N, you'd be breaking N-1 -> N compatibility | 15:12 |
| stephenfin | that gives people the duration of N to get off $deprecatedthing | 15:13 |
| dansmith | isn't N-1->N+1 the same thing I said, but with N->N+2? | 15:13 |
| *** ttsiouts has quit IRC | 15:14 | |
| dansmith | oh, you're saying we deprecated it in U so people in U should know to get off of it so it can be gone in V? I guess technically that's right, but I really thought it was supposed to be two releases | 15:14 |
| stephenfin | yes :) you had me confused there | 15:15 |
| stephenfin | docs say the same https://docs.openstack.org/nova/latest/contributor/process#smooth-upgrades | 15:15 |
| stephenfin | i.e. continue to support and test features for at least one release before they are removed | 15:15 |
| sean-k-mooney | ya so 1 release is the miniume we usuall take 2-3 to remove it | 15:16 |
| *** mlavalle has quit IRC | 15:16 | |
| stephenfin | definitely not black and white | 15:16 |
| sean-k-mooney | depeneing on how much of a pain it is | 15:16 |
| stephenfin | sean-k-mooney: if we ever remove it :) | 15:16 |
| dansmith | given it could break people's glance setup without warning, | 15:16 |
| dansmith | I'd say it's pretty high impact | 15:16 |
| stephenfin | "will be removed in a future release" <-- my boilerplate | 15:16 |
| sean-k-mooney | we would need a nova-status check at a minium right | 15:16 |
| stephenfin | I wouldn't say without warning, since we do have a big "this thing is deprecated" warning | 15:17 |
| sean-k-mooney | also ooo would have to be modified proably | 15:17 |
| dansmith | sean-k-mooney: for something like this, yeah I'd think so | 15:17 |
| stephenfin | I guess it comes down to how difficult the migration is | 15:18 |
| stephenfin | from $old_way to $new_way | 15:18 |
| stephenfin | I don't know about that so I'll defer to others | 15:18 |
| sean-k-mooney | old way is template a config option. new way is either add the service to a load blancer or deploy a new loadbalncer and add it to that | 15:25 |
| sean-k-mooney | i think the only thing we use this for in ooo wa rabbitmq but i could be wrong about that | 15:26 |
| sean-k-mooney | i think in ooo we go to glance vai haproxy | 15:27 |
| sean-k-mooney | but in general people could have been using this for any of the api endpoint? | 15:27 |
| *** mgariepy has joined #openstack-nova | 15:28 | |
| *** slaweq has quit IRC | 15:33 | |
| *** huaqiang has joined #openstack-nova | 15:34 | |
| *** mlavalle has joined #openstack-nova | 15:35 | |
| *** hamalq has joined #openstack-nova | 15:39 | |
| *** ociuhandu has joined #openstack-nova | 15:40 | |
| *** links has quit IRC | 15:44 | |
| *** hamalq has quit IRC | 15:45 | |
| gibi | nova weekly meeting will start in 12 minutes on #openstack-meeting-3 | 15:47 |
| mordred | sean-k-mooney: well - we wouldnt' be removing this for rabbit | 15:49 |
| *** bhagyashris is now known as bhagyashris|away | 15:49 | |
| mordred | sean-k-mooney: this is only about talkig to openstack api services | 15:49 |
| mordred | it's possible we've not been properly clear about that | 15:49 |
| *** dlbewley has quit IRC | 15:50 | |
| sean-k-mooney | mordred: why would we remove it for one and not the other | 15:50 |
| *** dlbewley has joined #openstack-nova | 15:50 | |
| sean-k-mooney | its equally bad in both cases | 15:50 |
| sean-k-mooney | or good depending on your view | 15:50 |
| mordred | sean-k-mooney: "use services from the keystone catalog and via the normal api consumption channels" is the driver | 15:50 |
| *** gyee has joined #openstack-nova | 15:50 | |
| mordred | rabbit doesn't get configured via the catalog nor consumed using a standard set of connection information | 15:51 |
| *** priteau has joined #openstack-nova | 15:53 | |
| sean-k-mooney | mordred: have you tought about a keystone feature to allow the same behavior :P | 15:53 |
| mordred | yes - it was _strongly_ opposed by the keystone team - becaues they did not want to build a bad load balancer | 15:54 |
| sean-k-mooney | but ok if its the openstack services only im not sure if that changes the edge configuration | 15:54 |
| mordred | yeah - and really, it's ONLY glance | 15:54 |
| mordred | that's the only place this option exists | 15:54 |
| *** spatel has quit IRC | 15:55 | |
| mordred | so it's not even that there is a generalized mechanism for talking to api services with a list of endpoits in nova -- it's _just_ for glance | 15:55 |
| *** spatel has joined #openstack-nova | 15:55 | |
| mordred | and I'm pretty sure the reason it's special is that way back in the day glance wasn't considered user-facing, so why would you have it in your load balancer | 15:55 |
| sean-k-mooney | well in the edge configurtion you want your edge site to use a local glance api | 15:56 |
| sean-k-mooney | and leavge the glance multistore feature | 15:56 |
| sean-k-mooney | now i dont know if that needs this | 15:57 |
| sean-k-mooney | or can we configure the glance store seperatly and have multiple endpoints in keystone | 15:57 |
| sean-k-mooney | but i feel like we would need the later? | 15:58 |
| sean-k-mooney | you dont want just one endpoint as you would have to do anycast routing tricks if you wanted to hit the local api | 15:59 |
| *** Luzi has quit IRC | 16:01 | |
| *** jobewan has quit IRC | 16:09 | |
| mordred | sean-k-mooney: well - for those you can totally still use endpoint override | 16:10 |
| mordred | I think the thing is that in general for a consumer the idea is that an openstack service has "an endpoint" - and that concept is pretty baked in to many things | 16:10 |
| mordred | but it's always possible for a consumer to override "the endpoint" that keystone tells it | 16:11 |
| sean-k-mooney | isnt that what the config option is for | 16:11 |
| mordred | if we wanted to do a thing like your second case - it's obviously possible - but it would really take a coordinated design and implementation in a few places | 16:11 |
| sean-k-mooney | i know it can do the loadbalncing tooo | 16:11 |
| mordred | no - the config option that we want to remove is for giving a _list_ of overrides | 16:12 |
| openstackgerrit | Merged openstack/nova master: Remove hacking rules for python 2/3 compatibility https://review.opendev.org/733987 | 16:12 |
| mordred | and that concept exists in one and only one place - inside of nova for overriding glance | 16:12 |
| sean-k-mooney | right but do we have a way to locally override without that | 16:12 |
| mordred | yes | 16:12 |
| sean-k-mooney | for glance | 16:12 |
| sean-k-mooney | ok | 16:12 |
| mordred | that doesn't go away | 16:12 |
| sean-k-mooney | right i just was not sure if we had two way for glance | 16:13 |
| mordred | that just gets to use the normal mechanism and nova can remove a pile of special case code and we can go through and simplify several things | 16:13 |
| sean-k-mooney | on that supports a list and another that is a single url | 16:13 |
| mordred | yup. that's what you have currently | 16:13 |
| *** sangeet has joined #openstack-nova | 16:17 | |
| *** belmoreira has quit IRC | 16:17 | |
| sangeet | I have SSL enabled for keystone. My compute service fails to come up due to SSL 'certificate verify failed'. I am not sure if I am placing the certifcate at the correct place. Where should the certifciate go? I tried to palce it in /etc/nova/certs/ca.crt. it did not work. Then I set CA_CERTS, it still did not work. Any help will be highly apprecaited. I am running Stein | 16:19 |
| sean-k-mooney | sangeet: you need to add it to the openerating systems certificat store | 16:20 |
| sean-k-mooney | i dont think we support passing a ca directly to nova since now does not realy know anything about tls | 16:21 |
| sangeet | nova-api is working fine for me. I placed the certs in /etc/nova/certs folder | 16:21 |
| *** songwenping_ has joined #openstack-nova | 16:22 | |
| sean-k-mooney | is the api runnign on the host that generated teh cert | 16:22 |
| sangeet | No | 16:23 |
| sangeet | But I converted api to wsgi | 16:23 |
| sean-k-mooney | the only ca config option i see are related to rabbitmq and we dont mention this in https://docs.openstack.org/nova/latest/admin/security.html | 16:23 |
| sean-k-mooney | actully we have options for vendor data too | 16:24 |
| *** bbowen_ has quit IRC | 16:24 | |
| sangeet | Issue is compuet is trying to get token from keystone and since keystone supports SSL it faile | 16:24 |
| sean-k-mooney | sangeet: right but thats becasue you have not added the private ca to the operating systsms cert store | 16:25 |
| *** songwenping__ has quit IRC | 16:25 | |
| sean-k-mooney | you should be adding the ca cert in /usr/local/share/ca-certificates/ and then do sudo update-ca-certificates | 16:27 |
| *** jmlowe has quit IRC | 16:27 | |
| *** jmlowe has joined #openstack-nova | 16:29 | |
| *** hamalq has joined #openstack-nova | 16:31 | |
| *** hamalq has quit IRC | 16:32 | |
| *** hamalq has joined #openstack-nova | 16:33 | |
| *** salmankhan has quit IRC | 16:34 | |
| *** hamalq_ has joined #openstack-nova | 16:44 | |
| *** hamalq has quit IRC | 16:47 | |
| *** avolkov has quit IRC | 16:51 | |
| *** bbowen has joined #openstack-nova | 16:58 | |
| *** rpittau is now known as rpittau|afk | 17:00 | |
| *** derekh has quit IRC | 17:02 | |
| *** dlbewley has quit IRC | 17:02 | |
| *** dlbewley has joined #openstack-nova | 17:02 | |
| *** dtantsur is now known as dtantsur|afk | 17:10 | |
| *** tesseract has quit IRC | 17:21 | |
| sangeet | sean-k-mooney .. sorry had to run for an appointment. Should the cert be in /usr/local/share/ca-certificates/ or /var/lib/openstack/lib/python3.6/site-packages/certifi/cacert.pem? certifi.where() shows later. Also name "ca.crt" is the correct name? | 17:50 |
| mordred | do we not expose the keystoneauth session ssl options? | 17:53 |
| sean-k-mooney | mordred: maybe but that would be documented in keystoneauth | 17:54 |
| sean-k-mooney | not nova | 17:54 |
| sean-k-mooney | sangeet: it should be in /usr/local/share/ca-certificates/ | 17:54 |
| sean-k-mooney | since they are your own addtional CA certs not one packaged by the distro | 17:55 |
| mordred | sean-k-mooney: that won't necessarily work | 17:55 |
| *** dlbewley has quit IRC | 17:55 | |
| mordred | sean-k-mooney: python requests bundles a CA bundle and doesn;t' use system cas | 17:55 |
| mordred | because MONKEYS | 17:55 |
| *** dlbewley has joined #openstack-nova | 17:55 | |
| sean-k-mooney | ... | 17:56 |
| mordred | don't even get me started | 17:56 |
| sean-k-mooney | ok so do we have docs for how to configure keystone with tls somehwere | 17:56 |
| mordred | which is why it's important to be able to pass a path to a CA in config - I believe nova is doing a register_conf_options from ksa - so it should be possible to pass cafile in nova.conf ... | 17:56 |
| * mordred goes looking | 17:56 | |
| sean-k-mooney | its not in the nova security docs | 17:56 |
| mordred | well - that might be a bug - but let me see what I can find | 17:56 |
| sean-k-mooney | mordred: ya i think we do | 17:57 |
| *** ralonsoh has quit IRC | 17:57 | |
| sean-k-mooney | i guess it could be in the install docs i just did a google search and didnt find them | 17:57 |
| mordred | register_ksa_opts | 17:57 |
| mordred | that's in nova/conf/utils - and calls ks_loading.register_session_conf_options - which will register an option "cafile" | 17:58 |
| mordred | it's going to be per-service - so I think it would be [identity]cafile=/etc/nova/certs/ca.crt - but really it would be best to put it in [networking]cafile= and [image]cafile and [placement]cafile= too ... | 18:01 |
| mordred | I'm not 100% sure what the story would be if keystone and a given service need _different_ ca's | 18:01 |
| mordred | sangeet: ^^ | 18:01 |
| sangeet | thanks mordred .. so I onlty need to change the conf file as you suggested and it will be used by ksa automatically? | 18:02 |
| mordred | sangeet: yes. at least I hope so - that's the theory :) | 18:02 |
| sangeet | Let me try that .. my system is up. Thanks | 18:03 |
| mordred | woot! | 18:03 |
| mordred | we should really make a general [session] config section - having those options repeat for each service is a little weird | 18:03 |
| mordred | sean-k-mooney: I feel like I should update the nova docs to include this information - but I'm not really sure where would be a good idea for that | 18:04 |
| mordred | efried: isn't there docs somewhere about the ksa conf options | 18:06 |
| mordred | ? | 18:06 |
| efried | ... | 18:06 |
| mordred | efried: like - now that you can configure session adapter stuff via service-types-authority names and the ksa options ... do we have docs about that in the nova docs? | 18:07 |
| * mordred can't find - but doesn't know where to look | 18:07 | |
| *** mriedem has joined #openstack-nova | 18:07 | |
| efried | I think I understand the question, I'm just trying to swap that back in from tape. | 18:08 |
| efried | mordred: https://docs.openstack.org/nova/latest/configuration/config.html | 18:09 |
| efried | If you search for e.g. `cafile` you'll find an entry for each $service that uses ksa. | 18:09 |
| efried | six entries for `endpoint_override` | 18:09 |
| mordred | efried: ah - oh, that's probably generated from ksa by sphinx | 18:09 |
| efried | exactly | 18:09 |
| mordred | so a git grep wasn't finding it - that makes sense | 18:10 |
| mordred | efried: thanks! | 18:10 |
| efried | yw | 18:10 |
| efried | not sure about 'sphinx', but generated by doc build, yes. | 18:10 |
| *** factor__ has quit IRC | 18:11 | |
| efried | ...and I think it comes in by virtue of the `list_opts()` methods in conf/*, e.g. https://github.com/openstack/nova/blob/master/nova/conf/glance.py#L173 | 18:11 |
| efried | ...which as you can see uses ksa's methods for generating those | 18:11 |
| *** factor__ has joined #openstack-nova | 18:11 | |
| mordred | efried: yah. \o/ yay | 18:13 |
| *** rmart04 has quit IRC | 18:14 | |
| sangeet | ten thousand thansk modred .. it worked. I am so exicted. QQ Do I need to put it under idenity also or neutron, glance and placement is enough? | 18:19 |
| sangeet | Sorry *mordred ^^ | 18:19 |
| sean-k-mooney | mordred: i was expecting to find it here https://docs.openstack.org/nova/latest/admin/security.html | 18:21 |
| sean-k-mooney | mordred: although the install guide would make sense | 18:21 |
| sean-k-mooney | efried: mordred i dont think the config guide is really helpful in this case | 18:22 |
| sean-k-mooney | that is where i started but i did not find it mainly because i was looking for ca_ | 18:22 |
| sean-k-mooney | but it was not obvious | 18:22 |
| efried | I don't think we should describe the optinos in depth in the security guide, but it would be sane to refer to the config docs from there. | 18:23 |
| sean-k-mooney | yep that is what i was thinking too altough i think having a secting in the install guide would make sense | 18:24 |
| sean-k-mooney | e.g. how to isntall with tls? | 18:24 |
| sangeet | I agree .. that would be an excellent idea | 18:27 |
| mordred | sangeet: you'll likely need to put it in for each service you're using | 18:28 |
| mordred | sangeet: so - yeah - I'd do identity I think | 18:29 |
| *** bbowen has quit IRC | 18:37 | |
| *** bbowen has joined #openstack-nova | 18:37 | |
| sangeet | Thank you mordred | 18:38 |
| *** songwenping__ has joined #openstack-nova | 18:41 | |
| *** songwenping_ has quit IRC | 18:44 | |
| *** maciejjozefczyk has quit IRC | 18:52 | |
| *** vishalmanchanda has quit IRC | 18:55 | |
| *** songwenping_ has joined #openstack-nova | 19:00 | |
| *** jdillaman has quit IRC | 19:01 | |
| *** songwenping__ has quit IRC | 19:03 | |
| sangeet | mordred .. oops now my nova-conductor is not liking it when I try to create a server. I have set cafiles as we discussed above. "OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/nova/certs/ca.crt" | 19:12 |
| sangeet | It seems conductor is expecting the CA file to be at some other location | 19:13 |
| mordred | sangeet: are those on the same machine? | 19:17 |
| sangeet | differnt pods | 19:17 |
| sangeet | the file exist | 19:18 |
| mordred | hrm. I'm not sure about that one - maybe someone else will know | 19:18 |
| sangeet | sean-k-mooney efried .. please help ^^ | 19:19 |
| efried | I'm no expert here, so this is just a guess: | 19:20 |
| efried | If you put this in [identity], it means all the nova services will try to use it when talking to keystone. | 19:20 |
| sangeet | I am up for trying anything | 19:20 |
| efried | So you need it on every node that's running any nova service (conductor, compute, scheduler, whatever) | 19:21 |
| efried | it == the crt fil. | 19:21 |
| efried | file | 19:21 |
| sangeet | so put cafile=/etc/nova/certs/ca.crt under identity | 19:21 |
| efried | eh? I thought that's what you did, and it didn't work | 19:21 |
| efried | Let's back up. | 19:22 |
| efried | What change did you make that's leading to this error? | 19:22 |
| sangeet | I ut it in compute and not in conductore | 19:22 |
| sangeet | Let me try to put it under identity also | 19:22 |
| *** gmann is now known as gmann_afk | 19:22 | |
| efried | waitwait | 19:22 |
| efried | I haven't been following this conversation, so I don't want to lead you down a rabbit hole. | 19:22 |
| efried | What exactly have you changed so far? | 19:22 |
| sangeet | I have Keystone deployed with SSL. In my nova.conf I set cafile=/etc/nova/certs/ca.crt for [neutron], [glance], [keystone_authtoken], [placement] | 19:25 |
| efried | Hm, okay, I'm not sure about [keystone_authtoken] -- that's to set up the server side of the keystone service. | 19:26 |
| efried | But nova talks to some of those services from multiple places -- conductor, scheduler, compute. | 19:27 |
| efried | I don't remember offhand which ones talk to which ones from where. | 19:27 |
| efried | Also note that conductor and compute use different config files by default (unless that's changed, or unless you're on an old release), so if you're running both, you'll want to set up both files. | 19:29 |
| efried | But honestly, beyond that, I'm really out of my depth. sean-k-mooney would probably be a better resource. He is EU, so maybe try again "tomorrow". | 19:29 |
| sangeet | I am using stein and I do have different configs as they are running on a different pods | 19:30 |
| sean-k-mooney | efried: is that a hint i should go get dinner because it should be just cooked :) | 19:31 |
| efried | sean-k-mooney: yes, you should not be working right now. | 19:32 |
| efried | sangeet: TL;DR: I suspect you need the crt file available/accessible on the same file system as the nova.conf file you're editing. | 19:32 |
| efried | same file system in the same container etc. | 19:33 |
| sean-k-mooney | efried: hehe if i had started at 11 like i normally do i would argue but since i started at 8am today, night all o/ | 19:33 |
| efried | But I'm also really not sure what mucking with [keystone_authtoken] did for you. | 19:33 |
| *** xinranwang_ has quit IRC | 19:33 | |
| *** jsuchome has quit IRC | 19:34 | |
| *** gmann_afk is now known as gmann | 19:37 | |
| *** lvdombrkr has quit IRC | 19:40 | |
| sangeet | efried .. yes cafile needed to be set in identity section on conductor. It works now. | 20:08 |
| efried | \o/ | 20:08 |
| efried | Not to confuse things, but is it possible it's *only* required in [identity]? | 20:09 |
| efried | I don't know whether/where we talk to keystone as an actual client; we may just be using it to set up the connection to the other services. But... I really don't know how that all works. | 20:10 |
| *** gmann is now known as gmann_afk | 20:10 | |
| *** nweinber has quit IRC | 20:14 | |
| mordred | efried: yeah - I'm gonna try to do an audit through of that | 20:17 |
| efried | cool | 20:17 |
| mordred | efried: cause right now I think it's ... well, if you don't know and I don't know - then likely nobody knows | 20:17 |
| efried | lbragstad might :P | 20:18 |
| efried | or cmurphy. Or other people who are no longer stacking. | 20:18 |
| *** slaweq has joined #openstack-nova | 20:23 | |
| lbragstad | i actually didn't realize nova had an identity section *and* a keystone_authtoken section | 20:24 |
| efried | IIUC, the former is for talking to keystone as a client, and the latter is for setting nova up as a server. | 20:24 |
| lbragstad | (the keystone_authtoken section comes from keystonemiddleware) | 20:24 |
| efried | tbh, I'm not sure where we're actually using the former from nova. But we must be, or sangeet's change wouldn't have fixed the problem... | 20:25 |
| lbragstad | yeah - iirc (and i'm probably dated here) [keystone_authtoken] is only invoked by keystonemiddleware to fetch information about tokens out of keystone | 20:27 |
| lbragstad | ahh - it looks like the [identity] section is used to validate project IDs | 20:31 |
| efried | "used" -- by nova? | 20:31 |
| efried | or by, like, everyone who has one? | 20:32 |
| lbragstad | https://opendev.org/openstack/nova/src/branch/master/nova/api/openstack/identity.py | 20:32 |
| efried | Interesting. | 20:33 |
| lbragstad | maybe? i might not be following this properly | 20:33 |
| lbragstad | i think this is what i was looking at https://docs.openstack.org/nova/latest/configuration/config.html#keystone | 20:34 |
| efried | Oh, that's for sure a place where we're using the [identity] section to create a ksa adapter that talks as a client to the keystone service's API. | 20:34 |
| *** priteau has quit IRC | 20:34 | |
| lbragstad | https://docs.openstack.org/nova/latest/configuration/config.html#keystone-authtoken is the keystonemiddleware section | 20:35 |
| efried | Yeah, some layer of this translates $service_name to $project_name or vice versa | 20:35 |
| efried | yes. | 20:35 |
| lbragstad | i don't actually see an [identity] section | 20:35 |
| lbragstad | in the configuration reference | 20:35 |
| lbragstad | i see [keystone] and [keystone_authtoken] | 20:36 |
| efried | I believe you're allowed to use either [identity] or [keystone] for anything we load up via get_{sdk|ksa}_adapter. | 20:36 |
| efried | I mean, [$service] or [$project] | 20:36 |
| lbragstad | oh - so service types and service names are interchangable? | 20:36 |
| lbragstad | interchangeable* | 20:36 |
| efried | only if we're using get_{sdk|ksa}_adapter. | 20:36 |
| efried | Which I think we are for everything except cinder at this point. | 20:37 |
| lbragstad | ok | 20:37 |
| efried | And by "we" I mean "they". I don't work here anymore :P | 20:37 |
| lbragstad | :) | 20:38 |
| *** haleyb has quit IRC | 20:47 | |
| *** maciejjozefczyk has joined #openstack-nova | 20:48 | |
| *** haleyb has joined #openstack-nova | 20:55 | |
| *** xek has quit IRC | 20:56 | |
| mordred | efried: I was about to ask you about cinder | 21:02 |
| mordred | efried: I'm guessing I should put that on my list too | 21:02 |
| efried | I made several attempts at it over the years, but they never quiiite landed. | 21:02 |
| mordred | (mostly noticed that there's a weird os-region-name option) | 21:02 |
| efried | I think there's still an open change under my name | 21:02 |
| mordred | efried: cool. maybe I'll finish that - with your name on it it'll have more credibility | 21:03 |
| efried | There were significant difficulties with the (in)compatibility between ksa and cinderclient. | 21:03 |
| efried | hah! | 21:03 |
| mordred | in fact, I might just push up changes for this forging the author to say they're from you | 21:03 |
| mordred | efried: "surprising" | 21:03 |
| efried | https://review.opendev.org/508345 | 21:04 |
| efried | https://review.opendev.org/655985 | 21:04 |
| efried | (I haven't opened those, they're just the ones on my dashboard with 'cinder' in the title) | 21:04 |
| *** raildo has quit IRC | 21:10 | |
| *** mloza has quit IRC | 21:15 | |
| *** hamalq has joined #openstack-nova | 21:16 | |
| *** hamalq__ has joined #openstack-nova | 21:17 | |
| *** hamalq_ has quit IRC | 21:18 | |
| *** hamalq has quit IRC | 21:20 | |
| melwitt | dansmith: left comments on the bottom two rbd multi store patches. bottom one is a docstring update needed, next patch looks to be one small case of missing test coverage and a couple doc-related things | 21:26 |
| dansmith | melwitt: thanks | 21:39 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Plumb image import functionality through our glance module https://review.opendev.org/731550 | 21:45 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make libvirt able to trigger a backend image copy when needed https://review.opendev.org/656998 | 21:45 |
| *** hamalq__ has quit IRC | 21:49 | |
| *** spatel has quit IRC | 21:52 | |
| *** rcernin_ has joined #openstack-nova | 21:54 | |
| *** markvoelker has joined #openstack-nova | 21:58 | |
| *** markvoelker has quit IRC | 22:02 | |
| *** gmann_afk is now known as gmann | 22:04 | |
| *** tonyb has joined #openstack-nova | 22:05 | |
| *** eharney has quit IRC | 22:08 | |
| *** rcernin_ has quit IRC | 22:13 | |
| *** slaweq has quit IRC | 22:13 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/nova master: libvirt: check for AMD SEV only on x86-64 https://review.opendev.org/714425 | 22:18 |
| *** rchurch has quit IRC | 22:24 | |
| *** tosky has quit IRC | 22:25 | |
| *** dlbewley has quit IRC | 22:25 | |
| *** dlbewley has joined #openstack-nova | 22:26 | |
| *** rchurch has joined #openstack-nova | 22:27 | |
| melwitt | gmann: is/was there a change yet to make stable/stein grenade n-v? wondering if it's safe to recheck changes yet | 22:37 |
| melwitt | also note that nova-live-migration is failing on stable/stein too | 22:37 |
| melwitt | saying virtualenv missing | 22:38 |
| gmann | melwitt: i am trying to get working on train with this and tomorrow I am planning for making it n-v on stein - https://review.opendev.org/#/c/736284/9 | 22:38 |
| melwitt | ack | 22:38 |
| gmann | nova-live-migration is difficult one as that is legacy job, on devstack, grenade we fixed that error - https://review.opendev.org/#/c/736750/1 | 22:39 |
| gmann | melwitt: but there are multiple updates happening on devstack/grenade and image side. so let's see how it behaves once we get all those in | 22:41 |
| gmann | clark is updating images also which were not present due to disk full. | 22:41 |
| *** sangeet has quit IRC | 22:59 | |
| *** tkajinam has joined #openstack-nova | 23:00 | |
| *** rcernin_ has joined #openstack-nova | 23:15 | |
| *** rcernin_ has quit IRC | 23:15 | |
| *** rcernin has joined #openstack-nova | 23:16 | |
| *** dlbewley has quit IRC | 23:32 | |
| *** dlbewley has joined #openstack-nova | 23:33 | |
| *** tetsuro has joined #openstack-nova | 23:47 | |
| *** dlbewley has quit IRC | 23:53 | |
| *** dlbewley has joined #openstack-nova | 23:53 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!