| mnasiadka | SvenKieske: not blocking, but still we need a reviewer outside StackHPC :) | 04:49 |
|---|---|---|
| opendevreview | Merged openstack/kolla master: dev-mode: Run kolla_install_projects using sudo https://review.opendev.org/c/openstack/kolla/+/930559 | 06:07 |
| kevko | mnasiadka: reviewed | 06:28 |
| kevko | morning | 06:28 |
| kevko | btw .... my message from yesterday 'mnasiadka: btw, you mentioned yesterday that you will check my patches tomorrow (today) ...did you have a time ?' | 06:29 |
| kevko | >> https://review.opendev.org/c/openstack/kolla/+/928956 anybody for review .. another review approved waiting for this to be merged :/ | 06:43 |
| kevko | anybody here ? or is it already saturday ? :D | 07:40 |
| SvenKieske | o/ I'm following along with half an eye if something really urgent crops up. I will make room for some hours over the weekend to catch up on at least some of the normal reviews, though I'm unfortunately also time constrained there. | 08:04 |
| sylvr | Good morning ! Is there a project to reintroduce ceph deployment with kolla-ansible/kayobe ? if yes where can I follow the development ? thanks :) | 08:25 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Bump previous release to 2024.1 in Dalmatian https://review.opendev.org/c/openstack/kayobe/+/930277 | 08:43 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Add support for Ubuntu Noble Numbat (24.04) LTS https://review.opendev.org/c/openstack/kayobe/+/930026 | 08:45 |
| opendevreview | Bartosz Bezak proposed openstack/kolla master: Revert "Pin OpenSearch Dashboards to 2.15" https://review.opendev.org/c/openstack/kolla/+/930685 | 09:11 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Bump previous release to 2024.1 in Dalmatian https://review.opendev.org/c/openstack/kayobe/+/930277 | 09:12 |
| opendevreview | Bartosz Bezak proposed openstack/kolla stable/2024.1: Revert "Pin OpenSearch Dashboards to 2.15" https://review.opendev.org/c/openstack/kolla/+/930686 | 09:28 |
| opendevreview | Bartosz Bezak proposed openstack/kolla stable/2023.2: Revert "Pin OpenSearch Dashboards to 2.15" https://review.opendev.org/c/openstack/kolla/+/930687 | 09:29 |
| opendevreview | Bartosz Bezak proposed openstack/kolla stable/2023.1: Revert "Pin OpenSearch Dashboards to 2.15" https://review.opendev.org/c/openstack/kolla/+/930688 | 09:29 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: CI: Use threads = ansible_processor_vcpus https://review.opendev.org/c/openstack/kolla/+/930689 | 09:30 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Bump previous release to 2024.1 in Dalmatian https://review.opendev.org/c/openstack/kayobe/+/930277 | 09:38 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Add support for Ubuntu Noble Numbat (24.04) LTS https://review.opendev.org/c/openstack/kayobe/+/930026 | 09:38 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Get the list of ironic nodes - use correct scope https://review.opendev.org/c/openstack/kayobe/+/930691 | 10:06 |
| PrzemekK | How to add to kolla-ansible flat network ? On compute its ens256 (physnet4?) In /etc/kolla/neutron-server/ml2_conf.ini i see [ml2_type_flat] flat_networks = physnet1 | 10:06 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Get the list of ironic nodes - use correct scope https://review.opendev.org/c/openstack/kayobe/+/930691 | 10:15 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Bump previous release to 2024.1 in Dalmatian https://review.opendev.org/c/openstack/kayobe/+/930277 | 10:16 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Add support for Ubuntu Noble Numbat (24.04) LTS https://review.opendev.org/c/openstack/kayobe/+/930026 | 10:17 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Add support for Ubuntu Noble Numbat (24.04) LTS https://review.opendev.org/c/openstack/kayobe/+/930026 | 10:18 |
| kevko | frickler: replied | 10:52 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Add size limits to Fluentd buffers https://review.opendev.org/c/openstack/kolla-ansible/+/924359 | 11:23 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe master: kolla-build: Add support for cross-arch builds https://review.opendev.org/c/openstack/kayobe/+/930204 | 11:24 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe master: kolla-build: Add support for cross-arch builds https://review.opendev.org/c/openstack/kayobe/+/930204 | 11:25 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Get the list of ironic nodes - use correct scope https://review.opendev.org/c/openstack/kayobe/+/930691 | 11:26 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Bump previous release to 2024.1 in Dalmatian https://review.opendev.org/c/openstack/kayobe/+/930277 | 11:26 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Add support for Ubuntu Noble Numbat (24.04) LTS https://review.opendev.org/c/openstack/kayobe/+/930026 | 11:26 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe master: kolla-build: Add support for cross-arch builds https://review.opendev.org/c/openstack/kayobe/+/930204 | 11:33 |
| opendevreview | Merged openstack/kolla master: Revert "Pin OpenSearch Dashboards to 2.15" https://review.opendev.org/c/openstack/kolla/+/930685 | 12:15 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe master: kolla-build: Add support for cross-arch builds https://review.opendev.org/c/openstack/kayobe/+/930204 | 12:28 |
| Eldiabolo_ | Hi people, would it be okay, to ask for help here regarding pci-passthrough problems with kolla? | 12:52 |
| SvenKieske | Eldiabolo: sure, but it's friday. in general kolla is not that special, did you review the docs already? https://docs.openstack.org/kolla-ansible/latest/reference/networking/sriov.html what do you want to know? | 13:21 |
| SvenKieske | ah sorry, misspelled your name, Eldiabolo_ :D | 13:22 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Get the list of ironic nodes - use correct scope https://review.opendev.org/c/openstack/kayobe/+/930691 | 13:28 |
| kevko | SvenKieske: any time for some review :) ? | 13:40 |
| Eldiabolo_ | Thank you Sven. Docs are reviewed. I'm not even at the stage of SRIOV, i want to passthrough a whole PCIe Device (Nvidia A100). GPU is bound to VFIO and also picked up by openstack. Shows up with "openstack allocation candidate list --resource CUSTOM_PCI_10DE_20B0=1". | 13:41 |
| Eldiabolo_ | just when spawning an instance with the gpu flavor, nova-scheduler logs "Dropped 4 device(s) due to mismatched PCI attribute(s)" (theres 4 A100 in the host, so makes sense). I'm note sure whey they are dropped, the PCI IDs match 100% with what is configured in the nova confs | 13:42 |
| Eldiabolo_ | If you need better overview, i also started a reddit thread: https://www.reddit.com/r/openstack/comments/1fqkoyk/nova_dropping_pci_devices_due_to_missmatched/ | 13:44 |
| SvenKieske | Eldiabolo_ do you have the necessary nova scheduler filter enabled? "PciPassthroughFilter" (it's also mentioned in above sriov docs) | 13:47 |
| SvenKieske | depending on your kernel and nvidia drivers installed you might also need to change some kernel parameters so the GPU is correctly configured. | 13:48 |
| Eldiabolo_ | Yes, Filter is enabled. Driver shouldnt be necessary at this stage as the GPUs are bound to VFIO. | 13:49 |
| SvenKieske | what does "sudo lspci -vv" report for the NVIDIA card wrt to "Kernel driver in use:"? it should probably read "vfio-pci" | 13:49 |
| SvenKieske | the kernel default driver might be wrong | 13:49 |
| Eldiabolo_ | yes, does read vfio: # for PCI_ID in $(lspci | grep NVIDIA | cut -d" " -f1); do sudo lspci -s ${PCI_ID} -k; done 01:00.0 3D controller: NVIDIA Corporation GA100 [A100 SXM4 40GB] (rev a1) Subsystem: NVIDIA Corporation GA100 [A100 SXM4 40GB] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau 41:00.0 3D controller: NVIDIA Corporation GA100 [A100 SXM4 40GB] (rev a1) Subsystem: NVIDIA Corporation GA100 [A100 SX | 13:49 |
| SvenKieske | mhm ok | 13:50 |
| SvenKieske | ah sorry, I look at reddit, you have all the info there it seems | 13:51 |
| SvenKieske | on which release do you test this? | 13:54 |
| Eldiabolo_ | 2024.1, so recent one. | 13:55 |
| SvenKieske | ok, the code in question also seems to not have changed recently, so this is a pci device pool "filter" function it seems (I'm not very deep into nova code): https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L648 | 13:57 |
| SvenKieske | I guess somehow nova thinks it can't fullfill your request for this specific pci device because after _filter_pools_for_spec the count is too low, maybe there is an error in your spec wrt to the pci device settings? | 13:58 |
| Eldiabolo_ | yeah, I checked the code as well, but i'm not a proper programmer, so it didnt help me much... | 13:59 |
| SvenKieske | the definition of the filter function is here: https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L351 | 13:59 |
| SvenKieske | it matches on vendor_id product_id etc, so I would double check those first | 13:59 |
| Eldiabolo_ | what do you mean by "spec wrt to the pci device setting" | 13:59 |
| SvenKieske | well if vendor_id etc are all really "correct" for your device, I don't know. | 14:00 |
| SvenKieske | it's maybe a better question for #openstack-nova channel or the mailing list. I'm pretty sure there are people around who got these cards working. But I don't have played with those personally. | 14:00 |
| Eldiabolo_ | Like I said, i check 50x at least, i'll do so one more time :D | 14:00 |
| Eldiabolo_ | Thanks, fair point, i'll ask there as well! | 14:01 |
| SvenKieske | it's always good to c&p and let the computer do the comparison. I have made so many typos leading to such bugs, it's not funny anymore :D | 14:01 |
| SvenKieske | also check not only the values but the proper names | 14:01 |
| SvenKieske | sometimes the variable names have typos in the code and you need to actually use typoed variables until it's fixed :D (I guess not in this case) | 14:02 |
| SvenKieske | as a last resort you can enable nova scheduler debug logging, maybe you get more information that way | 14:02 |
| SvenKieske | HTH & good luck! and keep the reddit or mailing list updated when you have a solution, so the next poor soul can find it :D | 14:03 |
| Eldiabolo_ | yes definitely! thanks for your time! | 14:03 |
| opendevreview | Merged openstack/kolla stable/2023.1: Revert "Pin OpenSearch Dashboards to 2.15" https://review.opendev.org/c/openstack/kolla/+/930688 | 14:11 |
| opendevreview | Merged openstack/kolla stable/2023.2: Revert "Pin OpenSearch Dashboards to 2.15" https://review.opendev.org/c/openstack/kolla/+/930687 | 14:13 |
| opendevreview | Merged openstack/kolla stable/2024.1: Revert "Pin OpenSearch Dashboards to 2.15" https://review.opendev.org/c/openstack/kolla/+/930686 | 14:17 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Get the list of ironic nodes - use correct scope https://review.opendev.org/c/openstack/kayobe/+/930691 | 14:26 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Get the list of ironic nodes - use correct scope https://review.opendev.org/c/openstack/kayobe/+/930691 | 15:45 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: CI: Build aarch64 images on x86 https://review.opendev.org/c/openstack/kolla/+/930571 | 15:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move pre tasks into roles https://review.opendev.org/c/openstack/kolla/+/920590 | 15:57 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Backup from active mariadb server https://review.opendev.org/c/openstack/kolla/+/928956 | 16:21 |
| opendevreview | Merged openstack/kolla-ansible master: CI: increase timeout during server resize https://review.opendev.org/c/openstack/kolla-ansible/+/928377 | 16:49 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [DNM] Add apache-base image https://review.opendev.org/c/openstack/kolla/+/930753 | 16:53 |
| opendevreview | Merged openstack/kolla-ansible master: Automate prometheus blackbox configuration https://review.opendev.org/c/openstack/kolla-ansible/+/912420 | 17:03 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Get the list of ironic nodes - use correct scope https://review.opendev.org/c/openstack/kayobe/+/930691 | 17:34 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [DNM] Add apache-base image https://review.opendev.org/c/openstack/kolla/+/930753 | 17:41 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Get the list of ironic nodes - use correct scope https://review.opendev.org/c/openstack/kayobe/+/930691 | 18:46 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [DNM] Add apache-base image https://review.opendev.org/c/openstack/kolla/+/930753 | 18:49 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Try to switch withoud KOLLA_DISTRO https://review.opendev.org/c/openstack/kolla-ansible/+/930782 | 21:17 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!