| noonedeadpunk | mornings | 06:30 |
|---|---|---|
| karni | Hi there | 06:53 |
| hamidlotfi_ | mornings | 06:53 |
| karni | Does OSA have supports for DPDK? I searching but couldn't find any related link. | 06:54 |
| karni | I'm* | 06:55 |
| noonedeadpunk | karni: yup it does | 07:26 |
| noonedeadpunk | https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch-dpdk.html | 07:26 |
| jrosser | good morning | 07:54 |
| jrosser | karni: i think you need a dpdk enabled application in your vm for it to be worth the effort of doing dpdk on the host | 07:54 |
| admin1 | karni, your app must also support dpdk and utilize it .. else its no use .. sr-iov might be better if your app is not aware of dpdk | 08:11 |
| karni | What do you mean by the "App"? Isn't OVS the app itself here? | 09:16 |
| jrosser | DPDK works by making a shared memory area that packets land in from the NIC | 09:17 |
| jrosser | a DPDK enabled application will deal directly with that shared memory rather than "conventional" network stack | 09:17 |
| karni | We want to maximize the traffic throughput on Computes while it's limited to 1G on 10G cards! I implimented it manually and I get 6X enhancement | 09:17 |
| jrosser | i think if you are seeing 1G on 10G cards then that points to there being some other issue | 09:19 |
| karni | jrosser: Then Open vSwitch can deal with that hugepges space, can't it? | 09:19 |
| jrosser | imho something else is wrong if you only get 1G performance with a 10G nic | 09:19 |
| karni | `ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-hugepage-dir=/path/to/hugepages/mountpoint` | 09:20 |
| karni | jrosser: I think the Kernel can not handle the traffic well on HP G8 servers. | 09:22 |
| noonedeadpunk | karni: 1g on 10g cards is vif-multique | 09:22 |
| noonedeadpunk | i'm 99% sure about that | 09:22 |
| noonedeadpunk | https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html | 09:24 |
| noonedeadpunk | you can set either `hw_vif_multiqueue_enabled` per image in it's properties or hw:vif_multiqueue_enabled per flavor (https://docs.openstack.org/nova/latest/configuration/extra-specs.html#hw:vif_multiqueue_enabled) | 09:25 |
| noonedeadpunk | as default behaviour is that traffic is being processed by 1 core and virtio-net multiqueue do release this | 09:26 |
| opendevreview | Merged openstack/ansible-hardening master: Fix linters and metadata https://review.opendev.org/c/openstack/ansible-hardening/+/888682 | 09:31 |
| karni | noonedeadpunk: interesting, I'll go to study around it | 09:36 |
| opendevreview | Merged openstack/openstack-ansible-os_designate master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/888550 | 10:34 |
| opendevreview | Merged openstack/openstack-ansible-os_swift master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/888672 | 10:36 |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/890126 | 10:36 |
| opendevreview | Merged openstack/openstack-ansible-os_glance master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/888567 | 10:37 |
| opendevreview | Merged openstack/openstack-ansible-os_murano master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/888601 | 10:38 |
| opendevreview | Merged openstack/openstack-ansible-os_barbican master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/888472 | 10:39 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/888585 | 10:45 |
| opendevreview | Merged openstack/openstack-ansible-os_mistral master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/890125 | 10:47 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/888534 | 10:48 |
| opendevreview | Merged openstack/openstack-ansible-os_designate master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/890098 | 10:50 |
| opendevreview | Merged openstack/openstack-ansible-os_tacker master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/888609 | 10:50 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/888687 | 10:51 |
| opendevreview | Merged openstack/openstack-ansible-os_trove master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/888614 | 10:51 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/888667 | 10:52 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/888613 | 10:52 |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/888600 | 10:52 |
| opendevreview | Merged openstack/openstack-ansible-os_barbican master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/890095 | 10:53 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/890089 | 10:53 |
| opendevreview | Merged openstack/openstack-ansible-os_zun master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/890134 | 10:54 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/888579 | 10:55 |
| opendevreview | Merged openstack/openstack-ansible-os_cloudkitty master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/888549 | 10:55 |
| opendevreview | Merged openstack/openstack-ansible-os_heat master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/888573 | 10:55 |
| opendevreview | Merged openstack/openstack-ansible-os_heat master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/890122 | 10:55 |
| opendevreview | Merged openstack/openstack-ansible-os_blazar master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/890096 | 10:56 |
| opendevreview | Merged openstack/openstack-ansible-os_blazar master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/888473 | 10:56 |
| opendevreview | Merged openstack/openstack-ansible-os_magnum master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/888592 | 10:56 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/890091 | 11:02 |
| opendevreview | Merged openstack/openstack-ansible-os_zun master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/888616 | 11:02 |
| opendevreview | Merged openstack/openstack-ansible-os_placement master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/890130 | 11:05 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/888692 | 11:19 |
| opendevreview | Merged openstack/openstack-ansible-ceph_client stable/zed: Use correct index of previous task results https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/891106 | 11:23 |
| opendevreview | Merged openstack/openstack-ansible-os_sahara master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/888608 | 11:29 |
| opendevreview | Merged openstack/openstack-ansible-os_glance master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/890099 | 11:43 |
| opendevreview | Merged openstack/openstack-ansible-ceph_client stable/zed: Define libvirt secrets from keyring files in ceph_extra_confs https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/891108 | 12:00 |
| karni | What does the following statement mean: | 12:11 |
| karni | " A PCI slot typically corresponds to a single NUMA node. For optimal performance, a DPDK NIC and any instance utilizing the NIC should be restricted to the same NUMA node and its respective memory...; Cores associated with the other NUMA node should be made unavailable to instances." | 12:11 |
| karni | Src: https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch-dpdk.html#numa-topology | 12:11 |
| jrosser | karni: PCI lanes are connected to a CPU | 12:11 |
| jrosser | if you have more than one CPU then the PCI lanes come from one, or the other | 12:11 |
| noonedeadpunk | karni: I'm pretty sure you don't need dpdk with only 10Gbit cards... | 12:12 |
| jrosser | so to get optimum performance for DPDK you must ensure that instances only schedule to the CPU connected directly to the NIC | 12:12 |
| jrosser | this is one of the "horrible" things with trying to use DPDK | 12:12 |
| noonedeadpunk | and it's not the only one unfortunatelly | 12:12 |
| karni | noonedeadpunk: I will definitely take what you said into consideration and look into it deeply. But now halfway through DPDK, I'd like to learn it properly (even if it's for my CV!) | 12:15 |
| karni | jrosser: So, NIC1 should be bound to NUMA0; and NUMA1 would be useless, right? | 12:18 |
| jrosser | perhaps :) | 12:18 |
| jrosser | i don't now specifics as DPDK has always been something i avoided at all costs | 12:19 |
| jrosser | and also i have no dual socket compute nodes also, to avoid NUMA troubles | 12:19 |
| jrosser | there are other people here who have looked in more detail, jamesdenton and spatel in particular but I don't believe anyone we know here in #openstack-ansible went past lab tests to actual production deployment with DPDK | 12:21 |
| jrosser | karni: see this https://satishdotpatel.github.io/openstack-dpdk-with-intel-x550-nic-part1/ | 12:23 |
| karni | jrosser: Thanks | 12:29 |
| karni | And I appreciate having jamesdenton and spatel's comments (if they're online) | 12:29 |
| jamesdenton | good morning | 12:32 |
| jamesdenton | jrosser is right, we never implemented DPDK in production and haven't really had a reason to. It's been a few years since I messed with it. spatel probably has more recent experience | 12:35 |
| noonedeadpunk | and spatel has abandined dpdk in favor of SR-IOV just in case ;) | 12:40 |
| noonedeadpunk | karni: you can read his blogpost https://satishdotpatel.github.io/openstack-dpdk-with-intel-x550-nic-part1/ | 12:41 |
| admin1 | i also did dpdk on client request .. turns out his app does not know of dpdk .. then used sr-iov instead | 12:42 |
| admin1 | his app was realtime gaming so very small udp packets that benefitted from sr-iov | 12:43 |
| karni | Oh, there are lots of things to be considered. | 12:45 |
| karni | admin1: You mean if I enable DPDK, customers who launches instances based on dpdk-aware flavors may not get benifit of DPDK based on their useage in the instances? | 12:45 |
| karni | benefit* | 12:46 |
| karni | I haven't looked into real world applications. My tests are based on stuffing the bandwidth with something like `ipref` | 12:49 |
| jamesdenton | i don't recall having an issue hitting > 1Gbps on G8 hardware. That was with the X520 LOM or PCI | 12:53 |
| admin1 | did anything related to keystone changed between 26.1.1 -> 27.0.1 .. after upgrade, when i login as a tenant, i cannot list instances or images or anything | 13:14 |
| jrosser | admin1: when you login to...... horizon? or the cli? | 13:15 |
| admin1 | horizon | 13:15 |
| admin1 | have not tried tenant on cli yet | 13:15 |
| jrosser | can you check with the cli too | 13:15 |
| admin1 | on it | 13:15 |
| admin1 | Failed to discover available identity versions when contacting https://cloud.domain.com. Attempting to parse version from URL. | 13:18 |
| admin1 | as an admin, i can | 13:19 |
| admin1 | checking further .. | 13:19 |
| jrosser | i don't know if it is the same thing at all tbh, but we had this https://bugs.launchpad.net/openstack-ansible/+bug/2029486 | 13:19 |
| admin1 | so osa defaults to _member_ , but i have to change it to member | 13:23 |
| admin1 | i will give it a try | 13:23 |
| admin1 | jrosser, that was it | 13:23 |
| jrosser | can you leave a comment on the bug about what happened and how you fixed it? | 13:24 |
| admin1 | i have a user with 2 projects, i let it stay as _member_ in one, but changed member to another .. the one with member worked | 13:24 |
| jrosser | also if this was fresh or upgrade? | 13:24 |
| admin1 | upgrade | 13:24 |
| admin1 | 26.1.1 -> 27.0.1 | 13:24 |
| jrosser | ^ noonedeadpunk admin1 also had the _member_ issue we did | 13:24 |
| noonedeadpunk | yeah | 13:25 |
| noonedeadpunk | I was going to look into upgrade script to make _member_ implied. Despite that won't solve application credentials :( | 13:25 |
| admin1 | its better to write a note and ask people to use member instead of _member_ .. makes it more inline and its can be scripted | 13:26 |
| noonedeadpunk | I bet it's already written in keystone notes one day... | 13:26 |
| admin1 | updated with my findings | 13:30 |
| admin1 | note. admin has no role in service, so had to make it "member" in service project to allow it to list the amphora images | 13:31 |
| admin1 | cli is still borken .. Failed to discover available identity versions when contacting https://cloud.domain.com. Attempting to parse version from URL. | 13:58 |
| opendevreview | Merged openstack/openstack-ansible-os_masakari master: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/890124 | 14:10 |
| noonedeadpunk | admin1: jsut add _member_ as implied to member | 14:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant stable/2023.1: Install mysqlclient devel package https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/891279 | 14:37 |
| admin1 | Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Invalid Response - Bad version data returned: | 17:27 |
| admin1 | someting else | 17:27 |
| NeilHanlon | hmm https://mitogen.networkgenomics.com/ansible_detailed.html | 20:16 |
| Karni | > admin1: your app must also support dpdk and utilize it | 20:53 |
| Karni | What happens if the application inside the OpenStack instance is not compatible with dpdk? | 20:56 |
| Karni | What path does the packet take? OVS is configured to work with dpdk and bypass the kernel (DPDK concept). What does it matter where the packet comes from (here, a software unfamiliar with dpdk)? | 20:58 |
| Karni | jrosser: ^ | 20:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!