| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: Remove unnecessary pki step in haproxy install https://review.opendev.org/c/openstack/openstack-ansible/+/812361 | 07:35 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Fix haproxy Let's Encrypt SSL path https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/811985 | 07:49 |
| spatel | jamesdenton morning | 12:46 |
| jamesdenton | hello | 12:48 |
| spatel | I am dealing with one strange issue with OVS let me show you what is going on | 12:48 |
| spatel | jamesdenton i am setup dpdk compute node and trying to spin up VM but vm not getting any IP address from DHCP, that means somewhere its networking issue. all other compute nodes are working fine and i can spin up VM etc.. | 12:51 |
| spatel | This is dpdk node status, not able to ping outside and get ip from DHCP - https://paste.opendev.org/show/809790/ | 12:52 |
| spatel | what else i should check or troubleshoot, how do i run tcpdump on ovs bridge/interface? | 12:52 |
| jamesdenton | you might try setting up some mirror ports to see what you see on the tap and the bond | 12:53 |
| jamesdenton | i have an example at the bottom: https://www.jimmdenton.com/proliant-intel-dpdk/ | 12:53 |
| spatel | so no direct way to sniff traffic? | 12:53 |
| jamesdenton | no | 12:53 |
| jamesdenton | well, mirror port off the ovs bridge | 12:54 |
| spatel | let me try your way | 12:54 |
| jamesdenton | and have you checked to see if the bond is established properly? | 12:55 |
| spatel | Yes i did check bond was working before i setup dpdk, i meant i configure bonding in kernel before to check and verify. also i tried setting up with single physical nic but that also didn't help | 12:56 |
| spatel | let me do traffic sniff to understand flow | 12:56 |
| spatel | jamesdenton i tap mirror on br-int and i can see my VM asking for ARP but no reply - https://paste.opendev.org/show/809792/ | 13:08 |
| jamesdenton | ok, how about the provider bridge? | 13:09 |
| spatel | nothing on br-provider | 13:09 |
| spatel | in my case i have br-vlan | 13:09 |
| jamesdenton | right | 13:10 |
| spatel | do you think its problem of connection between br-vlan----br-int ? | 13:10 |
| spatel | I can see patch between them - phy-br-vlan | 13:11 |
| spatel | let me sniff on that patch | 13:11 |
| jamesdenton | i doubt it. did it work before bonding? | 13:12 |
| spatel | yes bonding was working before i install openstack | 13:14 |
| spatel | by default my all server has bonding.. | 13:14 |
| spatel | let me do one things. create bonding in kernel and setup | 13:14 |
| spatel | quick question can i mirror int-br-vlan interface? because my command doesn't working | 13:15 |
| spatel | ovs-vsctl add-port int-br-vlan tap0 -- --id=@p get port tap0 -- --id=@m create mirror name=m0 select-all=true output-port=@p -- set bridge int-br-vlan mirrors=@m | 13:16 |
| spatel | i tried with bridge br-int also | 13:16 |
| jamesdenton | sorry, i meant did it work with dpdk before you setup the bond | 13:16 |
| jamesdenton | you have to add the tap to the bridge (int-br-vlan is a port, not the bridge) | 13:16 |
| jamesdenton | br-vlan is the bridge | 13:17 |
| spatel | i am curious why ARP packet not hitting br-vlan (provider bridge) ? | 13:17 |
| spatel | like this - > ovs-vsctl add-port br-int tap0 -- --id=@p get port tap0 -- --id=@m create mirror name=m0 select-all=true output-port=@p -- set bridge int-br-vlan mirrors=@m | 13:18 |
| spatel | i tried couple of combo and none of them work | 13:18 |
| jamesdenton | https://paste.opendev.org/show/809793/ | 13:20 |
| spatel | that works but i was looking to tap specific interface phy-br-vlan which is patch with int-br-vlan | 13:22 |
| jamesdenton | yeah, i'm not entirely sure how to do that. lemme see | 13:22 |
| spatel | hey i can see ARP packet on br-vlan | 13:23 |
| spatel | let me remove bonding and see | 13:24 |
| jamesdenton | vlan 2001 is trunked, right? | 13:24 |
| jamesdenton | spatel http://arthurchiao.art/blog/traffic-mirror-with-ovs/ | 13:30 |
| *** dpawlik2 is now known as dpawlik | 13:31 | |
| spatel | yes 2001 is my public VLAN | 13:31 |
| spatel | awesome!! | 13:31 |
| spatel | jamesdenton i have removed bonding and just added single physical nic but still not luck so let me see what is going on - https://paste.opendev.org/show/809795/ | 13:34 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-haproxy_server master: Fix haproxy Let's Encrypt SSL path https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/811985 | 14:51 |
| spatel | jamesdenton bummer!!! | 15:01 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:02 |
| opendevmeet | Meeting started Tue Oct 5 15:02:17 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:02 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:02 |
| noonedeadpunk | #topic rollcall | 15:02 |
| noonedeadpunk | \o/ | 15:02 |
| spatel | we are renting these servers and turnout that these switches disable port if no LACP packet received | 15:02 |
| spatel | anyway meeting time :) | 15:02 |
| jamesdenton | o/ | 15:02 |
| noonedeadpunk | huh, feels ike topic not working anymore? | 15:02 |
| noonedeadpunk | or it's just me who haven't noticed that before?:) | 15:03 |
| jamesdenton | spatel yeah, they would need to be in passive more and no suspend-individual (or equiv). OK i'll be quiet. | 15:03 |
| noonedeadpunk | #topic office hours | 15:07 |
| noonedeadpunk | So, it's high time we've populated our PTG etherpad | 15:07 |
| noonedeadpunk | #link https://etherpad.opendev.org/p/osa-yoga-ptg | 15:07 |
| noonedeadpunk | However I can't say we really proceeded a lot with agreed items from previous one | 15:08 |
| noonedeadpunk | #link https://etherpad.opendev.org/p/osa-xena-ptg | 15:08 |
| noonedeadpunk | But anyway, PTG is super close now :( | 15:10 |
| noonedeadpunk | Another thing is that we have tons of patches for review | 15:11 |
| noonedeadpunk | we haven't merged a thing for quite a while :( | 15:11 |
| spatel | This is pretty long list | 15:13 |
| jamesdenton | i can try to take a stab at some of those this week | 15:13 |
| andrewbonney | I'll try and take a look at some this week. Are there any particularly high priority or blocking things? | 15:13 |
| jamesdenton | ^^^ | 15:13 |
| mgariepy | I can also. | 15:14 |
| spatel | I can help you out some of stuff for testing or fixing in my capacity | 15:14 |
| noonedeadpunk | Well specifically would be awesome to land murano fix https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/781239 | 15:15 |
| noonedeadpunk | but there;s also whole topic https://review.opendev.org/q/topic:%22osa%252Fgalera_pki%22+(status:open%20OR%20status:merged) | 15:15 |
| noonedeadpunk | Which actually shows that we have bunch of broken roles ... | 15:16 |
| noonedeadpunk | Also I guess we need to merge ansible bump https://review.opendev.org/c/openstack/openstack-ansible/+/807316 | 15:17 |
| noonedeadpunk | But I guess none are reall blockers... | 15:20 |
| noonedeadpunk | well, except murano | 15:20 |
| noonedeadpunk | I just got a bit burried in internal stuff during this cycle ( | 15:20 |
| jamesdenton | same | 15:21 |
| TK_ | Hello Guys | 15:28 |
| TK_ | I am using Openstack Wallaby on Kolla-ansible | 15:28 |
| TK_ | I would like to have the masakari default Instance recovery to be process_all_instances¶ but I am afraid I dont know how to change the default policy ... | 15:29 |
| TK_ | Any help will be highly appreciates | 15:29 |
| TK_ | appreciated | 15:29 |
| spatel | TK_ sorry this channel is for openstack-ansible try channel #kolla-ansible | 15:30 |
| TK_ | Thanks Patel | 15:31 |
| noonedeadpunk | ok, I guess I don't have anything else to say, to will end up meeting then | 15:36 |
| noonedeadpunk | *so | 15:37 |
| noonedeadpunk | #endmeeting | 15:37 |
| opendevmeet | Meeting ended Tue Oct 5 15:37:08 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:37 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-10-05-15.02.html | 15:37 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-10-05-15.02.txt | 15:37 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-10-05-15.02.log.html | 15:37 |
| noonedeadpunk | damn ,how to trace back interface on compute back to vxlan interface? Because I kind of understood how to find back to qvo, but not sure how to find patched stuff from br-int to br-tun... | 15:38 |
| noonedeadpunk | ovs still feels for me a bit overcomplicated related to lxb... | 15:42 |
| noonedeadpunk | esp when it comes to the need of debugging and finding if everything is present at least | 15:44 |
| mgariepy | nothing like having issues and needing to investigate on the fly .. :P | 15:50 |
| spatel | +1 noonedeadpunk agreed, ovs is not easy to handle :) | 15:52 |
| spatel | but i think its good time to learn OVS ahead of time because soon it will be everywhere. | 15:53 |
| spatel | jamesdenton bonding is working now after i configure bonding using LACP active | 15:54 |
| mgariepy | the good news is that with learning ovs then ovn becomes much more simpler. | 15:54 |
| mgariepy | even more if you do use the ovs flows instead of the iptables stuff for ovs. | 15:54 |
| spatel | yes OVN is very simple if you know how OVS work. (and ovs isn't that difficult except it has very complicated commands to remember) | 15:55 |
| noonedeadpunk | so any advices then how to map qvo to vxlan interfaces?:) | 15:57 |
| noonedeadpunk | (qvo is another part of veth pair from bridge) | 15:58 |
| spatel | https://docs.openstack.org/neutron/latest/contributor/internals/openvswitch_agent.html | 15:58 |
| spatel | br-tun which deal with vxlan not qvo. i may be wrong | 15:59 |
| noonedeadpunk | Well I guess https://docs.openstack.org/neutron/latest/admin/deploy-ovs-selfservice.html is a bit better describing architecture? | 16:00 |
| noonedeadpunk | so, what I'm trying to do is to map VM interface with underlying VXLAN and understand if vxlan is even present and has valid options | 16:01 |
| noonedeadpunk | sounds pretty simple... but I cant really recall how that is done... | 16:01 |
| noonedeadpunk | (tended to use lxb :)) | 16:01 |
| spatel | why do you want to do that, that all stuff get auto setup | 16:02 |
| noonedeadpunk | because I can't ping VM from l3 namespace... | 16:05 |
| noonedeadpunk | Only when VM is on single compute | 16:06 |
| noonedeadpunk | and no errors.... | 16:06 |
| spatel | This guy is awesome if you want to check how all these pieces connected - https://www.youtube.com/watch?v=tnSkHhsLqpM&ab_channel=DavidMahler | 16:08 |
| spatel | what is your ovs-vsctl show saying? | 16:10 |
| noonedeadpunk | it's pretty fair as for me... | 16:13 |
| noonedeadpunk | well, dunno, maybe it's not networking thing after all... | 16:14 |
| noonedeadpunk | I guess I will just create another VM I will be owning in the same network and same compute... | 16:14 |
| opendevreview | Merged openstack/openstack-ansible master: Bump OpenStack-Ansible master https://review.opendev.org/c/openstack/openstack-ansible/+/811742 | 16:28 |
| jamesdenton | all of the vxlan traffic traverses br-tun, and the flow rules will dictate whether traffic heads from br-int -> br-ex or br-int -> to br-tun, local segment ids (on br-int) will be translated to VNIs and send to the respective vtep based on MAC (i think?). The flow rules handle all of that | 16:40 |
| opendevreview | Matthew Thode proposed openstack/openstack-ansible-lxc_hosts master: allow release info change on apt update https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/812528 | 16:59 |
| mgariepy | anyone here is speaking inronic ? | 18:41 |
| mgariepy | TL;DR: https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/810210 is broken because of : https://review.opendev.org/c/openstack/ironic/+/789382 | 18:42 |
| mgariepy | so all `deploy: iscsi` in https://github.com/openstack/openstack-ansible-os_ironic/blob/master/vars/main.yml#L22-L141 need to go.. | 18:43 |
| mgariepy | noonedeadpunk, should i replace all by ramdisk or it really needs to be specific per driver and so on? | 18:44 |
| opendevreview | Merged openstack/openstack-ansible master: Include openstack_services for murano role https://review.opendev.org/c/openstack/openstack-ansible/+/805373 | 18:47 |
| opendevreview | Merged openstack/ansible-role-pki master: Ensure certificate installation path exists https://review.opendev.org/c/openstack/ansible-role-pki/+/807771 | 18:54 |
| spatel | jamesdenton X550T is beast.. | 19:02 |
| jamesdenton | yeah? | 19:02 |
| spatel | I can see its attached to both NUMA | 19:02 |
| spatel | NUMA0 and NUMA1 and performance of DPDK is really good | 19:02 |
| spatel | Intel 82599 is OK.. | 19:04 |
| spatel | but X550T is amazing not a single drop of packet even i push 10G udp traffic | 19:04 |
| jamesdenton | 82599 may as well be 486 :D | 19:05 |
| spatel | hehe.. no doubt | 19:05 |
| jamesdenton | it's just old | 19:05 |
| jamesdenton | so, your DPDK woes are solved then? | 19:05 |
| spatel | i can see big big difference between both nic | 19:05 |
| spatel | I am now going full DPDK deployment in production :) | 19:05 |
| jamesdenton | :o | 19:06 |
| jamesdenton | glad to hear it! | 19:06 |
| spatel | my bonding is working great with DPDK so look like all my problems has been resolved. | 19:06 |
| spatel | Now question is go with legacy OVS or OVN+OVS | 19:07 |
| spatel | I love OVN but i don't want to take risk in production.. | 19:07 |
| jamesdenton | what version of OSA are you using? | 19:08 |
| spatel | i found some strange bug with DPDK related TSO offloading issue in 2.13.3 version (that issue has been resolved in 2.16.x _ | 19:08 |
| spatel | currently i am running - 2.13.3 | 19:08 |
| spatel | I have OVN lab which is running 2.16.x (because of TSO bug) | 19:09 |
| jamesdenton | you get what Canonical gives you. And you'll like it. | 19:09 |
| spatel | Yes i like distribution version instead of hacking and building own package | 19:09 |
| spatel | here is the bug, just incase you run over - http://patchwork.ozlabs.org/project/openvswitch/patch/162548620436.40409.579366497986013480.stgit@wsfd-netdev64.ntdv.lab.eng.bos.redhat.com/ | 19:10 |
| spatel | if you planning to use DPDK with OVN then 2.13.3 no go.. you have to use 2.16.x | 19:11 |
| spatel | without this patch OVS will keep crashing.. | 19:11 |
| jamesdenton | thank you | 19:12 |
| spatel | how is your testing going with OVN ? | 19:14 |
| jamesdenton | well, good, i think. I have a VM behind a router, metadata working | 19:15 |
| jamesdenton | just needed to run a 'repair' to sync eveything from neutron->ovn | 19:15 |
| spatel | i love neutron->ovn sync script :) | 19:16 |
| spatel | you don't need to take backup of ovn database | 19:16 |
| spatel | did you get a time to test ovn clustering | 19:16 |
| jamesdenton | um, i didn't do anything special to set it up, i have not killed any controllers, yet, to make sure it actually works | 19:18 |
| spatel | no worry! next thing i would like to test is upgrade plan, how does that impact in prod traffic etc.. | 19:20 |
| jamesdenton | agreed | 19:20 |
| spatel | jamesdenton i found issue why DPDK wasn't working | 21:10 |
| spatel | this package is missing libdpdk-dev | 21:10 |
| spatel | without this package DPDK port failed to bind | 21:11 |
| jamesdenton | oh really. lemme give that a go, too | 21:50 |
| jamesdenton | was that not a dependency with the dpdk package/ | 21:50 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!