| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Bump OpenStack-Ansible Wallaby https://review.opendev.org/c/openstack/openstack-ansible/+/806688 | 08:42 |
|---|---|---|
| agemuend | Hi All. We're having a hard time with our vxlan setup. Does someone have some hints on how to debug that properly? | 10:52 |
| jrosser | agemuend: can you be a little more specific? is it DHCP related for example? | 10:53 |
| agemuend | Well yes, dhcp fails already | 11:02 |
| agemuend | But I have the feeling that packets dont get forwarded at all | 11:02 |
| agemuend | If I tcpdump on the tap interface, I see the dhcp packages | 11:03 |
| agemuend | But then, should I be able to see them on the vxlan-$i interface, or on the master interface? I'm not sure about that | 11:03 |
| agemuend | We use the linuxbridges backend, pretty much following the documentation examples, e.g. https://docs.openstack.org/project-deploy-guide/openstack-ansible/newton/app-networking.html#network-appendix | 11:04 |
| agemuend | oh the link is from newton, but we're using victoria, sorry | 11:05 |
| noonedeadpunk | with linux bridge you should see actually bridge to be created on compute with vxlan id in it and interface that attached to the VM | 11:17 |
| noonedeadpunk | I wonder if that is properly created... | 11:19 |
| noonedeadpunk | but if you see packages on tap interface, I think that it is | 11:20 |
| agemuend | Yes I see a brq.... bridge being created with the tap interface and a vxlan- interface | 11:22 |
| agemuend | ip -d link show vxlan-4 also shows the following: | 11:25 |
| agemuend | vxlan id 4 group 239.1.1.1 dev br-vxlan srcport 0 0 dstport 8472 ttl 32 ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx | 11:25 |
| jrosser | you should also be able to see the ecapsulated vxlan packets with tcpdump in the interface | 11:25 |
| jrosser | and you can then look for those same packets on the interface of your network (infra?) node | 11:26 |
| agemuend | you mean on the final physical? I was hoping I could go step by step | 11:26 |
| agemuend | But I can check the physical, sec | 11:26 |
| jrosser | if they don't make it to your network node then you need to look at multicast config on your switched | 11:26 |
| jrosser | *switcheas | 11:26 |
| jrosser | arg :) | 11:26 |
| jrosser | broadcast/unknown/multicast packets from the VM in the vxlan tunnel become multicast on the underlay network | 11:27 |
| jrosser | so failed dhcp can be due to lack of correct multicast/igmp config on your network hardware | 11:27 |
| agemuend | I see lots of ARP requests, but the bootp message doesnt come through apparently | 11:28 |
| agemuend | I think they dont even leave the compute node, thats my point | 11:28 |
| agemuend | I dont see it on the physical if | 11:29 |
| noonedeadpunk | So eventua;ly that's why I asked if bridge that neutron created for VM contain vxlan interface? | 11:31 |
| noonedeadpunk | also, br-vxlan should have assigned IP address from vxlan network and you should be able to reach infra/net node with it | 11:32 |
| agemuend | Yes, the vxlan-4 interface is created and its member of the brq... bridge | 11:33 |
| agemuend | the vxlan-4 interface has the vxlan params, I pasted that | 11:34 |
| agemuend | br-vxlan has an ip and can reach the infra nodes | 11:34 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Fix service removal condition https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/801910 | 11:36 |
| agemuend | btw. already thanks for your help | 11:36 |
| agemuend | If you have any other idea what to test to debug this, that would be very appreciated | 11:53 |
| opendevreview | Merged openstack/openstack-ansible master: Bump OpenStack-Ansible master https://review.opendev.org/c/openstack/openstack-ansible/+/806345 | 12:11 |
| opendevreview | Merged openstack/openstack-ansible master: Replace deprecated include statement https://review.opendev.org/c/openstack/openstack-ansible/+/805587 | 12:11 |
| noonedeadpunk | agemuend: so on net infra nodes you can execute smth like ping or tcpdump from inside of the namespaces where router or dnsmasq is run | 12:18 |
| noonedeadpunk | I think that is another thing that could be tested | 12:19 |
| agemuend | I tried that already, it doesn't work | 12:19 |
| noonedeadpunk | I think you can also add some veth pair and add it manually to bridge (with some static ip) just to verify connectivity for vxlans | 12:19 |
| noonedeadpunk | but for me it seems that your net equipment just filter out vxlans tbh | 12:19 |
| noonedeadpunk | I can recall some cases where ppl had to explicitly add some rules for their Cisco switches so that they passed traffic | 12:21 |
| noonedeadpunk | I guess it at least requires UDP port 4789 | 12:21 |
| noonedeadpunk | and I guess multicast? not sure here | 12:23 |
| agemuend | Mhm, I've enabled multicast globally on the switch and tried with and without igmp snooping | 12:26 |
| agemuend | but as I said, I fear that the package does not even leave the compute node because there is something weird going on with the bridges or vxlan parent relationship | 12:26 |
| noonedeadpunk | there's super simple thing actually | 12:28 |
| noonedeadpunk | so if interfaces are in bridge, bridge exists, and br-vxlan has IP that is reachable - it should work and never saw any issues there | 12:30 |
| noonedeadpunk | Well, actually, you can remove complexity a bit if you wish, because br-vxlan can be just an interface and not required to be a real bridge | 12:30 |
| noonedeadpunk | But I doubt that woudl change anything | 12:31 |
| noonedeadpunk | actually, what you can do - manually create another vxlan on top of br-vxlan and check if it works between compute and net node | 12:32 |
| noonedeadpunk | just to ensure that there's no traffic filtering in place or smth like that | 12:33 |
| jrosser | agemuend: can you double check which ports are being used for the vxlan tunnel? | 12:54 |
| jrosser | 4789 vs 8472 for example | 12:55 |
| jrosser | you can also use ssmping to validate the multicast | 12:56 |
| agemuend | Sorry for the late answer | 13:14 |
| agemuend | > [14:32] <noonedeadpunk> actually, what you can do - manually create another vxlan on top of br-vxlan and check if it works between compute and net node - How would I do that? | 13:15 |
| jrosser | agemuend: https://vincent.bernat.ch/en/blog/2012-multicast-vxlan | 13:17 |
| jrosser | imho you have a layered set of things you can check out from the lowest layers upward | 13:18 |
| jrosser | unicast routing, multicast routing first | 13:18 |
| agemuend | jrosser: on the compute node, it uses dstport 8472 | 13:22 |
| agemuend | Not sure how to check on the infranode | 13:22 |
| jrosser | agemuend: https://paste.opendev.org/show/808458/ | 13:28 |
| jrosser | you can see all of the ports / vtep device / group address with that | 13:29 |
| agemuend | I have pasted that above somewhere | 13:34 |
| agemuend | imho looks good | 13:34 |
| spatel | jrosser did we release 23.2.0 yet or still hanging pieces? | 13:53 |
| noonedeadpunk | spatel: 23.1.0 but no we haven't yet | 14:16 |
| jrosser | hmm https://github.com/openstack/releases/blob/master/deliverables/wallaby/openstack-ansible.yaml | 14:16 |
| noonedeadpunk | we need to merge that bump for that https://review.opendev.org/c/openstack/openstack-ansible/+/806688 | 14:17 |
| spatel | cool! i am waiting for that to release so i can go to 23.2.0 :) | 14:18 |
| spatel | then we can start work on X | 14:18 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Aug 31 15:00:19 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| spatel | \o/ | 15:01 |
| noonedeadpunk | #topic office hours | 15:16 |
| noonedeadpunk | so. there're 2 days left to publish release highlights | 15:16 |
| jrosser | hello | 15:17 |
| noonedeadpunk | I suggest posting at least: 1. Ceph Pacific 2. Ansible-core 2.12 3. mariadb 10.6? | 15:18 |
| noonedeadpunk | not sure about last part... | 15:18 |
| jrosser | OVN is much improved | 15:18 |
| noonedeadpunk | oh, yes | 15:18 |
| jrosser | i guess we *plan* to finish up PKI stuff | 15:18 |
| noonedeadpunk | we also have a vault role, but not sure if we will do anything with it | 15:19 |
| noonedeadpunk | well, I have os_freezer role but not fully tested it yet | 15:20 |
| noonedeadpunk | https://github.com/noonedeadpunk/openstack-ansible-os_freezer/ | 15:20 |
| noonedeadpunk | oh, switch mariadb balancing from haproxy | 15:22 |
| noonedeadpunk | to ProxySQL | 15:25 |
| noonedeadpunk | And I think that's it for sure. Not sure if I should mention mariadb and freezer... | 15:25 |
| spatel | Planning to deploy OVN on 10 compute nodes to run some workload/benchmark.. | 15:28 |
| spatel | I need to update OSA - OVN doc for SRIOV implementation | 15:29 |
| spatel | currently trying to deploy DPDK with OVN | 15:29 |
| spatel | I can see big difference in OVN vs neutron agent. OVN is very fast and quick to spin up VM | 15:31 |
| noonedeadpunk | And I hope reliable when it comes to maintenances and need to upgrade "net nodes" :D | 15:48 |
| noonedeadpunk | #endmeeting | 16:10 |
| opendevmeet | Meeting ended Tue Aug 31 16:10:08 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:10 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-08-31-15.00.html | 16:10 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-08-31-15.00.txt | 16:10 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-08-31-15.00.log.html | 16:10 |
| spatel | noonedeadpunk are you ask me | 16:10 |
| noonedeadpunk | nah, it was kind of fact | 16:27 |
| spatel | if more people start using OVN then we have more stability because of more bug report etc.. | 16:50 |
| opendevreview | Merged openstack/openstack-ansible stable/wallaby: Bump OpenStack-Ansible Wallaby https://review.opendev.org/c/openstack/openstack-ansible/+/806688 | 16:54 |
| mgariepy | tripleo does use ovn by default also. | 16:55 |
| noonedeadpunk | well, tempest does as well | 16:59 |
| noonedeadpunk | I guess we should have switched one day as well | 16:59 |
| mgariepy | it would be nice to have a patch to upgrade from lxb to ovn and some point. | 17:05 |
| noonedeadpunk | oh.. I'm not sure that is possible at all... | 17:31 |
| noonedeadpunk | there're only migration from ovs afaik | 17:31 |
| mgariepy | if we need to do : lxb > ovs > ovn then it's ok as well it would be just nice to have an upgrade path. | 17:34 |
| spatel | Devstack also default use OVN | 19:50 |
| spatel | its very easy to add any new feature in OVN model | 19:51 |
| spatel | Logical flow and openflow make things very easy. (example: add DHCP option just using openflow) | 19:53 |
| spatel | loadbalancer and source base routing other features. | 19:53 |
| spatel | mgariepy there is already upgrade path from OVS to OVN | 19:55 |
| spatel | but yes not tested in OSA | 19:55 |
| *** sshnaidm is now known as sshnaidm|afk | 23:34 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!