| *** acidfu has joined #openvswitch | 00:02 | |
| *** acidfu has quit IRC | 00:06 | |
| *** acidfu has joined #openvswitch | 00:12 | |
| bdonnahue2 | larsks, i was playing with ovs and im having an issue with my vxlan. I manually created an adaptor and connected it to the vxlan bridge. i was wondeing if i need to set mtu for the adaptor? | 00:46 |
|---|---|---|
| larsks | bdonnahue2: that's a good question. You may want to adjust the MTU of your virtual machine interfaces to be lower than the MTU of the physical NICs that are transporting the vxlan traffic. https://oswalt.dev/2014/03/mtu-considerations-for-vxlan/ has some thoughts on the topic. | 02:38 |
| larsks | I'm going to run some tests with my setup to see if I can produce failures due to mtu issues. | 02:39 |
| bdonnahue2 | larsks, not to send you on a goos chase: i am tesing out proxmox's ceph cluster functionality. the nodes in the cluster compaling that they cannot talk to each other. i can ping the other nodes, i can see tcpdump traffic on the ports they comaplain about, i see them listening on the right ip with netstat. completely lost here. will have a look at th earticle you posted. | 02:41 |
| larsks | Try varying the packet size you use with ping (e.g. `ping -s 1400`, `ping -s 1500`) and see when it fails. | 02:43 |
| bdonnahue2 | https://pastebin.com/DZ8uRuQU | 02:45 |
| bdonnahue2 | thats some notes i took ^^ | 02:45 |
| bdonnahue2 | ahh thats an interesting idea... so if the ping fails with a given size, what does that mean in terms of vxlan/ovs? | 02:46 |
| bdonnahue2 | (just waiting for some vms to finish installing and i will revert my configs and test the ping) | 02:47 |
| larsks | I ping fails with a packet size approaching your MTU, that suggests you might need to lower the MTU of your vm interfaces (or increase the mtu of the host interfaces). | 02:47 |
| bdonnahue2 | larsks, i am super new to MTU. i understand how it works, but not is set for my interfaces by default. any idea how i can check this? | 02:57 |
| larsks | Running 'ip link show <device>' will show you the MTU for the device. ("3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc ...", where "mtu" is just after the flags) | 02:59 |
| larsks | Running 'ip link set mtu <mtu> dev <device>' will set the mtu (until you reboot) | 02:59 |
| larsks | Setting the MTU persistently differents between distributions and distribution versions. | 02:59 |
| larsks | I've transitioned my setup from using vxlan tunnels to playing with OVN, but I find I have to set the interface mtu *inside the vms* to 1440 for things to work successfully. | 03:00 |
| larsks | (OVN uses geneve tunnels rather than vxlan, but the principal is the same) | 03:01 |
| bdonnahue2 | larsks, i never noticed the mtu in the output before, thanks for pointing that out. | 03:02 |
| bdonnahue2 | interesting re. the 1440 mtu | 03:02 |
| *** rcernin has quit IRC | 03:05 | |
| larsks | It is often easier to *increase* the MTU on your physical hosts, rather than lowering the MTU in all your virtual machines (because you typically have a static set of physical hosts, while virtual machines are constantly being created). | 03:05 |
| bdonnahue2 | larsks, is the a best value to set or a max value to avoid? | 03:06 |
| larsks | I'm not sure I'm reading that question correctly. Can you rephrase it? | 03:06 |
| *** rcernin has joined #openvswitch | 03:07 | |
| bdonnahue2 | sure, so if i change the physical nic mtu... i could go with 1550, 1600, 999999999? whats the normal "larger" mtu one would select? | 03:07 |
| larsks | The maximum values depends on your network infrastructure (e.g., what is the maximum frame size supported by your switches?). For strictly internal communication you can usually safely pick an MTU up to 9000 bytes, but now that I think about it I'm actually not sure what the implications are for traffic that will be leaving your local network. I would probably just select something as close to | 03:10 |
| larsks | 1500 as you can get while still allowing your tunnels to work. | 03:10 |
| larsks | And maybe consult someone with greater network knowledge for a more authoritative answer. | 03:10 |
| bdonnahue2 | gotcha, ok makes sense | 03:11 |
| *** rcernin has quit IRC | 03:23 | |
| *** rcernin has joined #openvswitch | 03:26 | |
| *** atpa8a has quit IRC | 03:44 | |
| bdonnahue2 | larsks, the mtu did the trick! much appreciated. I would have never gotten past that one | 03:45 |
| bdonnahue2 | the performance of the VMs with disks hosted on the cluster is not great... very slow | 03:46 |
| bdonnahue2 | will look into this | 03:46 |
| bdonnahue2 | may be a networking issue, may be i underprovisionged cpu | 03:46 |
| bdonnahue2 | looks like i spoke too soon, some osd's went offline | 03:47 |
| bdonnahue2 | got em back online... wonder in a transient network surge took down the cluster | 03:49 |
| bdonnahue2 | the cluster is complaining about a slow heartbeat on the vxlan tunnel... maybe i should reduce the mtu even further | 03:52 |
| *** acidfu has quit IRC | 03:54 | |
| *** acidfu has joined #openvswitch | 04:03 | |
| larsks | bdonnahue2: I don't know that reducing the mtu is going to impact performance (it may ultimately have a negative effect, because as you make the mtu smaller you're increasing the % of your traffic that is just metadata (because the header sizes stay the same, so you're reducing the amount of data in each frame). That's why, for example, network storage is often served over a dedicated network | 04:18 |
| larsks | using large (9000+ bytes) mtus. | 04:18 |
| * larsks 😴 | 04:19 | |
| bdonnahue2 | larsks, i was just reading a post about that. Trying to figure out how to determine what mtu my switches support | 04:20 |
| *** dholler has joined #openvswitch | 05:33 | |
| *** links has joined #openvswitch | 05:56 | |
| *** slaweq has quit IRC | 06:16 | |
| *** slaweq has joined #openvswitch | 06:19 | |
| *** amorenoz_ has joined #openvswitch | 06:20 | |
| *** ralonsoh has joined #openvswitch | 06:32 | |
| *** eelco has joined #openvswitch | 06:42 | |
| *** amorenoz_ has quit IRC | 06:50 | |
| *** rcernin has quit IRC | 06:52 | |
| *** bdonnahue2 has quit IRC | 06:54 | |
| *** amorenoz has joined #openvswitch | 06:55 | |
| *** bdonnahue1 has joined #openvswitch | 07:04 | |
| *** rcernin has joined #openvswitch | 07:08 | |
| *** mdgray has joined #openvswitch | 07:14 | |
| *** elvira has joined #openvswitch | 07:37 | |
| *** rcernin has quit IRC | 07:43 | |
| *** rcernin has joined #openvswitch | 07:49 | |
| *** rcernin has quit IRC | 07:59 | |
| *** rcernin has joined #openvswitch | 08:31 | |
| *** ktraynor has joined #openvswitch | 08:33 | |
| *** rcernin has quit IRC | 08:38 | |
| *** biyiklioglu has joined #openvswitch | 08:46 | |
| *** imaximets_ has joined #openvswitch | 08:47 | |
| *** imaximets has quit IRC | 08:49 | |
| *** imaximets_ is now known as imaximets | 08:50 | |
| *** biyiklioglu has quit IRC | 08:56 | |
| *** amorenoz has quit IRC | 09:13 | |
| *** amorenoz has joined #openvswitch | 09:24 | |
| *** istokes has joined #openvswitch | 09:25 | |
| *** cpaelzer has quit IRC | 09:40 | |
| *** yamamoto has quit IRC | 09:59 | |
| *** deadalnix_ has joined #openvswitch | 10:01 | |
| *** cpaelzer__ has joined #openvswitch | 10:03 | |
| *** anilvenkata has joined #openvswitch | 10:08 | |
| *** deadalnix_ has quit IRC | 10:19 | |
| *** yamamoto has joined #openvswitch | 10:19 | |
| *** deadalnix_ has joined #openvswitch | 10:28 | |
| *** rcernin has joined #openvswitch | 10:37 | |
| *** anilvenkata has quit IRC | 10:37 | |
| *** rcernin has quit IRC | 11:04 | |
| *** rcernin has joined #openvswitch | 11:28 | |
| *** yamamoto has quit IRC | 11:31 | |
| *** rcernin has quit IRC | 11:33 | |
| *** yamamoto has joined #openvswitch | 12:03 | |
| *** fbl has joined #openvswitch | 12:05 | |
| *** yamamoto has quit IRC | 12:08 | |
| *** ricolin has quit IRC | 12:15 | |
| *** ricolin has joined #openvswitch | 12:15 | |
| *** tbachman has joined #openvswitch | 12:29 | |
| *** bostondriver has joined #openvswitch | 12:53 | |
| *** acidfoo has joined #openvswitch | 13:14 | |
| *** acidfu has quit IRC | 13:16 | |
| *** fdangelo__ has quit IRC | 13:30 | |
| *** rcernin has joined #openvswitch | 13:35 | |
| *** dcbw has joined #openvswitch | 13:38 | |
| *** rcernin has quit IRC | 13:41 | |
| *** ricolin has quit IRC | 13:49 | |
| *** ricolin has joined #openvswitch | 13:50 | |
| *** rfolco has quit IRC | 13:56 | |
| *** rfolco has joined #openvswitch | 14:01 | |
| *** fdangelo__ has joined #openvswitch | 14:21 | |
| *** lablt has quit IRC | 14:33 | |
| *** lablt has joined #openvswitch | 14:33 | |
| *** ralonsoh_ has joined #openvswitch | 15:06 | |
| *** ralonsoh has quit IRC | 15:06 | |
| *** ralonsoh_ has quit IRC | 15:07 | |
| *** ralonsoh has joined #openvswitch | 15:08 | |
| *** anilvenkata has joined #openvswitch | 15:23 | |
| bdonnahue1 | larsks, i think there is some complexity to vxlan that i am not understanding. for some reacon i cannot ping the other vxlan nodes on the interface connected to the bridge | 15:25 |
| bdonnahue1 | how can i troubleshoot errors ovs might be having while establishign the connection? | 15:26 |
| bdonnahue1 | hmm i found some logs in /var/log/openvswitch... i see erorr indicating the service has failed | 15:27 |
| bdonnahue1 | idk if the service is what keeps the tunnel open | 15:27 |
| *** Tobbbles has joined #openvswitch | 15:31 | |
| *** rcernin has joined #openvswitch | 15:36 | |
| larsks | vxlan is operationally pretty simple. I suspect the failed service may be your problem. | 15:38 |
| bdonnahue1 | i think openvswitch might have borked... i see processes running (ovsdb and ovs-vswitched) but the openvswitch-service says its ative and exited | 15:39 |
| *** lablt has quit IRC | 15:40 | |
| *** lablt has joined #openvswitch | 15:40 | |
| bdonnahue1 | i only see warn but no error in the log | 15:40 |
| bdonnahue1 | |WARN|received packet on unknown port 3 on bridge vmbr1 | 15:40 |
| bdonnahue1 | |WARN|receive tunnel port not found | 15:41 |
| bdonnahue1 | so the tunnel is having issues (vmbr1 is the bridge for the tunnel) | 15:41 |
| *** rcernin has quit IRC | 15:41 | |
| *** anilvenkata has quit IRC | 15:43 | |
| bdonnahue1 | maybe the service being down is unrelated | 15:45 |
| bdonnahue1 | i can ping on of the two other nodes | 15:45 |
| *** deadalnix__ has joined #openvswitch | 15:46 | |
| *** deadalnix_ has quit IRC | 15:49 | |
| bdonnahue1 | something interesting... nodes A and C can both ping B, bot not eachother | 15:49 |
| bdonnahue1 | i restarted (systemctl restart openvswitch-nonetwork) and temporarily saw a few pings go through | 15:49 |
| *** eelco has quit IRC | 15:52 | |
| *** links has quit IRC | 15:55 | |
| bdonnahue1 | now theyre all offline | 16:00 |
| larsks | Maybe try running 2.15? It's been stable for me so far. | 16:00 |
| *** elvira has quit IRC | 16:03 | |
| *** moldorcoder7 has quit IRC | 16:04 | |
| bdonnahue1 | i think im on 2.12, will have a look | 16:08 |
| bdonnahue1 | proxmox says openvswitch-switch is already the newest version (2.12.3-1). | 16:09 |
| bdonnahue1 | larsks, trying to update the package now | 16:17 |
| bdonnahue1 | larsks, i dont think 2.15 is built for debian | 16:20 |
| bdonnahue1 | i see proxmox is debian 10 and there is a ovs 2.15 on debian sid which is unstable | 16:23 |
| *** dholler has quit IRC | 16:24 | |
| *** moldorcoder7 has joined #openvswitch | 16:25 | |
| *** Tobbbles has quit IRC | 16:27 | |
| *** lablt has quit IRC | 16:33 | |
| bdonnahue1 | i am updating my kernel etc to see if that helps | 16:33 |
| *** lablt has joined #openvswitch | 16:35 | |
| *** tbachman has quit IRC | 16:50 | |
| *** tbachman has joined #openvswitch | 16:51 | |
| *** moldorcoder7 has quit IRC | 16:51 | |
| bdonnahue1 | upgrading did not help... i am thinkign there must be something wrong with the ovs configuration or a bug in this version | 16:52 |
| bdonnahue1 | larsks, does anything look wrong with the decorators in my config: https://pastebin.com/jSMb8DMu | 16:53 |
| bdonnahue1 | its odd i am not seeing errors in the logs... | 16:53 |
| *** moldorcoder7 has joined #openvswitch | 16:59 | |
| *** ralonsoh has quit IRC | 17:15 | |
| *** ihrachys has quit IRC | 17:23 | |
| *** rcernin has joined #openvswitch | 17:37 | |
| *** rcernin has quit IRC | 17:42 | |
| *** lablt has quit IRC | 18:07 | |
| *** lablt has joined #openvswitch | 18:07 | |
| *** deadalnix__ has quit IRC | 18:20 | |
| *** lablt has quit IRC | 18:32 | |
| *** lablt has joined #openvswitch | 18:38 | |
| *** istokes has quit IRC | 18:42 | |
| *** lablt has quit IRC | 18:47 | |
| *** lablt has joined #openvswitch | 18:49 | |
| *** yamamoto has joined #openvswitch | 18:55 | |
| *** yamamoto has quit IRC | 18:59 | |
| *** moldorcoder7 has quit IRC | 19:02 | |
| *** slaweq has quit IRC | 19:13 | |
| *** lablt has quit IRC | 19:16 | |
| *** moldorcoder7 has joined #openvswitch | 19:17 | |
| *** tbachman has quit IRC | 19:19 | |
| bdonnahue1 | i wonder if this is my issue: https://github.com/yyang13/ovs_nsh_patches/issues/2 | 19:25 |
| *** mdgray has quit IRC | 19:29 | |
| *** slaweq has joined #openvswitch | 19:32 | |
| *** lablt has joined #openvswitch | 19:34 | |
| *** rcernin has joined #openvswitch | 19:38 | |
| bdonnahue1 | can anyone tell me where to look for error messages from openvswitch if the vxlan tunnel is crashing? | 19:38 |
| *** rcernin has quit IRC | 19:43 | |
| *** tbachman has joined #openvswitch | 19:58 | |
| *** lablt has quit IRC | 19:59 | |
| *** lablt has joined #openvswitch | 20:05 | |
| *** slaweq_ has joined #openvswitch | 20:22 | |
| *** fdangelo__ has quit IRC | 20:31 | |
| *** moldorcoder7 has quit IRC | 20:32 | |
| *** acidfoo has quit IRC | 20:33 | |
| *** ygk_12345_ has quit IRC | 20:37 | |
| *** edwarnicke has quit IRC | 20:37 | |
| *** fkautz has quit IRC | 20:38 | |
| *** ygk_12345_ has joined #openvswitch | 20:38 | |
| *** fkautz has joined #openvswitch | 20:38 | |
| *** d1rewolf has quit IRC | 20:38 | |
| *** mnasiadka has quit IRC | 20:38 | |
| *** edwarnicke has joined #openvswitch | 20:38 | |
| *** mnasiadka has joined #openvswitch | 20:39 | |
| *** d1rewolf has joined #openvswitch | 20:39 | |
| *** moldorcoder7 has joined #openvswitch | 20:43 | |
| *** tbachman has quit IRC | 20:43 | |
| *** rfolco has quit IRC | 20:46 | |
| *** lablt has quit IRC | 21:10 | |
| *** lablt has joined #openvswitch | 21:11 | |
| *** lablt has quit IRC | 21:22 | |
| *** lablt has joined #openvswitch | 21:23 | |
| *** rcernin has joined #openvswitch | 21:38 | |
| *** rcernin has quit IRC | 21:44 | |
| *** rcernin has joined #openvswitch | 21:51 | |
| *** rcernin has quit IRC | 21:57 | |
| *** fdangelo__ has joined #openvswitch | 22:00 | |
| *** yamamoto has joined #openvswitch | 22:01 | |
| *** rcernin has joined #openvswitch | 22:12 | |
| *** rcernin has quit IRC | 22:14 | |
| *** rcernin has joined #openvswitch | 22:14 | |
| *** tbachman has joined #openvswitch | 22:15 | |
| *** yamamoto has quit IRC | 22:41 | |
| *** yamamoto has joined #openvswitch | 22:42 | |
| *** matteo| has joined #openvswitch | 22:51 | |
| *** matteo has quit IRC | 22:53 | |
| *** deadalnix has joined #openvswitch | 23:16 | |
| *** deadalnix_ has joined #openvswitch | 23:29 | |
| *** tbachman has quit IRC | 23:31 | |
| *** tbachman_ has joined #openvswitch | 23:31 | |
| *** deadalnix has quit IRC | 23:33 | |
| *** tbachman has joined #openvswitch | 23:37 | |
| *** thaller has quit IRC | 23:37 | |
| *** thaller has joined #openvswitch | 23:37 | |
| *** tbachman_ has quit IRC | 23:39 | |
| *** bostondriver has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!