| *** thumpba has joined #openstack-neutron-ovn | 00:15 | |
| *** thumpba has quit IRC | 00:20 | |
| *** dslev has joined #openstack-neutron-ovn | 00:34 | |
| *** chenli has joined #openstack-neutron-ovn | 00:38 | |
| *** dslev has quit IRC | 01:05 | |
| *** lrichard has quit IRC | 01:14 | |
| *** fzdarsky__ has joined #openstack-neutron-ovn | 02:12 | |
| *** fzdarsky_ has quit IRC | 02:15 | |
| *** thumpba has joined #openstack-neutron-ovn | 02:18 | |
| *** yamamoto has joined #openstack-neutron-ovn | 02:20 | |
| *** thumpba has quit IRC | 02:23 | |
| *** yamamoto has quit IRC | 02:31 | |
| *** thumpba has joined #openstack-neutron-ovn | 02:54 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 03:23 | |
| *** mamulsow has quit IRC | 03:30 | |
| *** gangil has joined #openstack-neutron-ovn | 03:52 | |
| *** gangil has joined #openstack-neutron-ovn | 03:52 | |
| *** dslev has joined #openstack-neutron-ovn | 04:01 | |
| *** dslev has quit IRC | 04:06 | |
| *** thumpba has quit IRC | 05:59 | |
| *** gangil has quit IRC | 06:00 | |
| *** gangil has joined #openstack-neutron-ovn | 06:08 | |
| *** gangil has joined #openstack-neutron-ovn | 06:08 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 07:18 | |
| *** fzdarsky__ has quit IRC | 07:34 | |
| *** fzdarsky has joined #openstack-neutron-ovn | 07:36 | |
| *** palexster has joined #openstack-neutron-ovn | 08:25 | |
| *** gangil has quit IRC | 08:31 | |
| *** thumpba has joined #openstack-neutron-ovn | 08:31 | |
| *** thumpba has quit IRC | 08:36 | |
| *** salv-orlando has quit IRC | 09:04 | |
| *** flaviof has quit IRC | 09:22 | |
| *** chenli has quit IRC | 09:34 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 10:05 | |
| *** salv-orlando has quit IRC | 10:12 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 10:22 | |
| *** thumpba has joined #openstack-neutron-ovn | 10:33 | |
| *** thumpba has quit IRC | 10:39 | |
| *** yamamoto_ has quit IRC | 11:12 | |
| *** roeyc has joined #openstack-neutron-ovn | 11:34 | |
| *** yamamoto has joined #openstack-neutron-ovn | 11:58 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 11:59 | |
| *** yamamoto has quit IRC | 12:03 | |
| *** yamamoto_ has quit IRC | 12:17 | |
| *** rtheis has joined #openstack-neutron-ovn | 12:19 | |
| *** yamamoto has joined #openstack-neutron-ovn | 12:26 | |
| *** thumpba has joined #openstack-neutron-ovn | 12:50 | |
| *** thumpba has quit IRC | 12:55 | |
| *** salv-orlando has quit IRC | 13:02 | |
| *** yamamoto has quit IRC | 13:03 | |
| *** yamamoto has joined #openstack-neutron-ovn | 13:04 | |
| *** yamamoto has quit IRC | 13:08 | |
| *** lrichard has joined #openstack-neutron-ovn | 13:12 | |
| *** yamamoto has joined #openstack-neutron-ovn | 13:17 | |
| *** flaviof has joined #openstack-neutron-ovn | 13:22 | |
| *** dslev has joined #openstack-neutron-ovn | 13:52 | |
| *** dslev_ has joined #openstack-neutron-ovn | 13:53 | |
| *** gongysh has joined #openstack-neutron-ovn | 13:53 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 13:57 | |
| *** dslev has quit IRC | 13:57 | |
| *** yamamoto has quit IRC | 14:02 | |
| *** thumpba has joined #openstack-neutron-ovn | 14:10 | |
| openstackgerrit | Merged openstack/networking-ovn: Vagrant: Fix issue with boxes https://review.openstack.org/271494 | 14:32 |
|---|---|---|
| openstackgerrit | Merged openstack/networking-ovn: Vagrant: Adjust HOST_IP for compute nodes https://review.openstack.org/271495 | 14:33 |
| *** gongysh has quit IRC | 14:34 | |
| russellb | tempest totally broken for us right now, fix is: https://review.openstack.org/#/c/272082/1 | 15:02 |
| *** shettyg has joined #openstack-neutron-ovn | 15:16 | |
| *** shettyg has quit IRC | 15:17 | |
| *** dslev_ has quit IRC | 15:45 | |
| Sam-I-Am | russellb: moo? | 15:49 |
| russellb | quack. | 15:50 |
| Sam-I-Am | russellb: got some time today? | 15:50 |
| russellb | yes | 15:50 |
| Sam-I-Am | excellent. think friday ran out on us. | 15:50 |
| Sam-I-Am | are you east coast? | 15:51 |
| *** regXboi has joined #openstack-neutron-ovn | 15:51 | |
| Sam-I-Am | here comes trouble | 15:51 |
| Sam-I-Am | oh hai regXboi | 15:52 |
| regXboi | Sam-I-Am: not trouble yet - I think I've figured out my headache, which is that I'm missing a log | 15:53 |
| regXboi | what is happening is A->B->C and I'm looking at the logs of A and C, without the log of B | 15:54 |
| regXboi | which makes understanding the process somewhat challenging | 15:54 |
| Sam-I-Am | yeo | 15:55 |
| Sam-I-Am | yep | 15:55 |
| regXboi | otherwise, morning all | 16:02 |
| *** salv-orl_ has joined #openstack-neutron-ovn | 16:05 | |
| Sam-I-Am | regXboi: ramping this mtu testing up was sort of similar | 16:06 |
| Sam-I-Am | 'crap i should have tested X here too' | 16:06 |
| *** salv-orlando has quit IRC | 16:08 | |
| russellb | yes i'm east coast | 16:12 |
| *** shettyg has joined #openstack-neutron-ovn | 16:12 | |
| Sam-I-Am | russellb: whens good for you today? | 16:14 |
| mestery | russellb: Sad about the tempest brokeness :( | 16:42 |
| *** flaviof has quit IRC | 16:59 | |
| *** ajo has quit IRC | 17:20 | |
| russellb | mestery: all your fault! | 17:24 |
| russellb | Sam-I-Am: i have a call at 1, so 2pm or later (eastern) | 17:24 |
| Sam-I-Am | russellb: lets do 2 eastern | 17:27 |
| russellb | k | 17:27 |
| *** ajo has joined #openstack-neutron-ovn | 17:33 | |
| *** azbiswas has joined #openstack-neutron-ovn | 17:35 | |
| *** azbiswas_ has joined #openstack-neutron-ovn | 17:40 | |
| *** azbiswas has quit IRC | 17:43 | |
| *** azbiswas_ has quit IRC | 17:48 | |
| *** azbiswas_ has joined #openstack-neutron-ovn | 17:49 | |
| *** shettyg has quit IRC | 18:03 | |
| russellb | mestery: could use a review on this once jenkins signs off https://review.openstack.org/#/c/272082/ | 18:07 |
| * mestery looks | 18:07 | |
| russellb | networking-ovn busted on that | 18:07 |
| mestery | Done! | 18:08 |
| mestery | WAiting on Jenkins | 18:09 |
| *** gangil has joined #openstack-neutron-ovn | 18:09 | |
| *** gangil has joined #openstack-neutron-ovn | 18:09 | |
| *** chandrav has joined #openstack-neutron-ovn | 18:30 | |
| *** salv-orl_ has quit IRC | 18:35 | |
| *** shettyg has joined #openstack-neutron-ovn | 18:40 | |
| *** pcaruana has joined #openstack-neutron-ovn | 18:51 | |
| regXboi | russellb: does it make sense to add code to the devstack/plugin to treat ovsdb-server log the same way as the other logs? Right now, there is only one copy kept around which caused me some grief after a restack :( | 18:54 |
| russellb | regXboi: yes, and ovs-vswitchd while we're at it | 18:54 |
| regXboi | russellb: ok, I'll spin a patch out in the near future to address both issues | 18:54 |
| russellb | ack thanks | 18:54 |
| *** gangil has quit IRC | 18:55 | |
| Sam-I-Am | russellb: can we push our chat 30 minutes? | 18:57 |
| russellb | Sam-I-Am: yes, i'm reading your dev posts now anyway | 18:57 |
| Sam-I-Am | russellb: oh, they're really exciting. | 19:00 |
| Sam-I-Am | the one from this past friday is the most 'accurate' since i had a real lab with jumbo frames and didn't have to munge a bunch of stuff | 19:00 |
| *** openstackgerrit has quit IRC | 19:02 | |
| *** openstackgerrit has joined #openstack-neutron-ovn | 19:02 | |
| *** pcaruana has quit IRC | 19:32 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 19:35 | |
| *** pcaruana has joined #openstack-neutron-ovn | 19:36 | |
| Sam-I-Am | russellb: how goes the reading? | 19:39 |
| russellb | well, i read it | 19:39 |
| russellb | i can't say i have my head wrapped around it all though | 19:39 |
| Sam-I-Am | where can i help? | 19:40 |
| russellb | i probably just have to spend more time understanding things ... most helpful thing probably is to spoon feed me/ovn any recommendations you have | 19:40 |
| *** pcaruana has quit IRC | 19:40 | |
| russellb | like "add option X to make life easier" | 19:41 |
| *** mamulsow has joined #openstack-neutron-ovn | 19:41 | |
| Sam-I-Am | i'm betting that ovn with the neutron l3 agent is going to experience at least some of these problems | 19:41 |
| Sam-I-Am | but i'm particularly curious about ovn w/ native l3 | 19:41 |
| Sam-I-Am | thats what i'm testing here in a bit | 19:41 |
| russellb | yes, native L3 is what i care about most | 19:41 |
| russellb | l3 agent is a temp hack anyway | 19:41 |
| russellb | ok cool | 19:41 |
| Sam-I-Am | also things like dhcp... any namespacey stuff | 19:42 |
| mamulsow | so jumping in the middle of a conversation I didn't see the beginning of, but.. | 19:42 |
| mamulsow | anything in particular you're looking for in the native l3 routing? | 19:43 |
| mamulsow | or dhcp stuff | 19:43 |
| *** pcaruana has joined #openstack-neutron-ovn | 19:43 | |
| mamulsow | I've been hammering on that stuff pretty hard | 19:43 |
| Sam-I-Am | mamulsow: in conventional neutron, we have a problem with mtu when VMs reside on overlay networks | 19:43 |
| Sam-I-Am | people have been throwing random fixes at it for a while, but none of them actually work, so i ran some experiments to see whats up | 19:44 |
| mamulsow | hmm, so we were seeing some issues that seemed to be MTU related, but I bumped MTU to 9000 on all our physical layer stuff and haven't seen MTU issues since then | 19:45 |
| *** salv-orlando has quit IRC | 19:46 | |
| Sam-I-Am | mamulsow: this wouldn't be performance issues... it would be more like things breaking issues | 19:46 |
| Sam-I-Am | let's say your physical layer has mtu 9000 | 19:47 |
| mamulsow | yes, I'm not 100% sure it was MTU caused, but when the compute nodes had MTU 1500 we were seeing dropped traffic and weird cases of only being able to get 1 way communication | 19:47 |
| Sam-I-Am | a packet from the outside world with an mtu of 9000 heads to the VM... which has a smaller mtu due to the overlay protocol overhead | 19:47 |
| Sam-I-Am | ohhh.... thats sort of smells like mtu problems | 19:47 |
| Sam-I-Am | mtu disparities are normal... e.g., the internet doesnt have a 9000 mtu, yet hosts with 9000 mtu seem to communicate fine over it. | 19:48 |
| Sam-I-Am | the problem is where the disparity occurs | 19:48 |
| mamulsow | yep | 19:49 |
| Sam-I-Am | negotiating mtu is a layer-3 operation that requires icmp... so if the disparity happens on a purely layer-2 device (such as a switch, bridge, or veth pair), packets with a larger mtu are silently dropped. | 19:49 |
| mamulsow | so for our tests VMs had MTU of 1500 and physical layer had MTU 1500, so with encap overhead we should have been splitting any large packets, but for some cases it seemed like packets were getting silently dropped | 19:50 |
| mamulsow | but it's entirely possible I had something misconfigured that was causing that issue | 19:52 |
| Sam-I-Am | this is what i'm looking to find out | 19:52 |
| Sam-I-Am | so your fix was just increasing the phys net mtu to 9000? | 19:53 |
| mamulsow | yep, and haven't seen issues since then | 19:53 |
| Sam-I-Am | ok, thats also what my conventional ovs testing reveals | 19:53 |
| Sam-I-Am | did you try just setting the vm's mtu to 1500 minus overlay overhead? | 19:54 |
| Sam-I-Am | the vm side is the easy part. we need them to know the mtu of their overlay link. | 19:54 |
| Sam-I-Am | the not so easy side is what comes in from the outside world | 19:55 |
| mamulsow | not this time, but in the past I believe we did that with conventional OVS and got good results | 19:55 |
| mamulsow | ie. VM MTU 1450 with vxlan encapsulation over physical layer at 1500 | 19:56 |
| Sam-I-Am | yeah | 19:56 |
| mamulsow | I believe that worked well | 19:56 |
| *** pcaruana has quit IRC | 19:56 | |
| Sam-I-Am | it sort of works | 19:56 |
| *** flaviof has joined #openstack-neutron-ovn | 19:56 | |
| mamulsow | but then we had the problem of what if a customer uploads an image with a different MTU | 19:56 |
| Sam-I-Am | hopefully people are using dhcp... or they're aware that mtu is 'a thing' and they can obtain the value from the api (or something) | 19:57 |
| Sam-I-Am | mamulsow: can you see in the code for the native l3 agent whether or not the ports plugged into it (assuming one on either network) account for the mtu of those networks? | 20:00 |
| Sam-I-Am | or if ovn/ovs generate icmp fragmentation-needed or packet-too-large messages in general | 20:01 |
| mamulsow | not off the top of my head, but I can go check | 20:02 |
| mamulsow | I suspect that it is not, give what we've been seeing | 20:02 |
| Sam-I-Am | so far, my testing with conventional ovs reveals that ovs itself Just Works with any arbitrary mtu | 20:04 |
| Sam-I-Am | its all the non-ovs stuff thats broken | 20:04 |
| Sam-I-Am | we'll always have a router between a provider net and a private net... and that's where the mtu change needs to occur, with icmp messages | 20:06 |
| *** lrichard has quit IRC | 20:07 | |
| Sam-I-Am | the mtu value for the private net should be calculated from the mtu value of the phys net it rides on | 20:07 |
| Sam-I-Am | mamulsow: is there a way to glean mtu values of ovs ports? | 20:08 |
| *** lrichard has joined #openstack-neutron-ovn | 20:11 | |
| *** flaviof has quit IRC | 20:13 | |
| *** flaviof_ has joined #openstack-neutron-ovn | 20:13 | |
| *** flaviof_ has quit IRC | 20:14 | |
| *** flaviof has joined #openstack-neutron-ovn | 20:15 | |
| russellb | Sam-I-Am: i like "just works"! | 20:20 |
| Sam-I-Am | russellb: yeah, hence why i'm curious whats going on inside | 20:20 |
| russellb | i wouldn't know.. | 20:21 |
| Sam-I-Am | inside ovs? | 20:21 |
| russellb | yes, i don't know ovs internals that well | 20:25 |
| russellb | Sam-I-Am: thanks for digging into this in so much detail, i do really appreciate it | 20:27 |
| Sam-I-Am | russellb: can you point me at someone for ovs internals? | 20:29 |
| Sam-I-Am | russellb: its a long-standing issue that needs resolution... lets get it right the first time with ovn :) | 20:29 |
| russellb | ovs discuss mailing list is probably the best audience | 20:32 |
| *** salv-orlando has joined #openstack-neutron-ovn | 20:32 | |
| *** rtheis has quit IRC | 20:38 | |
| *** chandrav has quit IRC | 20:38 | |
| *** thumpba has quit IRC | 20:46 | |
| openstackgerrit | Ryan Moats proposed openstack/networking-ovn: Make OVS log multi-stack friendly https://review.openstack.org/272265 | 20:54 |
| regXboi | russellb: ^^^^^ (there you go) | 20:55 |
| russellb | ack thanks | 20:55 |
| *** yamamoto has joined #openstack-neutron-ovn | 20:57 | |
| russellb | regXboi: i thought you were going to do ovsdb-server as well? | 21:00 |
| russellb | and i'm not actually sure what this is doing really | 21:00 |
| regXboi | russellb - this patch was for this one log | 21:00 |
| russellb | what about running ovsdb-server and ovs-vswitchd in screen with the rest? | 21:01 |
| russellb | run_process ovsdb-server .. | 21:01 |
| regXboi | russellb: well, now that's an interesting idea - I was thinking of that in a separate patch set | 21:01 |
| russellb | ok | 21:01 |
| regXboi | but -1 this and I'll merge it all togethre | 21:01 |
| russellb | k | 21:02 |
| russellb | thanks for working on it | 21:02 |
| regXboi | sure. no worries | 21:07 |
| regXboi | I got caught by it not working that way over the weekend and I don't want to lose work in the future | 21:07 |
| *** manand has joined #openstack-neutron-ovn | 21:07 | |
| regXboi | russellb: I see why I got confused - it looks like both ovs-vswitchd and ovsdb-server log to the same place | 21:09 |
| regXboi | which I can also fix in the next PS | 21:09 |
| Sam-I-Am | yeah... they can (and probably should) be different | 21:11 |
| Sam-I-Am | distros usually package them as such | 21:12 |
| *** gongysh has joined #openstack-neutron-ovn | 21:30 | |
| *** chandrav has joined #openstack-neutron-ovn | 21:47 | |
| *** roeyc has quit IRC | 21:50 | |
| *** roeyc has joined #openstack-neutron-ovn | 21:51 | |
| *** Guest66973 has joined #openstack-neutron-ovn | 21:56 | |
| *** gongysh has quit IRC | 22:01 | |
| *** regXboi has quit IRC | 22:01 | |
| *** gongysh has joined #openstack-neutron-ovn | 22:01 | |
| *** roeyc has quit IRC | 22:01 | |
| *** roeyc has joined #openstack-neutron-ovn | 22:02 | |
| *** roeyc has quit IRC | 22:03 | |
| *** salv-orl_ has joined #openstack-neutron-ovn | 22:06 | |
| *** roeyc has joined #openstack-neutron-ovn | 22:06 | |
| *** gongysh has quit IRC | 22:07 | |
| *** salv-orlando has quit IRC | 22:09 | |
| *** yamamoto has quit IRC | 22:10 | |
| *** flaviof has quit IRC | 22:22 | |
| *** s3wong has joined #openstack-neutron-ovn | 22:34 | |
| *** chandrav has quit IRC | 23:26 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!