russellb | azbiswas: yes, i did | 00:36 |
---|---|---|
russellb | azbiswas: the problems i was hitting at the time were a combination of 1) a bug in ovs+conntrack support. and 2) we needed to add some custom ACLs allowing DHCP. | 00:36 |
russellb | are you having trouble? | 00:36 |
azbiswas | we are having trouble with DHCP requests reaching the DHCP server | 00:39 |
azbiswas | for provider networks | 00:39 |
azbiswas | if the dhcp namespace is the only port in the network on a host, then the DHCP request does reach it, but if there are other VMs on the same host, then it's not reaching | 00:41 |
russellb | interesting | 00:41 |
russellb | is this a VM reaching a remote DHCP server? | 00:41 |
azbiswas | We have bonding enabled as well, so it could be this - https://bugzilla.redhat.com/show_bug.cgi?id=1267291 | 00:41 |
openstack | bugzilla.redhat.com bug 1267291 in openvswitch "[Openvswitch] balance-tcp bond mode causing issues with OSP Deployments" [High,New] - Assigned to lrichard | 00:41 |
russellb | a DHCP server on another host | 00:41 |
russellb | ok, i'm not familiar with bonding and OVS | 00:42 |
russellb | you could try disabling that to eliminate it from the equation | 00:42 |
azbiswas | right | 00:42 |
azbiswas | i'm not in charge of the environment :), i can only observe and see what's happening. | 00:43 |
russellb | OK | 00:43 |
russellb | any digging yet? or just looking around to see if it was a known issue? | 00:44 |
azbiswas | the frame was reaching the br-pub but not reaching the dhcp agent unless that dhcp agent was the only port on that network in the host. That indicated a broadcast problem. | 00:45 |
russellb | and this is using up to date networking-ovn and ovs/ovn ? | 00:45 |
russellb | this was a problem with provider networks several weeks ago | 00:45 |
russellb | more than 1 port on the same provider net on the same host was broken | 00:46 |
azbiswas | it's pretty recent i.e. the ebay/han zhou patch is there | 00:46 |
russellb | ok, sounds recent enough. | 00:46 |
russellb | (brb) | 00:46 |
russellb | if you know it's hitting the remote host, then the next thing i'd be doing is trying to debug the flow table ... | 00:48 |
russellb | i'd have a client generating a DHCP request regularly (every second or something) | 00:49 |
russellb | and then i'd watch the counters in the flow table to see what's getting hit | 00:49 |
russellb | that usually helps me spot where a packet is going | 00:49 |
russellb | i really like this command, or some variant ... watch -d -n 0.5 "sudo ovs-ofctl -O OpenFlow13 dump-flows br-int | cut -f3- -d','" | 00:49 |
russellb | you could also grep for "metadata=0x1" (or whatever the right number is) to only look at flows for a certain datapath | 00:50 |
azbiswas | we were doing something along that line, it was hitting the broadcast flow but not the one after. | 00:52 |
azbiswas | The watch command is something I'll keep in mind. | 00:52 |
russellb | sounds like you've narrowed it down a good bit | 00:53 |
azbiswas | We don't seem to hitting this in other cloud systems, so it's not consistent either. There could be other stability issues. I wanted to confirm that the conntrack/ovs bug you found was fixed. | 00:53 |
russellb | yes, the bug i was hitting back then was fixed | 00:54 |
russellb | azbiswas: i believe it was https://github.com/openvswitch/ovs/commit/e37b8437e915a02a88116e5ea7af1a7e716bd597 | 00:58 |
azbiswas | russellb: thanks | 00:59 |
russellb | np | 01:04 |
Sam-I-Am | russellb: weeeeekennnndddd | 01:06 |
russellb | i have beer | 01:06 |
Sam-I-Am | but you also have computer | 01:07 |
russellb | it's true | 01:07 |
Sam-I-Am | so, halfway there :) | 01:07 |
Sam-I-Am | i need to weekend hard | 01:07 |
Sam-I-Am | its been... rough | 01:07 |
russellb | sorry to hear that | 01:07 |
Sam-I-Am | new bike showed up today, so there's that | 01:07 |
russellb | nice | 01:07 |
Sam-I-Am | going to see how it rides tomorrow | 01:07 |
russellb | i went to a bike store today for one of those tow your kid around trailers | 01:07 |
Sam-I-Am | oh, fun | 01:08 |
Sam-I-Am | thats good exercise | 01:08 |
russellb | yeah, kid was scared of it first try, but we'll keep at it :) | 01:08 |
Sam-I-Am | i'm scared of you | 01:08 |
russellb | makes sense | 01:08 |
Sam-I-Am | its a thing | 01:09 |
russellb | it has been mid-70s here | 01:09 |
russellb | <3 | 01:09 |
Sam-I-Am | its been lovely here | 01:10 |
Sam-I-Am | not getting out as much as i should be | 01:10 |
Sam-I-Am | figure now that my road bike showed up, winter will make another appearance | 01:10 |
Sam-I-Am | what really worries me is dry march-april here leads to fires | 01:10 |
*** chandrav has quit IRC | 01:11 | |
Sam-I-Am | i remember riding home from work in 85 degrees in march 2012... and the whole state lit up that year. | 01:11 |
azbiswas | There are storms all over the left coast here, so there should be residual over the rockies | 01:11 |
Sam-I-Am | stuff has sort of gone around here | 01:12 |
Sam-I-Am | the mountains arent doing too bad, actually | 01:12 |
Sam-I-Am | its the plains that are dry | 01:12 |
Sam-I-Am | russellb: i think this beer thing is a good idea | 01:12 |
Sam-I-Am | perhaps with a side of pizza | 01:12 |
russellb | looks like the valley should be nice for my visit next week | 01:12 |
* azbiswas thirsty | 01:13 | |
russellb | sunny, upper 60s/70s | 01:13 |
russellb | i had some nachos. | 01:13 |
Sam-I-Am | russellb: with a side of donut? | 01:13 |
russellb | no! | 01:13 |
Sam-I-Am | a few weeks ago the donut shop had nutella-stuffed glazed donuts | 01:13 |
Sam-I-Am | if you're going to donut, you might as well go all in | 01:14 |
russellb | i thought i might code a little | 01:19 |
russellb | but... | 01:19 |
Sam-I-Am | heh | 01:21 |
Sam-I-Am | i'm trying to avoid my roommate while he watches the ku baylor game | 01:21 |
Sam-I-Am | things are going to get real here | 01:21 |
russellb | ha | 01:22 |
russellb | i'm almost due for another beer | 01:22 |
Sam-I-Am | i just got one | 01:22 |
russellb | and at that point there is 0% chance of productivity | 01:22 |
Sam-I-Am | well, i dunno about that. killing off the weak brain cells let the others work more efficiencly. | 01:23 |
Sam-I-Am | thats why you feel smarter after drinking beer | 01:23 |
*** asti has joined #openstack-neutron-ovn | 01:24 | |
russellb | you're on to something | 01:24 |
Sam-I-Am | code reviews go much better with beer | 01:24 |
Sam-I-Am | i need to uncomputer pretty hard this weekend. bikes. airplanes. in particular, washing an airplane. its pleasantly mindless, and when i get done, i have a feeling of accomplishment. | 01:25 |
Sam-I-Am | might break out the RC helicopters after mothballing them for the winter | 01:25 |
*** banix has joined #openstack-neutron-ovn | 01:27 | |
*** jckasper has joined #openstack-neutron-ovn | 01:33 | |
russellb | Sam-I-Am: sounds fun! | 01:36 |
*** jckasper has quit IRC | 01:38 | |
Sam-I-Am | russellb: yep. its amazing how liberating washing and waxing an airplane is. | 01:38 |
*** asti has quit IRC | 01:40 | |
*** fzdarsky|afk has quit IRC | 02:22 | |
*** fzdarsky has joined #openstack-neutron-ovn | 02:28 | |
*** gongysh has joined #openstack-neutron-ovn | 02:43 | |
*** s3wong has quit IRC | 03:03 | |
*** azbiswas has quit IRC | 03:08 | |
*** irenab has quit IRC | 03:24 | |
*** irenab has joined #openstack-neutron-ovn | 03:28 | |
*** azbiswas has joined #openstack-neutron-ovn | 04:09 | |
*** jckasper has joined #openstack-neutron-ovn | 04:17 | |
*** azbiswas has quit IRC | 04:24 | |
*** azbiswas has joined #openstack-neutron-ovn | 04:24 | |
*** azbiswas has quit IRC | 04:29 | |
*** banix has quit IRC | 04:30 | |
*** jckasper has quit IRC | 04:31 | |
*** jckasper has joined #openstack-neutron-ovn | 04:32 | |
*** jckasper has quit IRC | 04:40 | |
*** jckasper has joined #openstack-neutron-ovn | 04:40 | |
*** jckasper has quit IRC | 04:41 | |
*** jckasper has joined #openstack-neutron-ovn | 04:41 | |
*** azbiswas has joined #openstack-neutron-ovn | 05:27 | |
*** azbiswas has quit IRC | 05:27 | |
*** jckasper has quit IRC | 05:31 | |
*** jckasper has joined #openstack-neutron-ovn | 05:31 | |
*** jckasper has quit IRC | 05:33 | |
*** jckasper has joined #openstack-neutron-ovn | 05:33 | |
*** armax has quit IRC | 05:38 | |
*** jckasper has quit IRC | 06:06 | |
*** jckasper has joined #openstack-neutron-ovn | 06:06 | |
*** jckasper has joined #openstack-neutron-ovn | 06:07 | |
*** jckasper has quit IRC | 06:19 | |
*** jckasper has joined #openstack-neutron-ovn | 06:19 | |
*** jckasper has quit IRC | 06:19 | |
*** jckasper has joined #openstack-neutron-ovn | 06:20 | |
*** jckasper has quit IRC | 06:38 | |
*** jckasper has joined #openstack-neutron-ovn | 06:38 | |
*** jckasper has quit IRC | 06:43 | |
*** jckasper has joined #openstack-neutron-ovn | 06:45 | |
*** gangil has quit IRC | 07:15 | |
*** gangil has joined #openstack-neutron-ovn | 07:17 | |
*** gangil has joined #openstack-neutron-ovn | 07:17 | |
*** jckasper has quit IRC | 07:27 | |
*** jckasper has joined #openstack-neutron-ovn | 07:28 | |
*** gongysh has quit IRC | 07:29 | |
*** jckasper has quit IRC | 07:32 | |
*** gangil has quit IRC | 08:02 | |
*** yamamoto has joined #openstack-neutron-ovn | 10:19 | |
*** yamamoto has quit IRC | 10:19 | |
*** yamamoto has joined #openstack-neutron-ovn | 11:20 | |
*** yamamoto has quit IRC | 11:22 | |
*** yamamoto has joined #openstack-neutron-ovn | 11:22 | |
*** yamamoto has quit IRC | 11:22 | |
*** jckasper has joined #openstack-neutron-ovn | 11:29 | |
*** jckasper has quit IRC | 11:33 | |
*** yamamoto has joined #openstack-neutron-ovn | 11:50 | |
*** yamamoto has quit IRC | 12:15 | |
*** yamamoto has joined #openstack-neutron-ovn | 12:22 | |
*** yamamoto has quit IRC | 12:36 | |
*** banix has joined #openstack-neutron-ovn | 12:52 | |
*** yamamoto has joined #openstack-neutron-ovn | 13:14 | |
*** yamamoto has quit IRC | 13:41 | |
*** numans has joined #openstack-neutron-ovn | 13:41 | |
*** yamamoto has joined #openstack-neutron-ovn | 13:43 | |
*** yamamoto has quit IRC | 13:43 | |
*** yamamoto has joined #openstack-neutron-ovn | 13:44 | |
*** numans has quit IRC | 13:46 | |
*** banix has quit IRC | 13:50 | |
*** asti has joined #openstack-neutron-ovn | 13:54 | |
*** yamamoto has quit IRC | 13:58 | |
*** yamamoto has joined #openstack-neutron-ovn | 13:59 | |
*** asti has quit IRC | 14:07 | |
*** banix has joined #openstack-neutron-ovn | 14:09 | |
*** banix has quit IRC | 14:23 | |
*** banix has joined #openstack-neutron-ovn | 14:24 | |
*** banix has quit IRC | 14:56 | |
*** yamamoto has quit IRC | 15:11 | |
*** yamamoto has joined #openstack-neutron-ovn | 15:12 | |
*** yamamoto has quit IRC | 15:16 | |
*** irenab has quit IRC | 15:27 | |
*** irenab has joined #openstack-neutron-ovn | 15:29 | |
*** openstackgerrit_ has quit IRC | 15:30 | |
*** jckasper has joined #openstack-neutron-ovn | 15:30 | |
*** jckasper has quit IRC | 15:35 | |
*** yamamoto has joined #openstack-neutron-ovn | 15:40 | |
*** openstackgerrit_ has joined #openstack-neutron-ovn | 15:43 | |
*** obondarev has joined #openstack-neutron-ovn | 16:08 | |
*** obondarev_ has quit IRC | 16:12 | |
*** markmcclain has quit IRC | 16:12 | |
*** markmcclain has joined #openstack-neutron-ovn | 16:14 | |
*** gangil has joined #openstack-neutron-ovn | 16:27 | |
*** gangil has joined #openstack-neutron-ovn | 16:27 | |
*** gangil has quit IRC | 16:31 | |
*** armax has joined #openstack-neutron-ovn | 16:39 | |
*** yamamoto has quit IRC | 17:00 | |
*** dlundquist has quit IRC | 17:32 | |
*** yamamoto has joined #openstack-neutron-ovn | 18:00 | |
*** yamamoto has quit IRC | 18:11 | |
*** armax has quit IRC | 19:01 | |
*** jckasper has joined #openstack-neutron-ovn | 19:32 | |
*** jckasper has quit IRC | 19:37 | |
*** banix has joined #openstack-neutron-ovn | 20:02 | |
*** banix has quit IRC | 20:06 | |
openstackgerrit | Amitabha Biswas proposed openstack/networking-ovn: Reduce number of OVSDB row changes during ACL updates https://review.openstack.org/282488 | 22:14 |
*** jckasper has joined #openstack-neutron-ovn | 23:34 | |
*** jckasper has quit IRC | 23:38 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!