| *** acidfu has joined #openvswitch | 00:14 | |
| *** dcbw has quit IRC | 00:34 | |
| *** fdangelo has quit IRC | 01:24 | |
| *** Anticimex has quit IRC | 01:56 | |
| *** Anticimex has joined #openvswitch | 01:59 | |
| *** rcernin has quit IRC | 02:10 | |
| *** rcernin has joined #openvswitch | 02:36 | |
| *** fdangelo has joined #openvswitch | 02:37 | |
| *** acidfu has quit IRC | 03:03 | |
| *** rcernin has quit IRC | 03:34 | |
| *** rcernin has joined #openvswitch | 03:44 | |
| *** rcernin has quit IRC | 03:55 | |
| *** rcernin has joined #openvswitch | 03:55 | |
| *** rcernin has quit IRC | 04:13 | |
| *** fdangelo has quit IRC | 04:15 | |
| *** osmanlicilegi has left #openvswitch | 04:18 | |
| *** ygk_12345 has joined #openvswitch | 04:41 | |
| *** donhw has quit IRC | 04:47 | |
| *** donhw has joined #openvswitch | 04:49 | |
| *** rcernin has joined #openvswitch | 04:55 | |
| *** rcernin has quit IRC | 05:10 | |
| *** donhw has quit IRC | 05:43 | |
| *** ygk_12345 has quit IRC | 05:43 | |
| *** donhw has joined #openvswitch | 05:43 | |
| *** rcernin has joined #openvswitch | 05:49 | |
| *** rcernin has joined #openvswitch | 05:50 | |
| *** slaweq has quit IRC | 05:55 | |
| *** slaweq_ has joined #openvswitch | 05:55 | |
| *** slaweq_ is now known as slaweq | 05:55 | |
| *** rcernin has quit IRC | 06:10 | |
| *** ralonsoh has joined #openvswitch | 06:10 | |
| *** osmanlicilegi has joined #openvswitch | 06:16 | |
| *** slaweq_ has joined #openvswitch | 06:21 | |
| *** slaweq_ has quit IRC | 06:27 | |
| *** eelco has joined #openvswitch | 06:27 | |
| *** rcernin has joined #openvswitch | 06:28 | |
| *** slaweq_ has joined #openvswitch | 06:29 | |
| *** slaweq_ has quit IRC | 06:29 | |
| *** rcernin has quit IRC | 06:29 | |
| *** slaweq_ has joined #openvswitch | 06:30 | |
| *** rcernin has joined #openvswitch | 06:30 | |
| *** slaweq has quit IRC | 06:36 | |
| *** slaweq_ is now known as slaweq | 06:36 | |
| *** dholler has joined #openvswitch | 06:39 | |
| *** amorenoz has joined #openvswitch | 06:40 | |
| *** ygk_12345 has joined #openvswitch | 06:46 | |
| *** ygk_12345 has quit IRC | 06:52 | |
| *** ygk_12345 has joined #openvswitch | 06:53 | |
| *** ygk_12345 has joined #openvswitch | 06:53 | |
| *** ygk_12345 has quit IRC | 06:56 | |
| *** jaicaa has quit IRC | 07:10 | |
| *** jaicaa has joined #openvswitch | 07:13 | |
| *** mdgray has joined #openvswitch | 07:17 | |
| *** mdgray has quit IRC | 07:23 | |
| *** mdgray has joined #openvswitch | 07:27 | |
| *** ygk_12345 has joined #openvswitch | 07:33 | |
| *** ygk_12345 has quit IRC | 07:34 | |
| *** ygk_12345 has joined #openvswitch | 07:35 | |
| *** elvira has joined #openvswitch | 07:46 | |
| *** ygk_12345 has quit IRC | 07:51 | |
| *** ygk_12345 has joined #openvswitch | 07:56 | |
| *** ygk has joined #openvswitch | 08:00 | |
| *** ygk has left #openvswitch | 08:04 | |
| *** ygk_12345 has left #openvswitch | 08:05 | |
| *** ygk_12345_ has joined #openvswitch | 08:08 | |
| *** rcernin has quit IRC | 08:37 | |
| *** slaweq has quit IRC | 08:51 | |
| *** slaweq has joined #openvswitch | 08:51 | |
| *** istokes has joined #openvswitch | 08:54 | |
| *** psahoo has joined #openvswitch | 10:12 | |
| *** fdangelo has joined #openvswitch | 10:54 | |
| *** deadalnix has joined #openvswitch | 11:18 | |
| *** psahoo_ has joined #openvswitch | 11:28 | |
| *** psahoo has quit IRC | 11:31 | |
| *** yamamoto has quit IRC | 11:51 | |
| *** rfolco has joined #openvswitch | 11:56 | |
| *** yamamoto has joined #openvswitch | 11:59 | |
| *** yamamoto has quit IRC | 12:04 | |
| *** bostondriver has joined #openvswitch | 12:04 | |
| *** yamamoto has joined #openvswitch | 12:46 | |
| *** yamamoto has quit IRC | 12:57 | |
| *** acidfu has joined #openvswitch | 13:17 | |
| *** istokes has quit IRC | 13:32 | |
| *** istokes has joined #openvswitch | 13:35 | |
| *** ygk_1234555 has joined #openvswitch | 13:42 | |
| *** fdangelo has quit IRC | 13:58 | |
| *** fdangelo has joined #openvswitch | 14:04 | |
| *** links has joined #openvswitch | 14:29 | |
| *** thaller_ has joined #openvswitch | 14:31 | |
| *** thaller has quit IRC | 14:34 | |
| *** eelco has quit IRC | 14:55 | |
| *** ygk_1234555 has quit IRC | 15:17 | |
| *** ygk_1234581 has joined #openvswitch | 15:26 | |
| *** ygk_1234581 has quit IRC | 15:26 | |
| *** ygk_1234584 has joined #openvswitch | 15:27 | |
| *** ygk_1234584 has quit IRC | 15:32 | |
| *** psahoo_ has quit IRC | 15:53 | |
| *** psahoo has joined #openvswitch | 15:57 | |
| *** elvira has quit IRC | 16:16 | |
| *** psahoo has quit IRC | 16:18 | |
| *** fdangelo has quit IRC | 16:25 | |
| *** fdangelo has joined #openvswitch | 16:25 | |
| *** psahoo has joined #openvswitch | 16:32 | |
| *** links has quit IRC | 16:40 | |
| *** psahoo has quit IRC | 16:52 | |
| *** yamamoto has joined #openvswitch | 16:53 | |
| *** deadalnix has quit IRC | 16:54 | |
| *** eelco has joined #openvswitch | 17:00 | |
| *** deadalnix has joined #openvswitch | 17:07 | |
| *** mdgray has quit IRC | 17:08 | |
| *** ralonsoh has quit IRC | 17:12 | |
| *** yamamoto has quit IRC | 17:18 | |
| *** KpuCko has quit IRC | 17:19 | |
| mmichelson | Hi everyone, sorry for being late on the meeting | 17:21 |
|---|---|---|
| mmichelson | Got caught up in other things | 17:21 |
| mmichelson | #startmeeting | 17:21 |
| openstack | mmichelson: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 17:21 |
| _lore_ | hi all | 17:21 |
| mmichelson | #startmeeting ovn_community_development_meeting | 17:21 |
| openstack | Meeting started Thu Apr 15 17:21:22 2021 UTC and is due to finish in 60 minutes. The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:21 |
| imaximets | hi | 17:21 |
| mmichelson | uh, openstack? | 17:21 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:21 |
| openstack | The meeting name has been set to 'ovn_community_development_meeting' | 17:21 |
| mmichelson | ah there we go | 17:21 |
| *** zhouhan has joined #openvswitch | 17:22 | |
| *** zhouhan has quit IRC | 17:25 | |
| *** zhouhan_hzhou8 has joined #openvswitch | 17:25 | |
| mmichelson | OK, so, I finally have a new patch series up for the floating IP issue that I started working on a long time ago | 17:26 |
| mmichelson | It's gone from a 2 patch series to a 5 patch series in an attempt to both fix the issue and to make things more efficient by no longer requiring ARPs to be sent in cases where they shouldn't be necessary | 17:26 |
| mmichelson | We also released new versions of OVN 20.03 and 20.06 for Ubuntu's purposes. | 17:27 |
| numans | Hello | 17:27 |
| mmichelson | I was hoping blp would be here today so I could ask about the bug report I sent for ovn-northd-ddlog, but I guess I'll just need to bump the email I sent. | 17:27 |
| mmichelson | Um...I think that's it for me. | 17:28 |
| numans | I can go real fast | 17:28 |
| numans | I've submitted a couple of patches for review related to conntrack improvement and on usage of ct.inv. | 17:29 |
| *** tbachman has joined #openvswitch | 17:29 | |
| numans | zhouhan_hzhou8, ^ appreciate if you can take a look. | 17:29 |
| numans | I also addressed zhouhan_hzhou8's comments and submitted another version of physical flow split patch. | 17:29 |
| numans | I did some code reviews. | 17:29 |
| numans | That's it from me. | 17:29 |
| imaximets | I have a small update too. | 17:30 |
| imaximets | I finished and posted v2 of stream record/replay functionality with integration to ovsdb-server. | 17:31 |
| imaximets | #link https://patchwork.ozlabs.org/project/openvswitch/list/?series=238830 | 17:31 |
| imaximets | Once this accepted to OVS, we will be able to integrate into ovn daemons, i.e. northd or ovn-controller. for debugging and performance testing purposes. | 17:32 |
| imaximets | Dumitru and zhouhan_hzhou8 reviewed bundles support for ofctrl. So, I guess, the patch is good to go now. :) | 17:33 |
| imaximets | that's it from my side. | 17:33 |
| _lore_ | can I go next? quite fast | 17:33 |
| _lore_ | this week I worked on skip_force_snat patch, thx mark for the review | 17:34 |
| _lore_ | then I started working on 2 items and I would like to have your opinion: | 17:34 |
| _lore_ | 1- I noticed whenever we wake ovn-controller main thread from pinctrl we run all the handlers even if just one will do some goodput | 17:35 |
| _lore_ | does it worth to just run the related handler in pinctrl_run()? | 17:36 |
| _lore_ | any opinions? | 17:37 |
| mmichelson | _lore_, so you want to add incremental processing for pinctrl, essentially? | 17:38 |
| _lore_ | mmichelson: nope | 17:39 |
| imaximets | _lore_, I'm not sure about the idea (I just do not know that code), but I'm also not sure how thread-safe ovn-controller is. Is it? | 17:39 |
| _lore_ | there is a mutex between ovn-controller main thread and pinctrl_thread | 17:39 |
| _lore_ | imaximets: ..^ | 17:39 |
| imaximets | _lore_, ack. | 17:40 |
| _lore_ | mmichelson: what I mean is having a mask to run just the handlers in pinctl_run() that has been set in pinctrl_thread() | 17:40 |
| _lore_ | it is just an idea, I need to review the code to see if it is feasible | 17:41 |
| *** zhouhan has joined #openvswitch | 17:41 | |
| mmichelson | _lore_, If it's causing a noticeable problem, then sure I'd say to go ahead and pursue that as a possible option. I'm not sure how much of a problem this actually poses right now though. | 17:42 |
| _lore_ | mmichelson: I do not have any report, I just noticed lookig at the code | 17:42 |
| numans | I agree with mmichelson. | 17:43 |
| * zhouhan finally got passwd back on a new computer | 17:43 | |
| _lore_ | numans: mmichelson: ok, I will see if it easy to implement | 17:43 |
| numans | zhouhan, cool | 17:43 |
| _lore_ | 2- we have an issue related to garp that can make ovn-northd clocks at 100% cpu | 17:44 |
| zhouhan | _lore_ I don't remember exactly but shouldn't pinctrl run just one handler based on the message type? | 17:44 |
| _lore_ | zhouhan: what I mean is not pinctrl_thread | 17:44 |
| _lore_ | I mean if we can optmize main thread | 17:45 |
| _lore_ | but it is just an idea | 17:45 |
| _lore_ | I am not sure | 17:45 |
| _lore_ | related to 2, I was wondering if we can implement something specific to arp reply to limit the rate of wakes or we can just use a meter on the action | 17:45 |
| _lore_ | what do you think? | 17:46 |
| zhouhan | _lore_ oh, sorry, just re-read your message and I got your point now. Yes, you are right, but the most costly part is in I-P and since there is no input change, the I-P engine will not do any compute. | 17:46 |
| _lore_ | zhouhan: maybe there is no difference, I spotted this just looking at the code | 17:46 |
| zhouhan | _lore_ If there are handlers too costly to be run on each main thread wakeup, then we should put it in I-P eng | 17:46 |
| _lore_ | ack | 17:46 |
| _lore_ | any opinion about point 2? | 17:47 |
| mmichelson | _lore_, can you be more specific about why the garps are causing ovn-northd to run at 100% cpu? | 17:48 |
| mmichelson | is it because mac_bindings are being updated too often? | 17:48 |
| _lore_ | according to the bz keepalived keeps moving the vip from one place to another and send garps to it | 17:49 |
| _lore_ | yes | 17:49 |
| _lore_ | mac_bindings is updated very opten by the garp | 17:49 |
| _lore_ | this will end-up in a sb update | 17:49 |
| zhouhan | but mac_bindings are not handled by northd, right? | 17:49 |
| zhouhan | ok, so northd just got wake up, and there is no I-P there in the C version :) | 17:50 |
| _lore_ | yes | 17:50 |
| _lore_ | correct | 17:50 |
| zhouhan | Oh, wait, northd shouldn't even monitor mac_binding | 17:50 |
| mmichelson | zhouhan, northd monitors mac_bindings and deletes them if the logical port has been deleted. | 17:51 |
| mmichelson | zhouhan, see cleanup_mac_bindings() | 17:51 |
| zhouhan | oh, ok, I recall this been added sometime ago | 17:51 |
| _lore_ | this is the related bz: https://bugzilla.redhat.com/show_bug.cgi?id=1947913 | 17:51 |
| openstack | bugzilla.redhat.com bug 1947913 in ovn2.13 "[OVN][RFE] Add protection mechanism against gARPs / flapping ports" [High,New] - Assigned to lorenzo.bianconi | 17:51 |
| mmichelson | What's happening here is that the MAC_Bindings are being updated, but since no ports are being changed, it means that northd is doing useless work. | 17:51 |
| _lore_ | yes | 17:52 |
| zhouhan | Does the ddlog northd help? | 17:52 |
| mmichelson | So a possible workaround that dceara discussed at one point was making ovn-northd stop prematurely if the only change was a MAC_Binding. But that's kind of a poor man's I-P | 17:52 |
| mmichelson | zhouhan, presumably, it would. But that's not being used in production | 17:53 |
| _lore_ | the bz even say ovs-db is quite loaded so maybe better to filter them in ovn-controller? | 17:53 |
| mmichelson | But the flip side to this is that the garps themselves should probably be metered. And I think that's the part you're bringing up here _lore_ | 17:53 |
| _lore_ | mmichelson: correct | 17:53 |
| _lore_ | my question is: | 17:53 |
| _lore_ | is it better a meter for the action (need to check the code) or to implement something in c for mac_binding? | 17:54 |
| zhouhan | I remember dumitru started some work 1 - 2 years ago for control plane ratelimiting in general | 17:57 |
| mmichelson | _lore_, I think probably both are good ideas, personally. | 17:57 |
| _lore_ | ok, I will come up with something | 17:58 |
| zhouhan | but can't remember how did it go. In general, there was a dilemma for ratelimiting. Either it could block real request or it could require huge amount of meters causing performance problem. | 17:59 |
| imaximets | mmichelson, _lore_: about rate limiting, don't we need to just ban certain garps that causes problems instead of limiting all of them? We will loose some valid binding events due to limiting and that may be bad. | 17:59 |
| _lore_ | imaximets: yes, this is what I would like to do in c | 18:00 |
| _lore_ | in pinctrl code | 18:00 |
| _lore_ | zhouhan: maybe it is better to just ratelimit a given IP | 18:00 |
| _lore_ | not all the IPs | 18:00 |
| zhouhan | _lore_ hmm, but how do you know which IP to meter? Through config? | 18:01 |
| _lore_ | in the GARP we have the src mac and IP | 18:01 |
| _lore_ | right? | 18:01 |
| _lore_ | or in the arp reply in general | 18:01 |
| _lore_ | I will try to come up with a PoC | 18:02 |
| mmichelson | But how do you know how to program which IPs to meter on? Don't you have to know that before the garps arrive? | 18:02 |
| *** eelco has quit IRC | 18:02 | |
| _lore_ | and then we can continue the discussion on the ml | 18:02 |
| zhouhan | I mean, we can't predict what IPs could appear in GARPs, i.e. which GARPs to apply rateliimiting | 18:02 |
| zhouhan | (same as what mmichelson said :) | 18:04 |
| _lore_ | I mean when we receive the first request we can have a map for them | 18:05 |
| _lore_ | and so keep track of the incoming requests | 18:05 |
| _lore_ | it is just an idea | 18:05 |
| mmichelson | _lore_, if you can put together a PoC, then I agree that we can continue the discussion on the mailing list | 18:06 |
| _lore_ | ack thx | 18:06 |
| _lore_ | :) | 18:06 |
| zhouhan | _lore_: so when there are lots of new GARP comes with different IPs, that map can increase which would require huge amount of meters, right? | 18:07 |
| zhouhan | yep, maybe a POC is great. Some tradeoffs could be made | 18:07 |
| _lore_ | I will not use meters | 18:07 |
| _lore_ | I mean, I will implement the logic of a meter, but in pinctrl thread | 18:08 |
| _lore_ | and the map will have a max size | 18:08 |
| _lore_ | we the map is full we will discard the new packets since we are under attack | 18:08 |
| _lore_ | right? | 18:08 |
| zhouhan | _lore_: ok, but discarding new packets basically is blocking "healthy" ones (as a result of DDoS) | 18:09 |
| _lore_ | zhouhan: yes, but if the maps is full it means somthing not right is happening, so it is better just to discard packets and not run at 100% cpu right | 18:10 |
| _lore_ | ? | 18:10 |
| _lore_ | it is a tradeoff | 18:10 |
| *** dholler has quit IRC | 18:10 | |
| _lore_ | let me review the code better and we can continue discussing about it | 18:11 |
| zhouhan | yes, I believe some tradeoff has to be made, to solve the dilemma | 18:11 |
| zhouhan | sorry I have to run for another meeting. ttyl | 18:11 |
| mmichelson | _lore_, but if the map gets full, and we stop processing new garps, then wouldn't that still result in a DoS since we're not processing garps any longer? | 18:11 |
| _lore_ | zhouhan: this is a perfect world :D | 18:11 |
| mmichelson | I'll wait for a PoC since it's hard to know exactly how this will work. | 18:12 |
| _lore_ | mmichelson: it is better to not processing just arp but the system continue working, right? | 18:12 |
| *** zhouhan_hzhou8 has quit IRC | 18:12 | |
| *** zhouhan has quit IRC | 18:12 | |
| mmichelson | _lore_, Ok I see what you're saying | 18:12 |
| _lore_ | I will keep you updated :) | 18:13 |
| _lore_ | that's all from me | 18:13 |
| _lore_ | thx | 18:13 |
| mmichelson | Anybody else? | 18:14 |
| mmichelson | OK I guess that's it for today | 18:15 |
| mmichelson | #endmeeting | 18:15 |
| openstack | Meeting ended Thu Apr 15 18:15:23 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 18:15 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-04-15-17.21.html | 18:15 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-04-15-17.21.txt | 18:15 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-04-15-17.21.log.html | 18:15 |
| _lore_ | bye all | 18:15 |
| *** KpuCko has joined #openvswitch | 18:45 | |
| *** donhw has quit IRC | 18:57 | |
| *** donhw has joined #openvswitch | 18:58 | |
| *** dcbw has joined #openvswitch | 18:59 | |
| *** istokes has quit IRC | 19:56 | |
| *** dcbw has quit IRC | 21:36 | |
| *** fdangelo has quit IRC | 21:39 | |
| *** bostondriver has quit IRC | 21:43 | |
| *** yamamoto has joined #openvswitch | 22:31 | |
| *** yamamoto has quit IRC | 22:35 | |
| *** yamamoto has joined #openvswitch | 22:36 | |
| *** rcernin has joined #openvswitch | 22:51 | |
| bdonnahue2 | hey folks can anyone help me setup a vxlan tunnel on a proxmox host? | 23:16 |
| bdonnahue2 | cant seem to get the configs right | 23:16 |
| *** KpuCko has quit IRC | 23:23 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!