| *** acidfoo has quit IRC | 00:25 | |
| *** zhouhan_ has joined #openvswitch | 00:31 | |
| *** zhouhan has quit IRC | 00:34 | |
| *** acidfoo has joined #openvswitch | 00:36 | |
| *** acidfoo has quit IRC | 00:42 | |
| *** acidfoo has joined #openvswitch | 01:07 | |
| *** armax has quit IRC | 01:41 | |
| *** armax has joined #openvswitch | 02:03 | |
| *** Madkiss_ has quit IRC | 02:28 | |
| *** Madkiss has joined #openvswitch | 02:28 | |
| *** Madkiss has joined #openvswitch | 02:28 | |
| *** rcernin has quit IRC | 02:29 | |
| *** links has joined #openvswitch | 02:29 | |
| *** armax has quit IRC | 02:35 | |
| *** rcernin has joined #openvswitch | 02:52 | |
| *** rcernin has quit IRC | 02:56 | |
| *** rcernin has joined #openvswitch | 02:56 | |
| *** armax has joined #openvswitch | 03:03 | |
| *** acidfoo has quit IRC | 03:06 | |
| *** acidfoo has joined #openvswitch | 03:17 | |
| *** armax has quit IRC | 03:19 | |
| *** acidfoo has quit IRC | 03:21 | |
| *** psahoo has joined #openvswitch | 03:57 | |
| *** anilvenkata_ has quit IRC | 04:23 | |
| *** psahoo has quit IRC | 04:42 | |
| *** psahoo has joined #openvswitch | 04:55 | |
| *** psahoo has quit IRC | 05:18 | |
| *** anilvenkata_ has joined #openvswitch | 05:26 | |
| *** psahoo has joined #openvswitch | 05:35 | |
| *** eelco has joined #openvswitch | 05:52 | |
| *** ralonsoh has joined #openvswitch | 06:14 | |
| *** dholler has joined #openvswitch | 06:26 | |
| *** slaweq has joined #openvswitch | 06:32 | |
| *** psahoo has quit IRC | 06:43 | |
| *** psahoo has joined #openvswitch | 06:49 | |
| *** yamamoto has quit IRC | 07:12 | |
| *** yamamoto has joined #openvswitch | 07:13 | |
| *** __lore__ is now known as _lore_ | 07:22 | |
| *** rcernin has quit IRC | 07:28 | |
| *** dholler has quit IRC | 07:37 | |
| *** dholler has joined #openvswitch | 07:50 | |
| *** yamamoto has quit IRC | 08:15 | |
| *** yamamoto has joined #openvswitch | 08:25 | |
| *** anilvenkata_ has quit IRC | 08:46 | |
| *** anilvenkata_ has joined #openvswitch | 08:47 | |
| *** anilvenkata_ has quit IRC | 09:19 | |
| *** mishiranu has quit IRC | 09:23 | |
| *** yamamoto has quit IRC | 09:24 | |
| *** mishiranu has joined #openvswitch | 09:29 | |
| *** yamamoto has joined #openvswitch | 09:53 | |
| *** yamamoto has quit IRC | 10:05 | |
| *** anilvenkata_ has joined #openvswitch | 10:14 | |
| *** dswebb has joined #openvswitch | 10:21 | |
| *** yamamoto has joined #openvswitch | 10:36 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** yamamoto has joined #openvswitch | 10:40 | |
| *** jaicaa has quit IRC | 10:41 | |
| *** jaicaa has joined #openvswitch | 10:44 | |
| *** yamamoto has quit IRC | 11:01 | |
| *** yamamoto has joined #openvswitch | 11:12 | |
| *** yamamoto has joined #openvswitch | 11:35 | |
| *** thaller_ is now known as thaller | 12:21 | |
| *** acidfoo has joined #openvswitch | 13:14 | |
| *** yamamoto has quit IRC | 13:23 | |
| *** yangyapeng has quit IRC | 14:14 | |
| *** yangyapeng has joined #openvswitch | 14:28 | |
| *** anilvenkata_ has quit IRC | 14:36 | |
| *** dcbw has joined #openvswitch | 14:40 | |
| *** armax has joined #openvswitch | 14:45 | |
| *** links has quit IRC | 14:53 | |
| *** yangyapeng has quit IRC | 15:05 | |
| *** yangyapeng has joined #openvswitch | 15:09 | |
| *** eelco has quit IRC | 15:15 | |
| *** yamamoto has joined #openvswitch | 15:24 | |
| *** yamamoto has quit IRC | 15:29 | |
| *** dholler has quit IRC | 15:36 | |
| *** psahoo has quit IRC | 16:02 | |
| *** JamesBenson has joined #openvswitch | 16:06 | |
| *** fbl_ has joined #openvswitch | 16:07 | |
| *** fbl has quit IRC | 16:08 | |
| *** fbl_ has quit IRC | 16:15 | |
| *** fbl_ has joined #openvswitch | 16:15 | |
| *** fbl_ has quit IRC | 16:15 | |
| *** fbl has joined #openvswitch | 16:16 | |
| *** zhouhan_ has quit IRC | 16:30 | |
| *** zhouhan has joined #openvswitch | 16:31 | |
| *** dceara has joined #openvswitch | 16:38 | |
| *** yamamoto has joined #openvswitch | 16:48 | |
| *** yamamoto has quit IRC | 16:53 | |
| *** ralonsoh has quit IRC | 17:12 | |
| imaximets | Hi. | 17:17 |
|---|---|---|
| zhouhan | Helloe | 17:17 |
| dceara | Hi | 17:17 |
| imaximets | mmichelson and, probably, numans are not here today. | 17:18 |
| _lore_ | hi all | 17:18 |
| imaximets | So, I'll host this meeting. | 17:18 |
| imaximets | #startmeeting ovn_community_development_discussion | 17:18 |
| openstack | Meeting started Thu Aug 27 17:18:55 2020 UTC and is due to finish in 60 minutes. The chair is imaximets. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:18 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:18 |
| openstack | The meeting name has been set to 'ovn_community_development_discussion' | 17:18 |
| imaximets | I do not have much to say. Whou wants to go first? | 17:19 |
| dceara | I can start | 17:19 |
| imaximets | dceara, ok. | 17:20 |
| dceara | thanks zhouhan for reviewing the conntrack bypass patch, I'll probably not have time for a v3 until next week though. | 17:20 |
| dceara | except for that I wanted to schedule a run on our scale setup with zhouhan | 17:21 |
| dceara | 's I-P patches | 17:21 |
| dceara | hopefully I can manage to do that tomorrow. | 17:21 |
| zhouhan | dceara: thanks dceara | 17:21 |
| dceara | And I sent a couple of bug fix patches today. One of which should probably be backported all the way until 20.03. | 17:21 |
| dceara | That's it on my side, thanks! | 17:22 |
| imaximets | dceara, thanks! | 17:22 |
| zhouhan | I can go next | 17:23 |
| imaximets | zhouhan, sure. | 17:23 |
| zhouhan | I sent the series of incremental processing for flow installation: #link https://patchwork.ozlabs.org/project/openvswitch/list/?series=197009 | 17:23 |
| zhouhan | The CPU cost reduced around 40% for a scale of 1200 HV with 12K ports | 17:24 |
| zhouhan | It also solves a bug when conjunction combination is used. | 17:24 |
| zhouhan | (that may need back port as well) | 17:25 |
| zhouhan | I also did more scale test for 3k HV with 30k ports. | 17:25 |
| dceara | zhouhan: regarding the conjunction bug (I didn't look at the patches yet), but would it be possible to move it earlier in the series to make backporting easier? | 17:26 |
| zhouhan | It ran successfully. However, current ovn-nbctl --wait=hv mechanism is not accurate for measuring the end to end latency, because the updates of nbcfg from all HVs actually contributes the most cost. | 17:27 |
| zhouhan | dceara: I think the earlier patches are required by the bug fix. (The bug fix is actually a big part of the series) | 17:28 |
| dceara | zhouhan: ack, thanks, i'll try to have a closer look too. | 17:28 |
| zhouhan | To measure the latency more accurately, I think I need to improve the nb_cfg mechanism, to include a timestamp field. I will work on it. | 17:29 |
| zhouhan | But overall, by manually checking the latency, it seems a port binding can finish within 4 - 5 sec at that scale. | 17:30 |
| zhouhan | In addition, I did some code reviews. imaximets: could you take a look at this one as well? | 17:31 |
| zhouhan | https://patchwork.ozlabs.org/project/openvswitch/patch/20200813205259.5036-1-zhewang@nvidia.com/ | 17:31 |
| zhouhan | That's it from me | 17:31 |
| dceara | zhouhan: in our tests we wait until the port can ping its gateway (or an external host) and we see it taking >10sec in some cases. I didn't try with your patches yet though. | 17:32 |
| imaximets | zhouhan, yeah, I looked a this patch and I'm thinking if it's possible to fix the issues from the inside of idl/jsonrpc, without requirement for CMS to call special functions. | 17:33 |
| zhouhan | dceara: do you ping from all the VMs? I guess that action itself may take a lot of overhead. | 17:33 |
| dceara | zhouhan: only from the new fake vm (netns) until it is successful | 17:34 |
| zhouhan | imaximets: that would be better, if it can be supported. | 17:34 |
| zhouhan | dceara: but there will be 30k of them? And we need to make sure the slowest one can ping ... | 17:35 |
| zhouhan | dceara: or do you just ping from a random VM and assume most of the VMs got the flow installed at similar latency? | 17:36 |
| dceara | zhouhan: in our tests we don't advance to create the next port until the current one can ping its own gateway. | 17:36 |
| dceara | zhouhan: We also don't batch port add operations. This in order to try to see the worst case scenario latency for flow installation. | 17:37 |
| zhouhan | dceara: Oh, I think that's a different scenario. I am testing when the whole scale is built up, then create and bind a new port, and see how long it takes for this new change to get processed in all the HVs (meaning the new port can reach all other ports) | 17:37 |
| dceara | zhouhan: I see, ok, I can set try to set our scenario in a similar way too, thanks. | 17:38 |
| dceara | s/can set try/can try/ | 17:38 |
| imaximets | zhouhan, I do not know yet, how to make re-balancing of connections work from the inside of idl, I will likely reply to ML with some ideas a bit later, if any. | 17:39 |
| zhouhan | dceara: to make sure *all* HVs has processed the new change, I am utilizing the --wait=hv feature. Now I realized that this mechanism itself was a bottleneck (even after solving the flooding problem). | 17:39 |
| dceara | zhouhan: ack, we decided to go for the ping approach exactly to avoid --wait=hv | 17:40 |
| zhouhan | dceara: So I am thinking about posting a timestamp from each ovn-controller while reporting the nb_cfg number it processed, so the nbctl can finally rely on the timestamp to calculate the time spent for the slowest HV | 17:41 |
| zhouhan | imaximets: ok, thanks! But do you think that could be a follow-up improvement, independent of the command provided by that patch? | 17:42 |
| zhouhan | (of course, with that improvement, the current patch provided won't be as useful any more) | 17:42 |
| imaximets | zhouhan, in general, I'd like to avoid introduction of new commands if possible, especially if we can fix the issue in general. | 17:43 |
| imaximets | zhouhan, how these commands supposed to be used? Will CMS just re-disribute all the clients by itself, or will nominate only part of them for re-connection? | 17:44 |
| zhouhan | imaximets: agree in general. But I feel this command does provide some value for operational need. | 17:45 |
| *** Ankur1 has joined #openvswitch | 17:45 | |
| Ankur1 | Hi | 17:45 |
| zhouhan | imaximets: I think the typical case is when failover happend and the node recovered, the newly recovered node has no connections. So operator can use the command to instruct some of the HVs to connect back to the recovered node. | 17:46 |
| *** Ankur171 has joined #openvswitch | 17:46 | |
| Ankur1 | hi | 17:47 |
| Ankur171 | ab | 17:47 |
| *** Ankur1 has left #openvswitch | 17:47 | |
| *** Ankur171 has quit IRC | 17:47 | |
| zhouhan | imaximets: but it may also be useful if someone wants to adjust (fine tune) the load of different servers but moving clients from one server to another. | 17:47 |
| imaximets | zhouhan, I see. Let me think a little bit. I will reply on ML, or just apply the patch if there will be no clever ideas from my side. :) | 17:48 |
| zhouhan | s/but moving/by moving | 17:48 |
| zhouhan | thanks imaximets | 17:48 |
| dceara | zhouhan: I think there were some similar discussions on the ML at some point about adding a timestamp to the sync mechanism. We can probably continue the chassis.nb_cfg discussion there. | 17:49 |
| zhouhan | dceara: sure | 17:49 |
| zhouhan | dceara: maybe I will try a POC first | 17:50 |
| dceara | zhouhan: cool | 17:50 |
| imaximets | OK. Anyone else wants to share some updates? | 17:53 |
| imaximets | So, I think, we could call it now. | 17:55 |
| imaximets | #endmeeting | 17:56 |
| openstack | Meeting ended Thu Aug 27 17:56:31 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:56 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-08-27-17.18.html | 17:56 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-08-27-17.18.txt | 17:56 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-08-27-17.18.log.html | 17:56 |
| zhouhan | bye | 17:56 |
| imaximets | Bye. | 17:57 |
| _lore_ | bye | 17:58 |
| dceara | Bye | 17:59 |
| *** acidfoo has quit IRC | 18:03 | |
| *** dceara has quit IRC | 18:20 | |
| *** thaller has quit IRC | 18:50 | |
| *** thaller has joined #openvswitch | 18:50 | |
| *** fbl has quit IRC | 19:19 | |
| *** fbl has joined #openvswitch | 19:23 | |
| *** acidfoo has joined #openvswitch | 20:11 | |
| *** fbl has quit IRC | 20:56 | |
| *** slaweq has quit IRC | 21:22 | |
| *** yamamoto has joined #openvswitch | 21:30 | |
| *** JamesBenson has quit IRC | 21:42 | |
| *** imaximets has quit IRC | 22:01 | |
| *** imaximets has joined #openvswitch | 22:01 | |
| *** yamamoto has quit IRC | 22:12 | |
| *** rcernin has joined #openvswitch | 22:45 | |
| *** armax has quit IRC | 22:52 | |
| *** yangyapeng has quit IRC | 23:00 | |
| *** yangyapeng has joined #openvswitch | 23:04 | |
| *** armax has joined #openvswitch | 23:08 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!