Thursday, 2020-04-23

*** mTeK has quit IRC00:24
*** mTeK has joined #openvswitch00:24
*** inflatador has quit IRC00:40
*** timthowtdi has quit IRC01:53
*** timthowtdi has joined #openvswitch01:55
*** acidfoo has joined #openvswitch02:15
*** acidfu has quit IRC02:16
*** acidfoo has quit IRC02:29
*** dholler has quit IRC02:30
*** mishiran- has joined #openvswitch02:41
*** mishiranu has quit IRC02:42
*** dholler has joined #openvswitch02:43
*** thaller_ has joined #openvswitch03:07
*** thaller has quit IRC03:10
*** psahoo has joined #openvswitch03:13
*** troulouliou_div2 has joined #openvswitch03:29
*** yamamoto has quit IRC03:36
*** yamamoto has joined #openvswitch03:40
*** imaximets has quit IRC03:51
*** yamamoto has quit IRC03:53
*** gmg has quit IRC03:55
*** imaximets has joined #openvswitch04:00
*** gmg1 has joined #openvswitch04:13
*** gmg1 has quit IRC04:17
*** yamamoto has joined #openvswitch04:26
*** thaller__ has joined #openvswitch04:32
*** yamamoto has quit IRC04:33
*** thaller_ has quit IRC04:34
*** yamamoto has joined #openvswitch04:38
*** yamamoto has quit IRC04:47
*** yamamoto has joined #openvswitch04:50
*** anilvenkata has joined #openvswitch05:40
*** slaweq has joined #openvswitch05:48
*** yamamoto has quit IRC06:06
*** yamamoto has joined #openvswitch06:07
*** jaicaa has quit IRC06:29
*** jaicaa has joined #openvswitch06:32
*** mmirecki has joined #openvswitch06:45
*** mmirecki has quit IRC06:52
*** mmirecki has joined #openvswitch06:56
*** eelco has joined #openvswitch07:09
*** mbarroso has joined #openvswitch07:11
*** dceara has joined #openvswitch07:59
*** links has joined #openvswitch08:11
*** igordc has quit IRC08:13
*** yamamoto has quit IRC08:18
*** yamamoto has joined #openvswitch08:19
*** thaller__ is now known as thaller08:44
*** yamamoto has quit IRC08:48
*** yamamoto has joined #openvswitch08:49
*** yamamoto has joined #openvswitch08:50
*** mishiran- has quit IRC09:04
*** mishiranu has joined #openvswitch09:06
*** zhouhan has quit IRC09:09
*** zhouhan has joined #openvswitch09:09
*** mishiranu has quit IRC09:31
*** timothy has joined #openvswitch09:31
*** mishiranu has joined #openvswitch09:32
*** numans has quit IRC09:42
*** numans has joined #openvswitch09:43
*** yamamoto has quit IRC09:55
*** rcernin has quit IRC10:03
*** FH_thecat has quit IRC10:12
*** yamamoto has joined #openvswitch10:42
*** links has quit IRC10:52
*** links has joined #openvswitch11:02
*** ndim has quit IRC11:02
*** troulouliou_div2 has quit IRC12:03
*** jraju__ has joined #openvswitch12:41
*** links has quit IRC12:41
*** bostondriver has joined #openvswitch12:43
*** inflatador has joined #openvswitch12:44
*** acidfoo has joined #openvswitch12:54
*** dholler has quit IRC12:58
*** ndimitrij has joined #openvswitch13:03
*** igordc has joined #openvswitch13:07
*** yamamoto has quit IRC13:09
*** yamamoto has joined #openvswitch13:09
*** dholler has joined #openvswitch13:10
*** ndimitrij is now known as ndim13:43
*** inflatador_ has joined #openvswitch13:52
*** inflatador has quit IRC13:55
*** inflatador_ is now known as inflatador13:55
*** eelco has quit IRC14:28
*** eelco has joined #openvswitch14:32
*** eelco has quit IRC14:48
*** eelco has joined #openvswitch14:48
*** eelco has quit IRC14:49
*** eelco has joined #openvswitch14:49
*** eelco has joined #openvswitch14:50
*** gmg has joined #openvswitch14:54
*** acidfoo has quit IRC15:16
*** acidfoo has joined #openvswitch15:18
*** yamamoto has quit IRC15:18
*** yamamoto has joined #openvswitch15:19
*** yamamoto has quit IRC15:21
*** yamamoto has joined #openvswitch15:21
*** eelco has quit IRC15:30
*** dholler has quit IRC15:33
*** gmg2 has joined #openvswitch16:06
*** gmg has quit IRC16:08
*** mmirecki has quit IRC16:09
*** gmg2 has quit IRC16:11
*** atpa8a has quit IRC16:22
*** acidfoo has quit IRC16:26
*** mbarroso has quit IRC16:27
*** acidfoo has joined #openvswitch16:28
*** gmg has joined #openvswitch16:30
*** donhw_ has joined #openvswitch17:06
*** donhw has quit IRC17:08
*** aginwala has joined #openvswitch17:14
*** jraju__ has quit IRC17:14
* flaviof hears Yeah Ant soundtrack in the making.... http://dig.ccmixter.org/files/speck/4210017:15
numansHello17:20
flaviofhi there!17:20
numansflaviof, howz the track ?17:20
flavioflol. rocking!17:20
numansIts time for OVN meeting17:20
numans#startmeeting ovn-community-development-discussion17:21
openstackMeeting started Thu Apr 23 17:21:03 2020 UTC and is due to finish in 60 minutes.  The chair is numans. Information about MeetBot at http://wiki.debian.org/MeetBot.17:21
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.17:21
openstackThe meeting name has been set to 'ovn_community_development_discussion'17:21
numansHello17:21
numansLets start the meeting.17:21
zhouhanhi17:21
numanszhouhan, Hi17:21
numansmmichelson is not joining today.17:21
numansWho wants to start17:21
numansI can go real quick.17:22
dcearaHi17:22
numansLast week I submitted the v3 of I-P patches17:22
numansdceara, Hi17:22
numansThen I started working on the OVN load balancer hashing.17:23
numansI submitted a patch to enable openflow 1.5, since we cannot set the selection_method unless we use oflow 1.517:23
numansI'm still working on providing the option for the CMS to chose the hash fields - like ip_src, ip_dst, tp_src etc17:24
numanswhich will be used for the hashing.17:24
numansI'm working on another issue. Recently I submitted a patch to address an issue in tcp_reset action17:24
numansAnd to support this, I added llflows to by pass tcp rst pkts from conntrack.17:24
numansAnd this is causing the conntrack entries to be established state even after the client/server closes the connection by sending tcp rst pkt17:25
numansI'm working on it.17:25
numansThat17:25
numansThat's it for me.17:25
numanszhouhan, dceara It will be great if you can take a look at the I-P patches whenever you can.17:25
flaviofNice, numans! 2 follow up questions, if I may.17:25
numansflaviof, sure.17:26
dcearanumans, ack17:26
_lore_hi all17:26
zhouhannumans: thanks a lot. I will review them17:26
numanszhouhan, thanks.17:26
flaviof1) does the hashing for lp require a specific verion of the kernel? or anything that can do ovs1.5 is enough?17:26
numansflaviof, No. it doesn't depend the kernel version.17:26
numansOVS supports 2 hashing methods - dp_hash and hash.17:26
numansdp_hash is calculated by the datapath.17:27
numansfor the latter, ovs calculates the hash.17:27
flaviofnumans: ah. ack. I heard from Maiciej about it today. Interesting find!17:27
flaviof2) can you tell me if the conntrack issue is a regression or has it always been there?17:27
zhouhannumans: dp_hash should still ensure that same 5-tuple is hashed to same value, right?17:27
numanszhouhan, that's not the case in my testing.17:28
numanszhouhan,for a given 5tuple , I see that ovs is not choosing the same bucket all the time17:28
numansflaviof, I don't think its a regression.17:29
zhouhannumans: that's strange. For what I observed in the past it seems always the same bucket. Maybe I can do more test and confirm17:29
numanszhouhan, that would be great if you could confirm17:29
flaviofzhouhan: maybe you are using OF1.5 ?17:29
zhouhannumans: usually I use ping (ICMP) to test. Does that impact the result?17:30
numansflaviof, starting from ovs 2.10, the defualt selection method changed from hash to dp_hash in ovs.17:30
numanszhouhan, I used ncat17:30
numansand specified the source port.17:30
numansflaviof, ovn was always selecting dp_hash, but since we are not using oflow1.5, that never reached ovs and it uses the default if no selection_method is set17:31
flaviofack, understood. ty17:31
zhouhannumans: do you know by any chance what's the dp_hash algorithm? In the code comment it mentioned Webster method, but I never heard of it17:31
numanszhouhan, I'd suggest to use ncat with src port specified in your testing17:31
numanszhouhan, it is calculated by kernel17:31
numanszhouhan, and from what I understand it uses skb_get_hash() for that17:32
* numans merely knows datapath and I can be wrong.17:32
*** atpa8a has joined #openvswitch17:32
zhouhanok, thanks!17:32
imaximetsnumans, zhouhan: IIRC, dp_hash is likely an RSS hash if available.17:32
numanszhouhan, from the code I saw, it uses webster to allocate the hashes to buckets.17:33
numansimaximets, ok17:33
imaximetsthere might be difference while making upcall, because vswitchd will calculate 5tuple hash by itself if RSS hash is not present.17:33
zhouhannumans: any links to "webster" would be helpful. Google just give me the dictionary :(17:34
numanszhouhan, :). Even I'm not aware of it. I just saw the comments. But sure, I'll share if I come across17:34
zhouhanimaximets: RSS hash should still make sure same flow hashed to same bucket, right? So I still don't understand how could same flow end up in different bucket17:35
imaximetszhouhan, yes, RSS should ensure.17:35
numansimaximets, zhouhan, I'm pretty certain on that. I did some prints, and the value of dp_hash was always different17:36
imaximetsthe issue might be if you're calculating dp_hash in userspace for the first packet and using RSS in datapath for subsequent ones.  That is the only case I can think of.17:36
numanshttps://github.com/openvswitch/ovs/blob/master/ofproto/ofproto-dpif-xlate.c#L460417:37
numansI put a print here and the dp_hash was different.17:37
numansok.17:37
imaximetsnumans, if it's always different, this might be the bug in kernel. Maybe you're reading incorrect memory, or your hashing algorithm uses more fields than 5tuple.17:38
numansimaximets, ovn doesn't set the selection_method so the default is used17:38
numansimaximets, actually ovn sets, but it nevers gets encoded when ovn sends the group_mod message17:38
numansI think we can continue further in the ML.17:39
imaximetsnumans, ok.17:39
numanszhouhan, I've replied to maciej's email. If you can take a look at that and reply to that after the testing that would be great.17:39
numansimaximets, zhouhan thanks for the discussion.17:39
numansI'm done. If someone wants to go next.17:39
zhouhannumans: sure, will do17:40
_lore_can I go next?17:40
numans_lore_, sure.17:40
flaviof#link   https://github.com/openvswitch/ovs/commit/2e3fd24c7c440f87d7a24fbfce1474237de7e1cf pick_dp_hash_select_group  ref on hash17:40
_lore_this week I worked on a issue related to QoS ovn metering17:40
_lore_in particular if you create 2 meters with same values for rate and burst on the same hv they will be mapped to the same kernel meter and they will share the bandwidth17:41
_lore_I am wondering if it is an intended behaviour or not17:41
_lore_any idea?17:41
_lore_maybe not :)17:42
flaviofsorry _lore_ i don't know17:43
numans_lore_, not sure on that.17:43
_lore_actually I posted a RFC patch to make the meters unique, I tested it and it works fine17:43
numans_lore_, but I think we can fix that if that is causing incorrect meter allocation17:43
_lore_it has been tested even by osp folks and they are fine with it17:43
_lore_so I would send it as normal patch17:44
numans_lore_, sounds good to me.17:44
_lore_and in case we have some regression takes care of it17:44
_lore_ack17:44
_lore_moreover I posted some ipv6 pd trivial fixes17:44
_lore_zhouhan: I read your reply about my ovn-scale-test PR but I did not get it exactly17:45
_lore_what do you mean?17:45
zhouhan_lore_: Oh, I mean, the port-groups and ACLs are better to be configured, instead of hard-coded in the implementation17:46
_lore_ah ok17:46
zhouhan_lore_: for ovn-scale-test, it should be able to test different port-groups and ACLs settings, without update the code everytime.17:46
_lore_so you mean to generalize it adding the possibility to read the configuration from json file17:47
_lore_instead of hard code ACL and so on17:47
_lore_right?17:47
zhouhan_lore_: I also posted an issue in ovn-k8s to discuss the reason why multiple default group is used instead of one. I think in ovn-scale-test we can test the scalability difference.17:48
zhouhanyes17:48
zhouhanthat's right17:48
zhouhanThen we can change the json file and test different scenarios and compare the results17:48
_lore_ack, I will look how to generalize it17:49
_lore_but the concept is ok for you, right?17:49
zhouhan_lore_: sorry, what concept?17:49
_lore_I mean to implement OpenShift network policy17:50
*** slaweq has quit IRC17:50
zhouhan_lore_: yes, of course17:51
_lore_ack fine17:51
_lore_that's all from my side17:51
numansOk. Thanks.17:51
numansWho wants to go next17:51
zhouhanI can go quickly17:52
numanssure17:52
zhouhanWe observed same problem dceara is fixing - the ovsdb missing updates, and spent lot of time debugging, until I recall what dceara has reported. So, thanks!17:53
zhouhanI am reviewing dceara's v3 patch17:53
dcearazhouhan, no worries :) Does the patch fix the issue for you?17:53
dcearazhouhan, there's a v4 (addressing Ilya's comments):  https://patchwork.ozlabs.org/project/openvswitch/list/?series=17210917:54
zhouhandceara: it is in production, so no chance to apply the patch. We just worked around by restarting all impacted ovn-controllers17:54
dcearazhouhan, ack17:54
*** dholler has joined #openvswitch17:55
zhouhandceara: however, I am concerned even with a fix because in our case 1/3 of the HVs had the issue, which may due to a problem in one of the 3 nodes in the raft cluster. If all of the HVs starts to do the clear and resync at the same time if may cause very high load of the server.17:56
zhouhanI will think more about it.17:56
zhouhanOther than this, I was simply following up some of the discussions and reviews.17:57
dcearazhouhan, you mean 1/3 of the nodes had the issue at the same time?17:57
zhouhan1/3 of the nodes were missing same flows and showing same warning logs at same time.17:57
*** psahoo has quit IRC17:58
dcearazhouhan, oh, I see, we only saw it occasionally17:58
zhouhanIt could be that, when there is a change to SB, e.g. creating a new DP, it causes all HVs changing the condition, and at the same time one of the SB server were disconnected thus causing some of the flow updates in the following SB transaction missing.17:59
imaximetsdceara, zhouhan: one possibility that appears in mind is that we could store 'last_id' for the previous successful cond_change and use it instead of 0 on re-connection if there was in-flight cond_change.18:00
zhouhanimaximets: yeah, great idea.18:01
dcearaimaximets, that might work yes18:01
zhouhanwell, I am done with my update18:01
flaviofmay I go next?18:01
dcearaimaximets, do you mind replying to the ML with the suggestion? I'll try it out tomorrow18:01
imaximetsdceara, ok.18:01
dcearaimaximets, thanks18:02
numansflaviof, sure18:02
flaviofnumans: thanks. I spent some time taking a closer look at Ankur's port-range changes in OVN. I was a little18:02
flaviofconfused about the usage of the word external, thinking it was meant for external ip, but I18:02
*** anilvenkata has quit IRC18:02
flaviofwas wrong. It really means the source port range visible to the 'external' side of the connection.18:02
flaviofAnd that may be the external_ip or the logical_ip, depending on the nat type (dnat vs snat).18:02
flaviofMore details on that are in the ML:18:03
flaviof#link https://mail.openvswitch.org/pipermail/ovs-discuss/2020-April/049959.html questions on port-range18:03
flaviofAnyways, I was able to test it some and verified that the nat rules are indeed populated properly18:03
flaviofall the way into conntrack. Tested with ncat(s)  ;)18:03
numanscool18:03
flaviofAlso, I tried out Numan's fix to make OVN test 85 pass consistently. No18:04
flaviofsurprise it worked great. ;)18:04
flaviofIn the process, I saw OVN test 78 failing and decided to look into it.18:04
flaviofThen, dceara told me that test 76 needed love too, and I was able to18:04
flaviofreproduce the issue and make it better.18:04
flaviofWith these changes I get it to pass every time in my system,18:04
flaviofbut would love to have folks here find out if this is not just me. ;)18:04
flaviofThis is basically the command I do to test the changes:18:04
flaviofCNT=0 ; while [ $? -eq 0 ]; do sleep 3 ; echo $(date +"%D %T %Z") -- cnt is $CNT ; \18:04
flaviofCNT=$((${CNT}+1)) ; make check TESTSUITEFLAGS="76 78 85" ; done18:04
flaviof#link https://patchwork.ozlabs.org/project/openvswitch/patch/20200417145737.1769111-1-numans@ovn.org/ Fix for OVN test 8518:05
flaviof#link https://patchwork.ozlabs.org/project/openvswitch/patch/20200417212501.23757-1-flavio@flaviof.com/ Fix for OVN test 7818:05
flaviof#link https://patchwork.ozlabs.org/project/openvswitch/patch/20200423123731.29123-1-flavio@flaviof.com/ Fix for OVN test 7618:05
flaviofthat is it fro me18:05
flaviof*from18:05
numansflaviof, Thanks. I'll take a look tomorrow. I applied one of your patch and tested a wrong case.18:05
flavioflol. all good.18:05
numansapplied as in locally :)18:05
flavioftiming things are tricky18:05
numansWho wants to go next.18:05
numansflaviof, yeah. And there're a few tests which fail 100% of the time in one of server I've access to.18:06
*** donhw_ has quit IRC18:06
numansI need to dig further.18:06
flaviofawesome!18:06
numansAnyway, if some one wants to go next/18:06
*** donhw has joined #openvswitch18:07
dcearaI'd just like to bring up these OVN2.12 backports https://patchwork.ozlabs.org/project/openvswitch/list/?series=168987 It would be nice to have them backported as we need them downstream.18:08
dceara#link https://patchwork.ozlabs.org/project/openvswitch/list/?series=16898718:08
dcearaflaviof, that's the way to share links right? :)18:08
dcearathat's it on my side18:08
numansdceara, yes.18:08
flaviofdceara: you got it. whatever comment you want to store, make it after the url18:09
dcearaflaviof, ah, now i see, ok, next time I know18:09
numansWho wants to go next.18:11
aginwalanm from my side but spent time with Han on debugging the flow miss issue since we have interconnection enabled now between two AZs, random pod/vm connectivity failures were reported by customers when reaching workloads to/from az1/az2. Hence, the fix by dceara helped us recall and audit results were surprising with 1/3 of the HVs missing the18:11
aginwalaupdates. Following the ML for fixes by you guys.18:11
*** dholler has quit IRC18:11
flaviofaginwala: cool news on the interconnection you guys do! Great to hear from you18:13
aginwalayo!18:14
numanscool.18:15
numansI thnk its time to end the meeting.18:16
zhouhannumans: In case you didn't know yet, I confirmed that we also had the disconnection between RAFT nodes due to probe timeout you reported before, when the servers are overloaded, even though the election timer is not timed out yet. And NVIDIA folks also reported that and sent a patch to disable the probe #link https://patchwork.ozlabs.org/project/openvswitch/patch/20200331002104.26230-1-zhewang@nvidia.com/18:16
* flaviof cues up Space Bazooka :) http://dig.ccmixter.org/files/Kirkoid/4300518:16
numanszhouhan, I saw that patch.18:16
numansOk.18:16
numanszhouhan, thanks for the update.18:17
zhouhannp18:17
numansI guess we can end the meeting ?18:17
numansOk. Bye everyone.18:18
flaviofbye all18:18
numans#endmeeting18:18
openstackMeeting ended Thu Apr 23 18:18:13 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)18:18
openstackMinutes:        http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-04-23-17.21.html18:18
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-04-23-17.21.txt18:18
openstackLog:            http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-04-23-17.21.log.html18:18
zhouhanbye all18:18
dcearabye everyone18:18
aginwalabye all. Stay safe!18:18
imaximetsbye18:18
*** aginwala has quit IRC18:19
*** zhouhan_ has joined #openvswitch18:51
*** zhouhan has quit IRC18:55
*** fbl has quit IRC18:58
*** yogananth has quit IRC19:08
*** imaximets has quit IRC19:13
*** imaximets has joined #openvswitch19:14
*** atpa8a has quit IRC19:14
*** atpa8a has joined #openvswitch19:36
*** igordc has quit IRC19:39
*** fbl has joined #openvswitch19:44
*** thaller has quit IRC20:06
*** mmirecki has joined #openvswitch20:08
*** mmirecki has quit IRC20:15
*** igordc has joined #openvswitch20:32
*** slaweq has joined #openvswitch20:33
*** igordc has quit IRC21:51
*** rcernin has joined #openvswitch22:10
*** slaweq has quit IRC22:14
*** rcernin has quit IRC22:15
*** rcernin has joined #openvswitch22:15
*** acidfoo has quit IRC22:18
*** acidfoo has joined #openvswitch22:18
*** igordc has joined #openvswitch22:26
*** gmg has quit IRC22:37
*** gmg has joined #openvswitch22:39
*** timothy has quit IRC22:42
*** igordc_tmp has joined #openvswitch23:05
*** igordc has quit IRC23:05
*** rcernin has quit IRC23:09
*** igordc_tmp has quit IRC23:13
*** rcernin has joined #openvswitch23:16
*** dceara has quit IRC23:18
*** dceara has joined #openvswitch23:19
*** dceara has quit IRC23:29
*** gmg has quit IRC23:32
*** gmg has joined #openvswitch23:37
*** dceara has joined #openvswitch23:38
*** gmg has quit IRC23:45
*** dceara has quit IRC23:47
*** gmg has joined #openvswitch23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!