Monday, 2020-11-09

*** macz_ has joined #openstack-neutron00:39
*** macz_ has quit IRC00:44
*** ociuhandu has joined #openstack-neutron00:48
*** ociuhandu has quit IRC00:52
*** spatel has joined #openstack-neutron00:54
*** marbindrakon has quit IRC01:21
*** spatel has quit IRC01:23
*** marbindrakon has joined #openstack-neutron01:25
*** sapd1 has joined #openstack-neutron01:28
*** macz_ has joined #openstack-neutron01:30
*** macz_ has quit IRC01:34
*** falencastro_ has quit IRC02:04
*** falencastro_ has joined #openstack-neutron02:05
*** sapd1 has quit IRC02:18
*** tbachman has joined #openstack-neutron02:20
*** ramishra has joined #openstack-neutron03:28
*** slaweq has quit IRC03:40
*** benj_ has quit IRC03:41
*** slaweq has joined #openstack-neutron03:42
*** benj_ has joined #openstack-neutron03:45
*** macz_ has joined #openstack-neutron03:54
*** amotoki has quit IRC03:56
*** amotoki has joined #openstack-neutron03:57
*** macz_ has quit IRC03:58
*** baojg has joined #openstack-neutron05:13
*** baojg has quit IRC05:16
*** baojg has joined #openstack-neutron05:17
*** ratailor has joined #openstack-neutron05:17
*** baojg has quit IRC05:51
*** baojg has joined #openstack-neutron05:52
*** gcheresh has joined #openstack-neutron05:53
*** zzzeek has quit IRC05:58
*** zzzeek has joined #openstack-neutron06:00
*** waleedm__ has joined #openstack-neutron06:16
*** lajoskatona has joined #openstack-neutron06:43
*** rcernin has quit IRC06:48
*** rcernin has joined #openstack-neutron06:49
*** viks____ has joined #openstack-neutron06:57
*** abdysn has joined #openstack-neutron07:02
*** numans_ has joined #openstack-neutron07:06
*** lpetrut has joined #openstack-neutron07:11
*** sapd1 has joined #openstack-neutron07:19
*** rcernin has quit IRC07:38
*** ccstone has quit IRC07:40
*** eandersson has quit IRC07:40
*** ccstone has joined #openstack-neutron07:41
*** eandersson has joined #openstack-neutron07:41
*** tmorin has joined #openstack-neutron07:46
*** jlibosva has joined #openstack-neutron07:48
openstackgerritMerged openstack/neutron master: Fix a typo in sample policy file  https://review.opendev.org/75981907:57
*** iurygregory has quit IRC07:58
*** baojg has quit IRC07:58
*** baojg has joined #openstack-neutron07:59
*** tesseract has joined #openstack-neutron08:08
*** bengates has joined #openstack-neutron08:18
*** bengates has joined #openstack-neutron08:19
*** elvira has joined #openstack-neutron08:23
*** waleedm has joined #openstack-neutron08:27
*** waleedm__ has quit IRC08:27
*** rpittau|afk is now known as rpittau08:30
*** iurygregory has joined #openstack-neutron08:30
*** sapd1 has quit IRC08:33
openstackgerritSlawek Kaplonski proposed openstack/neutron master: DNM - Just devstack test  https://review.opendev.org/76187108:54
*** lucasagomes has joined #openstack-neutron08:59
slaweqfrickler: hi, if You have few minutes, please check https://review.opendev.org/#/c/761178/09:00
openstackgerritSlawek Kaplonski proposed openstack/neutron-tempest-plugin master: Consolidate some tempest tests in the neutron-tempest-plugin jobs  https://review.opendev.org/76141009:07
*** waleedm has quit IRC09:07
*** ociuhandu has joined #openstack-neutron09:08
*** baojg has quit IRC09:11
*** gcheresh has quit IRC09:11
*** gcheresh has joined #openstack-neutron09:12
*** baojg has joined #openstack-neutron09:12
*** ociuhandu has quit IRC09:18
viks____hi, When we create instances, it creates virtual tap interface in the host. So my question is if there is any limit on number of virtual interfaces in the host? or does more number of virtual interfaces has any impact on overall performance? What is the genral guideline for this?09:18
slaweqviks____: hi, I don't know about overall limit of tap interfaces09:20
slaweqbut if You have many neutron ports on one compute, neutron-ovs-agent may be very slow, so e.g. restart of neutron-ovs-agent may take very long time to by synced09:21
*** waleedm has joined #openstack-neutron09:25
*** ociuhandu has joined #openstack-neutron09:26
slaweqlajoskatona: hi, can You maybe approve https://review.opendev.org/#/c/759679/ (You already gave +2 for it)09:27
slaweqpatch https://review.opendev.org/#/c/759555/ is already merged now09:27
viks____slaweq: Ok.. thnx.  i'm using linux-bridge-agent.. so that also should be having similar behaviour right?09:27
slaweqviks____: right09:28
*** tmorin has quit IRC09:28
*** ociuhandu has quit IRC09:28
*** ociuhandu has joined #openstack-neutron09:28
openstackgerritSlawek Kaplonski proposed openstack/neutron-tempest-plugin master: Unmark port_forwarding tests as unstable  https://review.opendev.org/75611409:30
viks____slaweq: is there any general guideline on like the optimal number of such ports?09:30
slaweqviks____: nope09:30
lajoskatonaslaweq: done09:30
slaweqlajoskatona: thx a lot09:30
*** ralonsoh has joined #openstack-neutron09:33
viks____slaweq: also is there any limits for neutron ports/network?09:33
slaweqin neutron You have just quota settings which can limit You09:34
slaweqbut that can be changed with API09:34
*** iurygregory has quit IRC09:35
*** iurygregory has joined #openstack-neutron09:36
viks____slaweq: ok... thanks... does more ports/network has any other performance impact other than the point you have mentioned above09:41
slaweqviks____: I'm not linuxbridge agent expert so it's hard to say for me09:41
slaweqI never used it on bigger scale09:41
slaweqlajoskatona: can You also check once again https://review.opendev.org/#/c/601336/ ?09:42
slaweqand let me know wdyt about my replies to Your comments there09:42
slaweqthx in advance09:42
viks____slaweq: oh ok.. in case of ovs agent, do you have some gauge on ports per network?09:45
viks____slaweq: anyay thanks a lot for you inputs09:53
lajoskatonaslaweq: I can check09:54
slaweqlajoskatona: thx09:54
*** tesseract has quit IRC09:54
slaweqviks____: if You have many ports which are using same SG then You shouldn't use rules with remote_group_id there as this may slow agent down09:57
*** tesseract has joined #openstack-neutron09:57
slaweqviks____: but I'm not really sure how this may imact linuxbridge agent, probably the same as for neutron-ovs-agent but I never tested it09:57
viks____slaweq: ok.. thanks a lot09:59
slaweqviks____: yw10:05
openstackgerritOleg Bondarev proposed openstack/neutron master: Auto-remove floating agent gw ports on net/subnet delete  https://review.opendev.org/76149810:05
*** ociuhandu has quit IRC10:28
openstackgerritMerged openstack/neutron master: [Docs] Add info about how multicast is treated by fw drivers  https://review.opendev.org/75967910:31
*** iurygregory_ has joined #openstack-neutron10:32
*** baojg has quit IRC10:34
*** sapd1 has joined #openstack-neutron10:34
*** baojg has joined #openstack-neutron10:34
*** iurygregory has quit IRC10:35
*** ociuhandu has joined #openstack-neutron10:36
*** ociuhandu has quit IRC10:37
*** ociuhandu has joined #openstack-neutron10:37
*** jangutter has joined #openstack-neutron10:38
*** jangutter_ has joined #openstack-neutron10:55
*** abdysn has quit IRC10:56
*** abdysn has joined #openstack-neutron10:56
*** jangutter has quit IRC10:58
openstackgerritSlawek Kaplonski proposed openstack/neutron-tempest-plugin master: Mark test_subport_connectivity as unstable for now  https://review.opendev.org/76188611:00
*** sapd1 has quit IRC11:12
*** ociuhandu has quit IRC11:17
*** ratailor_ has joined #openstack-neutron11:20
openstackgerritMerged openstack/neutron-tempest-plugin master: Skip WritableMtuTest for OVN  https://review.opendev.org/76044611:20
*** baojg has quit IRC11:20
*** baojg has joined #openstack-neutron11:21
*** ociuhandu has joined #openstack-neutron11:21
*** ratailor_ has quit IRC11:22
*** ratailor has quit IRC11:22
*** ratailor_ has joined #openstack-neutron11:23
*** sapd1 has joined #openstack-neutron11:32
openstackgerritMerged openstack/neutron-tempest-plugin master: Fix access to dictionary variable "server"  https://review.opendev.org/76171711:37
*** sapd1 has quit IRC11:46
*** Abdallahyas has joined #openstack-neutron11:48
*** ratailor_ has quit IRC11:50
*** abdysn has quit IRC11:52
*** Luzi has joined #openstack-neutron11:56
*** ociuhandu has quit IRC12:04
*** JamesBenson has joined #openstack-neutron12:10
openstackgerritLajos Katona proposed openstack/networking-bagpipe master: Add privsep boilerplate for bagpipe  https://review.opendev.org/73384812:13
*** ociuhandu has joined #openstack-neutron12:13
*** obondarev has joined #openstack-neutron12:14
*** ociuhandu has quit IRC12:18
*** ociuhandu has joined #openstack-neutron12:25
*** raildo has joined #openstack-neutron12:28
*** ociuhandu has quit IRC12:29
*** ociuhandu has joined #openstack-neutron12:30
*** dtantsur|afk is now known as dtantsur12:32
*** ociuhandu has quit IRC12:35
openstackgerritLajos Katona proposed openstack/neutron master: Remove left over code for admin_state_down_before_update  https://review.opendev.org/74490112:48
*** spatel has joined #openstack-neutron12:55
*** baojg has quit IRC12:59
*** spatel has quit IRC12:59
*** baojg has joined #openstack-neutron13:00
*** iurygregory_ is now known as iurygregory13:01
*** Abdallahyas has quit IRC13:10
*** priteau has joined #openstack-neutron13:18
*** Luzi has quit IRC13:25
*** ociuhandu has joined #openstack-neutron13:29
*** falencastro_ is now known as falencastro13:34
*** waleedm_ has joined #openstack-neutron13:41
*** waleedm has quit IRC13:45
*** waleedm_ has quit IRC13:53
*** baojg has quit IRC13:55
*** waleedm has joined #openstack-neutron14:06
*** obondarev has quit IRC14:06
*** ociuhandu has quit IRC14:06
*** nweinber has joined #openstack-neutron14:07
*** TheJulia has joined #openstack-neutron14:10
*** waleedm has quit IRC14:11
*** waleedm has joined #openstack-neutron14:11
*** waleedm_ has joined #openstack-neutron14:13
*** waleedm has quit IRC14:17
*** waleedm_ has quit IRC14:26
*** armax has joined #openstack-neutron14:30
*** sapd1 has joined #openstack-neutron14:35
*** ociuhandu has joined #openstack-neutron14:42
*** ociuhandu has quit IRC14:47
*** lpetrut has quit IRC15:02
coreycbhello neutron folks, can someone give this patch a review? https://review.opendev.org/#/c/752248/  we are getting a lot of heat on it and would like upstream's perspective on it.15:15
slaweqcoreycb: sure, I will check it today15:16
coreycbthank you very much slaweq15:16
*** bengates_ has joined #openstack-neutron15:18
*** bengates has quit IRC15:19
*** ociuhandu has joined #openstack-neutron15:22
otherwiseguyslaweq: are the logstash logs...spotty? It seems like 'build_change:753847 AND build_status:FAILURE AND build_name:neutron-ovn-tempest-ovs-release' should return results when I have that particular failed test recently. with build_name:neutron-ovn-tempest-slow it returns results. or dropping the build_status with the ovs-release test...15:26
otherwiseguyboth tests failed the same day.15:26
otherwiseguyI'm also seeing the timestamp field be 2020-11-08, but the message field shows Nov 03...so that seems weird too.15:27
*** baojg has joined #openstack-neutron15:38
*** iurygregory has quit IRC15:43
*** nweinber has quit IRC15:46
*** nweinber has joined #openstack-neutron15:48
*** hyang has joined #openstack-neutron15:51
*** rpittau is now known as rpittau|bbl15:52
*** hyang has quit IRC15:55
*** ociuhandu has quit IRC15:58
clarkbotherwiseguy: yes, indexing has had trouble due to large input files15:59
clarkbneutron was part of hte problem last I checked too though I think those issues may have been addressed. I just haven't had time to test melwitt's file stream updates to see if that helps or to track down other issues with everything else going on15:59
otherwiseguyclarkb: ok. it's all kind of part of a vicious cycle. lots of failures, lots of rechecks, hard to find patterns in logs because of log indexing issues, so hard to know whether to recheck... :)16:01
otherwiseguyclarkb: thanks for the info!16:01
clarkbotherwiseguy: right I'm just pointing out that the genesis of all these problems was neutron using debug logs to dump performance metrics for individuals functions whcih exploded the size of the input data which caused everything to break16:03
*** iurygregory has joined #openstack-neutron16:03
clarkbI asked neutron to fix it and it didn't get fixed after like a month so I stopped bothering to really try and keep the system working16:03
*** gcheresh has quit IRC16:04
otherwiseguyclarkb: thanks, I'm not up on the history of that change. Do you have an example of the logs that are causing the problem? I'm happy to take a look at it.16:05
* otherwiseguy sometimes gets lost in his little ovsdb-related world16:05
otherwiseguyAnd ovsdbapp tests run really quickly. :p (though have in the past been responsible for producing a stupid amount of logs, but I think we fixed that)16:06
clarkbI'll have to go look at irc logs or service logs to find it. Basically there was a function decorator added to things taht put performance metrics for that function in debug logs. If that function is called a lot the debug log size went from a few MB to a few hundred MB then you multiply that by the number of jobs we run ...16:06
*** mlavalle has joined #openstack-neutron16:06
otherwiseguyoh my16:06
clarkband it may be fixed now. I don't know. It just wasn't fixed by the time I last tried restarting things to roll forwarding on the indexing16:07
openstackgerritHang Yang proposed openstack/neutron master: Support remote address group in SG rules  https://review.opendev.org/75111016:07
openstackgerritHang Yang proposed openstack/neutron master: <WIP> Support address group in OVS firewall agent  https://review.opendev.org/75765016:07
otherwiseguyclarkb: so this stuff? https://docs.openstack.org/neutron/latest/contributor/internals/code_profiling.html enable_code_profiling?16:08
otherwiseguyclarkb: grepping, looks like maybe just the rally job has it enabled?16:09
otherwiseguyclarkb: are you saying that we should disable it for that job? Or maybe not index the rally job?16:10
clarkbno it was deep in neutron and happened for all jobs iirc16:11
clarkbsorry in a meeting so not able to grep through my logs yet16:11
*** jgwentworth is now known as melwitt16:12
*** baojg has quit IRC16:12
*** baojg has joined #openstack-neutron16:13
*** ircuser-1 has joined #openstack-neutron16:14
clarkbotherwiseguy: https://review.opendev.org/#/c/741540/2/neutron/objects/ports.py that was one of the first chagnes to start addressing the problem. Not sure if that decorator is used elsewhere but double checking on that would be great16:26
*** tmorin has joined #openstack-neutron16:26
otherwiseguyclarkb: thanks! will do.16:26
otherwiseguyclarkb: looks like only place in tree is most of the l3 api methods.16:29
otherwiseguyclarkb: I'll look at the logs produced. thanks again for the heads up!16:29
openstackgerritMerged openstack/neutron stable/victoria: Fix formatting error in agent/linux/external_process.py  https://review.opendev.org/76148516:34
*** slaweq has quit IRC16:37
*** ociuhandu has joined #openstack-neutron16:38
*** slaweq has joined #openstack-neutron16:40
otherwiseguyclarkb: grabbed all of the logs for a grenade job which I figured should be on the high end: https://paste.centos.org/view/d3173caf (699 Time-cost lines total, most in screen-q-l3.txt files)16:46
clarkbok so that isn't too bad there16:47
otherwiseguyclarkb: checking a functional job too just to be sure16:48
otherwiseguyclarkb: the download script is on file 70/1979 for that though, so it might be a bit.16:48
otherwiseguy(speaking of indexing files) :p16:49
*** ociuhandu has quit IRC16:50
openstackgerritMerged openstack/neutron master: Add property "noopfirewall_or_firewall_disabled"  https://review.opendev.org/75267616:56
*** ociuhandu has joined #openstack-neutron16:58
*** lucasagomes has quit IRC17:00
*** dsneddon has joined #openstack-neutron17:02
*** jlibosva has quit IRC17:03
*** ociuhandu has quit IRC17:03
*** ociuhandu has joined #openstack-neutron17:08
openstackgerritLajos Katona proposed openstack/neutron-tempest-plugin master: start using available_features cfg option from tempest  https://review.opendev.org/76143417:08
*** hamalq has joined #openstack-neutron17:11
openstackgerritRodolfo Alonso Hernandez proposed openstack/neutron master: Use new engine facade in L3 DB code  https://review.opendev.org/71531517:14
*** hamalq has quit IRC17:14
*** hamalq has joined #openstack-neutron17:15
otherwiseguyclarkb: yeah, looks like only 112 Time-cost lines in the functional tests.17:16
otherwiseguyclarkb: though the dsvm-functional-logs directory is 257MB17:17
*** bengates has joined #openstack-neutron17:17
*** bengates_ has quit IRC17:21
*** bengates has quit IRC17:21
*** rpittau|bbl is now known as rpittau17:22
*** viks____ has quit IRC17:25
*** lajoskatona has quit IRC17:28
*** ralonsoh has quit IRC17:34
otherwiseguyclarkb: looks like the ovn_sb is biggest file in most dirs. DBG logs on that seem pretty chatty. Looks like ~74MB in db logs for a functional test run.17:34
otherwiseguy(for nb/sb log files under dsvm-functional-logs)17:35
clarkbthat may be an issue if we try to index those files17:35
clarkbbut we're also working to deploy improvements on the indexing side so maybe wewait and see if those correct things17:35
clarkboh can you check syable branches for that decorator I linked?17:36
clarkbI wonder if it is stable jobs hitting us now17:36
otherwiseguyclarkb: back through train anyway, I only see timecost in neutron/agent/l3/agent.p y17:40
clarkbthanks17:41
*** lajoskatona has joined #openstack-neutron17:42
*** hjensas__ has joined #openstack-neutron17:43
otherwiseguyother biggish files are ./neutron.tests.functional.agent.linux.test_linuxbridge_arp_protect.LinuxBridgeARPSpoofTestCase.test_arp_correct_protection_allowed_address_pairs.txt which is just mostly strings of mac addresses in ebtables CLI calls. But that's the only one over 500k (it's 1.8M) that isn't an ovn log file.17:43
*** ociuhandu_ has joined #openstack-neutron17:43
otherwiseguy(some of the testrun.txt ovn files are also > 1mb but mostly it's dbs)17:44
*** elvira has quit IRC17:44
otherwiseguydb *log files* that is. Though the directories also store the actual .db files which are just essentially JSON text and definitely shouldn't be indexed either.17:46
otherwiseguyclarkb: ^17:46
*** hjensas_ has quit IRC17:47
*** ociuhandu has quit IRC17:47
*** ociuhandu_ has quit IRC17:48
clarkbya i doubt we index those17:50
*** nweinber has quit IRC17:51
*** lajoskatona has quit IRC18:17
*** dtantsur is now known as dtantsur|afk18:24
otherwiseguyclarkb: so is it https://opendev.org/opendev/base-jobs/commit/d568ace8486770bf3a745d6574a19a3cbead8f8f that would be reconsidered? Also, I don't see anything about dsvm-functional-logs/ files being indexed at all, so I'm guessing we're ok on the ovsdb log files.18:31
clarkbyes I think we can revert that now if the svc logs are no longergiant18:32
otherwiseguyclarkb: a spot check of a couple of different jobs, 23MB and 55MB.18:36
*** nsmeds has joined #openstack-neutron18:43
*** mlavalle has quit IRC18:43
*** mlavalle has joined #openstack-neutron18:44
*** rpittau is now known as rpittau|afk18:45
*** tesseract has quit IRC19:02
*** baojg has quit IRC19:24
*** baojg has joined #openstack-neutron19:25
*** jangutter has joined #openstack-neutron19:26
*** jangutter_ has quit IRC19:30
*** baojg has quit IRC20:24
*** baojg has joined #openstack-neutron20:24
*** tbachman has quit IRC20:50
*** tbachman has joined #openstack-neutron20:53
*** gcheresh has joined #openstack-neutron20:59
openstackgerritSlawek Kaplonski proposed openstack/neutron master: DVR: Add ARP responder flows for allowed_address_pair IP  https://review.opendev.org/60133621:05
*** sapd1 has quit IRC21:05
*** tmorin has quit IRC21:20
openstackgerritMiguel Lavalle proposed openstack/neutron master: Agent side push notifications for address groups  https://review.opendev.org/75911721:24
*** mgoddard has quit IRC21:25
*** mgoddard has joined #openstack-neutron21:26
*** rcernin has joined #openstack-neutron21:27
*** baojg has quit IRC21:33
*** baojg has joined #openstack-neutron21:34
*** baojg has quit IRC21:42
*** baojg has joined #openstack-neutron21:43
*** rcernin has quit IRC22:06
*** gcheresh has quit IRC22:10
*** rcernin has joined #openstack-neutron22:12
*** rcernin has quit IRC22:22
*** rcernin has joined #openstack-neutron22:22
*** takamatsu has quit IRC22:35
*** slaweq has quit IRC22:39
*** iurygregory has quit IRC23:04
*** iurygregory has joined #openstack-neutron23:16
*** takamatsu has joined #openstack-neutron23:20
*** rubasov has quit IRC23:22
*** rubasov has joined #openstack-neutron23:29
*** baojg has quit IRC23:29
*** baojg has joined #openstack-neutron23:30
*** baojg has quit IRC23:42
*** baojg has joined #openstack-neutron23:43

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!