Wednesday, 2020-04-08

*** ricolin has joined #openstack-meeting-301:58
*** artom has quit IRC02:03
*** apetrich has quit IRC02:11
*** yamamoto has joined #openstack-meeting-302:14
*** psachin has joined #openstack-meeting-303:28
*** yamamoto has quit IRC03:42
*** yamamoto has joined #openstack-meeting-303:44
*** yamamoto has quit IRC04:42
*** yamamoto has joined #openstack-meeting-304:47
*** igordc has quit IRC05:32
*** links has joined #openstack-meeting-306:20
*** yamamoto has quit IRC06:52
*** slaweq_ has joined #openstack-meeting-306:57
*** maciejjozefczyk has joined #openstack-meeting-307:00
*** amorin has joined #openstack-meeting-307:01
amorinhello07:01
*** yamamoto has joined #openstack-meeting-307:05
*** lpetrut has joined #openstack-meeting-307:09
*** belmoreira has joined #openstack-meeting-307:16
*** ralonsoh has joined #openstack-meeting-307:33
*** e0ne has joined #openstack-meeting-307:38
ttxo/08:00
ttx#startmeeting large_scale_sig08:00
openstackMeeting started Wed Apr  8 08:00:54 2020 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.08:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.08:00
*** openstack changes topic to " (Meeting topic: large_scale_sig)"08:00
openstackThe meeting name has been set to 'large_scale_sig'08:00
ttx#topic Rollcall08:01
*** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)"08:01
ttxHi! Who is here for the Large Scale SIG meeting ?08:01
ttxSome confusion due to change in meeting channel, I'm trying to people people there08:02
belmoreirahi08:02
belmoreiraindeed, I was in openstack-meeting. Thanks08:03
*** masahito has joined #openstack-meeting-308:03
masahitohi08:03
ttxSorry about that, I gave the wrong channel in the reminder email08:04
ttxOK... How is everyone doing in this confined world?08:04
belmoreiraFor now no symptoms. Hope you all are fine08:05
*** etp has joined #openstack-meeting-308:05
ttxNote very productive but fine08:05
ttxNot*08:05
masahitoI'm also fine :-)08:05
etpHi08:06
ttxamorin: you here?08:06
ttxOur agenda for today is at:08:06
ttx#link https://etherpad.openstack.org/p/large-scale-sig-meeting08:06
ttxLet's start with the second topic, waiting for amorin to rejoin (he was here earlier)08:07
ttx#topic Progress on "Scaling within one cluster" goal08:07
*** openstack changes topic to "Progress on "Scaling within one cluster" goal (Meeting topic: large_scale_sig)"08:07
amorinhye all08:07
ttx#link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling08:07
ttxI had a TODO to push for oslo.metric spec approval, which is now merged!08:07
ttxmasahito: what are the next steps?08:08
ttxI'd say get the code up?08:08
masahitonext action is publishing a poc code08:08
masahitoPreparing but not much progress now.08:09
ttxOK, once you are ready, I can help with setting up code repositories08:09
ttxJust keep us posted in progress there08:09
ttx#action masahito to prepare oslo.metric POC code release08:10
ttxoneswig had a TODO around contributing a scaling story on bare metal cluster scaling, but he hasn't been around lately08:10
ttxso I'll push that one back again08:11
ttx#action oneswig to contribute a scaling story on bare metal cluster scaling08:11
ttxAnything else on that topic ? Any other work in progress ?08:11
belmoreiraI can contribute on that as well. (scaling story on bare metal)08:12
ttxbelmoreira: *any* scaling story from CERN would be great :)08:12
belmoreirawhat's the expected format? Like a user story?08:12
ttxAt the moment we dump them on https://etherpad.openstack.org/p/scaling-stories08:12
ttxBut if we start collecting more, we'll have to set up something08:12
ttxI just did not want to put too much constraints on format or length08:13
ttxto encourage people to share08:13
belmoreiragreat, I will add our bare metal story08:13
ttxMaybe we could do a set of wiki pages08:13
ttxI can prepare something... But use the etherpad for now08:14
belmoreirai think is good the etherpad for now08:14
ttx#action belmoreira to submit a scaling story to https://etherpad.openstack.org/p/scaling-stories08:14
ttx#action ttx to copy scaling stories on the wiki for long-term storage08:15
ttxok next topic...08:15
ttx#topic Progress on "Documenting large scale operations" goal08:15
*** openstack changes topic to "Progress on "Documenting large scale operations" goal (Meeting topic: large_scale_sig)"08:15
ttx#link https://etherpad.openstack.org/p/large-scale-sig-documentation08:15
ttxWe had two TODOs for amorin on that08:15
ttx(create a wiki page for large scale documentation, propose patch against Nova doc)08:16
amorinyup08:16
amorinI created the wiki page, over here:08:16
amorinhttps://wiki.openstack.org/wiki/Large_Scale_Configuration_Guidelines08:16
amorinwhich is referenced in our main page08:16
ttxnice start!08:16
amorinFor now, it's almost empty, I need to fill the page :p08:17
amorinanyway it gives a good overview of what it will look like08:17
amorinI was also supposed to create a patch for nova documentation08:18
amorinbut I havnt been able to move forward with that yet08:18
ttxok, I'll push that back too08:18
amorinso we can add it back into my action08:18
amorinthanks08:18
ttx#action amorin propose patch against Nova doc08:18
ttxAlright, anything else on that topic ?08:18
amorinthat's all I think on my side08:19
ttx#topic Large scale operations now-virtual OpenDev track08:19
*** openstack changes topic to "Large scale operations now-virtual OpenDev track (Meeting topic: large_scale_sig)"08:19
ttx#link https://etherpad.openstack.org/p/LargeScaleOps_OpenDev08:19
ttxIt looks like the group has made great progress in turning that track into something virtual we could associate the SIG with08:19
ttxI have great hopes that can give us more input and participation08:20
ttxbelmoreira, masahito : what's the status on that? I heard from Allison that you had good ideas08:21
belmoreirawe have the proposal in the etherpad. People are invited to contribute08:22
ttxOK, let's continue to track that08:24
belmoreirawe didn't meet during the last 2 weeks. I think now Allison will get all this input for the session during the event08:24
ttxAnything else on that topic?08:24
ttx#topic Next meeting08:25
*** openstack changes topic to "Next meeting (Meeting topic: large_scale_sig)"08:25
ttxin 2 weeks ?08:25
amorinworks for me08:26
ttxI'm also fine skipping one if we think we won;t have that much progress to report on in two weeks08:26
belmoreirasounds good to me08:26
belmoreiraI mean, in 2 weeks works for me08:26
masahitoI'm fine.08:26
ttxOK, let's do it in two weeks, and make it short if we haven't much to report08:26
ttx#info next meeting: April 22, 8:00UTC08:26
ttxThanks everyone! Stay safe!08:27
amorinthanks,stay home :p08:27
etpthanks, stay safe08:27
ttx#endmeeting08:27
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"08:27
openstackMeeting ended Wed Apr  8 08:27:57 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)08:27
openstackMinutes:        http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-04-08-08.00.html08:28
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-04-08-08.00.txt08:28
openstackLog:            http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-04-08-08.00.log.html08:28
*** slaweq_ is now known as slaweq08:36
*** raymonddeng has joined #openstack-meeting-309:14
*** raymonddeng has quit IRC09:15
*** ricolin_ has quit IRC09:29
*** ricolin_ has joined #openstack-meeting-309:30
*** ricolin has quit IRC09:30
*** ricolin_ has quit IRC09:57
*** ricolin_ has joined #openstack-meeting-309:57
*** ricolin has joined #openstack-meeting-310:00
*** ricolin_ has quit IRC10:02
*** masahito has quit IRC10:04
*** links has quit IRC10:08
*** yamamoto has quit IRC10:13
*** slaweq has quit IRC10:23
*** slaweq has joined #openstack-meeting-310:25
*** links has joined #openstack-meeting-310:31
*** yamamoto has joined #openstack-meeting-310:33
*** yamamoto has quit IRC10:34
*** ricolin has quit IRC10:39
*** yamamoto has joined #openstack-meeting-310:53
*** e0ne has quit IRC10:58
*** e0ne has joined #openstack-meeting-310:58
*** apetrich has joined #openstack-meeting-311:16
*** links has quit IRC11:44
*** links has joined #openstack-meeting-311:45
*** raildo has joined #openstack-meeting-311:53
*** artom has joined #openstack-meeting-312:08
*** ricolin has joined #openstack-meeting-312:57
*** yamamoto has quit IRC13:50
*** jraju__ has joined #openstack-meeting-313:50
*** links has quit IRC13:52
*** liuyulong has joined #openstack-meeting-313:59
*** belmoreira has quit IRC14:02
*** yamamoto has joined #openstack-meeting-314:17
*** igordc has joined #openstack-meeting-314:45
*** lpetrut has quit IRC14:46
slaweq#startmeeting neutron_ci15:00
openstackMeeting started Wed Apr  8 15:00:16 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:00
openstackThe meeting name has been set to 'neutron_ci'15:00
ralonsohhi15:00
slaweqhi15:00
njohnstono/15:00
maciejjozefczyk\o15:00
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:01
slaweq#topic Actions from previous meetings15:01
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:01
bcafarelo/15:01
slaweqfirst one:15:01
slaweqslaweq to investigate fullstack SG test broken pipe failures15:01
slaweqI did some investigation on that one15:01
slaweqI have some theory why it could fail like that but I need to try to reproduce it locally to confirm that15:02
slaweqso I will continue this work next week too15:02
slaweq#action slaweq to continue investigation of fullstack SG test broken pipe failures15:02
slaweqnext one15:02
slaweqmaciejjozefczyk to take a look and report LP for failures in neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log15:02
maciejjozefczykI *think* that I found whats the problem15:03
maciejjozefczyk#link https://review.opendev.org/#/c/717704/15:03
maciejjozefczykI proposed to change the transaction timeout for updating OVN rows15:03
maciejjozefczykwith bumping up the timeout I cannot see any failure of that test in the link above.15:03
maciejjozefczykfor a while I changed the zuul config to run those jobs 20 times and in all cases the failure wasn't there.15:04
maciejjozefczykaccording to the logs in the LP report - I found the timeout errors there, so I think that should help15:04
slaweqbased on old comment there, it's not first time when we are changing this timeout for tests15:04
maciejjozefczykslaweq, yes, but the funny think is by *default* in the config we have a much bigger timeout there15:05
maciejjozefczykI don't know why it was set as 5 seconds for functionals before15:05
slaweqahh, ok15:05
maciejjozefczykhttps://docs.openstack.org/networking-ovn/latest/configuration/ml2_conf.html15:06
maciejjozefczykdefault is 180 seconds15:06
slaweqok, if that solves the problem, lets try that :)15:06
*** yamamoto has quit IRC15:06
slaweqthx maciejjozefczyk15:06
maciejjozefczykSo actually it wasn't that bad that sometimes we got Timeout after 5 seconds :)15:06
slaweqok, lets move on15:07
slaweqnext one15:07
slaweqralonsoh to open LP about issue with neutron.tests.functional.agent.linux.test_keepalived.KeepalivedManagerTestCase15:07
ralonsohpatch merged15:08
slaweq++15:08
slaweqthx ralonsoh15:08
ralonsoh#link  https://review.opendev.org/#/c/716944/15:08
slaweqI hope that with this one and maciejjozefczyk's patch functional jobs will be in better shape finally :)15:09
ralonsohsure!15:09
maciejjozefczyk++15:09
slaweqok, next one15:09
slaweqslaweq to check server termination on multicast test15:09
slaweqI again didn't had time for this15:09
slaweqand sadly it just hit as again in maciejjozefczyk's patch mentioned few minutes ago15:10
slaweq:/15:10
slaweqI will really dedicate some time this week to finally check that one15:10
slaweq#action slaweq to check server termination on multicast test15:10
slaweqnext one15:10
slaweqralonsoh to check failure in neutron.tests.functional.agent.l3.test_keepalived_state_change.TestMonitorDaemon15:10
ralonsohwhich one?15:11
ralonsohups, I wasn't aware of this one15:11
ralonsohsorry15:11
slaweqno problem15:12
maciejjozefczykslaweq, perhaps my patch needs rebase15:12
slaweqdo You want to check it this week?15:12
slaweqmaciejjozefczyk: just recheck :)15:12
ralonsohslaweq, do you have the links?15:12
ralonsohof the logs15:12
slaweqralonsoh: https://80137ce53930819135d8-42d904af0faa486c8226703976d821a0.ssl.cf2.rackcdn.com/704833/23/check/neutron-functional/17568d5/testr_results.html15:12
ralonsohslaweq, thanks!15:13
slaweqralonsoh: is this related https://bugs.launchpad.net/neutron/+bug/1870313 ?15:13
openstackLaunchpad bug 1870313 in neutron ""send_ip_addr_adv_notif" can't use eventlet when called from "keepalived_state_change"" [High,Fix released] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)15:13
ralonsohahh but is the same problem15:13
slaweqYou linked this bug last week in meeting's etherpad15:13
ralonsohyes, I was writing this15:13
ralonsohwe can remove it from the TODO list15:13
slaweqok, so done :)15:14
slaweqgreat, thx15:14
slaweqok, and the last one from previous week15:14
slaweqralonsoh to check issue with neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_router_fip_qos_after_admin_state_down_up15:14
ralonsohyes, the namespace drama15:14
ralonsoh#link https://review.opendev.org/#/c/717017/15:15
*** yamamoto has joined #openstack-meeting-315:15
*** yamamoto has quit IRC15:15
ralonsohit's merged, please review the commit message for more info15:15
slaweqLOL - "namespace drama" sounds like a good name for topic15:15
ralonsohor ping me in IRC15:15
ralonsohhehehehe15:15
slaweqahh, it's that one15:15
slaweqok15:15
maciejjozefczykheh15:16
slaweqok, that's all actions from last week15:16
slaweqlets move on15:16
slaweq#topic Stadium projects15:16
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:16
slaweqstandardize on zuul v315:16
slaweqEtherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop15:16
slaweqI just marked networking-bagpipe as done15:16
slaweqthx lajoskatona for work on this15:17
slaweqso we have left only networking-{midonet,odl} in the list15:17
slaweqnot bad :)15:17
bcafarelgetting there15:17
njohnston\o/15:17
slaweqIPv6-only CI15:17
slaweq    Etherpad https://etherpad.openstack.org/p/neutron-stadium-ipv6-testing15:17
slaweqno progress on my side with that one still15:18
slaweqand there is one more thing related to stadium projects for today15:19
slaweqmidonet UT failures: https://bugs.launchpad.net/networking-midonet/+bug/187156815:19
openstackLaunchpad bug 1871568 in networking-midonet "python3 unit tests jobs are failing on master branch" [Undecided,New]15:19
slaweqbasically midonet gate is broken now15:19
slaweqI just cloned it locally to check that one, but if someone wants to take it - feel free :)15:19
njohnstonDo we have any active midonet contributors left?15:20
ralonsohyamamoto?15:20
njohnstonI haven't seen yamamoto active recently15:20
slaweqnjohnston: yamamoto is the only one I'm aware of15:20
slaweqbut his activity is very limited :/15:20
slaweqI will send him email about that one - maybe he will be able to help with this15:21
slaweq#action slaweq to ping yamamoto about midonet gate problems15:22
slaweqok, anything else related to stadium for today?15:22
njohnstonI think it would be good from a stadium perspective to do that just to check the health of the midonet driver, I know they have struggled to keep up the last few cycles with Xenial->Bionic and py2715:22
slaweqnjohnston: yes, that's true15:22
slaweqmaybe we will need to discuss that (again) during the next PTG15:23
njohnstonand Stackalytics shows no activity for yamamoto since January15:23
slaweqok15:24
slaweqI will email him and we will see how it will be15:24
njohnston+115:25
ralonsoh+115:25
slaweqlets move on15:25
slaweq#topic Stable branches15:25
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)"15:25
slaweqas recently we have less issues to discuss e.g. with scenario jobs, I though that it would be good to add topic about stable branches to this meeting15:26
slaweqso we can all catch up on current ci state for stable brances15:26
bcafarel+10015:26
ralonsohperfect15:26
njohnstonI have not been updating the stable grafana dashboards, has anyone else?15:27
slaweqnjohnston: nope15:27
njohnstonah looks like they are good, excellent15:27
njohnstonhttp://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1 <- train15:27
njohnstonhttp://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1 <- stein15:28
slaweqI'm not sure if all jobs are up to date there15:28
bcafarelyes at least the branch name changes are there, did not check if jobs are up to date there15:28
slaweqbcafarel: do You want to check that this week? :)15:30
bcafarelsure!15:30
slaweqthx15:30
bcafarelfrom my "what recheck keyword did you type?" memory, most are designate, rally and the occasional non-neutron test failing15:30
slaweq#action bcafarel to check and update stable branches grafana dashboards15:30
slaweqone serious issue from today is this one with rally https://bugs.launchpad.net/neutron/+bug/187159615:31
openstackLaunchpad bug 1871596 in neutron "Rally job on stable branches is failing" [Critical,Confirmed] - Assigned to Slawek Kaplonski (slaweq)15:31
slaweqbut it seems it's already fixed in rally thanks to lucasgomes :)15:31
ralonsoh+115:31
bcafarelyep15:32
njohnstonbcafarel: looks like haleyb updated the stable dashboards when train was released15:32
slaweqgreat, so other than that it's fine for stable branches now, right?15:32
bcafarelfix is merged and from andreykurilin's comments there are not many rally calls from playbook (the part currently running from master) so we should be good15:33
bcafarelhttps://bugs.launchpad.net/bugs/1871327 also has fixes merged in all branches now15:33
openstackLaunchpad bug 1871327 in tempest "stable/stein tempest-full job fails with "tempest requires Python '>=3.6' but the running Python is 2.7.17"" [Undecided,New]15:33
slaweqok15:34
slaweqso I think we can move on to the next topic15:34
slaweq#topic Grafana15:34
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:34
slaweq#link http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:34
slaweqAverage number of rechecks in last weeks:15:34
slaweq    week 14 of 2020: 3.1315:34
slaweq    week 15 of 2020: 3.615:34
slaweqthat's my metrics - not so good this week :/15:35
ralonsohIMO, related to mentioned problems15:35
slaweqralonsoh: yes15:35
slaweqthat's for sure - I didn't noticed any new problems last week15:35
slaweqone thing which I want to mention is that Grenade jobs should be better soon as https://bugs.launchpad.net/bugs/1844929 has proposed patch https://review.opendev.org/71766215:36
openstackLaunchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,In progress] - Assigned to melanie witt (melwitt)15:36
slaweqmelwitt did great debugging on it and found finally what was the issue there15:36
bcafarelooooh nice! (I forgot grenade in the frequent recheck causes on stable))15:36
slaweqI also proposed patch https://review.opendev.org/718392 with updates for grafana dashboard15:38
slaweqand that's all about grafana from me15:39
slaweqanything else to add?15:39
ralonsoh(you need to rebase it)15:39
slaweqralonsoh: I will15:40
slaweqok, if not, let's move on15:40
slaweq#topic Tempest/Scenario15:41
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"15:41
slaweqI have one new issue only here15:41
slaweqTripleo based jobs are failing, like e.g.: https://933286ee423f4ed9028e-1eceb8a6fb7f917522f65bda64a8589f.ssl.cf5.rackcdn.com/717754/2/check/neutron-centos-8-tripleo-standalone/a5f2585/job-output.txt15:41
slaweqdo You maybe know why it can be like that? It seems that neutron rpm build fails in those jobs15:41
bcafarelsounds like https://review.rdoproject.org/r/2630515:42
maciejjozefczykslaweq, after we merged the ovn migration tools15:42
maciejjozefczykyes bcafarel15:42
slaweqI though that but I wanted to confirm that :)15:42
slaweqthx15:42
slaweqso should be good soon15:42
slaweqok, so that's all what I have for today15:43
slaweqanything else You want to talk about regarding ci?15:43
njohnstonglad things are looking pretty stable going into the U release15:44
slaweqnjohnston: yes, IMO it is in better shape recently15:45
slaweqwe don't have many new issues15:45
njohnstonnothing else from me15:45
slaweqand patches are generally merged (usually) pretty fast15:45
slaweqok, if there is nothing else, I will give You almost 15 minutes back :)15:46
slaweqthx for attending the meeting and for taking care of our CI15:46
slaweqo/15:46
slaweq#endmeeting15:46
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:46
openstackMeeting ended Wed Apr  8 15:46:31 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:46
njohnstono/15:46
maciejjozefczyk\o bb15:46
ralonsohbye15:46
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-04-08-15.00.html15:46
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-04-08-15.00.txt15:46
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-04-08-15.00.log.html15:46
bcafarelo/15:46
*** yamamoto has joined #openstack-meeting-315:55
*** yamamoto has quit IRC16:09
*** psachin has quit IRC16:24
*** ralonsoh has quit IRC18:01
*** diablo_rojo has quit IRC18:02
*** diablo_rojo has joined #openstack-meeting-318:06
*** e0ne has quit IRC18:21
*** diablo_rojo has quit IRC18:53
*** slaweq_ has joined #openstack-meeting-318:59
*** slaweq has quit IRC19:00
*** maciejjozefczyk has quit IRC19:24
*** yamamoto has joined #openstack-meeting-320:07
*** yamamoto has quit IRC20:13
*** igordc has quit IRC20:17
*** igordc has joined #openstack-meeting-320:18
*** jraju__ has quit IRC21:07
*** slaweq_ has quit IRC21:38
*** slaweq_ has joined #openstack-meeting-322:13

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!