Wednesday, 2020-04-22

*** apetrich has quit IRC02:09
*** ricolin has joined #openstack-meeting-302:33
*** psachin has joined #openstack-meeting-303:31
*** yamamoto has quit IRC04:18
*** negronjl has quit IRC04:22
*** negronjl has joined #openstack-meeting-304:24
*** yamamoto has joined #openstack-meeting-304:37
*** bnemec has quit IRC05:22
*** ricolin has quit IRC05:51
*** ricolin has joined #openstack-meeting-305:53
*** belmoreira has joined #openstack-meeting-306:10
*** belmoreira has quit IRC06:12
*** belmoreira has joined #openstack-meeting-306:23
*** psachin has quit IRC06:26
*** psachin has joined #openstack-meeting-306:44
*** slaweq has joined #openstack-meeting-306:58
*** slaweq has quit IRC07:02
*** apetrich has joined #openstack-meeting-307:04
*** slaweq has joined #openstack-meeting-307:05
*** ricolin has quit IRC07:06
*** maciejjozefczyk has joined #openstack-meeting-307:08
*** ricolin has joined #openstack-meeting-307:13
*** mdelavergne has joined #openstack-meeting-307:22
*** ralonsoh has joined #openstack-meeting-307:25
ttxo/08:00
mdelavergneHi!08:00
ttx#startmeeting large_scale_sig08:00
openstackMeeting started Wed Apr 22 08:00:12 2020 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.08:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.08:00
*** openstack changes topic to " (Meeting topic: large_scale_sig)"08:00
openstackThe meeting name has been set to 'large_scale_sig'08:00
ttx#topic Rollcall08:00
*** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)"08:00
ttxCalling all Large Scale SIG meeting participants!08:00
amorinhello!08:00
ttxmdelavergne: how are things?08:00
mdelavergnefine and you ?08:00
belmoreirao/08:00
ttxall fine!08:00
ttxOur agenda for today is at:08:01
ttx#link https://etherpad.openstack.org/p/large-scale-sig-meeting08:01
*** links has joined #openstack-meeting-308:01
ttxLet's wait a minute to give masahito a chance to join08:01
ttx#topic Progress on "Documenting large scale operations" goal08:02
*** openstack changes topic to "Progress on "Documenting large scale operations" goal (Meeting topic: large_scale_sig)"08:02
ttx#link https://etherpad.openstack.org/p/large-scale-sig-documentation08:02
ttxWe had one TODO left for amorin from last meeting (propose patch against Nova doc)08:02
ttxany progress there?08:02
amorinunfortunately no08:03
amorinI wasnt able to move forward on that topic08:03
ttxit's ok, I'll push it back08:03
ttx#action amorin to propose patch against Nova doc08:03
amorinthanks08:03
ttxanything else on that topic? things we should be doing? help needed?08:03
amorinhum08:04
amorinmaybe if some of you are interested in filling the wiki page08:04
ttxbelmoreira: I was wondering if CERN could share scrubbed config files, and if that would be interesting08:04
amorinthis one:08:05
amorinhttps://wiki.openstack.org/wiki/Large_Scale_Configuration_Guidelines08:05
mdelavergnethis would be really nice indeed08:05
ttxI don;t know if the "everything we do is public" extends to config files though :)08:05
belmoreiranot sure how interesting that is08:05
amorinmaybe not all config files, but the key elements that make your openstack scale in a better way08:06
belmoreirain my opinion it would be better to point out an issue and why we use a particular option08:06
amorinyes08:06
ttxok, so the "scaling story" is probably more interesting08:06
ttxwe'll get to that in next topic08:07
ttx#topic Progress on "Scaling within one cluster" goal08:07
*** openstack changes topic to "Progress on "Scaling within one cluster" goal (Meeting topic: large_scale_sig)"08:07
ttx#link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling08:07
belmoreiraI'm afraid we don't anything interesting in the config. Is more an architecture decision08:07
ttxmasahito is not around so I'll push back his work item08:08
ttx#action masahito to prepare oslo.metric POC code release08:08
ttxRegarding scaling stories, I did create a wiki page for long-term storage of submitted stories08:08
ttx#link https://wiki.openstack.org/wiki/Large_Scale_Scaling_Stories08:08
ttxPeople can/should still submit them to https://etherpad.openstack.org/p/scaling-stories, as it's simpler to dump content there08:08
ttxbelmoreira: do you still plan to submit one ?08:08
belmoreirayes, I added some notes, still WIP08:08
belmoreirait should be there08:09
ttxoh, I see it08:09
ttxI'll let you continue working on it before copying it to teh wiki08:09
belmoreirawe will also write a blog post about this08:09
ttx#action belmoreira to continue working on scaling story on https://etherpad.openstack.org/p/scaling-stories08:09
mdelavergnenice!08:10
ttxoneswig also had one story planned around bare metal cluster scaling08:10
ttxbut he has not been around much lately. I'll try to reach out to him off-meeting08:10
ttx#action ttx to talk to oneswig about his bare metal cluster scaling story work item08:11
ttxOK, anything else on that topic?08:11
ttxok, next up...08:12
ttx#topic Large scale operations now-virtual OpenDev track08:12
*** openstack changes topic to "Large scale operations now-virtual OpenDev track (Meeting topic: large_scale_sig)"08:12
ttx#link https://etherpad.openstack.org/p/LargeScaleOps_OpenDev08:12
ttxThe date is now set: June 29 - July 108:12
ttxSo please all block those dates in your calendar!08:12
ttxThe goal is really to start a longer discussion, and continue it in the SIG08:13
ttxsince we won;t be able to cover that much during the event08:13
ttxbelmoreira: anything to add?08:13
belmoreirano08:13
belmoreirathanks08:13
ttx#topic Next meeting08:14
*** openstack changes topic to "Next meeting (Meeting topic: large_scale_sig)"08:14
ttxNext sync point in two weeks ?08:14
amorinyup08:14
ttx#info next meeting: May 6, 8:00UTC08:14
ttxOh, I wanted to ask...08:14
ttxDo you find the meeting summaries and reminders I post to the mailing-list useful ?08:14
ttxIf not, I can stop writing them :)08:14
mdelavergneI read them, but if it's too much work I can read the pads :p08:15
amorinI dont receive them, I might have an issue with my rules08:15
mdelavergneI don't know if it attracts people though08:15
belmoreirattx they are really informative for everyone that doesn't attend the meeting08:15
amorinanyway it's a good idea I think08:15
ttxI think it's good to attract potential new members to teh SIG, but if that's the only goal maybe a monthly summary of large scale SIG progress would be enough08:16
ttxI don't want to create fatigue and train people to ignore them08:16
mdelavergneyep, one per month might be sufficient to attract people08:16
belmoreirattx I see a lot of value in them08:16
ttxOK so meeting summaries are useful08:17
ttxMaybe the Monday meeting reminder is too much08:17
ttxunless that is the way you remember the meeting :)08:17
mdelavergneMaybe if you do only one per month you could put the calendar for the next meetings, just in case people want to join in08:18
belmoreira:) if they don't create a lot of friction for you I think both are good08:18
ttxno it's not too much work. I'll keep them. Just doublechecking they are worth it :)08:18
ttxThat is all I had for today... anything else we should discuss?08:19
ttxI'll take that as a no.... Thanks everyone! Have a great day!08:20
ttx#endmeeting08:20
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"08:20
amorinthanks!08:20
openstackMeeting ended Wed Apr 22 08:20:34 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)08:20
openstackMinutes:        http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-04-22-08.00.html08:20
mdelavergneThanks everyone!08:20
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-04-22-08.00.txt08:20
openstackLog:            http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-04-22-08.00.log.html08:20
*** mdelavergne has quit IRC08:46
*** e0ne has joined #openstack-meeting-308:56
*** oneswig has joined #openstack-meeting-308:59
*** witek has joined #openstack-meeting-309:02
*** masahito has joined #openstack-meeting-309:05
*** ricolin has quit IRC09:22
*** ricolin has joined #openstack-meeting-309:23
*** ricolin has quit IRC09:24
*** ricolin has joined #openstack-meeting-309:25
*** ricolin has quit IRC09:25
*** ricolin_ has joined #openstack-meeting-309:25
*** e0ne has quit IRC09:34
*** e0ne has joined #openstack-meeting-309:35
*** ricolin_ is now known as ricolin09:36
*** yamamoto has quit IRC09:57
*** masahito has quit IRC10:28
*** yamamoto has joined #openstack-meeting-310:34
*** yamamoto has quit IRC10:44
*** yamamoto has joined #openstack-meeting-310:59
*** yamamoto has quit IRC11:03
*** njohnston has quit IRC11:14
*** njohnston has joined #openstack-meeting-311:18
*** yamamoto has joined #openstack-meeting-311:22
*** yamamoto has quit IRC11:25
*** yamamoto has joined #openstack-meeting-311:25
*** raildo has joined #openstack-meeting-311:53
*** oneswig has quit IRC12:02
*** yamamoto has quit IRC12:15
*** yamamoto has joined #openstack-meeting-312:48
*** yamamoto has quit IRC12:56
*** psachin has quit IRC13:14
*** e0ne has quit IRC13:44
*** e0ne has joined #openstack-meeting-313:44
*** yamamoto has joined #openstack-meeting-313:47
*** yamamoto has quit IRC13:58
*** e0ne has quit IRC14:00
*** e0ne has joined #openstack-meeting-314:01
*** yamamoto has joined #openstack-meeting-314:01
*** yamamoto has quit IRC14:01
*** liuyulong has joined #openstack-meeting-314:01
*** yamamoto has joined #openstack-meeting-314:01
*** bnemec has joined #openstack-meeting-314:01
*** yamamoto has quit IRC14:02
*** yamamoto has joined #openstack-meeting-314:03
*** liuyulong has quit IRC14:13
*** links has quit IRC14:49
slaweq#startmeeting neutron_ci15:00
openstackMeeting started Wed Apr 22 15:00:28 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
slaweqhi15:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:00
openstackThe meeting name has been set to 'neutron_ci'15:00
bcafarelo/15:01
ralonsohhi15:01
maciejjozefczykhey15:01
njohnstono/15:02
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:02
slaweqok, lets start15:02
slaweq#topic Actions from previous meetings15:02
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:02
slaweqralonsoh: check ovn jobs failure15:03
slaweqfix is already merged: https://review.opendev.org/#/c/720248/15:03
slaweqthx ralonsoh :)15:03
ralonsohyw15:03
slaweqsecond one15:03
slaweqslaweq: ping yamamoto about midonet gate problems15:03
slaweqI still don't have any reply from yamamoto15:03
slaweqI will keep trying and maybe take a look at those broken UTs if I will have few minutes15:04
slaweqand that's all for actions from last week15:04
slaweq#topic Stadium projects15:04
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:04
slaweqstandardize on zuul v315:04
slaweqEtherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop15:04
slaweqI don't think there was any update on this last week15:05
njohnstonhttps://review.opendev.org/#/c/672925 is failing on the networking-odl-functional jobs,. I am looking into it15:05
slaweqnjohnston: I just wanted to mention to try https://review.opendev.org/#/c/715439/3 but I think You already did15:05
slaweqso with this patch only midonet will still be not done, right?15:06
njohnstonLajos did that in PS 31 I think15:06
njohnstonslaweq: Yes I believe so15:06
slaweqthat's great15:07
slaweqthx njohnston for update15:07
slaweqaccording to CI issues in stadium projects, we have still this midonet bug15:07
slaweqother than that I think that stadium projects are running pretty fine15:08
slaweqdo You have anything to add/ask regarding stadium?15:08
njohnstonnothing here15:08
bcafarelsmall change in neutron-tempest-plugin, some jobs that were still using all-plugin tox target now use the standard "all"15:09
bcafarel(fixing this deprecated one was needed to make https://review.opendev.org/#/c/721277/ passing)15:09
slaweqbcafarel: and that will help us to not have issues like we had with Stein jobs recently, right?15:09
bcafarelyes, and hopefully should not have any side effect :)15:10
slaweqbcafarel++ thx15:10
slaweqok, and with that I think we can move to the next topic which is15:11
slaweq#topic Stable branches15:11
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)"15:11
slaweq:)15:11
slaweqTrain dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=115:11
slaweqStein dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=115:11
slaweqexcept this issue with neutron-tempest-plugin jobs on Stein it looks ok for me15:12
bcafarelyep and that stein fix is now in15:12
slaweqyes, I saw :) thx once again for taking care of this15:13
slaweqok, I think we can move on to next topic, right?15:14
njohnston+115:15
bcafarelyes15:15
slaweq#topic Grafana15:16
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:16
slaweqin overall I think that it looks ok here too15:17
njohnstonBTW what is with the points appearing in the "Number of integrated Tempest jobs runs (Gate queue)" for the "24 hours" line?  Seems like an error in the grafana config.15:17
slaweqnjohnston: are You asking about lack of data there?15:18
njohnston"24 hours" doesn't sound like a zuul job name15:19
*** links has joined #openstack-meeting-315:19
njohnstonso I don't know why it would be in the data set15:19
bcafarelhmm I don't have it, refreshing15:19
slaweqme neighter :)15:20
ralonsohnope15:20
njohnstonweird, I just refreshed it and it went away15:20
njohnstonthat is super weird15:20
njohnstonok, never mind :-)15:20
bcafarel:) you scared it away15:20
slaweqLOL15:20
ralonsohzuul! the source of problems and solutions15:20
slaweqok, so we at least solved one problem today ;P15:20
njohnstonLOL15:21
slaweqfrom the other things, I saw today that we have a lot of non-voting jobs in check queue15:21
slaweqmaybe we should think about promoting some of them to be voting?15:22
slaweqI'm not saying about doing it now but after we will cut stable/ussuri15:22
njohnstonI was going to suggest that openstacksdk-functional-devstack-networking might be a good candidate15:22
ralonsohright15:22
ralonsohcan we wait until V?15:22
slaweqralonsoh: yes :)15:22
njohnstonand openstack-tox-py38 is another good candidate.  No reason to move quickly though, I agree it would be good to wait until V15:23
ralonsoh+1 to the idea15:23
slaweqin next weeks I will prepare some data and proposals about what we can promote15:23
ralonsohagree15:23
bcafarelsounds good some of these look interesting15:23
slaweqthx15:23
bcafarellike neutron-tempest-with-uwsgi (just quick pick in the list)15:23
njohnstonneutron-ovn-tempest-slow has <10% failures for some time it looks like15:24
slaweqbcafarel: this one is actually already voting IIRC, we just still didn't merge https://review.opendev.org/#/c/718392/15:25
bcafareloh I thought I had seen it going in nvm then15:25
bcafarelok I just need to have my eyes sight, it was just Andreas' +215:25
slaweq:)15:26
slaweqI added frickler to it today so hopefully he will check it soon15:26
slaweqok, I think we can continue with other topics now15:27
slaweq#topic fullstack/functional15:27
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"15:27
slaweqtoday I found one new example of timeout in neutron.tests.functional.agent.linux.test_keepalived.KeepalivedManagerTestCase15:27
slaweqhttps://a1c2986e7388db3a1401-541b7a48fdccc7de277eccd1d7d5bce5.ssl.cf5.rackcdn.com/718690/2/check/neutron-functional/9f51d60/testr_results.html15:27
slaweqbut this time it's during creation of interface15:27
slaweqralonsoh: does it rings a bell for You?15:28
*** e0ne has quit IRC15:28
ralonsohthis is during the interface creation, not the namespace15:28
slaweqisn't that this GIL related issue which You were fixing recently?15:28
ralonsohbut I can take a look at it15:28
ralonsohnope15:28
*** e0ne has joined #openstack-meeting-315:28
slaweqok, I though that maybe it may be same/similar root cause15:29
slaweqthx for taking care of it :)15:29
slaweq#action ralonsoh to check timeout during interface creation in functional tests15:29
slaweqand I also found 2 ovn related issues in functional tests:15:30
slaweqhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_abb/717851/2/gate/neutron-functional/abb91cb/testr_results.html15:30
slaweqhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e2c/717083/6/check/neutron-functional/e2c63e4/testr_results.html15:30
slaweqmaciejjozefczyk: does it rings a bell for You maybe?15:30
maciejjozefczykslaweq, looks like the same story that the timeout change should solve15:31
slaweqmaciejjozefczyk: do You have link to patch?15:31
maciejjozefczyk#link https://review.opendev.org/#/c/717704/15:32
maciejjozefczykbut looks like it is still the case or something smiliar15:32
slaweqmaciejjozefczyk: yes, at least one of those failures is from this week15:33
slaweqso both probably have this timeouts patch already15:33
maciejjozefczykI'm gonna take a look on those two15:34
slaweqthx maciejjozefczyk15:34
maciejjozefczykand reopen bug: https://bugs.launchpad.net/neutron/+bug/186811015:34
openstackLaunchpad bug 1868110 in neutron "[OVN] neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log randomly fails" [High,Fix released] - Assigned to Maciej Jozefczyk (maciej.jozefczyk)15:34
slaweq#action maciejjozefczyk to take a look at ovn related functional test failures15:34
slaweqand that's all from me regarding functional/fullstack tests15:35
slaweqanything else You want to add?15:35
ralonsohno15:35
slaweqso lets move on15:37
slaweq#topic Tempest/Scenario15:37
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"15:37
slaweqfirst of all15:37
slaweqI proposed patch https://review.opendev.org/#/c/721805/ to enable l3_ha in scenario jobs15:37
slaweqIMO we lack of L3 HA coverage in our CI15:37
slaweqand that would be easy way to have it covered somehow15:37
slaweqeven if those are singlenode jobs, it will spawn keepalived, and all that stuff for each router15:38
ralonsohwe'll catch "functional" problems with keepalived15:38
ralonsoh+115:38
slaweqyep15:38
njohnstonvery good15:39
bcafarelsounds good, does that reduce coverage on non-l3 ha?15:39
slaweqin fact I got this idea when I did similar patch for tripleo standalone job yesterday15:39
slaweqand I found new bug with keepalived 2.x with it immediately :)15:39
bcafarel(not the code part I know the best)15:39
slaweqbcafarel: we still have tempest jobs for legacy, non-ha routers15:39
bcafarelok then no objection at all!15:40
slaweqbut IMO we should focus more on testing L3HA as IMO it's more used than legacy routers15:40
*** belmoreira has quit IRC15:40
njohnstonagreed, definitely15:40
slaweqthx for supporting this :)15:41
slaweqfrom other things related to scenario jobs, I found one issue recently15:41
slaweqin neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle - timeout while waiting for port to be ACTIVE15:41
slaweqhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d8c/717851/2/check/neutron-ovn-tempest-ovs-release/d8c0282/testr_results.html15:41
slaweqanyone wants to take a look into this?15:42
slaweqit's in ovn job15:42
maciejjozefczykslaweq, I think Jakub was checking it in d/s15:42
maciejjozefczykslaweq, I'll create a lp and check with Jakub15:43
slaweqmaciejjozefczyk: thx a lot15:43
slaweq#action maciejjozefczyk to report LP regarding failing neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle in neutron-ovn-tempest-ovs-release job15:44
slaweqok, and that's all what I have for today15:44
slaweqperiodic jobs are working fine15:44
slaweqanything else You want to discuss today?15:44
slaweqif not, I think we can finish a bit earlier today15:46
slaweqthx for attending15:46
slaweqo/15:46
bcafarelo/15:46
slaweq#endmeeting15:46
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:46
openstackMeeting ended Wed Apr 22 15:46:13 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:46
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-04-22-15.00.html15:46
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-04-22-15.00.txt15:46
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-04-22-15.00.log.html15:46
njohnstono/15:46
ralonsohbye15:46
maciejjozefczykbye15:46
*** links has quit IRC15:58
*** igordc has joined #openstack-meeting-316:12
*** links has joined #openstack-meeting-316:20
*** e0ne has quit IRC16:47
*** e0ne has joined #openstack-meeting-316:48
*** e0ne has quit IRC16:56
*** e0ne has joined #openstack-meeting-316:56
*** links has quit IRC17:10
*** witek has quit IRC17:38
*** ralonsoh has quit IRC17:56
*** e0ne has quit IRC18:34
*** e0ne has joined #openstack-meeting-318:34
*** e0ne_ has joined #openstack-meeting-318:43
*** e0ne has quit IRC18:43
*** e0ne_ has quit IRC19:04
*** diablo_rojo has joined #openstack-meeting-319:41
*** raildo has quit IRC21:37
*** haleyb has quit IRC21:47
*** haleyb has joined #openstack-meeting-321:50
*** slaweq has quit IRC21:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!