| *** jamesmcarthur has joined #openstack-meeting-3 | 00:37 | |
| *** jamesmcarthur has quit IRC | 01:17 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 02:33 | |
| *** baojg has joined #openstack-meeting-3 | 03:03 | |
| *** jamesmcarthur has quit IRC | 03:06 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 03:06 | |
| *** jamesmcarthur has quit IRC | 03:11 | |
| *** baojg has quit IRC | 03:18 | |
| *** dconde has joined #openstack-meeting-3 | 03:21 | |
| *** dconde has quit IRC | 03:21 | |
| *** dconde_ has joined #openstack-meeting-3 | 03:23 | |
| *** dconde has joined #openstack-meeting-3 | 03:27 | |
| *** dconde has left #openstack-meeting-3 | 03:27 | |
| *** dconde_ has quit IRC | 03:28 | |
| *** psachin has joined #openstack-meeting-3 | 03:36 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 03:41 | |
| *** openstack has joined #openstack-meeting-3 | 04:22 | |
| *** ChanServ sets mode: +o openstack | 04:23 | |
| *** jamesmcarthur has quit IRC | 05:04 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 05:05 | |
| *** jamesmcarthur has quit IRC | 05:11 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 05:11 | |
| *** diablo_rojo has quit IRC | 05:57 | |
| *** jamesmcarthur has quit IRC | 06:00 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 06:01 | |
| *** jamesmcarthur has quit IRC | 06:06 | |
| *** yamamoto has joined #openstack-meeting-3 | 06:18 | |
| *** yamamoto has quit IRC | 06:24 | |
| *** rubasov has quit IRC | 06:39 | |
| *** rubasov has joined #openstack-meeting-3 | 06:39 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 06:39 | |
| *** ralonsoh has joined #openstack-meeting-3 | 06:45 | |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 06:52 | |
| *** belmoreira has joined #openstack-meeting-3 | 06:53 | |
| *** slaweq has joined #openstack-meeting-3 | 06:56 | |
| *** lpetrut has joined #openstack-meeting-3 | 07:19 | |
| *** witek has joined #openstack-meeting-3 | 07:25 | |
| *** openstack has joined #openstack-meeting-3 | 07:49 | |
| *** ChanServ sets mode: +o openstack | 07:49 | |
| *** mdelavergne has joined #openstack-meeting-3 | 08:00 | |
| ttx | o/ | 08:01 |
|---|---|---|
| mdelavergne | Hi! | 08:01 |
| ttx | #startmeeting large_scale_sig | 08:01 |
| openstack | Meeting started Wed May 6 08:01:22 2020 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. | 08:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 08:01 |
| *** openstack changes topic to " (Meeting topic: large_scale_sig)" | 08:01 | |
| ttx | #topic Rollcall | 08:01 |
| openstack | The meeting name has been set to 'large_scale_sig' | 08:01 |
| *** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)" | 08:01 | |
| ttx | Who is here for the Large Scale SIG meeting ? | 08:01 |
| ttx | amorin, belmoreira maybe | 08:02 |
| *** masahito has joined #openstack-meeting-3 | 08:02 | |
| belmoreira | o/ | 08:02 |
| masahito | o/ | 08:02 |
| mdelavergne | and me, hi everyone | 08:02 |
| ttx | mdelavergne: hi! | 08:02 |
| ttx | Our agenda for today is at: | 08:02 |
| masahito | I completely forgot the time change at last meeting :-0 | 08:03 |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-meeting | 08:03 |
| ttx | masahito: it should not change anymore :) | 08:03 |
| ttx | #topic Progress on "Documenting large scale operations" goal | 08:03 |
| *** openstack changes topic to "Progress on "Documenting large scale operations" goal (Meeting topic: large_scale_sig)" | 08:03 | |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-documentation | 08:03 |
| ttx | We had one TODO left for amorin from last meeting (propose patch against Nova doc) | 08:04 |
| ttx | I don't think there was progress on that yet ? | 08:04 |
| ttx | amorin: what's blocking further progress on that goal? Is it more base information (like more scaling stories with config details), or more people to help with pushing the patches and writing the doc? | 08:06 |
| ttx | (trying to see what we should look for) | 08:06 |
| ttx | OK let's pass since amorin is not around, and discuss that next meeting | 08:08 |
| ttx | I'll push back the action item | 08:09 |
| ttx | #action amorin to propose patch against Nova doc | 08:09 |
| ttx | #topic Progress on "Scaling within one cluster" goal | 08:09 |
| *** openstack changes topic to "Progress on "Scaling within one cluster" goal (Meeting topic: large_scale_sig)" | 08:09 | |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling | 08:09 |
| ttx | masahito: progress on oslo.metrics POC code release ? | 08:09 |
| masahito | almost ready. | 08:09 |
| ttx | Great! | 08:09 |
| ttx | I'll push back the item | 08:10 |
| masahito | I'm wondering where I will push the oslo.metrics side code. | 08:10 |
| masahito | Is there any play ground repo about this activity? | 08:10 |
| ttx | So you can push it on github, and we can create the Opendev repository from it | 08:10 |
| ttx | It's usually the simplest | 08:10 |
| ttx | I could also create an empty repository and you would propose a single change | 08:11 |
| ttx | what would be your preference? | 08:11 |
| masahito | Okay, my preference is empty repo under openstack namespace because there is a co-auther in the oslo.metrics. It's easy to add his name in the patch :-) | 08:13 |
| ttx | ok, I can work to create that empty repo | 08:13 |
| masahito | Thank you, so much. | 08:13 |
| ttx | I'll have that ready by next meeting, or before if you think that will be ready before then | 08:13 |
| ttx | will keep you posted | 08:14 |
| masahito | depending on the date of next meeting :) | 08:14 |
| ttx | #action ttx to create an empty oslo-metrics repository | 08:14 |
| masahito | I will be able to push the code in next two week. | 08:14 |
| ttx | #action masahito to finalize oslo.metric POC code release | 08:14 |
| ttx | OK, then I'll just let you know when the repository is ready | 08:15 |
| ttx | I talked to oneswig about his bare metal cluster scaling story | 08:15 |
| ttx | That project has been delayed, he will come back to us once he made progress. | 08:15 |
| ttx | In the mean time we'll stop tracking it in the meeting | 08:15 |
| ttx | Anything else on this topic? | 08:15 |
| mdelavergne | not from me | 08:15 |
| ttx | OK, let's move to the virtual events... | 08:16 |
| ttx | #topic Opendev event on Large scale operations, June 29 - July 1 | 08:16 |
| *** openstack changes topic to "Opendev event on Large scale operations, June 29 - July 1 (Meeting topic: large_scale_sig)" | 08:16 | |
| ttx | #link https://etherpad.openstack.org/p/LargeScaleOps_OpenDev | 08:16 |
| ttx | belmoreira, masahito: Did you finalize the list of topics that will be discussed? | 08:16 |
| ttx | the etherpad has a bunch of suggestions | 08:16 |
| belmoreira | yes, we have a schedule. I think Allison will send the schedule today to the ML | 08:16 |
| ttx | How do you think we can best leverage that event to recruit new members for the SIG? | 08:17 |
| ttx | I was thinking... maybe we should organize followup discussions at each of the SIG meetings after the event, or something | 08:17 |
| ttx | Make the SIG the natural place to "continue the discussion" | 08:18 |
| belmoreira | I agree | 08:18 |
| belmoreira | During the event the SIG should also be mentioned | 08:19 |
| belmoreira | I think this will come naturally | 08:19 |
| ttx | So... during the event, it would be great to identify people that are the most engaged and potentially willing to help, and get them to leave their email addresses so that we can invite them to the followup discussions | 08:19 |
| ttx | I suspect some topics will be more... conducive to followup discussions than others | 08:20 |
| ttx | so maybe we won;t need to follow up on every opendev topic | 08:20 |
| ttx | will be good to identify the hottest topics that we should follow up on | 08:20 |
| belmoreira | I will see the engagement in each topic and we should focus in the popular ones | 08:21 |
| belmoreira | you write faster than me :) | 08:21 |
| ttx | we still have time to organize, it's two months from now | 08:21 |
| ttx | anything else on opendev? | 08:22 |
| ttx | #topic PTG participation | 08:22 |
| *** openstack changes topic to "PTG participation (Meeting topic: large_scale_sig)" | 08:22 | |
| ttx | A bit more urgent... | 08:22 |
| ttx | We have a virtual Project Teams Gathering coming up, to get teams to spend some (virtual) time together to plan future work and get things done | 08:22 |
| ttx | As a SIG we could sign up for some meeting slots if we think that would be useful | 08:23 |
| ttx | The deadline is end of this week | 08:23 |
| mdelavergne | So quite urgent :D | 08:23 |
| ttx | I'm not sure we need more meetings at this point, at least not before teh Opendev | 08:23 |
| ttx | the* | 08:23 |
| ttx | what's your take on it? | 08:24 |
| ttx | Would we benefit from a couple of hours in a videomeeting? | 08:24 |
| ttx | (June 1-5) | 08:24 |
| *** felixhuettner has joined #openstack-meeting-3 | 08:26 | |
| ttx | belmoreira, masahito, mdelavergne: thoughts? | 08:26 |
| mdelavergne | From my part I don't know if I will be able to participate in the upcoming events, so I prefer not to give my opinion on this | 08:26 |
| ttx | ok | 08:27 |
| belmoreira | I don't have a strong opinion, but I think it will be difficult to keep people engaged in more video calls | 08:27 |
| ttx | I think we should focus on Opendev, personally | 08:27 |
| masahito | Same for me. | 08:27 |
| ttx | It will be hard enough :) | 08:27 |
| ttx | and has much more potential for growing the group | 08:27 |
| ttx | #info Large Scale SIG does not need PTG space, will focus on Opendev | 08:28 |
| ttx | #topic Next meeting | 08:28 |
| *** openstack changes topic to "Next meeting (Meeting topic: large_scale_sig)" | 08:28 | |
| ttx | Two weeks from now places us on... | 08:28 |
| ttx | #info next meeting: May 20, 8:00UTC | 08:28 |
| ttx | I think that is all for today! | 08:28 |
| ttx | Last words, anyone? | 08:29 |
| ttx | Alright then, have a great week! Thanks for attending! | 08:29 |
| mdelavergne | thanks everyone! | 08:29 |
| *** felixhuettner has left #openstack-meeting-3 | 08:29 | |
| masahito | thanks. bye | 08:29 |
| belmoreira | Thanks | 08:30 |
| ttx | #endmeeting | 08:30 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 08:30 | |
| openstack | Meeting ended Wed May 6 08:30:01 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 08:30 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-05-06-08.01.html | 08:30 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-05-06-08.01.txt | 08:30 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-05-06-08.01.log.html | 08:30 |
| *** e0ne has joined #openstack-meeting-3 | 08:30 | |
| *** mdelavergne has quit IRC | 08:32 | |
| *** masahito has quit IRC | 08:34 | |
| *** baojg has joined #openstack-meeting-3 | 08:40 | |
| *** jamesmcarthur has quit IRC | 08:41 | |
| *** elod is now known as elod_pto | 08:43 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 09:16 | |
| *** jamesmcarthur has quit IRC | 09:26 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 09:57 | |
| *** jamesmcarthur has quit IRC | 10:07 | |
| *** baojg has quit IRC | 10:18 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 10:43 | |
| *** jamesmcarthur has quit IRC | 10:43 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 10:43 | |
| *** maciejjozefczyk_ has joined #openstack-meeting-3 | 10:43 | |
| *** jamesmcarthur has quit IRC | 10:44 | |
| *** maciejjozefczyk has quit IRC | 10:44 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 10:44 | |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 10:44 | |
| *** maciejjozefczyk_ has quit IRC | 10:48 | |
| *** maciejjozefczyk_ has joined #openstack-meeting-3 | 10:49 | |
| *** maciejjozefczyk has quit IRC | 10:50 | |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 10:50 | |
| *** jamesmcarthur has quit IRC | 10:51 | |
| *** maciejjozefczyk_ has quit IRC | 10:53 | |
| *** maciejjozefczyk_ has joined #openstack-meeting-3 | 10:55 | |
| *** maciejjozefczyk has quit IRC | 10:55 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 10:58 | |
| *** jamesmcarthur has quit IRC | 10:59 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 10:59 | |
| *** jamesmcarthur has quit IRC | 11:01 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 11:02 | |
| *** maciejjozefczyk_ has quit IRC | 11:04 | |
| *** maciejjozefczyk_ has joined #openstack-meeting-3 | 11:05 | |
| *** jamesmcarthur has quit IRC | 11:07 | |
| *** raildo has joined #openstack-meeting-3 | 11:37 | |
| *** witek has quit IRC | 11:39 | |
| *** yamamoto has joined #openstack-meeting-3 | 11:40 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 11:40 | |
| *** yamamoto has quit IRC | 11:42 | |
| *** jamesmcarthur has quit IRC | 12:25 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 13:01 | |
| *** ralonsoh has quit IRC | 13:08 | |
| *** ralonsoh has joined #openstack-meeting-3 | 13:11 | |
| *** jamesmcarthur has quit IRC | 13:21 | |
| *** maciejjozefczyk_ is now known as maciejjozefczyk | 13:37 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 14:01 | |
| *** jamesmcarthur has quit IRC | 14:22 | |
| *** belmoreira has quit IRC | 14:25 | |
| *** lpetrut has quit IRC | 14:30 | |
| *** belmoreira has joined #openstack-meeting-3 | 14:44 | |
| *** lajoskatona has joined #openstack-meeting-3 | 14:45 | |
| *** belmoreira has quit IRC | 14:54 | |
| *** belmoreira has joined #openstack-meeting-3 | 14:55 | |
| slaweq | #startmeeting neutron_ci | 15:00 |
| openstack | Meeting started Wed May 6 15:00:01 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| slaweq | hi | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:00 | |
| openstack | The meeting name has been set to 'neutron_ci' | 15:00 |
| njohnston | o/ | 15:00 |
| *** lpetrut has joined #openstack-meeting-3 | 15:00 | |
| bcafarel | o/ | 15:01 |
| ralonsoh | hi | 15:02 |
| slaweq | ok, I think we can start | 15:02 |
| slaweq | #topic Actions from previous meetings | 15:02 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 15:02 | |
| slaweq | first one | 15:02 |
| slaweq | maciejjozefczyk to take a look at ovn related functional test failures | 15:02 |
| lajoskatona | Hi | 15:02 |
| slaweq | I pinged maciejjozefczyk on neutron channel | 15:03 |
| slaweq | maybe he will join soon | 15:03 |
| maciejjozefczyk | hey | 15:04 |
| maciejjozefczyk | ok, so I took a look on those failures for functional tests | 15:04 |
| maciejjozefczyk | #link https://bugs.launchpad.net/neutron/+bug/1868110 | 15:06 |
| openstack | Launchpad bug 1868110 in neutron "[OVN] neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log randomly fails" [High,In progress] - Assigned to Maciej Jozefczyk (maciej.jozefczyk) | 15:06 |
| maciejjozefczyk | I found that the test from this class failed only once recently, on the same which is: TimeoutException | 15:07 |
| maciejjozefczyk | So i think we have 2 possible solution: let it be, as per 1 failure per week is not that much | 15:07 |
| maciejjozefczyk | or bump the timeout value | 15:07 |
| bcafarel | what's the current timeout? | 15:08 |
| slaweq | didn't we lower this timeout already to solve this problem? :) | 15:08 |
| maciejjozefczyk | I think that after bumping it from very litle value to 15 seconds we already have less occurance of this test failures | 15:08 |
| maciejjozefczyk | slaweq, bcafarel default in the neutron configuration is 180 seconds | 15:08 |
| bcafarel | oh right that was the one that was bumped | 15:09 |
| maciejjozefczyk | we had issues while it was 5 seconds set for functional tests, we updated to 15 seconds and I found only particular failure of this test class | 15:09 |
| slaweq | ahh, it was 5 seconds and You bumped it to 15, right? | 15:09 |
| slaweq | why not use default value? | 15:09 |
| maciejjozefczyk | slaweq, I think that the author a few years ago wanted to not have that high timeout value for functional tests because it causes the whole job to timeout | 15:09 |
| maciejjozefczyk | but we can try setting it to default | 15:10 |
| slaweq | so maybe we can try with e.g. 30 seconds for now and we will see :) | 15:10 |
| *** belmoreira has quit IRC | 15:11 | |
| maciejjozefczyk | slaweq, yes | 15:11 |
| maciejjozefczyk | but for now, as I said, I found only 1 failure of the test, at least I was checking week ago :) | 15:12 |
| slaweq | maciejjozefczyk: ok, IMO we can try to bump it a litte and see - maybe there will be no occurences anymore :) | 15:13 |
| maciejjozefczyk | slaweq, sure | 15:13 |
| slaweq | ralonsoh: njohnston bcafarel what do You think? | 15:13 |
| ralonsoh | +1 | 15:13 |
| ralonsoh | to increase the time | 15:13 |
| njohnston | I agree, it seems reasonable | 15:14 |
| bcafarel | yep, it is still an OK value | 15:14 |
| slaweq | ok, maciejjozefczyk will You take care of it? | 15:14 |
| maciejjozefczyk | slaweq, yes | 15:14 |
| slaweq | thx | 15:14 |
| slaweq | #action maciejjozefczyk will increase timeout in neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log | 15:15 |
| slaweq | ok, next one | 15:15 |
| slaweq | maciejjozefczyk to report LP regarding failing neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle in neutron-ovn-tempest-ovs-release job | 15:15 |
| maciejjozefczyk | #link https://bugs.launchpad.net/neutron/+bug/1874447 | 15:15 |
| openstack | Launchpad bug 1874447 in neutron "[OVN] Tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle fails randomly" [High,Confirmed] | 15:15 |
| slaweq | maciejjozefczyk: thx | 15:16 |
| slaweq | but I think Your comments to this LP are wrong :) | 15:16 |
| maciejjozefczyk | I was investigating it, but at that time I also haven't found any occurance of that failure (afair the logs are rotated after a week?) | 15:16 |
| slaweq | logs are gone after some time, I don't know exactly how long | 15:17 |
| njohnston | I've been doing some experimenting with trunks on the 2 changes you cite, but I don't think I have had enough CI runs on them to account for the statsu you mention. | 15:17 |
| njohnston | *stats | 15:17 |
| slaweq | if You would found similar issues, please link them to this LP | 15:19 |
| slaweq | we will see if that is serious issue | 15:19 |
| maciejjozefczyk | ok slaweq | 15:20 |
| njohnston | +1 | 15:20 |
| slaweq | thx | 15:20 |
| slaweq | ok, next one | 15:20 |
| slaweq | ralonsoh to check timeout during interface creation in functional tests | 15:20 |
| ralonsoh | #link https://review.opendev.org/#/c/722254/ | 15:20 |
| ralonsoh | and a patch in pyroute2 | 15:21 |
| ralonsoh | https://github.com/svinota/pyroute2/issues/702 | 15:21 |
| ralonsoh | now we can pass the PyDLL when creating a namespace context | 15:21 |
| ralonsoh | we need a new release in pyroyte2 0.5.13 | 15:21 |
| ralonsoh | same problem as when calling create/delete namespaces | 15:21 |
| ralonsoh | that's all | 15:22 |
| slaweq | thx a lot | 15:22 |
| ralonsoh | yw | 15:22 |
| slaweq | ok, so that's all from the last week | 15:23 |
| slaweq | lets move to the next topic | 15:23 |
| slaweq | #topic Stadium projects | 15:23 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:23 | |
| *** belmoreira has joined #openstack-meeting-3 | 15:23 | |
| slaweq | standardize on zuul v3 | 15:23 |
| slaweq | Etherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop | 15:23 |
| slaweq | only update which I have is that there is now zuulv3 grenade job | 15:23 |
| slaweq | so we can migrate our grenade jobs too | 15:24 |
| slaweq | it's already done for 2 of the neutron grenade jobs | 15:24 |
| slaweq | I will do the same for neutron-ovn job | 15:24 |
| njohnston | I like the idea of an OVN grenade job to complement the existing ml2/ovs one | 15:24 |
| slaweq | and there is also networking-odl grenade job to migrate | 15:25 |
| njohnston | so that just meanes the networking-odl-grenade job, and the regular zuulv3 transition for networking-midonet. Any word from yamamoto recently? | 15:25 |
| slaweq | njohnston: nope | 15:25 |
| lajoskatona | I started tat and got some advices from Luigi Toscano | 15:25 |
| njohnston | excellent | 15:25 |
| lajoskatona | that -^ | 15:25 |
| slaweq | he only told me that his time for openstack is "very limited" currently | 15:26 |
| slaweq | we will need to review list of stadium projects again during the ptg probably | 15:26 |
| slaweq | thx lajoskatona for taking care of networking-odl | 15:27 |
| lajoskatona | slaweq: np, we use it so its noral way of working ;) | 15:27 |
| lajoskatona | by the way: could you check these please : https://review.opendev.org/#/q/I3aed8031c1b784c16dd7055afd41456ed935b7fc | 15:28 |
| lajoskatona | as I know these are missing to make master and ussuri patches to be merged for reno etc... | 15:28 |
| slaweq | sure, I will | 15:29 |
| slaweq | ok, anything else regarding stadium projects' ci for today? | 15:29 |
| lajoskatona | nothing from me at least | 15:30 |
| njohnston | nope | 15:31 |
| ralonsoh | no | 15:31 |
| *** njohnston is now known as njohnston_ | 15:31 | |
| bcafarel | neutron-dynamic-routing fails tempest on train (though this was the first backport attempt): https://review.opendev.org/#/c/725790/ | 15:31 |
| *** njohnsto_ has joined #openstack-meeting-3 | 15:31 | |
| bcafarel | related to that series of monkeypatch eventlet fixes | 15:32 |
| slaweq | ModuleNotFoundError: No module named 'neutron_tempest_plugin' | 15:32 |
| slaweq | so IMO it's some issue with jobs definition | 15:34 |
| njohnsto_ | yeah | 15:35 |
| *** njohnsto_ is now known as njohnston | 15:35 | |
| bcafarel | yes, in train tests were still in-repo | 15:35 |
| slaweq | those are still legacy jobs | 15:35 |
| *** belmoreira has quit IRC | 15:36 | |
| slaweq | bcafarel: isn't that related to https://review.opendev.org/#/c/721277/3/devstack/plugin.sh ? | 15:37 |
| slaweq | maybe those jobs in stable/train should use some pinned version of neutron-tempest-plugin? | 15:38 |
| *** belmoreira has joined #openstack-meeting-3 | 15:39 | |
| bcafarel | hmm yes it's possible (so the legacy job was actually using the installed version of the plugin?) | 15:39 |
| slaweq | bcafarel: idk really :/ | 15:40 |
| slaweq | bcafarel: will You check that this week? | 15:40 |
| bcafarel | slaweq: I'll try at least :) and worst case fill a LP for it | 15:40 |
| slaweq | thx | 15:40 |
| slaweq | #action bcafarel to check neutron-dynamic-routing failurs in stable/train | 15:41 |
| slaweq | ok, lets move on | 15:41 |
| slaweq | #topic Stable branches | 15:41 |
| *** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)" | 15:41 | |
| slaweq | Train dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1 | 15:41 |
| slaweq | Stein dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1 | 15:41 |
| slaweq | IMHO dashboards looks more or less fine | 15:42 |
| bcafarel | most backports got in without too many rechecks yes | 15:43 |
| bcafarel | designate gate even behaved :) mostly rally/grenade rechecks | 15:43 |
| slaweq | for grenade it should be better when nova will backport fix for scheduler/placement | 15:44 |
| slaweq | in master it now works much better IMO | 15:44 |
| slaweq | ok, next topic then | 15:44 |
| slaweq | #topic Grafana | 15:44 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:44 | |
| slaweq | http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:45 |
| slaweq | here there is more results :) | 15:45 |
| slaweq | *more data | 15:45 |
| slaweq | and I see couple of problems there | 15:45 |
| slaweq | or potential problems | 15:45 |
| slaweq | neutron-ovn-tempest-slow (non-voting) | 15:46 |
| slaweq | neutron-ovn-tempest-full-multinode-ovs-master (non-voting) | 15:46 |
| slaweq | both jobs are going to high failure rate today | 15:46 |
| slaweq | I don't know really if that is some new problem | 15:46 |
| slaweq | in most cases where I see it both jobs are TIMED_OUT | 15:47 |
| slaweq | e.g. https://review.opendev.org/#/c/725836/ | 15:48 |
| slaweq | or https://review.opendev.org/#/c/724445/ | 15:48 |
| ralonsoh | I'll try to track those two jobs to see a possible problem | 15:49 |
| ralonsoh | maybe is just a matter of increasing the job timeout | 15:49 |
| slaweq | I now checked one such job https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_07c/717083/9/check/neutron-ovn-tempest-slow/07c6abc/job-output.txt | 15:49 |
| slaweq | it seems that all tests are failing due to "Failed to allocate the network(s), not rescheduling." | 15:50 |
| slaweq | so it seems like serious issue | 15:50 |
| maciejjozefczyk | I can try to investigate that | 15:50 |
| slaweq | and I think that it's due to something from today or yesterday | 15:50 |
| slaweq | https://zuul.opendev.org/t/openstack/build/07c6abc773a84930b20a5ea1b46f666e/log/controller/logs/screen-q-svc.txt?severity=4 | 15:51 |
| slaweq | I think that this can be culprit :) | 15:51 |
| slaweq | or maybe not :) but at least it's IMO worth to check | 15:52 |
| slaweq | ok, so ralonsoh maciejjozefczyk - to whom I should assign action to check that? :) | 15:53 |
| maciejjozefczyk | May 06 09:52:01.354607 ubuntu-bionic-rax-iad-0016398327 neutron-server[10171]: WARNING neutron.db.agents_db [None req-e6c8c261-b416-41da-beae-c05e137265bb None None] Agent healthcheck: found 1 dead agents out of 4: | 15:53 |
| maciejjozefczyk | May 06 09:52:01.354607 ubuntu-bionic-rax-iad-0016398327 neutron-server[10171]: Type Last heartbeat host | 15:53 |
| maciejjozefczyk | May 06 09:52:01.354607 ubuntu-bionic-rax-iad-0016398327 neutron-server[10171]: OVN Controller Gateway agent 2020-05-06 09:52:01.334533 ubuntu-bionic-rax-iad-0016398327 | 15:53 |
| maciejjozefczyk | ^ during one of failed tests | 15:53 |
| ralonsoh | I can take it | 15:54 |
| slaweq | thx ralonsoh | 15:54 |
| slaweq | and thx maciejjozefczyk for help with this | 15:54 |
| slaweq | #action ralonsoh to check ovn jobs timeouts | 15:54 |
| slaweq | ok, lets move on | 15:54 |
| slaweq | some recheck stats from last weeks | 15:55 |
| slaweq | Average number of rechecks in last weeks: | 15:55 |
| slaweq | week 17 of 2020: 2.62 | 15:55 |
| slaweq | week 18 of 2020: 1.4 | 15:55 |
| slaweq | week 19 of 2020: 1.71 | 15:55 |
| slaweq | so it seems not very bad recently :) | 15:55 |
| slaweq | ok, lets move quickly to the periodic jobs as we are almost out of time today | 15:55 |
| slaweq | #topic Periodic | 15:55 |
| *** openstack changes topic to "Periodic (Meeting topic: neutron_ci)" | 15:55 | |
| slaweq | neutron-ovn-tempest-ovs-master-fedora is failing since 26.04 | 15:56 |
| ralonsoh | again? | 15:56 |
| slaweq | first it was failing with error like: https://d11d69999bcdac29c8fb-3dea60a35fb0d38e41c535f13b48e895.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/b12ec3a/job-output.txt | 15:56 |
| slaweq | and in the next days it changed a bit: | 15:56 |
| slaweq | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dd7/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/dd76029/job-output.txt | 15:56 |
| slaweq | and | 15:56 |
| slaweq | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_71e/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/71e5c2f/job-output.txt | 15:56 |
| slaweq | ralonsoh: yeah - I don't like this fedora job ;) | 15:56 |
| ralonsoh | 2020-04-26 06:28:30.004222 | controller | Error: Unable to find a match: kernel-devel-5.5.17 | 15:57 |
| ralonsoh | but this is not the only error | 15:57 |
| slaweq | but in next days it was failing with: | 15:57 |
| slaweq | configure: error: Linux kernel in /lib/modules/5.6.6-200.fc31.x86_64/build is version 5.6.6, but version newer than 5.5.x is not supported (please refer to the FAQ for advice) | 15:57 |
| ralonsoh | and another one | 15:58 |
| ralonsoh | 2020-05-02 06:18:55.602437 | controller | configure: error: source dir /lib/modules/5.6.7-200.fc31.x86_64/build doesn't exist | 15:58 |
| slaweq | yes | 15:58 |
| slaweq | so basically it is failing on compilation of ovs | 15:58 |
| ralonsoh | exactly | 15:58 |
| ralonsoh | we can ping fedora guys in bugzilla | 15:58 |
| ralonsoh | (I think so) | 15:58 |
| maciejjozefczyk | we recently added compilation of ovs module while using qos | 15:58 |
| slaweq | maciejjozefczyk: You was playing with this complile of the ovs module | 15:59 |
| slaweq | can You take a look at this one? | 15:59 |
| maciejjozefczyk | yes, that was me | 15:59 |
| maciejjozefczyk | slaweq, ok | 15:59 |
| slaweq | thx a lot | 15:59 |
| slaweq | #action maciejjozefczyk to check failing compile of ovs in the fedora periodic job | 15:59 |
| slaweq | and we are out of time almost | 15:59 |
| slaweq | thx for attending | 16:00 |
| slaweq | #endmeeting | 16:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:00 | |
| openstack | Meeting ended Wed May 6 16:00:06 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-05-06-15.00.html | 16:00 |
| slaweq | o/ | 16:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-05-06-15.00.txt | 16:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-05-06-15.00.log.html | 16:00 |
| bcafarel | o/ | 16:00 |
| njohnston | o/ | 16:00 |
| ralonsoh | bye | 16:00 |
| maciejjozefczyk | bb! | 16:00 |
| *** maciejjozefczyk has quit IRC | 16:01 | |
| *** njohnston has quit IRC | 16:01 | |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 16:06 | |
| *** maciejjozefczyk_ has joined #openstack-meeting-3 | 16:07 | |
| *** lajoskatona has left #openstack-meeting-3 | 16:08 | |
| *** maciejjozefczyk has quit IRC | 16:11 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 16:20 | |
| *** psachin has quit IRC | 16:41 | |
| *** dconde has joined #openstack-meeting-3 | 16:54 | |
| *** dconde has quit IRC | 16:54 | |
| *** dconde has joined #openstack-meeting-3 | 16:54 | |
| *** jamesmcarthur has quit IRC | 16:55 | |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 17:18 | |
| *** maciejjozefczyk_ has quit IRC | 17:20 | |
| *** irclogbot_3 has quit IRC | 17:20 | |
| *** maciejjozefczyk has quit IRC | 17:21 | |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 17:21 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 17:21 | |
| *** maciejjozefczyk_ has joined #openstack-meeting-3 | 17:22 | |
| *** irclogbot_3 has joined #openstack-meeting-3 | 17:23 | |
| *** maciejjozefczyk_ has quit IRC | 17:23 | |
| *** maciejjozefczyk has quit IRC | 17:26 | |
| *** jamesmcarthur has quit IRC | 17:37 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 17:38 | |
| *** ralonsoh has quit IRC | 18:19 | |
| *** lpetrut has quit IRC | 18:22 | |
| *** e0ne has quit IRC | 18:34 | |
| *** jamesmcarthur has quit IRC | 19:25 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 19:30 | |
| *** belmoreira has quit IRC | 19:58 | |
| *** belmoreira has joined #openstack-meeting-3 | 20:10 | |
| *** diablo_rojo has joined #openstack-meeting-3 | 20:12 | |
| *** belmoreira has quit IRC | 20:25 | |
| *** jamesmcarthur has quit IRC | 20:50 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 20:51 | |
| *** jamesmcarthur has quit IRC | 21:10 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 21:16 | |
| *** slaweq has quit IRC | 21:26 | |
| *** e0ne has joined #openstack-meeting-3 | 21:35 | |
| *** e0ne has quit IRC | 21:44 | |
| *** raildo_ has joined #openstack-meeting-3 | 21:45 | |
| *** raildo has quit IRC | 21:48 | |
| *** jamesmcarthur has quit IRC | 21:55 | |
| *** raildo_ has quit IRC | 22:01 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!