*** yamamoto has joined #openstack-meeting-3 | 00:46 | |
*** yamamoto has quit IRC | 00:49 | |
*** yamamoto has joined #openstack-meeting-3 | 00:58 | |
*** markvoelker has joined #openstack-meeting-3 | 01:20 | |
*** markvoelker has quit IRC | 01:24 | |
*** ricolin_ has joined #openstack-meeting-3 | 01:43 | |
*** ricolin_ is now known as ricolin | 03:30 | |
*** psachin has joined #openstack-meeting-3 | 03:33 | |
*** markvoelker has joined #openstack-meeting-3 | 04:03 | |
*** markvoelker has quit IRC | 04:08 | |
*** apetrich has quit IRC | 04:43 | |
*** mugsie has quit IRC | 04:53 | |
*** mugsie has joined #openstack-meeting-3 | 04:57 | |
*** markvoelker has joined #openstack-meeting-3 | 05:03 | |
*** markvoelker has quit IRC | 05:07 | |
*** links has joined #openstack-meeting-3 | 05:34 | |
*** njohnston has quit IRC | 06:18 | |
*** markvoelker has joined #openstack-meeting-3 | 07:04 | |
*** markvoelker has quit IRC | 07:09 | |
*** slaweq has quit IRC | 07:12 | |
*** slaweq has joined #openstack-meeting-3 | 07:13 | |
*** markvoelker has joined #openstack-meeting-3 | 07:18 | |
*** markvoelker has quit IRC | 07:23 | |
*** ralonsoh has joined #openstack-meeting-3 | 07:33 | |
*** e0ne has joined #openstack-meeting-3 | 07:58 | |
*** apetrich has joined #openstack-meeting-3 | 08:48 | |
*** yamamoto has quit IRC | 09:37 | |
*** ricolin has quit IRC | 09:52 | |
*** yamamoto has joined #openstack-meeting-3 | 09:55 | |
*** yamamoto has quit IRC | 10:06 | |
*** yamamoto has joined #openstack-meeting-3 | 10:07 | |
*** apetrich has quit IRC | 10:12 | |
*** njohnston has joined #openstack-meeting-3 | 11:01 | |
*** markvoelker has joined #openstack-meeting-3 | 11:15 | |
*** markvoelker has quit IRC | 11:19 | |
*** raildo has joined #openstack-meeting-3 | 11:28 | |
*** apetrich has joined #openstack-meeting-3 | 11:45 | |
*** csatari has quit IRC | 11:53 | |
*** csatari has joined #openstack-meeting-3 | 11:56 | |
*** irclogbot_3 has quit IRC | 12:16 | |
*** irclogbot_0 has joined #openstack-meeting-3 | 12:17 | |
*** trident has quit IRC | 12:46 | |
*** trident has joined #openstack-meeting-3 | 12:49 | |
*** ricolin has joined #openstack-meeting-3 | 12:54 | |
*** psachin has quit IRC | 13:31 | |
*** liuyulong has joined #openstack-meeting-3 | 13:50 | |
*** yamamoto has quit IRC | 13:50 | |
*** yamamoto has joined #openstack-meeting-3 | 14:11 | |
*** yamamoto has quit IRC | 14:11 | |
*** yamamoto has joined #openstack-meeting-3 | 14:12 | |
*** yamamoto has quit IRC | 14:16 | |
*** maciejjozefczyk has joined #openstack-meeting-3 | 14:28 | |
*** links has quit IRC | 14:55 | |
slaweq | #startmeeting neutron_ci | 15:00 |
---|---|---|
openstack | Meeting started Wed Jul 1 15:00:27 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
*** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:00 | |
slaweq | hi | 15:00 |
openstack | The meeting name has been set to 'neutron_ci' | 15:00 |
ralonsoh | hi | 15:00 |
slaweq | lets wait few more minutes for others | 15:02 |
maciejjozefczyk | \p | 15:02 |
maciejjozefczyk | \o | 15:02 |
slaweq | bcafarel: njohnston: ping :) | 15:03 |
slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:04 |
bcafarel | o/ | 15:04 |
bcafarel | sorry I was listening to Edge session | 15:04 |
slaweq | ahh | 15:04 |
slaweq | ok | 15:04 |
slaweq | I forgot about it | 15:04 |
slaweq | ok, lets start | 15:04 |
slaweq | #topic Actions from previous meetings | 15:05 |
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 15:05 | |
slaweq | bcafarel to check gate status on rocky and queens (uwsgi problem) | 15:05 |
*** suryasingh has joined #openstack-meeting-3 | 15:05 | |
bcafarel | ok so patch in neutron itself is not enough, we need multiple fixes in devstack | 15:05 |
bcafarel | latest iteration in devstack itself looks good to go https://review.opendev.org/#/c/735615/ | 15:05 |
bcafarel | then we will need https://review.opendev.org/#/c/738851/ or something like that on top - with Depends-On I still see failures that should be fixed by the devstack one | 15:06 |
bcafarel | hopefully recheck once it is merged should be greener | 15:06 |
bcafarel | and once rocky is finally back on track, similar backports for older branches :) | 15:07 |
slaweq | ok | 15:07 |
slaweq | so it seems like we are still far from green gate for queens and rocky | 15:07 |
njohnston | o/ | 15:08 |
bcafarel | yup :/ but there is progress at least | 15:08 |
slaweq | thx bcafarel for taking care of it | 15:09 |
slaweq | ok, next one | 15:09 |
slaweq | maciejjozefczyk will check test_ovsdb_monitor.TestNBDbMonitorOverTcp.test_floatingip_mac_bindings failiure in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html | 15:09 |
maciejjozefczyk | fixed | 15:09 |
slaweq | thx maciejjozefczyk :) | 15:09 |
maciejjozefczyk | #link https://review.opendev.org/#/c/738415/ | 15:09 |
slaweq | so next one | 15:10 |
slaweq | ralonsoh will check get_datapath_id issues in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html | 15:10 |
slaweq | maciejjozefczyk: one question, should we backport it to ussuri? | 15:10 |
ralonsoh | slaweq, sorry, I didn't start debbuging | 15:10 |
maciejjozefczyk | slaweq, hmm we can | 15:11 |
slaweq | maciejjozefczyk: ok, can You propose backport? | 15:11 |
maciejjozefczyk | slaweq, clicked :D | 15:11 |
slaweq | thx | 15:11 |
slaweq | ralonsoh: sure, I know You were busy with other things | 15:12 |
slaweq | will You try to check that next week? | 15:12 |
ralonsoh | sure | 15:12 |
slaweq | #action ralonsoh will check get_datapath_id issues in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html | 15:12 |
slaweq | thx | 15:12 |
slaweq | ok, next one | 15:12 |
slaweq | slaweq will check errors with non existing interface in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html | 15:12 |
slaweq | I didn't have too much time to check this one | 15:12 |
slaweq | but from what I looked today it seems for me like maybe some tests are "overlapping" and other test cleanded some port/bridge | 15:13 |
slaweq | I will probably add some additional debug logging to be able to maybe investigate it more when it will happen again | 15:14 |
slaweq | and I will try to continue work on it this week | 15:14 |
slaweq | #action slaweq will check errors with non existing interface in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html | 15:14 |
slaweq | and the last one | 15:15 |
slaweq | maciejjozefczyk to check failing neutron-ovn-tempest-ovs-master-fedora periodic job | 15:15 |
maciejjozefczyk | yeah | 15:15 |
maciejjozefczyk | #link https://review.opendev.org/#/c/737984/ | 15:15 |
slaweq | I saw that ovn jobs were failing on this patch today morning | 15:15 |
maciejjozefczyk | the issue is trivial, ovs compilation code was duplicated in both ovn and ovs modules | 15:15 |
slaweq | so lets wait how it will be now :) | 15:15 |
maciejjozefczyk | yeah, slaweq, this should be fine now | 15:16 |
maciejjozefczyk | (I hope so) :D | 15:16 |
bcafarel | and I think that one may be interesting for focal transition too | 15:16 |
maciejjozefczyk | This is about cleaning and unifying the way we compile ovs/ovn, so next time we'll need to fix one function instead two | 15:16 |
maciejjozefczyk | bcafarel, indeed | 15:16 |
slaweq | ++ | 15:17 |
slaweq | thx | 15:17 |
slaweq | ok | 15:18 |
slaweq | that are all actions from last week | 15:18 |
slaweq | lets move on to the next topic | 15:18 |
slaweq | #topic Stadium projects | 15:18 |
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:18 | |
slaweq | with migration to zuul v3 stadium projects are actually good now | 15:18 |
slaweq | as only neutron-ovn-grenade job is still missing | 15:19 |
njohnston | \o/ | 15:19 |
slaweq | anything else You want to discuss about stadium projects today? | 15:19 |
slaweq | ok, so lets move on | 15:21 |
slaweq | #topic Stable branches | 15:21 |
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)" | 15:21 | |
slaweq | Ussuri dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1 | 15:21 |
slaweq | Train dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1 | 15:21 |
slaweq | from what I was seeing this week, branches which are not EM are running pretty good now | 15:22 |
bcafarel | I mostly looked at rocky that week, but I think ussuri to stein were OK | 15:22 |
slaweq | and EM are red due to known reasons | 15:22 |
slaweq | I thought about one small improvement to safe some gate resources: wdyt about moving all non-voting jobs to experimental queue in the EM branches? | 15:23 |
bcafarel | :) I was thinking about that when filling https://review.opendev.org/#/c/738851/ | 15:23 |
slaweq | :) | 15:24 |
slaweq | so there is at least 2 of us | 15:24 |
slaweq | that would safe about 4-6 jobs, mostly multinode | 15:24 |
slaweq | so pretty many vms spawned to test each patch | 15:24 |
bcafarel | probability is low that someone will work on fixing them in EM, and I don't think anyone checks their results in backports | 15:25 |
slaweq | exactly | 15:25 |
slaweq | almost noboday is checking non-voting jobs in master | 15:25 |
slaweq | ralonsoh: njohnston any thoughts? | 15:27 |
slaweq | if You are ok with this I will propose such patch this week | 15:27 |
ralonsoh | agree with removing them | 15:27 |
njohnston | I absolutely agree | 15:28 |
njohnston | There is no reason to be using those resources | 15:28 |
slaweq | ok | 15:28 |
slaweq | thx | 15:28 |
slaweq | so I will propose such change | 15:28 |
slaweq | #action slaweq to move non-voting jobs to experimental queue in EM branches | 15:29 |
slaweq | anything else regarding stable branches? or can we move on? | 15:29 |
bcafarel | nothing from me | 15:29 |
njohnston | nothing from me | 15:30 |
slaweq | ok, lets move on then | 15:30 |
slaweq | #topic Grafana | 15:30 |
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:30 | |
slaweq | #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:31 |
slaweq | I don't see any serious problems there | 15:32 |
slaweq | I pushed small patch to update dashboard https://review.opendev.org/738784 | 15:33 |
slaweq | ok, lets talk about some specific issues | 15:34 |
slaweq | #topic Tempest/Scenario | 15:34 |
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 15:34 | |
slaweq | I found few failures for which I opened LPs | 15:34 |
slaweq | first one is | 15:34 |
slaweq | tempest.api.network.admin.test_routers.RoutersAdminTest.test_create_router_set_gateway_with_fixed_ip | 15:35 |
slaweq | in job neutron-tempest-dvr-ha-multinode-full | 15:35 |
slaweq | it happens very often | 15:35 |
*** factor has joined #openstack-meeting-3 | 15:35 | |
slaweq | like e.g.: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2c4/734876/1/check/neutron-tempest-dvr-ha-multinode-full/2c466a0/testr_results.html | 15:35 |
slaweq | Bug reported https://bugs.launchpad.net/neutron/+bug/1885897 | 15:35 |
openstack | Launchpad bug 1885897 in neutron "Tempest test_create_router_set_gateway_with_fixed_ip test is failing often in dvr scenario job" [High,Confirmed] | 15:35 |
slaweq | in fact this is main reason of often failures of this job | 15:35 |
slaweq | any volunteer to check that one? | 15:36 |
slaweq | if not, I will try to check it this week | 15:36 |
ralonsoh | sorry, not this week | 15:36 |
slaweq | ok, I will check it | 15:37 |
slaweq | #action slaweq to investigate failures in test_create_router_set_gateway_with_fixed_ip test | 15:37 |
slaweq | I think that this may be some tempest cleaning issue maybe | 15:37 |
slaweq | or something like that | 15:37 |
slaweq | as it happens very often | 15:37 |
slaweq | next one is related to qos test | 15:38 |
slaweq | I found it in neutron-tempest-plugin-scenario-openvswitch job but I think it may happen also in other jobs | 15:38 |
slaweq | neutron_tempest_plugin.scenario.test_qos.QoSTest.test_qos_basic_and_update | 15:38 |
slaweq | https://20f4a85411442f4e3555-9f5a5e2736e26bdd8715596753fafe10.ssl.cf2.rackcdn.com/734876/1/check/neutron-tempest-plugin-scenario-openvswitch/a31f86b/testr_results.html | 15:38 |
slaweq | Bug reported https://bugs.launchpad.net/neutron/+bug/1885899 | 15:38 |
openstack | Launchpad bug 1885899 in neutron "test_qos_basic_and_update test is failing" [Critical,Confirmed] | 15:38 |
slaweq | seems like nc wasn't spawned properly - maybe we should add additional logging in https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_tempest_plugin/common/utils.py#L122 ? | 15:38 |
slaweq | anyone wants to check that? | 15:40 |
ralonsoh | but I remember you changed the way to spawn nc | 15:40 |
ralonsoh | making it more reliable | 15:40 |
slaweq | yes | 15:40 |
ralonsoh | sorry, but next week | 15:40 |
ralonsoh | not this one | 15:41 |
slaweq | ok, lets keep it unassigned, maybe someone will want to check it | 15:41 |
slaweq | I marked it as critical because it impacts voting jobs | 15:41 |
slaweq | ok, next one | 15:41 |
slaweq | this is related only to the ovn based jobs where test neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers is failing | 15:42 |
slaweq | like e.g. https://4ec598fcefc6b0367120-6910015cdc6b96c34eca0ab65a68e7f2.ssl.cf5.rackcdn.com/696926/18/check/neutron-ovn-tempest-full-multinode-ovs-master/c1c51ca/testr_results.html | 15:42 |
slaweq | Bug reported: https://bugs.launchpad.net/neutron/+bug/1885898 | 15:42 |
openstack | Launchpad bug 1885898 in neutron "test connectivity through 2 routers fails in neutron-ovn-tempest-full-multinode-ovs-master job" [High,Confirmed] | 15:42 |
slaweq | maciejjozefczyk: will You have time to take a look into this? | 15:42 |
maciejjozefczyk | slaweq, You have my sword | 15:43 |
slaweq | maciejjozefczyk: thx a lot | 15:43 |
bcafarel | :) | 15:43 |
slaweq | #action maciejjozefczyk to check neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers failure in ovn jobs | 15:43 |
slaweq | maciejjozefczyk: there is also another failure in ovn based jobs | 15:44 |
slaweq | neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle | 15:44 |
slaweq | https://7bea12b2d1429b68c6c8-10caedded388001c6bbc38619ca4b324.ssl.cf2.rackcdn.com/737047/8/check/neutron-ovn-tempest-full-multinode-ovs-master/592e31b/testr_results.html | 15:44 |
slaweq | Bug reported: https://bugs.launchpad.net/neutron/+bug/1885900 | 15:44 |
openstack | Launchpad bug 1885900 in neutron "test_trunk_subport_lifecycle is failing in ovn based jobs" [Critical,Confirmed] | 15:44 |
maciejjozefczyk | slaweq, yeah notiecd that bug :( | 15:44 |
maciejjozefczyk | that fails pretty often now? | 15:45 |
slaweq | You will probably don't have time to work on both this week but please keep it in mind :) | 15:45 |
slaweq | I saw couple of times at lease in last week | 15:45 |
slaweq | see http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22line%20240%2C%20in%20test_trunk_subport_lifecycle%5C%22 | 15:46 |
slaweq | and that's all what I had for today | 15:46 |
slaweq | I opened LP for each of failures so we can track them there | 15:47 |
maciejjozefczyk | slaweq, thanks for the link | 15:47 |
slaweq | maciejjozefczyk: yw :) | 15:47 |
slaweq | anything else You want to talk about today? | 15:47 |
* bcafarel looks for correct tab to copy link | 15:48 | |
bcafarel | https://review.opendev.org/#/c/738163/ I started to change a few bits for focal | 15:49 |
bcafarel | still early WIP but if you want to add stuff, please do so | 15:49 |
slaweq | thx bcafarel | 15:50 |
maciejjozefczyk | sure bcafarel thanks | 15:50 |
maciejjozefczyk | ovn fails in all the cases :D | 15:50 |
slaweq | I added myself to the reviewers to be up to date with this :) | 15:50 |
slaweq | damm ovn :P | 15:50 |
njohnston | +1 | 15:50 |
bcafarel | :) yes it needs more work on the skip ovs/ovn compilation bits | 15:50 |
slaweq | we should move it to the stadium project ;P | 15:51 |
bcafarel | ahah | 15:51 |
slaweq | maciejjozefczyk: wdyt? | 15:51 |
slaweq | :D | 15:51 |
njohnston | ROFL | 15:51 |
slaweq | ok, I think we can finish this meeting now | 15:53 |
slaweq | thx for attending | 15:53 |
slaweq | and see You next week :) | 15:53 |
slaweq | #endmeeting | 15:53 |
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:53 | |
openstack | Meeting ended Wed Jul 1 15:53:12 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:53 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-01-15.00.html | 15:53 |
ralonsoh | bye | 15:53 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-01-15.00.txt | 15:53 |
openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-01-15.00.log.html | 15:53 |
bcafarel | o/ | 15:53 |
maciejjozefczyk | lol :P | 15:54 |
maciejjozefczyk | bb! | 15:54 |
*** ricolin has quit IRC | 16:01 | |
*** liuyulong has quit IRC | 16:05 | |
*** etp has quit IRC | 16:16 | |
*** etp has joined #openstack-meeting-3 | 16:19 | |
*** etp has quit IRC | 16:24 | |
*** etp has joined #openstack-meeting-3 | 16:44 | |
*** factor has quit IRC | 16:51 | |
*** icarusfactor has joined #openstack-meeting-3 | 16:51 | |
*** ralonsoh has quit IRC | 17:10 | |
*** ralonsoh has joined #openstack-meeting-3 | 17:10 | |
*** bnemec has quit IRC | 17:15 | |
*** factor has joined #openstack-meeting-3 | 17:16 | |
*** icarusfactor has quit IRC | 17:16 | |
*** bnemec has joined #openstack-meeting-3 | 17:21 | |
*** tsmith2 has quit IRC | 17:26 | |
*** freerunner has quit IRC | 17:26 | |
*** hongbin has joined #openstack-meeting-3 | 17:28 | |
*** irclogbot_0 has quit IRC | 17:28 | |
*** tsmith2 has joined #openstack-meeting-3 | 17:29 | |
*** freerunner has joined #openstack-meeting-3 | 17:29 | |
*** irclogbot_2 has joined #openstack-meeting-3 | 17:30 | |
*** njohnston_ has joined #openstack-meeting-3 | 17:43 | |
*** njohnston has quit IRC | 17:43 | |
*** yamamoto has joined #openstack-meeting-3 | 17:44 | |
*** njohnston_ is now known as njohnston | 17:45 | |
*** yamamoto has quit IRC | 17:53 | |
*** e0ne has quit IRC | 17:59 | |
*** slaweq has quit IRC | 18:25 | |
*** hongbin has quit IRC | 18:44 | |
*** factor has quit IRC | 18:46 | |
*** factor has joined #openstack-meeting-3 | 18:46 | |
*** factor has quit IRC | 19:04 | |
*** factor has joined #openstack-meeting-3 | 19:04 | |
*** factor has quit IRC | 19:10 | |
*** hongbin has joined #openstack-meeting-3 | 19:18 | |
*** ralonsoh has quit IRC | 19:50 | |
*** hongbin has quit IRC | 20:06 | |
*** e0ne has joined #openstack-meeting-3 | 20:20 | |
*** slaweq has joined #openstack-meeting-3 | 20:25 | |
*** maciejjozefczyk has quit IRC | 20:36 | |
*** hongbin has joined #openstack-meeting-3 | 20:54 | |
*** e0ne has quit IRC | 21:07 | |
*** slaweq has quit IRC | 21:19 | |
*** factor has joined #openstack-meeting-3 | 21:29 | |
*** factor has quit IRC | 21:33 | |
*** factor has joined #openstack-meeting-3 | 21:33 | |
*** factor has quit IRC | 21:49 | |
*** factor has joined #openstack-meeting-3 | 21:50 | |
*** icarusfactor has joined #openstack-meeting-3 | 22:33 | |
*** factor has quit IRC | 22:36 | |
*** hongbin has quit IRC | 23:11 | |
*** hongbin has joined #openstack-meeting-3 | 23:16 | |
*** yamamoto has joined #openstack-meeting-3 | 23:52 | |
*** yamamoto has quit IRC | 23:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!