| *** ricolin has quit IRC | 02:15 | |
| *** ricolin has joined #openstack-meeting-3 | 02:21 | |
| *** artom has quit IRC | 02:35 | |
| *** psachin has joined #openstack-meeting-3 | 03:37 | |
| *** psahoo has joined #openstack-meeting-3 | 03:52 | |
| *** slaweq has joined #openstack-meeting-3 | 05:52 | |
| *** slaweq has quit IRC | 06:02 | |
| *** slaweq has joined #openstack-meeting-3 | 06:57 | |
| *** halali_ has joined #openstack-meeting-3 | 07:00 | |
| *** apetrich has joined #openstack-meeting-3 | 07:27 | |
| *** yangyapeng has joined #openstack-meeting-3 | 07:30 | |
| *** tosky has joined #openstack-meeting-3 | 07:39 | |
| *** ralonsoh has joined #openstack-meeting-3 | 07:40 | |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 08:14 | |
| *** e0ne has joined #openstack-meeting-3 | 08:24 | |
| *** psahoo has quit IRC | 10:44 | |
| *** ricolin has quit IRC | 10:52 | |
| *** psahoo has joined #openstack-meeting-3 | 10:56 | |
| *** psahoo has quit IRC | 11:37 | |
| *** markvoelker has joined #openstack-meeting-3 | 11:40 | |
| *** raildo has joined #openstack-meeting-3 | 11:44 | |
| *** markvoelker has quit IRC | 11:47 | |
| *** psahoo has joined #openstack-meeting-3 | 11:52 | |
| *** artom has joined #openstack-meeting-3 | 12:03 | |
| *** lpetrut has joined #openstack-meeting-3 | 12:44 | |
| *** psahoo has quit IRC | 12:45 | |
| *** markvoelker has joined #openstack-meeting-3 | 12:48 | |
| *** psahoo has joined #openstack-meeting-3 | 12:48 | |
| *** halali_ has quit IRC | 13:18 | |
| *** rubasov has quit IRC | 14:28 | |
| *** rubasov has joined #openstack-meeting-3 | 14:28 | |
| *** ricolin has joined #openstack-meeting-3 | 14:40 | |
| *** mlavalle has joined #openstack-meeting-3 | 14:40 | |
| *** lpetrut has quit IRC | 14:43 | |
| *** irclogbot_3 has quit IRC | 14:58 | |
| *** yangyapeng has quit IRC | 14:58 | |
| *** yangyapeng has joined #openstack-meeting-3 | 14:58 | |
| *** irclogbot_3 has joined #openstack-meeting-3 | 14:59 | |
| slaweq | #startmeeting neutron_ci | 15:00 |
|---|---|---|
| openstack | Meeting started Wed Jul 29 15:00:07 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| slaweq | hi | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:00 | |
| openstack | The meeting name has been set to 'neutron_ci' | 15:00 |
| ralonsoh | hi | 15:00 |
| bcafarel | o/ | 15:01 |
| slaweq | lets start | 15:02 |
| slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:02 |
| slaweq | we don't have any actions from last meeting for today | 15:02 |
| slaweq | so lets move directly to the next topic | 15:03 |
| slaweq | #topic Stadium projects | 15:03 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:03 | |
| slaweq | and first item there which is | 15:03 |
| slaweq | standardize on zuul v3 | 15:03 |
| slaweq | Etherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop | 15:03 |
| slaweq | based on recent mail from tosky I updated our etherpad a bit | 15:03 |
| slaweq | so we have a bit more things to do there | 15:03 |
| slaweq | I will try to send some patches related to it this and next week | 15:04 |
| slaweq | but if You want to help, feel free to send some patches and add links to them in the etherpad | 15:04 |
| bcafarel | oh I recognize my comment on networking-sfc from main etherpad :) | 15:05 |
| slaweq | :) | 15:06 |
| slaweq | anything else regarding stadium projects for today? | 15:06 |
| bcafarel | just general organization question should we add a focal migration section here? (or else in the meeting) | 15:07 |
| slaweq | there is separate topic for this in the agenda | 15:07 |
| bcafarel | I had started https://etherpad.opendev.org/p/neutron-victoria-switch_to_focal at some point close in style to zuulv3 transition one | 15:07 |
| slaweq | thx bcafarel | 15:08 |
| slaweq | that's great | 15:08 |
| slaweq | I think we need to focus more on that goal | 15:09 |
| slaweq | as we really need to do that soon IMO | 15:09 |
| bcafarel | +1 and at least from me I keep getting sidetracked on other stuff :( | 15:10 |
| tosky | yep, Victoria is not so far away | 15:10 |
| bcafarel | we have ovs installed from pip now but functional job timeouts after that | 15:10 |
| *** lajoskatona has joined #openstack-meeting-3 | 15:11 | |
| slaweq | bcafarel: ovs from the pip? You mean from the repo, right? | 15:11 |
| lajoskatona | o/, sorry for being late | 15:11 |
| ralonsoh | python-ovs | 15:12 |
| bcafarel | python-openvswith to be precise | 15:12 |
| bcafarel | ralonsoh: always faster :) | 15:12 |
| ralonsoh | exactly | 15:12 |
| slaweq | ahh, ok :) | 15:12 |
| slaweq | is it this patch https://review.opendev.org/#/c/734304/ ? | 15:13 |
| ralonsoh | it is | 15:13 |
| slaweq | it's failing due to some errors related to ovn probably: https://60633e52ebee9acccc9c-bc7527a6a4fcb9c26b6a927801c9ca9a.ssl.cf2.rackcdn.com/734304/8/check/neutron-functional/dd799fc/job-output.txt | 15:14 |
| slaweq | there is plenty of errors like " sqlite3.OperationalError: no such table: ovn_hash_ring" | 15:14 |
| slaweq | but not only that | 15:14 |
| slaweq | errors with ncat process | 15:14 |
| slaweq | and others | 15:14 |
| slaweq | I think someone needs to run locally vm with ubuntu focal and run those tests there to investigate | 15:15 |
| bcafarel | just rebased DNM https://review.opendev.org/#/c/738163/ to get fresh results on other jobs | 15:15 |
| ralonsoh | yes but this thing about the ovn_hash_ring table... | 15:16 |
| ralonsoh | I need to investigate that | 15:16 |
| slaweq | and I also see there issues related to new ebtables | 15:16 |
| slaweq | we had something similar in RH some time ago when we moved to RHEL/Centos 8 | 15:16 |
| slaweq | but we simply skipped those failing Linuxbridge related tests in d/s | 15:17 |
| slaweq | based on that I think we really need to give high priority for that as we have some real issues on this new OS | 15:18 |
| slaweq | I just increased timeout for functional job in https://review.opendev.org/#/c/734304/ | 15:19 |
| slaweq | I hope it will finish without timeout so we will have cleaner look on what tests are failing there | 15:19 |
| slaweq | and lets focus on that in next days | 15:20 |
| slaweq | ok, I think we can move on now | 15:22 |
| slaweq | #topic Stable branches | 15:22 |
| *** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)" | 15:22 | |
| slaweq | Ussuri dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1 | 15:22 |
| slaweq | Train dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1 | 15:22 |
| bcafarel | from what I saw in recent backports, we are back to stable CI for stable branches | 15:24 |
| slaweq | bcafarel: that's my understanding from the look for grafana :) | 15:24 |
| slaweq | do we have anything else related to CI of stable branches for today? | 15:26 |
| slaweq | or can we move on? | 15:26 |
| bcafarel | nothing from me at least | 15:26 |
| slaweq | ok, lets move on | 15:28 |
| slaweq | #topic Grafana | 15:28 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:28 | |
| slaweq | http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:28 |
| slaweq | first of all, I today proposed patch https://review.opendev.org/743729 with small improvement (IMHO) | 15:30 |
| slaweq | I proposed to move non-voting scenario jobs to separate graph | 15:30 |
| ralonsoh | agree with this | 15:30 |
| slaweq | as now we have 12 jobs on one graph and it's a bit hard to check quickly which job is failing often | 15:31 |
| slaweq | and if it's voting or non-voting job | 15:31 |
| slaweq | other than that I think we have a lot of functional jobs failures recently | 15:32 |
| ralonsoh | due to a problem in ovn | 15:32 |
| ralonsoh | https://review.opendev.org/#/c/743577/ | 15:32 |
| ralonsoh | (most of them) | 15:33 |
| slaweq | yes, I think that main problem is that one ralonsoh :) | 15:33 |
| slaweq | but second issue which happens pretty often is ovsdbapp timeout still | 15:33 |
| slaweq | like e.g. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_83a/729931/1/check/neutron-fullstack-with-uwsgi/83a80df/testr_results.html | 15:33 |
| slaweq | this one is from fullstack but I saw it also in functional tests | 15:34 |
| ralonsoh | yes... I'm aware | 15:34 |
| ralonsoh | and I don't know how to fix that | 15:34 |
| slaweq | and I know that we already know about it | 15:34 |
| slaweq | can't we maybe increase this timeout somehow in tests? | 15:34 |
| slaweq | to e.g. 30 seconds instead of 10? | 15:35 |
| ralonsoh | not a problem of timeout | 15:35 |
| slaweq | ahh | 15:35 |
| ralonsoh | but with eventlet and threads, the recurrent issue | 15:35 |
| slaweq | ahh | 15:35 |
| ralonsoh | for me this problem is like https://i.imgur.com/ofRuetq.jpg | 15:36 |
| *** psahoo has quit IRC | 15:36 | |
| slaweq | LOL | 15:36 |
| slaweq | so we can't do anything with that really? | 15:37 |
| ralonsoh | still thinking about this | 15:38 |
| slaweq | ralonsoh: and why we don't see it in e.g. scenario jobs but only in functional/fullstack? | 15:38 |
| slaweq | or we do? | 15:38 |
| ralonsoh | I think this is because the system is more busy during FT/fullstack | 15:38 |
| lajoskatona | is this only cpu power i.e., or OVS internals< | 15:39 |
| lajoskatona | ? | 15:39 |
| ralonsoh | this is just a guess | 15:40 |
| ralonsoh | but if you are waiting for a message and you use a "wait" method | 15:40 |
| ralonsoh | and then no other thread returns the GIL on time, you'll have a timeout | 15:40 |
| ralonsoh | is the solution to increase the timeout? I don't think so | 15:41 |
| ralonsoh | the solution is to find out where to make an active wait not returning the GIL | 15:41 |
| ralonsoh | (using evenlets) | 15:41 |
| ralonsoh | Am I still connected? | 15:45 |
| slaweq | ralonsoh: yes | 15:45 |
| ralonsoh | cool | 15:45 |
| slaweq | I just don't know what else I can add to this :) | 15:45 |
| ralonsoh | hehehe me neither | 15:46 |
| slaweq | ok, lets keep an eye on it, maybe someday someone will find solution :) | 15:46 |
| slaweq | lets go to the next topic for today | 15:47 |
| slaweq | #topic Tempest/Scenario | 15:47 |
| *** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 15:47 | |
| slaweq | I found 2 new (for me) failures in last days | 15:47 |
| slaweq | first: | 15:47 |
| slaweq | neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest | 15:47 |
| slaweq | https://bcbff5a63272be990f4a-3078f09a35fdfa0355b18572be8c3ad5.ssl.cf2.rackcdn.com/702197/5/check/neutron-tempest-plugin-api/068ce6c/testr_results.html | 15:47 |
| slaweq | but now I think it was related to the patch on which it was run: https://review.opendev.org/#/c/702197/ | 15:48 |
| slaweq | so please ignore it | 15:49 |
| slaweq | second one was: | 15:49 |
| slaweq | neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationAdminTests | 15:49 |
| slaweq | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_baf/740588/3/check/neutron-tempest-plugin-designate-scenario/baf5297/testr_results.html | 15:49 |
| slaweq | did You saw such failures before? | 15:50 |
| ralonsoh | the first one yes, not the second one | 15:50 |
| ralonsoh | (actually I saw the first one because that was my patch) | 15:50 |
| slaweq | ralonsoh: first one was on Your patch :P | 15:50 |
| ralonsoh | hehehehe | 15:50 |
| slaweq | second one failed due to error: | 15:51 |
| slaweq | designateclient.exceptions.Conflict: Duplicate Zone | 15:51 |
| slaweq | in neutron server logs | 15:51 |
| slaweq | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_baf/740588/3/check/neutron-tempest-plugin-designate-scenario/baf5297/controller/logs/screen-q-svc.txt | 15:51 |
| slaweq | I will keep an eye on this job to see if that will happen more | 15:51 |
| slaweq | and if so, I will open LP for that issue | 15:51 |
| slaweq | and that's basically all what I have for today | 15:52 |
| slaweq | do You have anything else for today's meeting? | 15:52 |
| ralonsoh | yes, a request for review | 15:53 |
| ralonsoh | related to fullstack tests | 15:53 |
| ralonsoh | https://review.opendev.org/#/c/738446/ | 15:53 |
| ralonsoh | that will reduce the problem we have sometimes with test clashes | 15:53 |
| slaweq | sure | 15:53 |
| slaweq | I will review it this week | 15:53 |
| slaweq | asap | 15:53 |
| * bcafarel adds to the pile | 15:54 | |
| bcafarel | one additional stable topic brought in https://review.opendev.org/#/c/742519/ comments | 15:54 |
| bcafarel | do you think jobs running against master branches of other projects are useful in stable branches? | 15:54 |
| bcafarel | neutron-tempest-with-os-ken-master / neutron-ovn-tempest-ovs-master-fedora periodic jobs specifically | 15:55 |
| slaweq | no | 15:55 |
| slaweq | I think we should drop such jobs in stable branches | 15:55 |
| maciejjozefczyk | +1 | 15:55 |
| slaweq | stable branches by definition aren't expected to run with other things from master | 15:55 |
| bcafarel | ok I will send cleanup patches (less periodic jobs to run, infra will be happy) | 15:56 |
| slaweq | thx bcafarel | 15:56 |
| slaweq | #action bcafarel to clean stable branches jobs from *-master jobs | 15:57 |
| slaweq | ^^ I had to add at least one today :P | 15:57 |
| bcafarel | all actions on me then :) | 15:57 |
| slaweq | :) | 15:57 |
| slaweq | ok, I think we can finish the meeting now | 15:58 |
| slaweq | thx for attending | 15:58 |
| ralonsoh | bye! | 15:58 |
| bcafarel | o/ | 15:58 |
| lajoskatona | bye | 15:58 |
| slaweq | and for taking care of ci in last 2 weeks :) | 15:58 |
| slaweq | o/ | 15:58 |
| slaweq | #endmeeting | 15:58 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:58 | |
| openstack | Meeting ended Wed Jul 29 15:58:22 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:58 |
| *** lajoskatona has left #openstack-meeting-3 | 15:58 | |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-29-15.00.html | 15:58 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-29-15.00.txt | 15:58 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-29-15.00.log.html | 15:58 |
| *** markvoelker has quit IRC | 16:13 | |
| *** akhil-g has joined #openstack-meeting-3 | 16:34 | |
| *** mlavalle has quit IRC | 16:46 | |
| *** mlavalle has joined #openstack-meeting-3 | 16:48 | |
| *** psachin has quit IRC | 16:53 | |
| *** ricolin has quit IRC | 17:01 | |
| *** artom has quit IRC | 17:45 | |
| *** artom has joined #openstack-meeting-3 | 17:46 | |
| *** artom has quit IRC | 17:52 | |
| *** artom has joined #openstack-meeting-3 | 18:07 | |
| *** akhil-g has quit IRC | 18:10 | |
| *** halali_ has joined #openstack-meeting-3 | 18:11 | |
| *** maciejjozefczyk has quit IRC | 18:24 | |
| *** artom has quit IRC | 18:31 | |
| *** artom has joined #openstack-meeting-3 | 18:48 | |
| *** artom has quit IRC | 18:48 | |
| *** artom has joined #openstack-meeting-3 | 18:48 | |
| *** ralonsoh has quit IRC | 18:53 | |
| *** slaweq has quit IRC | 21:07 | |
| *** raildo has quit IRC | 21:26 | |
| *** artom has quit IRC | 22:04 | |
| *** e0ne has quit IRC | 22:16 | |
| *** e0ne has joined #openstack-meeting-3 | 22:19 | |
| *** e0ne has quit IRC | 22:20 | |
| *** yangyapeng has quit IRC | 22:30 | |
| *** e0ne has joined #openstack-meeting-3 | 22:31 | |
| *** yangyapeng has joined #openstack-meeting-3 | 22:34 | |
| *** e0ne_ has joined #openstack-meeting-3 | 22:37 | |
| *** e0ne has quit IRC | 22:37 | |
| *** e0ne_ has quit IRC | 22:37 | |
| *** tosky has quit IRC | 23:21 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
