Tuesday, 2023-02-14

« Monday, 2023-02-13 Index Wednesday, 2023-02-15 »
opendevreviewMiguel Lavalle proposed openstack/neutron-tempest-plugin master: Use OVN version 22.06  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/84819800:40
opendevreviewMerged openstack/neutron-fwaas master: CI: add neutron and neutron-lib to docs required-projects  https://review.opendev.org/c/openstack/neutron-fwaas/+/87337600:58
opendevreviewMerged openstack/neutron-fwaas master: Add get_firewall_groups_count method  https://review.opendev.org/c/openstack/neutron-fwaas/+/87336001:01
opendevreviewliuyulong proposed openstack/neutron master: Pass physical bridge informations to OVS agent extension API  https://review.opendev.org/c/openstack/neutron/+/86663501:29
opendevreviewliuyulong proposed openstack/neutron master: Add host metadata haproxy manager  https://review.opendev.org/c/openstack/neutron/+/86464901:30
opendevreviewZhouHeng proposed openstack/neutron-fwaas master: [WIP]test multinode  https://review.opendev.org/c/openstack/neutron-fwaas/+/87308803:20
*** ministry is now known as __ministry03:22
opendevreviewMerged openstack/networking-bgpvpn master: Fix some pylint indentation warnings  https://review.opendev.org/c/openstack/networking-bgpvpn/+/86439703:28
opendevreviewZhouHeng proposed openstack/neutron-fwaas master: [WIP]test multinode  https://review.opendev.org/c/openstack/neutron-fwaas/+/87308806:17
opendevreviewZhouHeng proposed openstack/neutron-fwaas master: [WIP]test multinode  https://review.opendev.org/c/openstack/neutron-fwaas/+/87308808:09
opendevreviewSlawek Kaplonski proposed openstack/neutron-tempest-plugin master: [Secure RBAC] Add scope enforcement enabled job for master branch  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/86751808:17
opendevreviewyatin proposed openstack/neutron master: [DNM] Check ovn jammy ipv6 job failure  https://review.opendev.org/c/openstack/neutron/+/87368408:24
opendevreviewSlawek Kaplonski proposed openstack/neutron master: Remove tripleo related jobs from neutron queues  https://review.opendev.org/c/openstack/neutron/+/87368508:42
ralonsoh^^ reviewing08:43
opendevreviewRodolfo Alonso proposed openstack/neutron stable/zed: [OVN] Add missing LSP device_owner info in trunk driver  https://review.opendev.org/c/openstack/neutron/+/87360808:51
opendevreviewRodolfo Alonso proposed openstack/neutron stable/yoga: [OVN] Add missing LSP device_owner info in trunk driver  https://review.opendev.org/c/openstack/neutron/+/87360908:51
opendevreviewRodolfo Alonso proposed openstack/neutron stable/xena: [OVN] Add missing LSP device_owner info in trunk driver  https://review.opendev.org/c/openstack/neutron/+/87361008:52
opendevreviewRodolfo Alonso proposed openstack/neutron stable/wallaby: [OVN] Add missing LSP device_owner info in trunk driver  https://review.opendev.org/c/openstack/neutron/+/87361108:52
opendevreviewRodolfo Alonso proposed openstack/neutron stable/wallaby: [OVN] Bump the port revision number in trunk driver  https://review.opendev.org/c/openstack/neutron/+/87361608:52
opendevreviewRodolfo Alonso proposed openstack/neutron stable/zed: [OVN] Bump the port revision number in trunk driver  https://review.opendev.org/c/openstack/neutron/+/87361208:52
opendevreviewRodolfo Alonso proposed openstack/neutron stable/yoga: [OVN] Bump the port revision number in trunk driver  https://review.opendev.org/c/openstack/neutron/+/87361308:53
opendevreviewRodolfo Alonso proposed openstack/neutron stable/xena: [OVN] Bump the port revision number in trunk driver  https://review.opendev.org/c/openstack/neutron/+/87361408:53
*** gibi_ is now known as gibi09:00
opendevreviewLajos Katona proposed openstack/neutron master: WIP: desperate try to make DVR functional tests more stable  https://review.opendev.org/c/openstack/neutron/+/87311109:30
opendevreviewFrode Nordahl proposed openstack/neutron master: [ovn] Add helper for retrieving LR associated with LRP  https://review.opendev.org/c/openstack/neutron/+/87369811:23
opendevreviewFrode Nordahl proposed openstack/neutron master: [ovn] Apply soft anti-affinity for LRs with multiple LRPs when scheduling  https://review.opendev.org/c/openstack/neutron/+/87369911:23
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider master: Avoid use of ovn metadata port IP for HM checks  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/87342611:42
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider master: Avoid use of ovn metadata port IP for HM checks  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/87342611:48
*** dasm|off is now known as dasm13:10
opendevreviewArkady Shtempler proposed openstack/neutron-tempest-plugin master: Using "Designate API" in addition to DNS query client check  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/85987113:32
opendevreviewyatin proposed openstack/neutron-tempest-plugin master: [Wallaby] Fix tempest_exclude for ovs iptables_hybrid  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/87370813:56
ykarellucasagomes, hi, can you check https://bugs.launchpad.net/neutron/+bug/2007167 related to ovn dhcp baremetal14:00
ykarelthx in advance14:00
opendevreviewMerged openstack/neutron-tempest-plugin master: Clean definitions of the CI jobs for EOL'ed branches  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/87273614:01
ralonsohping bcafarel, elvira, frickler, mlavalle, mtomaska, obondarev, sahid, slawek, tobias-urdin, ykarel, lajoskatona, jlibosva, averdagu 14:01
ralonsoh#startmeeting networking14:01
opendevmeetMeeting started Tue Feb 14 14:01:27 2023 UTC and is due to finish in 60 minutes.  The chair is ralonsoh. Information about MeetBot at http://wiki.debian.org/MeetBot.14:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:01
opendevmeetThe meeting name has been set to 'networking'14:01
elvirao/14:01
ralonsohhello all14:01
obondarevhi14:01
lajoskatonao/14:01
frickler\o14:01
sahido/14:02
bcafarelo/14:02
ykarelo/14:02
ralonsohok, let's start14:02
* mlavalle is an internal company meeting. will lurk in the background14:02
ralonsoh#topic announcements14:02
ralonsohAntelope / 2023.1 schedule: https://releases.openstack.org/antelope/schedule.html14:03
ralonsohthis week is R-514:03
slaweqo/14:03
ralonsohand that is antelope-3 milestone14:03
ralonsohremember this he final release for client libraries14:03
ralonsohand the feature freeze14:03
ralonsohthe RC1 will be in 2 weeks14:04
ralonsohalso remember that you have time to propose your candidacy for Neutron PTL (and other projects)14:04
ralonsohplease check the calendar in https://governance.openstack.org/election/14:04
ralonsohand a couple of needed links for the next PTG14:05
ralonsohBobcat etherpad: https://etherpad.opendev.org/p/neutron-bobcat-ptg14:05
lajoskatona\o/14:05
ralonsohplease start adding your topics to the Bobcat Neutron etherpad14:05
lajoskatona+114:05
ralonsohduring the next meetings, we'll check them and make the schedule14:06
ralonsohI'll remember this link every week 14:06
ralonsohI also want to talk about the Vancouver meeting14:07
ralonsohbut, if you don't mind, I'll move this topic to the "on demand" section14:07
lajoskatonaok, I anyway added the advertisement for Vancouver forum proposals14:07
ralonsohlajoskatona, of course!14:07
lajoskatonaI mean I added to the on-demand agenda14:07
ralonsohright, I didn't see that14:08
ralonsohcool14:08
ralonsohand as usual, please check https://openinfra.dev/live/#all-episodes14:09
ralonsohok, I think I'm not missing anything14:10
ralonsohrelated to the release schedule or anything14:10
ralonsohok, let's move to the next topic then14:10
ralonsoh#topic bugs14:10
ralonsohlast week report is from elvira 14:11
ralonsoh#link https://lists.openstack.org/pipermail/openstack-discuss/2023-February/032167.html14:11
ralonsohso far, there are some bugs still not assigned14:11
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/200695314:11
ralonsohI think ykarel has sent a patch recently14:11
ykarelyes14:12
ralonsohright: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/87370814:12
ralonsohok thanks!14:12
ralonsohahhh ok, I see the issue, thanks!14:12
ralonsohthen next one is14:13
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/200668314:13
ralonsohbut I think ykarel sent another patch14:13
ykarelno lajoskatona sent it14:13
ralonsohah yes14:13
ralonsohand is merged now: https://review.opendev.org/c/openstack/neutron-fwaas/+/87336014:13
ralonsoh(cool)14:13
ykarelbut not full fix14:13
ralonsohwhat else is missing?14:14
ykareli commented there https://bugs.launchpad.net/neutron/+bug/2006683/comments/3, can you check if that makes sense14:14
lajoskatonaexactly, I saw this in the logs  but I am sure that it will not solve all issues, just cleans th log before the exact traceback14:14
lajoskatonaykarel: I check your comment later, I missed that14:15
ralonsohok so we are still seeing txn errors14:15
ralonsohdue to the slqalchemy migration14:15
lajoskatonayeah, but as I see sqalchemy2 will not be "included", at least the req bump patch is still red14:16
lajoskatonaand bauzas from Nova asked to skip it14:16
ykarellajoskatona, ack, we can check few more runs if it's sees again with your fix included14:16
ykarelyes the current failures are with sqlalchmey-1.414:16
ralonsohin any case, we need to check what is the txn that is failing, or the operation14:17
ralonsohlet's keep an eye on this one14:17
ykarel+114:17
lajoskatona+114:17
ralonsohthe last one I had is14:18
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/200614514:18
lajoskatonathis one is for neutron-dynamic-routing as I see14:18
ralonsohmaybe ltomasbo ^^ can help us on this one14:18
fricklerI'll take a look, too.14:19
ralonsohbut I would say that what they are testing could not be supported14:19
fricklerI see the use case, otoh the current system is kind of working as designed14:20
ralonsohthey are manually stopping the RPC communication14:20
ralonsohfrickler, it is?14:20
fricklerwell the current implementation assumes that rmq is always available14:21
ralonsohright14:21
fricklerthat's why we have HA for it14:21
fricklerbut if operators really need to shut it down for upgrades, we could try to adapt for that14:21
ralonsohso that could be a RFE, not a bug14:21
frickleryes, I agree to that14:22
ralonsohperfect, do you mind commenting on the bug?14:22
fricklerI also wonder how everything else in openstack handles an rmq outage14:22
ltomasbofrickler, ralonsoh: I know when using FRR you have some options to keep those routes. As NDR has its own speaker implementation, I suppose either that needs to be enabled (or added if missing)14:22
fricklerI'll have a look at the code and then comment, yes14:22
ralonsohother agents (OVS, DHCP, etc) keep running if no info from the server is needed14:23
* mlavalle back in the meeting14:23
ralonsohof course, if they need to request something from the server, the operation will stop14:23
fricklerdon't they do periodic refresh, too?14:23
ralonsohthey do heartbeats14:24
ralonsohbut that doesn't stop the agent14:24
ralonsohif the need to do a resync, then yes14:24
ralonsohin any case, I would need to check the ndr agent code too14:25
frickleryes, let's dig a bit and come back to it next week or outside the meeting14:25
ralonsohfrickler, ltomasbo thanks for the info14:25
ralonsohThis week ykarel is the deputy, next week will be bcafarel (mtomaska can't next week)14:26
ykarelyes14:26
bcafarelralonsoh: ack thanks for the reminder!14:26
ralonsohjust a heads-up: we are experiencing random issues in the CI, more frequent as before14:26
ralonsohwe will talk about this during the CI meeting14:27
ralonsohI know ykarel has addressed some of them during this week14:27
ralonsohbut we need, somehow, to stabilize the CI14:27
ralonsohok, let's move to the next topic14:28
ralonsoh#topic specs14:28
ralonsohthe only one open (an active) is 14:28
ralonsoh#link https://review.opendev.org/c/openstack/neutron-specs/+/87003014:28
ralonsohI've been following the discussion between Dmitrii and lajoskatona 14:28
ralonsohI still need to check some of the new API changes and the architecture14:29
ralonsohbut I don't think we should merge it in this cycle14:29
lajoskatonaas I see there are open topics for those items14:30
ralonsohright14:30
lajoskatonaagree, but the progress and responsibility is good as I see14:30
ralonsohfor sure and that will be attended during the next cycle without nay doubt14:30
ralonsohIMO, the spec is close to be ready14:31
ralonsohbut not this week14:31
ralonsohso I'll add a comment today14:31
ralonsohthat's all on this topic14:31
ralonsohlet's move to the next14:31
ralonsohnext one14:31
ralonsoh#14:32
ralonsoh(uffff)14:32
ralonsoh#topic community_goals14:32
ralonsoh1) Consistent and Secure Default RBAC 14:32
ralonsoh#link https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/86751814:32
ralonsohslaweq has done a fantastic job during this release14:32
ralonsohshould we merge this patch ^^ now?14:32
ralonsohor should we wait 2 weeks?14:33
ralonsohslaweq ^14:33
ralonsoh(I think he is in other meeting at the same time)14:34
bcafarelhow stable is the job? (or should be)14:34
bcafarelmostly thinking of CI stability these days :)14:34
ralonsohit is quite stable14:34
ralonsoha couple of failures in fwaas and bgp14:34
ralonsohbut seems stable14:35
lajoskatonaI think instability is not coming from this now but from general issues in all other jobs14:35
slaweqthis job is pretty stable14:35
slaweqthe same as all other similar jobs14:35
slaweqyes, I agree14:35
ralonsohlajoskatona, right14:35
ralonsohok, then I'll change my vote on this one14:35
slaweqthose neutron-tempest-plugin-xxx jobs are IMHO one of the most stable once recently :)14:35
ralonsohyes14:35
slaweq++14:35
bcafarelif it stable big +1 to have it running and seeing results already on these (before switching next cycle)14:35
slaweqthx14:35
ralonsohperfect14:36
slaweqwill be good to have it now so we will also have it in stable/anthelope later :)14:36
ralonsohok, I've increased the review priority14:36
slaweqin next cycle we will change it to be only job running with "old default policies" as we will switch to use new ones by default14:37
ralonsohperfect so new ones will be the default ones14:37
slaweqin 2023.2 yes14:38
slaweqin 2023.1 we will have one job which will test them14:38
ralonsohjust out of curiosity, what is the status of other projects?14:39
slaweqgmann knows it the best but AFAIK nova, glance, cinder are all good to switch also14:39
slaweqand keystone of course14:39
slaweqI don't really know much about other projects14:39
ralonsohgood to know that big projects are on track too14:40
slaweqhttps://review.opendev.org/q/topic:secure-rbac14:40
slaweqthere is many patches in that topic :)14:40
ralonsohok, in some weeks we'll swap to the new policies by default14:41
slaweqyes14:41
ralonsohperfect, we'll check at this time then14:41
ralonsohthanks!14:41
ralonsohthe next sub-topic is14:42
ralonsoh2) Neutron client deprecation 14:42
ralonsohlajoskatona, we have new releases, right?14:42
lajoskatonaActually no news from my side14:42
ralonsohso we can continue with some patches now14:42
lajoskatonaperhaps one is htat I was waiting for SDK release, but as it is now out the req bump is still missing14:42
ralonsohah right14:42
ralonsoh(I always forget that second step)14:43
ralonsohnever mind14:43
ralonsohok, thanks lajoskatona!14:43
lajoskatonaSo that is true for other libs also, we have releases finally but we still waiting for he u-c files to be upgraded14:43
ralonsohand the last topic for today is14:44
ralonsoh#topic on_demand14:44
ralonsohlajoskatona, please14:44
lajoskatonaI just wanted to advertise the possibilty to propose forum topics for the summit14:44
ralonsoh#link https://etherpad.opendev.org/p/neutron-bobcat-ptg14:45
lajoskatonaI think the forum schedule will be finalized mid April, and I suppose we have topropose before that14:45
lajoskatonahttps://openinfra.dev/summit/vancouver-2023/forum/14:45
slaweqgood point lajoskatona 14:45
mlavalleThanks for the reminder14:45
ralonsohyeah, aboyt this14:45
ralonsohone sec14:45
slaweqmaybe we can open another etherpad to collect some ideas there14:46
ralonsohI opened this second etherpad14:46
ralonsohhttps://etherpad.opendev.org/p/neutron-vancouver-202314:46
ralonsoh^^^14:46
slaweqralonsoh++14:46
lajoskatonathanks, +114:46
ralonsohso that's the point: what kind of topics are we going to discuss in Vancouver (after the virtual PTG)?14:46
ralonsohthe PTG is in March, Vancouver summit is in June (3 months)14:47
slaweqralonsoh I think that those are 2 different things: one is our "ptg" in vancouver and second are topics for the forum session, to e.g. collect feedback from operators, etc.14:47
slaweqbut we can use the same etherpad for now to collect ideas for both14:47
ralonsohright, we can open different topics in the agenda14:48
lajoskatonaagree, but I see the Vancouve mid-cycle meeting as a frre to use opportuninty for disccusion14:48
ralonsohin any case, I'll send a mail today with both links14:49
ralonsohjust to make it public for everyone 14:49
lajoskatonabut an agenda is good idea14:49
ralonsohto be honest (but this is my opinion) I don't see the relevance of the Vancouver PTG14:50
ralonsohanyway, as commented, I'll send a mail today with both agendas14:51
frickler+1, excluding the non-travelling part of the community isn't a good idea IMO14:51
lajoskatona+114:51
lajoskatonayeah but the meeting opprotunity is good, and I hope there will be option to have it hybrid14:52
ralonsohI agree on this14:52
ralonsohok folks, is there anything else to be discussed today?14:53
ralonsoh(please remember the CI meeting in 6 minutes)14:54
slaweq++14:54
slaweqsee You in few minutes on CI meeting14:54
ralonsohsee you there! bye14:54
ralonsoh#endmeeting14:54
opendevmeetMeeting ended Tue Feb 14 14:54:39 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:54
opendevmeetMinutes:        https://meetings.opendev.org/meetings/networking/2023/networking.2023-02-14-14.01.html14:54
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/networking/2023/networking.2023-02-14-14.01.txt14:54
opendevmeetLog:            https://meetings.opendev.org/meetings/networking/2023/networking.2023-02-14-14.01.log.html14:54
lajoskatonaBye14:54
ykarelo/14:54
ralonsohslaweq, video or IRC?14:54
ralonsoh(I never remember)14:54
mlavalleslaweq: video or irc today?14:54
slaweqirc today14:54
ralonsohperfect14:54
mlavalleack14:55
bcafarel:)14:55
ralonsohykarel, I'm going to send a neutron patch on top of https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/87370814:55
ralonsohCI is just checking docs and pep814:56
ykarelralonsoh, ohkk should work, but ok to confirm :)14:56
ykareljust in case14:56
opendevreviewRodolfo Alonso proposed openstack/neutron stable/wallaby: DNM Testing patch of https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/873708  https://review.opendev.org/c/openstack/neutron/+/87371714:58
ykarelthx14:59
slaweq#startmeeting neutron_ci15:00
opendevmeetMeeting started Tue Feb 14 15:00:13 2023 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'neutron_ci'15:00
mlavalleo/15:00
slaweqping bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva15:00
bcafarelo/15:00
ykarelo/15:00
mtomaskao/15:00
slaweqhi15:00
jlibosvao/15:00
ralonsohhi15:00
slaweqGrafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=115:00
slaweqok, lets start as we have many topics for today15:00
slaweq#topic Actions from previous meetings15:00
slaweqlajoskatona to check additional logs in failing dvr related functional tests15:01
lajoskatonao/15:01
lajoskatonayes, I checked and I am still not closer15:02
lajoskatonaI pushed a patch with tries for this issue, and today it is in this state:15:02
lajoskatonahttps://review.opendev.org/c/openstack/neutron/+/87311115:02
lajoskatonait is now to run part of these tests with concurrency=115:03
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider master: Avoid use of ovn metadata port IP for HM checks  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/87342615:04
lajoskatonawhen I checked the functional failures were for other failures and not for the usual DVR related issues, but I have to check more this topic15:04
slaweqwhat about execution time? it it much longer?15:04
lajoskatonagood question, I haven't checked it yet, but I see one issue which is "too many files open"15:05
lajoskatonanot always, but from this patch's 20 runs I saw once as I remember15:05
slaweq:/15:05
lajoskatonawhich is strange as in serial I would expect less load on such things15:06
slaweqmaybe that's another issue and some files aren't closed properly?15:06
ralonsohduring the config parsing, maybe15:06
lajoskatonapossible, I can check that15:07
slaweqthx15:07
ralonsohcan you open another bug for this issue?15:07
slaweqI will add AI for You for next week to remember about it15:07
lajoskatonaack15:07
slaweq#action lajoskatona to continue checking dvr functional tests issues15:07
slaweqnext one15:09
slaweq    ralonsoh to try to store journal log in UT job's results to debug "no such table" issues15:09
ralonsohno, I didn't work on this one15:09
ralonsohI was working on the FT ones15:09
slaweq#action ralonsoh to try to store journal log in UT job's results to debug "no such table" issues15:09
slaweqlets keep it for next week then15:09
slaweqnext one15:09
slaweqmtomaska to check failed test_restart_rpc_on_sighup_multiple_workers functional test15:09
mtomaskaI looked into it.Of course I am not able to reproduce it locally. I really think this failure happens because of stestr concurrency which I cant quite replicate on my dev machine. Adding more logs to the test wont help. Is there a way to mark the test with concurrency=1 permanently?15:10
ralonsohyou can move this test to the second FT execution15:10
ralonsohwhere we use concurrency one15:10
slaweqYou can do it in a way like lajoskatona in https://review.opendev.org/c/openstack/neutron/+/87311115:11
slaweqbut I would really like to avoid doing that for all tests :)15:11
ralonsohwe can move one single test only15:11
ralonsohsame as sql ones15:11
ralonsohor "test_get_all_devices"15:11
mtomaskaok cool. ralonsoh can I ping you on how to later?15:11
ralonsohsure15:11
slaweqbut if we will be sure that this is an issue15:11
slaweqnext one15:12
slaweqralonsoh to check Can't find port binding with logical port XXX error15:12
ralonsoh#link https://review.opendev.org/c/openstack/neutron/+/87312615:13
ralonsoh^ this is the patch15:13
slaweqgreat, at least one fixed :)15:13
slaweqthx15:14
slaweqnext one15:14
slaweqmlavalle to check LRP failed to bind failure15:14
mlavalleI checked it here: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-functional&branch=master&skip=015:14
mlavalledidn't happen again since the occurence you reported a week ago15:14
mlavalleso for now, we don't need to worry about it15:15
mlavalleit was a one of15:15
lajoskatona\o/15:15
lajoskatonagood news15:15
slaweqnot really15:15
slaweqI saw the same issue today :/15:15
slaweqhttps://32c2553a5b1b0d58bd31-9e074609c9fda723118db1a634ce1a12.ssl.cf5.rackcdn.com/873247/3/gate/neutron-functional-with-uwsgi/6195659/testr_results.html15:16
lajoskatonaohhhh15:16
mlavalleoh, I'm reporting from my last check last night15:16
mlavalleand that is a different job I think15:16
mlavallethan the one we looked at last week15:16
slaweqyes, it's different15:17
slaweqalso functional tests but it's from gate queue15:18
mlavalleyeap15:18
slaweqlast week it was in periodic15:18
mlavalleI was looking at periodic15:18
mlavalleso that tells us that test case is flaky15:19
slaweqyeap15:19
mlavalleacorss jobs15:19
mlavalleacross jobs15:19
slaweqwill You try to investigate again this week?15:19
mlavalleI'm sorry. I can't. I will be off on PTO starting tomorrow until March 1st15:19
mlavallebut if you are willing to wait, I'll be happy to look at it upon my return15:20
slaweqoh, ok15:20
slaweqI will open LP bug to not forget about it15:20
slaweqand we will see how often it will be occuring15:20
mlavalleassign it to me if nobody is going to look at it15:21
slaweq#action slaweq to report bug about failed to bind LRP in functional tests15:21
slaweqsure, thx mlavalle 15:21
mlavallethank you!15:21
slaweqand last one15:21
slaweqslaweq to check fullstack failures in test_multi_segs_network15:21
slaweqI spent some time investigating it15:21
slaweqI reported bug https://bugs.launchpad.net/neutron/+bug/200715215:21
slaweqit seems to be valid issue in the code, not test problem15:22
slaweqbut I don't know yet what's wrong exactly there15:22
slaweqthere is some race between creation of subnets in network and cleaning stale devices from the namespace15:22
slaweqI assigned this bug to myself and I will continue working on it15:23
slaweqany questions/comments?15:24
lajoskatonanothing from me15:25
slaweqif not, lets move on15:25
slaweq#topic Stable branches15:25
* bcafarel hides15:25
slaweqbcafarel any updates?15:25
slaweq:)15:25
bcafarelgood and bad this week :)15:25
bcafarelussuri grenade and train look good from a quick look thanks to ykarel, wallaby I opened a bug and also fix in progress by ykarel https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/87370815:26
bcafarelvictoria, we have 2 backports failing on py38/cover with " pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '<MagicMock name='execute().split().__getitem__().__getitem__()' id='140203724932048'>'"15:26
bcafarelhttps://zuul.opendev.org/t/openstack/build/8f14ad998131488680c8f2142d1eab9415:27
bcafarelthis one seems familiar but I could not find anything relevant, does it ring a bell? (if not will open a lp to track it)15:27
bcafarel(and other branches are good \o/)15:27
ralonsoh^^ we can force the _get_version() to return the value, not utils_exec15:29
ralonsohI'll push a patch15:29
slaweq++15:29
slaweqthx ralonsoh15:29
bcafarelralonsoh++ thanks15:29
slaweqand thx bcafarel for updates15:30
slaweqso lets move on15:31
slaweq#topic Stadium projects15:31
slaweqlajoskatona I see there is bunch of topics there in agenda15:31
slaweqare those added by You?15:31
lajoskatonamost of them seems ok15:31
lajoskatonano I think it was not me15:32
lajoskatonaI think it is for one of the not so green ones15:32
slaweqok, maybe ykarel added them15:32
lajoskatonafwaas job is failing, and it is causing failures as with tempest-plugin it is executed15:33
slaweqthere added links to failing fwaas tests15:33
ykarelfwaas one we discussed in previous meeting15:33
lajoskatonathis is the bug for this: https://bugs.launchpad.net/neutron/+bug/200668315:33
lajoskatonafor fwaas I mean15:33
slaweqwho is now fwaas liaison?15:34
lajoskatonathe other topic is for the cfg related issue (see https://review.opendev.org/c/openstack/neutron/+/872644  )15:34
lajoskatonaI think neutron-dynamic-routing and bgpvpn was the last affected by this, I have to check the fresh runs for them15:35
ralonsohI don't remember where this list is15:35
slaweqhttps://docs.openstack.org/neutron/latest/contributor/policies/neutron-teams.html15:36
slaweqfound it :)15:36
slaweqso it is zhouhenglc15:36
slaweqmaybe we should ping him to check those failures?15:36
lajoskatona+115:36
slaweqand temporary we can propose to make fwaas jobs to be non-voting if he will not reply15:36
lajoskatonaI will ping him15:37
ralonsohthanks15:37
slaweqlajoskatona++ thx15:37
slaweq#action lajoskatona to talk with zhouhenglc about fwaas jobs issues15:37
slaweqok, anything else regarding stadium today?15:37
lajoskatonanothing from me15:38
slaweqso lets move on15:38
slaweq#topic Grafana15:38
slaweqhere everything looks that are on high failure rate for me15:39
ralonsohdon't check too much the numbers15:39
ralonsohtempest ipv6: failing because of OVN issue in Jammy15:39
ralonsohFT: failign because of the test error I introduced (fixed now)15:39
slaweqyes, I also saw pretty many "DNM" or "WIP" patches where most of the jobs failed15:40
ralonsohand I've been playing with CIm using 20 FT jobs at the same time15:40
ralonsohyes15:40
ralonsohso please, don't focus this week on numbers15:40
slaweqso those results may be a bit missleading15:40
slaweqok, so lets move on to the rechecks then15:40
slaweq#topic Rechecks15:40
slaweq+---------+----------+... (full message at <https://matrix.org/_matrix/media/v3/download/matrix.org/flyUncZffilepRknBJrkPGIz>)15:40
slaweqlast week it was still very high: 2.0 rechecks in average to get patch merged15:41
ralonsohyes but always the same errors15:41
slaweqbut it's better in e.g. last 5 days or so15:41
slaweqso hopefully it will be going down once we will fix few issues mentioned already15:42
slaweqand some which I have to discuss still today :)15:42
slaweq#topic Unit tests15:42
slaweqcritical bug reported https://bugs.launchpad.net/neutron/+bug/200725415:42
ralonsohI'm on this one15:43
slaweqYou are working on it?15:43
ralonsohtyes15:43
ralonsohyes15:43
ralonsohI'll assign it to me15:43
slaweqok, please assign it to Yourself then :)15:43
slaweqthx15:43
ykarelk thx, i too was trying to reproduce it, but couldn't15:43
slaweqnext topic15:44
slaweq#topic fullstack/functional15:44
slaweqhere I found only one new issue15:44
slaweqhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_777/870081/6/check/neutron-functional-with-uwsgi/7775413/testr_results.html15:44
slaweqsomething related to mysql errors15:44
ralonsohagain? maybe those tests have been executed at the same time, using mysql15:45
ralonsohright, mysql15:45
ralonsohI would't investigate it, this is just a test clash15:46
mtomaskaa test called "test_1" should not be even considered as a real test :) . Thats just bad naming 15:46
slaweqmtomaska yeah :)15:47
slaweqralonsoh but both were run with "--concurrency 1"15:47
ralonsohmy bad when I pushed this patch15:47
slaweq2023-02-10 11:29:15.131934 | controller | ======... (full message at <https://matrix.org/_matrix/media/v3/download/matrix.org/AJmoLvYurxYfOWemkJFUhUBQ>)15:48
ralonsohslaweq, yes, you are right15:48
slaweqmy bet is that it was oom-killer what caused failure of first test15:49
slaweqso cleanup wasn't done15:49
slaweqand that's why second failed15:49
ralonsohyes, the 2nd is caused by the first one15:49
slaweqbut I didn't checked it in journal logs15:49
ralonsohand the first because of a timeout15:50
ralonsohI wouldn't spend too much time on this one15:50
slaweqyeah, lets move on to the other topics15:50
slaweq#topic grenade15:50
slaweqI added this today as we have pretty many failures there15:50
slaweqfirst, added by ykarel15:51
slaweqseeing pcp deploy issues in these jobs randomly, basically can be seen in jobs where dstat is enabled15:51
slaweqykarel do You think that disabling dstat in grenade jobs will fix that problem (or workaround)?15:51
ykarelslaweq, workaroud15:51
ykarelas actual fix would need in pcp package15:52
slaweqok15:52
slaweqwill You propose patch?15:52
ykarelsure will do that15:52
ralonsohykarel++15:52
slaweqthx a lot15:52
slaweqanother issue which I have seen in grenade this week was related to failed ping to server:15:52
slaweq    https://58140b717b0fd2108807-1424e0f18ed1d65bb7e7c00bb059b2d8.ssl.cf2.rackcdn.com/872830/1/gate/neutron-ovs-grenade-multinode/6b0f11b/controller/logs/grenade.sh_log.txt15:52
slaweq    https://bca8357d212b699db2ea-90e26954923e94ff0df2f7d3073d9fdf.ssl.cf2.rackcdn.com/871983/3/check/neutron-ovn-grenade-multinode-skip-level/fd1c1fa/controller/logs/grenade.sh_log.txt15:52
slaweq    https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_01d/873615/2/check/neutron-ovs-grenade-dvr-multinode/01d90b6/controller/logs/grenade.sh_log.txt15:52
slaweqI found at least those 3 occurences15:53
slaweqso I think we should investigate it15:53
slaweqanyone wants to check it?15:53
ralonsohsorry, not this week15:53
slaweqno need to sorry ralonsoh  :)15:53
slaweqif nobody have any cycles, I will report LP for it and we will see15:54
slaweq#action slaweq to report bug with failed ping in grenade jobs15:54
ykarel+115:54
slaweqnext topic15:54
slaweq#topic Tempest/Scenario15:54
slaweqhere I saw bunch of tests failing with ssh timeout15:55
slaweqlike e.g. tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic:15:55
slaweq    https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f1f/872737/3/gate/neutron-ovn-tempest-ipv6-only-ovs-release/f1f5212/testr_results.html15:55
slaweq    https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_18b/864649/7/check/neutron-ovn-tempest-ipv6-only-ovs-release/18b432e/testr_results.html15:55
slaweq    https://74b83773945845894408-fddd31f569f44d98bd401bfd88253d97.ssl.cf2.rackcdn.com/873553/1/check/neutron-ovn-tempest-ipv6-only-ovs-release/6800220/testr_results.html15:55
ralonsohykarel, founf the error15:55
ralonsohhttps://review.opendev.org/c/openstack/neutron/+/87368415:55
ralonsohI think we should mark it as unstable15:55
ralonsohuntil OVN is fixed in Jammy15:56
ykarelyes from logs i found that metadata flows were missing15:56
slaweqthat's great15:56
ykareland issue is not happening in focal15:56
ykarelso it's specific to ovn22.03.0 in ubuntu jammy15:56
slaweqcan we skip this one test only or it can happen in other tests too?15:56
ralonsohonly this one15:56
ralonsohso far15:56
slaweqso lets skip that test in the job's definiton temporary and keep job voting15:57
ykarelyes seen in this only tests, but can happen to others too15:57
slaweqif it will happen in other tests too, we can always mark job as non-voting temporary later15:57
slaweqwdyt?15:57
ralonsohwe can skip this test only, for now15:57
ykarel+1 to what ralonsoh said, and monitor the job15:58
slaweqok, ykarel will You propose patch for that?15:58
ykarelok will do15:58
slaweqthx15:58
slaweqok, we are almost on top of the hour15:58
slaweqbut I have one last thing for today15:58
slaweqthere is question from amorin in https://review.opendev.org/c/openstack/neutron/+/86974115:59
amorinhello!15:59
slaweqit's related to CI and neutron-ovs-tempest-dvr-ha-multinode-full15:59
ralonsohyes, I talked to him about this15:59
slaweqand I think it's valid question15:59
ralonsohdvr_snat is not valid in compute15:59
ralonsohbut we are still configuring it15:59
slaweqI would like obondarev to look into this one too15:59
slaweqIMO we can change L3 agent's mode to "dvr" in compute in that job16:00
ykarelfwiw this job used to be 3 node in past, when it was switched to 2 node dvr_snat was set16:00
slaweqand keep "ha=True" so it will be "ha" routers with only one node16:00
ralonsohyeah...16:00
slaweqthe same as we are doing in neutron-tempest-plugin- jobs16:00
slaweqcode path for ha will be tested16:00
slaweqthere will be no failover only really but that's not something what we are testing really in this job16:01
slaweqok, we are out of time16:01
slaweqplease comment on this patch and we can discuss there16:01
ralonsohok16:01
slaweqthx for attending the meeting today16:01
slaweq#endmeeting16:01
opendevmeetMeeting ended Tue Feb 14 16:01:43 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:01
opendevmeetMinutes:        https://meetings.opendev.org/meetings/neutron_ci/2023/neutron_ci.2023-02-14-15.00.html16:01
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/neutron_ci/2023/neutron_ci.2023-02-14-15.00.txt16:01
opendevmeetLog:            https://meetings.opendev.org/meetings/neutron_ci/2023/neutron_ci.2023-02-14-15.00.log.html16:01
ralonsohbye16:01
mlavalleo/16:01
bcafarelo/16:01
ykarelo/16:01
lajoskatonao/16:02
mtomaskao/16:03
opendevreviewLajos Katona proposed openstack/networking-bagpipe master: CI: Add periodic weekly job with sqlalchemy master  https://review.opendev.org/c/openstack/networking-bagpipe/+/87240816:07
opendevreviewLajos Katona proposed openstack/networking-bgpvpn master: CI: add oslo_master and sqlalchemy to periodic weekly  https://review.opendev.org/c/openstack/networking-bgpvpn/+/86196016:08
opendevreviewMerged openstack/neutron master: Remove tripleo related jobs from neutron queues  https://review.opendev.org/c/openstack/neutron/+/87368516:25
opendevreviewyatin proposed openstack/neutron master: [ovn-ipv6] Temporary exclude test_hotplug_nic  https://review.opendev.org/c/openstack/neutron/+/87368416:28
ykarelralonsoh, slaweq when u get chance a backport https://review.opendev.org/c/openstack/neutron/+/87198916:32
ykarelthx in advance16:32
ralonsohsure16:35
opendevreviewRodolfo Alonso proposed openstack/neutron master: Mock ``IptablesManager._get_version`` in ``test_floating_forward_rules``  https://review.opendev.org/c/openstack/neutron/+/87373417:24
opendevreviewRodolfo Alonso proposed openstack/neutron master: [UT] Mock the retry decorator when creating duplicated NDPProxy  https://review.opendev.org/c/openstack/neutron/+/87373918:02
gmannslaweq: on RBAC default switch update, nova has done, glance patch under review but cinder is not known. I think they might do in next cycle as there were some system scope things to cleanup18:14
opendevreviewBrian Haley proposed openstack/neutron stable/xena: Fix deprecation warnings in pkg_resources  https://review.opendev.org/c/openstack/neutron/+/87358519:19
opendevreviewBrian Haley proposed openstack/neutron stable/wallaby: Fix deprecation warnings in pkg_resources  https://review.opendev.org/c/openstack/neutron/+/87358619:19
opendevreviewBrian Haley proposed openstack/neutron stable/victoria: Fix deprecation warnings in pkg_resources  https://review.opendev.org/c/openstack/neutron/+/87358719:23
opendevreviewBrian Haley proposed openstack/ovsdbapp master: Add more IDL condition comparisons  https://review.opendev.org/c/openstack/ovsdbapp/+/87374819:56
opendevreviewMerged openstack/neutron stable/ussuri: Never raise an exception in notify()  https://review.opendev.org/c/openstack/neutron/+/87198922:56
« Monday, 2023-02-13 Index Wednesday, 2023-02-15 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!