Tuesday, 2021-12-21

opendevreviewMerged openstack/neutron-vpnaas master: Add Python3 yoga unit tests  https://review.opendev.org/c/openstack/neutron-vpnaas/+/80979702:03
qinhaizhong#openstack-dev02:56
qinhaizhongjoin #openstack-dev02:56
qinhaizhong#openstack-dev02:56
qinhaizhongchange #openstack-dev02:58
opendevreviewyatin proposed openstack/neutron-tempest-plugin master: [DNM] Test switch  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/82196504:54
opendevreviewyatin proposed openstack/neutron-tempest-plugin master: [DNM] Test switch  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/82196505:01
opendevreviewyatin proposed openstack/neutron-tempest-plugin master: [DNM] Test switch  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/82196505:15
opendevreviewyatin proposed openstack/neutron-tempest-plugin master: [DNM] Test switch  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/82196505:20
opendevreviewyatin proposed openstack/neutron-tempest-plugin master: [DNM] Test switch  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/82196505:24
opendevreviewyatin proposed openstack/neutron-tempest-plugin master: [DNM] Test switch  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/82196505:31
opendevreviewyatin proposed openstack/neutron-tempest-plugin master: [DNM] test override  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/82238305:34
*** ykarel__ is now known as ykarel05:39
qinhaizhongHi,05:49
qinhaizhong I noticed that the neutron-fwaas project has been deprecated,but the firewall project is a necessary function when the project is delivered. The lack of firewall function after switching OVN is not acceptable to customers. We intend to maintain this project and develop the fwaas driver based on ovn. Whether the neutron-fwaas project can be reactivated, What should I do ?05:49
opendevreviewMerged openstack/neutron master: [Functional] Fix expected number of the enqueue_state_change calls  https://review.opendev.org/c/openstack/neutron/+/82205108:01
opendevreviewTakashi Kajinami proposed openstack/neutron stable/ussuri: bw-limit: Pass int parameters to Open vSwitch  https://review.opendev.org/c/openstack/neutron/+/82241309:03
opendevreviewJorhson Deng proposed openstack/python-neutronclient master: Update python testing classifier  https://review.opendev.org/c/openstack/python-neutronclient/+/82244609:34
opendevreviewJaganathan Palanisamy proposed openstack/neutron master: Missing OvS DPDK nodes in ovn-controllers  https://review.opendev.org/c/openstack/neutron/+/82246510:24
slaweqralonsoh: hi, can You check https://review.opendev.org/c/openstack/neutron/+/822098 ?10:45
slaweqthx in advance10:45
ralonsohsure10:45
slaweqthank You10:53
qinhaizhongslaweq: Hi,I noticed that the neutron-fwaas project has been deprecated,but the firewall project is a necessary function when the project is delivered. The lack of firewall function after switching OVN is not acceptable to customers. We intend to maintain this project and develop the fwaas driver based on ovn. Whether the neutron-fwaas project can be reactivated, What should I do ?11:36
opendevreviewSlawek Kaplonski proposed openstack/neutron master: Allocate IPs in bulk requests in separate transactions  https://review.opendev.org/c/openstack/neutron/+/82172711:56
slaweqralonsoh: if You have some time, please check ^^ once again11:57
ralonsohsure11:57
slaweqqinhaizhong: hi, first of all You should probably talk with lajoskatona about it as he is our PTL now11:58
slaweqbut probably You will need to revive project as an unofficial one, in the x/ namespace first11:58
slaweqand maintain it there11:58
slaweqlater we can talk about eventually include it in the neutron stadium again11:59
slaweqbut that will need to be discussed with whole drivers team11:59
slaweqfor now, for sure doing it in the x/ namespace would be the fastest way to start working on it11:59
qinhaizhongThank you, I will communicate with lajoskatona,but how to revive the neutron-fwaas project to x/ namespace?12:02
slaweqqinhaizhong: TBH I don't know, I never did anything like that12:03
slaweqplease send email to the OpenStack discuss ML or ask on the #openstack-dev channel - there should be people who knows how to do it12:04
qinhaizhongslaweq: I would send an email, thx12:05
qinhaizhongslaweq: thank you12:06
labedzhi guys, quick question: is it on purpose that for DVR L3 agent there is no notification on VM port status change ACTIVE->DOWN and DOWN->ACTIVE which leads that stopped VM (so no tap interface) still has qrouter-* namespace configured?12:17
slaweqlabedz: I'm not sure if it's on purpose but in fact IMO it makes sense - such qrouter namespace shouldn't consume a lot of resources and router will be ready when You will start vm again12:22
labedzthen if you have related router then there gonna be more then one qrouter namespace with maybe some fip namespaces plus some snat-* machinery on dvr-snat nodes... all for that one stopped VM ;)12:27
dulekHi! I think we started to see elevated times to see ports moved to ACTIVE on Neutron's master. Can you check if there's anything unusual about port b07aa905-201b-4ced-9605-b5ca8e070ca5 in these logs: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4bc/818814/3/check/kuryr-kubernetes-e2e-np/4bc5772/controller/logs/screen-q-svc.txt12:34
dulek?12:34
dulekcc gryf 12:34
opendevreviewMerged openstack/neutron master: Change tobiko CI job in the periodic queue  https://review.opendev.org/c/openstack/neutron/+/81397713:32
opendevreviewMerged openstack/neutron master: Don't fail subnet validation if gw_ip is actually not changed  https://review.opendev.org/c/openstack/neutron/+/82209813:32
slaweqlabedz:  I think that it's like that because when You stop vm its port isn't really removed by nova. So it's still bound to host. I don't know how much changes would You need to do to clean qrouter in such case. Maybe You can open bug for that and it can be discussed with folks like obondarev or liuyulong who knows dvr code better than me13:34
opendevreviewMerged openstack/neutron-tempest-plugin master: Increase guest os ready timeout  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/81961214:05
slaweq#startmeeting neutron_ci15:00
opendevmeetMeeting started Tue Dec 21 15:00:42 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'neutron_ci'15:00
slaweqhi15:00
ralonsohhi15:00
slaweqthis week we should have meeting on video15:01
slaweqbut I can't open https://meetpad.opendev.org/neutron-ci-meetings15:01
slaweqis it working for You?15:01
ralonsohno, it isn't15:01
slaweq:/15:01
slaweqat least it's not something on my side15:01
bcafarel:)15:01
slaweqok, so lets do meeting on irc only15:01
slaweqand lets do it quick15:01
mlavalleok15:02
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:02
ykarelo/15:02
slaweqI think we can start as lajoskatona is on pto already and I also don't see obondarev to be online15:02
slaweq#topic Actions from previous meetings15:02
slaweq    slaweq to add some extra logs to the test, related to https://bugs.launchpad.net/neutron/+bug/1954751 to help further debugging15:03
slaweqI proposed fix for that https://review.opendev.org/c/openstack/neutron/+/822051 and it's merged now15:03
slaweqit also added some extra logs15:03
slaweqbcafarel: I will look to which versions we should backport that change15:03
slaweq:)15:03
bcafarelthanks :)15:03
slaweqnext one15:04
slaweqmlavalle to check failing neutron-ovn-tempest-ovs-master-fedora job15:04
mlavalleI did some fact finding15:04
mlavalleit seems we have a problem with mariadb15:04
mlavallelet me share with you some data...15:04
mlavallehttps://paste.openstack.org/show/811785/15:05
mlavallelook at lines 20 to 2815:05
mlavallecommand 'sudo mysqladmin -u root password secretdatabase' is failing15:06
mlavalleso later, when trying to create a user in the DB, root doesn't have privilege15:06
mlavallelet me show you the same thing from the last succesful run:15:07
mlavallehttps://paste.openstack.org/show/811784/15:07
mlavallelook at lines 26 to 3115:07
mlavallesame commands succeed15:07
mlavalleit's the same Fedora release, 3415:08
ralonsohproblem could be devstack script, method "configure_database_mysql"15:08
mlavallehowever, if you look at both pastes, in the lines above, it is not installing the same thing15:08
ralonsohwe are checking this 15:08
ralonsohif is_ubuntu && [ "$MYSQL_SERVICE_NAME" == "mariadb" ]; then15:08
mlavallein the successful case we install mariadb-3:10.5.12-1.fc34.x86_6415:09
mlavallewhereas is the fail case we install community-mysql-8.0.27-1.fc34.x86_64 15:10
mlavalleisn't that weird15:10
mlavalle?15:10
ralonsohno15:10
ralonsohthere are different instalation methods15:10
ralonsohdepending on the service15:10
slaweqhmm15:10
ralonsohand I think we don't capture the OS correctly in devstack15:10
fricklerdevstack+fedora seems currently broken15:10
fricklerwas just mentioning that in -qa, too15:10
mlavallefrickler: is that in general, not only neutron?15:11
frickleryes15:11
mlavalleis someone fixing it?15:11
mlavalleor is there a reported bug?15:12
fricklerwell currently no. needs some investment from redhat or whoever cares about it15:12
mlavalleok, that's us I think15:12
slaweqmlavalle: probably we can invest some time in it :)15:12
mlavalleyeap15:13
mlavalleI'll create a bug in devstack15:13
slaweqbut it should be at least easy to reproduce locally if it's such general problem with devstack on fedora15:13
slaweqmlavalle++ thx15:13
mlavalleslaweq: yeah, I will try to reproduce it locally 15:13
slaweqthx15:13
slaweqand thx for investigation on that one15:13
slaweqok, I think we can move on for now15:14
mlavalleyes, thx15:14
slaweq#topic Stable branches15:14
slaweqbcafarel: any updates/issues?15:15
ralonsohussuri is broken15:15
bcafarelwe have https://bugs.launchpad.net/tempest/+bug/1955418 on train (gmann++ on it)15:15
ralonsohat least grenade15:15
gmannyeah, stable/train is also broken #link http://lists.openstack.org/pipermail/openstack-discuss/2021-December/026405.html15:15
gmannI am on fixes which seems working, octavia passed. waiting for neutron testing patch to pass15:15
bcafarelthanks for working on it :) and thanks ralonsoh for reporting ussuri, I did not have time yet to check that15:15
gmannis there any bug for ussuri?15:16
ralonsohhttps://bugs.launchpad.net/neutron/+bug/195548615:16
ralonsoha problem with oslo.utils15:16
ralonsohand the version used in Ussuri15:16
ralonsohthe problem is I don't know how to cap this or implement a fix for stable tempest releases15:17
ralonsohbecause we don't have this15:17
gmannralonsoh: i see, that is one I am fixing for stable/train using tempest 28.0.015:18
slaweqright, if it's grenade in ussuri, it first tries to install train15:18
gmannand as stable/ussuri greande use stable/train tempest version it happen there15:18
slaweqso the issue can be the same :)15:18
ralonsohwhy are we using the version in ussuri?15:18
ralonsohok then15:18
gmannso same fix will work there, I will add testing patch for stable/ussuri to verify15:18
slaweqgmann++ thx a lot15:19
gmannthese two fixes #link https://review.opendev.org/c/openstack/devstack/+/822380 #link https://review.opendev.org/c/openstack/tempest/+/82233915:19
ralonsohI'll mark this bug as a duplicate15:19
gmannsure.15:19
slaweqfrom good news regarding stable/train, we have merged https://review.opendev.org/c/openstack/requirements/+/82197215:19
ralonsohyes15:20
slaweqthx ralonsoh for fixing that15:20
slaweqbcafarel: other than that issue, we should be good for stable branches' ci, right?15:21
bcafarelindeed15:21
slaweqgood :)15:21
slaweqthx15:21
slaweqso lets move on to the next topic15:21
slaweqas lajoskatona is away already, lets skip stadium projects today15:22
slaweqand go directly to the next topic15:22
slaweq#topic Grafana15:22
slaweqhttp://grafana.openstack.org/dashboard/db/neutron-failure-rate15:22
slaweqTBH graphs looks really ok for me15:23
slaweqmaybe it's some kind of Christmas gift for us :D15:23
mlavalleSanta came early to Neutrontown \o/15:24
slaweqLOL, yeah!15:24
bcafarel:)15:24
slaweqand also my weekly check of the failed jobs confirmed that15:25
slaweqI didn't saw many errors not related to the patches on which it was run15:25
slaweq#topic Tempest/Scenario15:25
slaweqagain I saw couple of timeouted jobs:15:25
slaweq    https://4b558001c4cbf62a997f-78e633edf2b137bdab04e16fad5df952.ssl.cf1.rackcdn.com/821433/2/check/neutron-tempest-plugin-scenario-linuxbridge/315a4ed/job-output.txt... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/klqERVtiMhWBIRMNARsRvTLN)15:25
slaweqinteresting thing is that all of those issues which I see are in the neutron-tempest-plugin-scenario- jobs15:26
slaweqwhich IIRC are moved to use nested virtualization now, right ykarel ?15:27
ralonsohyes15:27
opendevreviewGhanshyam proposed openstack/neutron stable/ussuri: DNM: test tempest train-last tag  https://review.opendev.org/c/openstack/neutron/+/82250415:27
gmannralonsoh: slaweq ^^ testing grenade job15:27
slaweqcan it be somehow related?15:27
ralonsohthanks15:27
ykarelslaweq, out of 4 i see 3 are older than 7 days15:28
ykareland not using nested virt nodes15:28
ykarel4th one is on nested virt node for which you mentioned it didn't finished devstack15:28
ralonsohthis is the one I'm checking15:29
ralonsohand seems to be too slow executing any command15:29
slaweqok, so maybe going for nested virt will actually improve it15:29
slaweqralonsoh: yes, that one can be just slow node15:29
ralonsohright15:29
slaweqbut in other cases the slowest were tests actually15:30
slaweqand for that nested virt may help15:30
slaweqok, lets keep an eye on it for next week(s) and we will see15:30
ykarelyes15:30
slaweqnext issue15:31
slaweqI opened new bug https://bugs.launchpad.net/neutron/+bug/195547815:31
slaweqI found such issue only once so far but though it would be good to have it recoreded somewhere :)15:31
slaweqas it seems that it may happen more times (maybe I just missed it somewhere else)15:31
slaweqso if anyone has some time and wants to investigate that, that would be great15:32
slaweqand that's basically all what I had for today15:33
ykarelas per logstash seems there are many such failures15:33
slaweqykarel: can You share link to logstash query?15:34
ykarelhttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%2017%2F20%3A%20up%20%5C%2215:34
ykarelfailure in different jobs, different nodes related to metadata fail15:35
slaweqbut most of them related to OVS (or even all, I'm not sure about non-neutron jobs)15:36
slaweqso it seems that it's more critical issue than I thought :)15:36
ykarelpuppet jobs are also OVS ones, so can say all are openvswitch15:37
slaweqok, so all ovs related15:38
slaweqwe need to investigate that15:38
slaweqI will add it as action item on me but I don't know if I will be able to do that this week15:38
slaweq#action slaweq to check https://bugs.launchpad.net/neutron/+bug/1955478 and broken metadata connectivity issues15:38
slaweqok, that's basically all what I had for today15:39
slaweqwe had to continue discussion about improvements ideas today15:40
slaweqbut as we don't have video meeting, and some folks are already off, lets skip it and get back to that next year15:40
slaweq#topic On Demand15:41
slaweqdo You have any other topics to discuss today?15:41
ralonsohno thanks15:41
bcafarelnothing from me15:41
ykarelnone from me too15:41
mlavallenothing from me either15:41
slaweqif not, I just want to say that this is our last meeting this year15:41
slaweqso thx all for Your hard work keeping neutron ci green(ish)15:42
slaweqhave a great holiday season and see You all in 2022 :)15:42
ralonsohsee you next year!15:42
slaweq#endmeeting15:42
opendevmeetMeeting ended Tue Dec 21 15:42:48 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:42
opendevmeetMinutes:        https://meetings.opendev.org/meetings/neutron_ci/2021/neutron_ci.2021-12-21-15.00.html15:42
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/neutron_ci/2021/neutron_ci.2021-12-21-15.00.txt15:42
opendevmeetLog:            https://meetings.opendev.org/meetings/neutron_ci/2021/neutron_ci.2021-12-21-15.00.log.html15:42
mlavalleHappy Holydays!!!15:42
bcafarellast meeting of 2021 then :)15:43
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/xena: Don't fail subnet validation if gw_ip is actually not changed  https://review.opendev.org/c/openstack/neutron/+/82251215:44
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/wallaby: Don't fail subnet validation if gw_ip is actually not changed  https://review.opendev.org/c/openstack/neutron/+/82251315:44
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/victoria: Implement "IPAllocation" router ports allocated retrieval  https://review.opendev.org/c/openstack/neutron/+/82250715:50
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/victoria: Don't fail subnet validation if gw_ip is actually not changed  https://review.opendev.org/c/openstack/neutron/+/82250815:50
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/ussuri: Implement "IPAllocation" router ports allocated retrieval  https://review.opendev.org/c/openstack/neutron/+/82251415:52
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/ussuri: Don't fail subnet validation if gw_ip is actually not changed  https://review.opendev.org/c/openstack/neutron/+/82251515:52
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/train: Implement "IPAllocation" router ports allocated retrieval  https://review.opendev.org/c/openstack/neutron/+/82250915:54
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/train: Don't fail subnet validation if gw_ip is actually not changed  https://review.opendev.org/c/openstack/neutron/+/82253015:54
opendevreviewMerged openstack/neutron stable/rocky: Populate self.floating_ips_dict using "ip rule" information  https://review.opendev.org/c/openstack/neutron/+/81042716:10
opendevreviewJaganathan Palanisamy proposed openstack/neutron master: Missing OvS DPDK nodes in ovn-controllers  https://review.opendev.org/c/openstack/neutron/+/82246517:22
opendevreviewJaganathan Palanisamy proposed openstack/neutron master: Missing OvS DPDK nodes in ovn-controllers  https://review.opendev.org/c/openstack/neutron/+/82246517:32
opendevreviewGhanshyam proposed openstack/neutron stable/train: DNM: test tempest 27.0.0 tag  https://review.opendev.org/c/openstack/neutron/+/81659718:21
opendevreviewMerged openstack/neutron stable/xena: Bump OVN version for functional job to 21.06  https://review.opendev.org/c/openstack/neutron/+/82143918:37
opendevreviewMerged openstack/neutron-tempest-plugin master: Add jobs definitions for Xena branch  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/81100318:52
*** gmann is now known as gmann_afk19:51
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/wallaby: Fix ObjectChangeHandler thread usage  https://review.opendev.org/c/openstack/neutron/+/82256121:13
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/victoria: Fix ObjectChangeHandler thread usage  https://review.opendev.org/c/openstack/neutron/+/82252021:13
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/ussuri: Fix ObjectChangeHandler thread usage  https://review.opendev.org/c/openstack/neutron/+/82252121:14
opendevreviewSlawek Kaplonski proposed openstack/neutron stable/train: Fix ObjectChangeHandler thread usage  https://review.opendev.org/c/openstack/neutron/+/82252221:14
opendevreviewMaximilian Stinsky proposed openstack/neutron master: Reduce iptables version check from 1.6.2 to 1.6.0  https://review.opendev.org/c/openstack/neutron/+/82256221:34
*** kmasterson` is now known as kmasterson21:40
opendevreviewMerged openstack/neutron master: Missing OvS DPDK nodes in ovn-controllers  https://review.opendev.org/c/openstack/neutron/+/82246522:02
opendevreviewMerged openstack/neutron stable/xena: Do not announce any DNS resolver if "0.0.0.0" or "::" provided  https://review.opendev.org/c/openstack/neutron/+/82229323:11
opendevreviewMerged openstack/neutron stable/victoria: Do not announce any DNS resolver if "0.0.0.0" or "::" provided  https://review.opendev.org/c/openstack/neutron/+/82229623:52
opendevreviewMerged openstack/neutron master: Use Port_Binding up column to set Neutron port status  https://review.opendev.org/c/openstack/neutron/+/82154423:52
opendevreviewMerged openstack/neutron master: Tweak port metadata test to be more reliable  https://review.opendev.org/c/openstack/neutron/+/82177923:52

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!