Tuesday, 2023-06-20

slaweqhi ralonsoh ykarel lajoskatona can You check https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/873380 once You will have few minutes? Thx in advance07:22
ralonsohsure, let me check07:23
ykarelslaweq, ack07:27
ykarelohkk that i recall depends on multiple unmerged patches07:27
lajoskatonaslaweq: sure07:32
lajoskatonaslaweq, ralonsoh: welcome back to the old continent ;)07:33
slaweqlajoskatona ykarel thx07:33
ralonsohthanks!07:33
slaweqlajoskatona yeah, welcome back07:34
slaweqI'm still not feeling well today after 2 weeks on the other side of the globe :)07:34
opendevreviewMerged openstack/ovn-bgp-agent master: Config register_opts for tests in base class  https://review.opendev.org/c/openstack/ovn-bgp-agent/+/88598307:34
lajoskatonaralonsoh: sorry for disturbing your jetlag, but could you check these 2 bugs when you have some free time: https://bugs.launchpad.net/neutron/+bug/2024160 & https://bugs.launchpad.net/neutron/+bug/202363407:34
ralonsohlajoskatona, I read the first one07:35
lajoskatonaralonsoh: both seems to be related to the recent OVN verison bump and https://review.opendev.org/c/openstack/neutron/+/88368107:35
ralonsohthis patch is not the culprit https://review.opendev.org/c/openstack/neutron/+/883681. This patch is only affecting the vritual ports07:35
ralonsohthe new OVN version changes how the portbinding registers are updated07:36
lajoskatonaralonsoh: but I was not sure if we have to revert for quick solution or there some smoother solution for the issue with PortBindingUpdateVirtualPortsEvent07:36
ralonsohactually, if I'm not wrong, instead of updating the register, it deletes and then creates again them07:36
lajoskatonaralonsoh: ahh, ok07:36
ralonsohthe secong one is legit, we need a way to fix this test07:36
ralonsohbut the trunks port issue is something different07:37
lajoskatonaralonsoh: ok, I wokred with the functional failure and in the logs I saw the same07:37
ralonsohArnau (not here now), a RH folk, is working on improving the trunk ports binding07:37
lajoskatonaralonsoh: ack, thanks for the background I refresh than my patch for the functional test failure07:37
ralonsohI'll check with him this afternoon07:37
lajoskatonaack, we can check it on the meeting, the trunk issue affects nova gate, and we can give feedback after the meeting07:38
sahidihti[m]: thanks ! have you already suffered strange behavior form the optimizer?09:00
sahidwe had an outage, agents was stuck trying to update their states, this put the server completly buzy, some guys mentioned the fact that it's the optimizer which was blocked09:03
sahidsince we have a certain number of rbacs I'm expecting avoid such issue in the future by configuring that option09:03
opendevreviewMaximilian Sesterhenn proposed openstack/ovn-bgp-agent master: [WIP] Implement L2 EVPN functionality  https://review.opendev.org/c/openstack/ovn-bgp-agent/+/88609009:48
amoralejhi, in the last days, the ovn scenarios in puppet-openstack-integration has been failing with (pymysql.err.OperationalError) (1040, 'Too many connections')10:07
ralonsohamoralej, do you have a link? 10:08
amoralejafter some debugging, i'm suspecting that some of the two last commits in neutron may be related10:08
amoralejyes10:08
amoraleji.e. https://logserver.rdoproject.org/openstack-periodic-daily/review.rdoproject.org/rdoinfo/master/weirdo-bobcat-promote-puppet-scenario005-centos-stream-9/2d9518b/logs/weirdo-project/logs/10:09
ralonsohamoralej, what commits are you referring?10:10
amoralejhttps://github.com/openstack/neutron/commit/cbb89fdb1414a1b3a8e8b3a9a4154ef627bb9d1a https://github.com/openstack/neutron/commit/db12fc400e926ee4901e3e50412992a1aeed79a110:10
amoralejcould those affect somehow to the number of opened connections to the db ?10:10
amoralejfrom the last passing to the first failing, i don't see other changes than that and a commit in glance that does not seems related at all10:11
ralonsohthe second one most probably is not affecting10:11
ralonsohbut maybe the first one10:11
amoralejand the fact that only is failing in ovn scenarios ...10:11
ralonsohlet me check10:11
amoralejlet me check the max connections defined in devstack jobs10:12
ralonsohamoralej, I think (just checking the code 1 min) that we are commiting a SQL operation without opening a txn10:14
ralonsohamoralej, exactly, this is what is happening10:15
ralonsohamoralej, thanks for raising the problem here! you found a bug10:15
amoralej:D damn10:15
ralonsohhehehe10:15
amoraleji see that devstack is setting max_connection to 1024, is that why it does not uncover it?10:16
amoralejp-o-i sets a much lower number10:16
ralonsohthis should be not a problem, or at least the issue is different10:17
ralonsohlet me open a LP bug and ping lucasagomes 10:17
amoralejsure, thanks10:17
amoralejif it's going to take time, we could pin to the last known-good commit in rdo10:18
ralonsohamoralej, that will fixed today for sure10:18
amoralejgreat!! thanks ralonsoh++10:19
lucasagomesralonsoh, amoralej will take a look o/10:22
ralonsohlucasagomes, I have a patch10:22
ralonsohlet me push it now10:22
lucasagomesralonsoh, sure, I will review it straight away10:22
opendevreviewRodolfo Alonso proposed openstack/neutron master: [OVN] The all() and count() methods should be inside a DB txn  https://review.opendev.org/c/openstack/neutron/+/88645710:25
ralonsohlucasagomes, ^10:25
lucasagomesralonsoh, ops! makes sene!10:27
lucasagomessense*10:27
ralonsohlajoskatona, if you have 1 min: https://review.opendev.org/c/openstack/neutron/+/88645710:53
ralonsohthanks!10:53
ralonsohthis is blocking the puppet CI10:54
*** amoralej is now known as amoralej|lunch11:26
sahid`11:30
ihti[m]sahid: No we haven't had any side effect of disabling the `derived_merge` optimization. For us it helped in our case with ~4000 network rbac rules.12:02
*** amoralej|lunch is now known as amoralej12:14
ralonsohihti[m], are you aware that this issue is solved?12:20
ralonsohhttps://review.opendev.org/q/topic:bug%252F191814512:20
ralonsohamoralej, https://review.opendev.org/c/openstack/neutron/+/886457 it will be merged today12:22
amoralejthanks, i'll check next promotion pipelines runs12:23
amoralejprobably tomorrow12:23
ihti[m]ralonsoh: Aah didn't knew that, thanks for the info12:25
ralonsohihti[m], what version are you using?12:25
ralonsohdon't forget to update both Neutron and n-lib12:25
ihti[m]Yoga12:26
ralonsohthere are backports for both projects12:26
ralonsohthe patches are quite simple and non intrusive12:26
ihti[m]I see, thanks!12:26
sahidACK thanks for the feedback ihti[m] 12:36
ralonsohmlavalle, hi! if you have a couple of mins, please check https://review.opendev.org/q/I1962fbc844bb67180e9071bcee01f8e95853bdda13:08
ralonsohthanks13:08
fricklerlajoskatona: I noticed ceilometer is also still using neutronclient, should I add it to https://etherpad.opendev.org/p/python-neutronclient_deprecation or did you exclude it intentionally?13:23
lajoskatonafrickler: please add it, I missed that to tell the truth13:25
lajoskatonafrickler: thanks for the info13:28
fricklerlajoskatona: added, thx. luckily it seems to be all bundled within a single fil13:28
frickler+e13:28
lajoskatonafrickler: thanks, I have to go back to that topic13:34
ralonsohPing list: bcafarel, elvira, frickler, mlavalle, mtomaska, obondarev, slawek, tobias-urdin, ykarel, lajoskatona, jlibosva, averdagu, amotoki 14:00
mlavalleo/14:00
opendevreviewMerged openstack/ovn-bgp-agent master: Ensure FIPs are exposed as part of cr-lrp binding events  https://review.opendev.org/c/openstack/ovn-bgp-agent/+/88598114:00
ralonsoh#startmeeting networking14:00
opendevmeetMeeting started Tue Jun 20 14:00:42 2023 UTC and is due to finish in 60 minutes.  The chair is ralonsoh. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'networking'14:00
ralonsohhello all14:00
obondarevhi14:00
lajoskatonao/14:00
ralonsohslaweq, is not going to attend today14:01
isabeko/14:01
ralonsohok, let's start14:02
ralonsoh#topic announcements14:02
ralonsoh#link https://releases.openstack.org/bobcat/schedule.html14:02
bcafarellate o/14:02
ralonsohin 2 weeks, we have bobcat-2 milestone14:02
ralonsohwe officially don't have a spec freeze14:03
ralonsohbut we should do the same as Nova, using this week as spec freeze14:03
ralonsohbut we'll talk later about this14:03
matfechnero/14:03
ralonsohand the PTG, of course!14:03
ralonsohthis is my summary: https://lists.openstack.org/pipermail/openstack-discuss/2023-June/034161.html14:03
lajoskatonathanks for the summary14:04
ralonsohmany people attending the forum and talk sessions14:04
ralonsohand quite a few joining us, slaweq and me (the only Neutron core reviewers) in the PTG table14:04
ralonsohthat was very interesting. Of course, that was more a issue/problems forum rather than a program PTG14:05
ralonsohbut we had it 2 months ago14:05
ralonsohand that's all, anything else missing here?14:05
ralonsohok, let's move to the next topic14:06
ralonsoh#topic bugs14:06
ralonsohwe have 2 reports14:06
ralonsohfrom bcafarel: https://lists.openstack.org/pipermail/openstack-discuss/2023-June/034086.html14:06
ralonsohfrom lajoskatona: https://lists.openstack.org/pipermail/openstack-discuss/2023-June/034147.html14:06
opendevreviewLajos Katona proposed openstack/python-neutronclient master: OSC: Remove FWAAS V2 calls to neutronclient  https://review.opendev.org/c/openstack/python-neutronclient/+/88062914:06
bcafarelfor once I had a quiet week :)14:06
ralonsohyeah hehehe I noticed that14:07
lajoskatonaagree14:07
ralonsohwe have 2 pending bugs to be discussed14:07
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/202425114:07
ralonsohIf no one is taking it, I'll assign it to myself but for the next week14:08
ralonsohit seems that is easy to reproduce it14:08
ralonsohthe most important bug we have this week is14:09
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/202416014:09
ralonsohit seems that a new version of OVN is affecting the OVN ports14:09
ralonsohin particular the subports, that are never set as ACTIVE14:09
ralonsohbut if I'm not wrong, this is affecting any subport, before being migrated14:10
ralonsohlajoskatona, right?14:10
lajoskatonayes thats visible from the bug14:11
lajoskatonaanother occurance is this bug:https://bugs.launchpad.net/neutron/+bug/202363414:11
ralonsohyeah, this is related to the OVN version14:11
lajoskatonathis appears only in our functional job, and it seems changing the test is enough14:12
ralonsohbut this code https://review.opendev.org/c/openstack/neutron/+/883681 is affecting only to the virtual ports14:12
ralonsohOVN has changed how the DB port binding registers are updated14:12
ralonsohI'll check the virtual port feature but seems to be not affected14:13
ralonsohonly the test14:13
ralonsohbut the issue with the subports is legit and is critical14:13
lajoskatonayes that seems from the functiona ltests, sometimes 2 delete event arrives for the same port14:13
ralonsohright ^14:13
lajoskatonayes and the trunk/subport issue affects other projects so attention is bigger14:13
ralonsohI think that your patch could add this comment there, https://review.opendev.org/c/openstack/neutron/+/886167/1/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py14:14
ralonsohjust to let anyone know that this assert changes depending on the OVN version14:14
ralonsohbut this fix for the FT test is OK IMO14:14
ralonsohI'll start working today to figure out what is happening with the subports14:14
ralonsohok, any other bug to be discussed here?14:15
fricklerfor my bgp bug, it would be nice if someone with more DB insight could take a look at https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/88599314:15
ralonsohfrickler, let me check the bug first14:15
fricklerthat's https://launchpad.net/bugs/2023632 just ftr14:16
ralonsohbut it seems to be a problem with the IP parsing14:17
ralonsohwell, IP version14:17
fricklerralonsoh: that may be either another issue in os-ken or maybe n-d-r, but the v4 prefix should not get announced at all in that scenario14:19
fricklerbecause it doesn't have the matching address-scope14:19
fricklerand afaict n-d-r also doesn't do MP-BGP at all yet14:19
ralonsohbut this os-ken call is a callback?14:19
ralonsohor are you calling it from n-d-r?14:19
fricklerI would have to check in detail, but I think it is called from n-d-r14:20
ralonsohin that case, this is easier if you filter that from n-d-r14:20
ralonsohwell, that's easy to say14:20
fricklermy patch filters it in the DB query14:21
fricklerwhich is already happening in the usual workflow. the bug only seems to happen if a router is updated with multiple interfaces14:21
fricklerI'll work on a tempest test to expose the bug, too14:21
ralonsoh^^ cool. Can you print the output of this query? as debug, temporarily14:22
frickleranyway my patch fixes the issue in my deployment where the bug was discovered, I just need a bit of confirmation that it looks correct14:22
ralonsohperfect14:22
fricklerI can try to add in some debugging, too, yes14:23
ralonsohperfect, I'll wait too for this tempest test, if possible14:23
ralonsohthanks!14:23
ralonsohthis week slaweq is the deputy, next week will be haleyb14:24
ralonsoh(I'll ping slaweq tomorrow)14:24
ralonsohhaleyb, ?14:24
haleyback14:24
ralonsohthanks114:24
ralonsohso let's jump to our next topic14:24
ralonsoh#topic specs14:24
ralonsohas commented before, we should start closing the spec reviews14:25
ralonsohwe have one pending right now14:25
ralonsohhttps://review.opendev.org/q/project:openstack%252Fneutron-specs+status:open14:25
ralonsoh#link https://review.opendev.org/c/openstack/neutron-specs/+/88532414:25
ralonsohok, if I'm not wrong, is ready for review, mlavalle right?14:25
mlavalleit is14:26
ralonsohperfect then, I'll most probably check it this week on Friday morning14:26
mlavallethanks!14:26
ralonsohplease, spend some time reviewing it in order to merge it, if possible, in the next 2 weeks14:26
lajoskatonaack14:27
ralonsohthanks folks14:27
ralonsohnext topic14:27
ralonsoh#topic community_goals14:27
ralonsoh1) Consistent and Secure Default RBAC 14:27
ralonsohas commented last meeting, slaweq is working now in the service to service role14:28
ralonsohbecause this is something new and still under study, I'll skip any update of this topic until new advice (from slaweq)14:28
ralonsohthe next one is14:28
ralonsoh2) Neutron client deprecation 14:28
ralonsohI see new/updated patches, right?14:28
lajoskatonathe usual etherpad: https://etherpad.opendev.org/p/python-neutronclient_deprecation14:29
lajoskatonayes, 2 are open:14:29
lajoskatonahttps://review.opendev.org/c/openstack/python-neutronclient/+/88062914:29
lajoskatonahttps://review.opendev.org/c/openstack/python-neutronclient/+/88418014:30
ralonsohthe first one is straight forward, I think14:30
ralonsohdo we have the sdk version released?14:30
lajoskatonayes, I hope it will be green, as all deppendencies are merged now14:31
ralonsohperfect14:31
ralonsohif the CI is happy, I'll be too14:31
lajoskatonaI have to check the sdk release14:31
ralonsohyeah, maybe bump it in the requiremtns14:32
ralonsohsorry no, not in the neutron client reqs14:32
lajoskatonathat's it for this topic from me14:33
ralonsohthanks!14:33
ralonsohok, and that's I'll I have14:34
ralonsoh#topic on_demand14:34
ralonsohdo you have any topic?14:34
fricklerI have a question about draining subnets14:34
ralonsohsure14:34
fricklerlike when you increase your public pool from /24 to /23 by adding a new subnet and then gradually empty the old subnet14:35
fricklerso you want to stop allocating router gateway and FIPs from it, but leave the existing ones working for a while14:35
fricklerI found that this can be achieved by setting service-type for the old subnet to some weird value like network:dummy14:36
fricklerbut I feel like I'm abusing that feature and wonder if there is a better solution14:36
frickleror if it should be implemented14:36
opendevreviewNickKush proposed openstack/neutron master: Handle fixed_ip delete in port with FIP  https://review.opendev.org/c/openstack/neutron/+/88599914:36
ralonsohto be honest, I don't know how to do this14:38
ralonsohI mean, how to enforce the IPAM module to use one of the subnets14:38
fricklerwell that's what the subnet service-type extension kind of does14:39
fricklerexcept I'm now telling it to not use one of the subnets14:39
obondarevthe workaround with service-type does not look so ugly imo, like setting it to network:no_use or like that14:39
lajoskatonaagree, perhaps the doc should be updated with the example or similar14:40
ralonsohI'm trying to find the service type for GW ports or FIPs or external networks14:40
ralonsohif any14:40
ralonsohdo you have some reference?14:41
fricklerthose are all nicely listed here https://docs.openstack.org/neutron/latest/admin/config-service-subnets.html14:41
fricklerthere is a check in place that prevents some random device owner being used. like "no-use" is denied14:42
fricklerbut "neutron:no_use" is kind of cheating a bit and allowed14:42
fricklerbecause there is no detailed list of allowed owners14:42
opendevreviewNickKush proposed openstack/neutron master: Handle fixed_ip delete in port with FIP  https://review.opendev.org/c/openstack/neutron/+/88599914:43
ralonsohmaybe we can make it "official", having a neutron:no_use constant for this purpose14:43
ralonsohthe subnet will be valid but won't be scheduled by the IPAM module14:43
fricklerso add that to that doc page or what else to make it official?14:44
ralonsohand make that official in the code: if no_use service type, this subnet is not usable 14:44
ralonsoh_validate_segment will reject it14:44
fricklerwell it should not reject existing FIPs, we'd have to be careful there14:45
ralonsohno no, just the IP allocation, that is what you want, right?14:45
ralonsohjust prevent new IPs to be allocated in the old subnet14:45
frickleryes, so that customers can migrate to new addresses at their own pace14:45
ralonsohbut any existing IP will be valid14:45
ralonsohI think this is a valid use for the service types14:46
ralonsohlet's open it as RFE, comment it in the neutron drivers meeting and most probably implement it without spec14:46
ralonsohjust the needed documentation14:46
ralonsohand the code14:46
haleybs/no_use/do_not_use? but we can talk in the RFE about the details14:47
fricklero.k., I'll do an RFE, ack14:47
ralonsohexactly, we can define the constant in the review or the meeting14:47
ralonsohcool!14:47
fricklerthx14:47
ralonsohany other topic?14:47
ralonsohremember today we don't have CI meeting14:48
ralonsohthank you all for attending and see you online14:48
ralonsoh#endmeeting14:48
opendevmeetMeeting ended Tue Jun 20 14:48:39 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:48
opendevmeetMinutes:        https://meetings.opendev.org/meetings/networking/2023/networking.2023-06-20-14.00.html14:48
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/networking/2023/networking.2023-06-20-14.00.txt14:48
opendevmeetLog:            https://meetings.opendev.org/meetings/networking/2023/networking.2023-06-20-14.00.log.html14:48
ralonsohbye14:48
mlavalleo/14:48
bcafarelo/14:48
obondarevo/14:48
frickler\o14:48
haleybo/14:48
lajoskatonao/14:48
opendevreviewLajos Katona proposed openstack/neutron master: Functional: assert multiple calls for update_virtual_port_host  https://review.opendev.org/c/openstack/neutron/+/88616715:07
opendevreviewRodolfo Alonso proposed openstack/neutron master: [OVS][QoS] Add QoS support for Trunk service, OVS driver  https://review.opendev.org/c/openstack/neutron/+/83952315:12
opendevreviewRodolfo Alonso proposed openstack/neutron master: [qos] _validate_create_network_callback return in no network  https://review.opendev.org/c/openstack/neutron/+/87604015:14
opendevreviewRodolfo Alonso proposed openstack/neutron master: [sqlalchemy-20] Remove redundant indexes from some tables  https://review.opendev.org/c/openstack/neutron/+/88621316:08
opendevreviewMerged openstack/neutron stable/yoga: Make DB migration creating indexes in RBACs conditional  https://review.opendev.org/c/openstack/neutron/+/88573516:25
*** amoralej is now known as amoralej|off16:47
opendevreviewMerged openstack/neutron stable/2023.1: Make DB migration creating indexes in RBACs conditional  https://review.opendev.org/c/openstack/neutron/+/88573317:50
opendevreviewJuan Pablo Suazo proposed openstack/tap-as-a-service master: Replaces Deprecated Bridge-utils Command  https://review.opendev.org/c/openstack/tap-as-a-service/+/88653221:21
opendevreviewMerged openstack/neutron master: Delete the "Chassis_Private" register when deleting an agent  https://review.opendev.org/c/openstack/neutron/+/88574422:38

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!