Tuesday, 2025-01-07

opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: dnm: just to trigger CI  https://review.opendev.org/c/openstack/neutron/+/93848707:13
ralonsohykarel, hello! I'm checking the n-d-r issue. So far, I see the issue is most of the times related to "delete_mac_binding_entries_by_mac"08:06
ralonsohhttps://github.com/openstack/neutron/blob/f2c3e3e68c0fa77cff5cdd22d847e3f6ea7cdd84/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L1897-L191308:07
ralonsohykarel, no sorry, this is red herring08:13
lajoskatonaralonsoh: Hi, ykarel has a patch to decrease the wsgi process number: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/938466 we can use that as a workaroundin the shor term08:13
ralonsohyes, I saw it08:13
ralonsohbut this is just a workaround08:14
ralonsohthe issue is that, for example with the "remove_router_interface" API call08:15
ralonsohthe Neutron API returns OK 20008:15
ralonsohfor example: https://paste.opendev.org/show/bXncGrCKtq5pydxIt1iH/08:16
ralonsohfrom https://7147d7e07a0a4e67ee4e-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf2.rackcdn.com/938399/1/check/neutron-tempest-plugin-dynamic-routing/7b5324c/controller/logs/screen-q-dr-agent.txt08:16
ralonsohsorry no08:16
ralonsohfrom https://7147d7e07a0a4e67ee4e-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf2.rackcdn.com/938399/1/check/neutron-tempest-plugin-dynamic-routing/7b5324c/controller/logs/screen-q-dr-agent.txt08:16
fricklerralonsoh: that was twice the same link, this is where your paste is from https://7147d7e07a0a4e67ee4e-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf2.rackcdn.com/938399/1/check/neutron-tempest-plugin-dynamic-routing/7b5324c/controller/logs/screen-neutron-api.txt08:20
ralonsohright...08:20
fricklerI'm confused though as to how this is related to bgp, the failure seems to be in a completely different section of code08:21
slaweqhi all, HNY!08:21
ralonsohfrickler, I'm not saying this is related to bgp08:22
ralonsohslaweq, HNY!08:22
slaweqralonsoh I am looking at your patch https://review.opendev.org/c/openstack/neutron/+/934418 now and have 1 question about it08:22
slaweqdoes this mean that we are effectively dropping support for ovn < 23.06 with this patch? Or it was already like that earlier and I just can't find it? 08:22
ralonsohslaweq, it wasn't working before for physnet LSP ports08:23
ralonsohso with ovn<23.06 we have the API but we don't have any QOS enforcement08:24
slaweqahh, ok then08:24
slaweqthx for explanation08:24
slaweqmaybe you could add some note about this in the the qos documentation maybe to make sure we have this written somewhere that this feature works only with OVN 23.06 or newer08:26
slaweqbut this can be added in the follow up patch for sure08:27
lajoskatonaHappy New Year to Everybody :-)08:29
ralonsohslaweq, you are right, I'll amend the release note08:30
ralonsohslaweq, if you have less than 1 min: https://review.opendev.org/c/openstack/neutron/+/938390/108:38
ralonsohthanks!08:38
opendevreviewRodolfo Alonso proposed openstack/neutron master: [L3] Reduce to one single thread the event processing  https://review.opendev.org/c/openstack/neutron/+/93840608:46
opendevreviewRodolfo Alonso proposed openstack/neutron master: [eventlet-deprecation] Replace ``eventlet.spawn_n`` usage  https://review.opendev.org/c/openstack/neutron/+/93841108:53
ykarelralonsoh, isn't the actual issue those DB errors that i included in the bug comment?09:00
ykarelthe success after 5 minute may be just some uwsgi fallback?09:00
fricklerlajoskatona: if you could tack an approval onto https://review.opendev.org/c/x/tap-as-a-service-dashboard/+/923819 and https://review.opendev.org/c/x/tap-as-a-service-tempest-plugin/+/923821 I would feel more confident force-merging them09:11
lajoskatonafrickler: ack, for these old unused things I feel it is ok 09:16
lajoskatonafrickler: done09:17
ralonsohykarel, yes, this seems to be the problem but I don't know why this is happening, why always in the router interface deletion09:19
fricklerty, will proceed with those in a bit09:19
opendevreviewRodolfo Alonso proposed openstack/neutron master: [eventlet-deprecation] Implement a metadata proxy without eventlet  https://review.opendev.org/c/openstack/neutron/+/93754509:23
opendevreviewRodolfo Alonso proposed openstack/neutron master: [eventlet-deprecation] Use non-eventlet metadata proxy in OVN metadata agent  https://review.opendev.org/c/openstack/neutron/+/93839309:24
opendevreviewRodolfo Alonso proposed openstack/neutron master: [eventlet-deprecation] Replace ``eventlet.spawn_n`` usage  https://review.opendev.org/c/openstack/neutron/+/93854109:31
slaweqralonsoh can you take a look at https://review.opendev.org/c/openstack/ovn-octavia-provider/+/929039 when you will have some time?09:31
ralonsohslaweq, sure09:31
slaweqthx09:31
M9d0cd7d2[m]Hi guys, I need some help while configuring OVN on a Charmed Openstack deployment. Can I address those doubts here, or there is a dedicated channel?09:33
M9d0cd7d2[m]Ok, saw that for "support" #_oftc_#openstack:matrix.org is recommended09:34
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: common: fix wait_until_true to support native thread  https://review.opendev.org/c/openstack/neutron/+/93784309:39
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: remove the usage of eventlet in the OVS agent  https://review.opendev.org/c/openstack/neutron/+/93776509:39
opendevreviewMerged openstack/neutron unmaintained/yoga: [unmaintained only] Drop sdk job  https://review.opendev.org/c/openstack/neutron/+/93847809:41
opendevreviewMerged openstack/neutron unmaintained/xena: [unmaintained only] Drop sdk job  https://review.opendev.org/c/openstack/neutron/+/93847909:43
opendevreviewliuyulong proposed openstack/neutron master: Add meter bandwidth limit support  https://review.opendev.org/c/openstack/neutron/+/86076709:54
opendevreviewMerged openstack/neutron unmaintained/wallaby: [unmaintained only] Drop sdk job  https://review.opendev.org/c/openstack/neutron/+/93848011:27
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: common: fix wait_until_true to support native thread  https://review.opendev.org/c/openstack/neutron/+/93784313:56
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: remove the usage of eventlet in the OVS agent  https://review.opendev.org/c/openstack/neutron/+/93776513:56
mlavalle#startmeeting networking14:00
opendevmeetMeeting started Tue Jan  7 14:00:10 2025 UTC and is due to finish in 60 minutes.  The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'networking'14:00
mlavallePing list: bcafarel, elvira, frickler, mlavalle, mtomaska, obondarev, slaweq, tobias-urdin, ykarel, lajoskatona, jlibosva, averdagu, amotoki, haleyb, ralonsoh14:00
slaweqo/14:00
ralonsohhello14:00
frickler\o14:00
bcafarelo/14:01
s3rj1khi all14:01
mlavalle#announcements14:01
mlavalle#link https://releases.openstack.org/epoxy/schedule.html14:02
lajoskatonao/14:02
mlavalleWe are currently in week R-1214:02
cbuggyo/14:02
mlavalleand it is the Epoxy-2 milestone14:03
ykarelo/14:03
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: common: fix wait_until_true to support native thread  https://review.opendev.org/c/openstack/neutron/+/93784314:03
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: remove the usage of eventlet in the OVS agent  https://review.opendev.org/c/openstack/neutron/+/93776514:03
mlavalleReminder: If you have a topic for the drivers meeting on Friday, please add it to the wiki @ https://wiki.openstack.org/wiki/Meetings/NeutronDrivers14:03
mlavalleand haleyb|out wants us to continue using the priorities dashboard for patches in the "ready to merge" state (weekly reminder)14:05
mlavalleFinally, have a very sucessful and happy 202514:07
mlavalleAny other announcements?14:07
lajoskatonamlavalle: +1 for 2025, and Happy New Year everybody :-)14:08
slaweqHNY!14:08
rubasovlate o/14:09
mlavalle#topic Bugs14:09
bcafarelHNY all :)14:09
mlavalleLast week the bug deputy was obondarev and the one before it was jlibosva's turn14:10
mlavallebut apparently nobody told them, since I didn't see any emails from them14:11
mlavalleso last report we have is from ralonsoh: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/STPSYHHAAYJTJFD4DBWAAC2L2M2MUEM2/14:11
ralonsohin any case, tomorrow morning I'll send a report for the last 2 weeks14:12
mlavalleanything you want to highlight ralonsoh from that report?14:12
ralonsohwe need to keep the list updated14:12
ralonsohno, we have 1 bug not assigned14:12
ralonsohhttps://bugs.launchpad.net/neutron/+bug/209217414:12
ralonsohbut this is not a priority14:12
ralonsohanyone is welcome to take it14:13
ralonsohand, as commented, tomorrow morning I'll send an updated bug mail14:13
mlavalleWe have 9 bugs accumulated since the last report, starting with https://bugs.launchpad.net/neutron/+bug/209240714:13
slaweqI can take it for sure, but this should be maybe discussed first within the team14:13
slaweqmaybe others don't want to deprecate this config option, maybe there is some use case for it which I am not aware about14:14
mlavallehow about sending a message to the ML and then discuss it in the drivers meeting?14:15
slaweqmlavalle sure, I will do that14:15
ralonsoh+114:15
slaweqand will add this to the Friday's meeting agenda14:16
lajoskatona+1 for mail14:16
mlavalleI would say give a couple of weeks between the message to the ML and discussion in the drivers meeting. That way we give opportunity to the community to give feedback14:17
slaweq++14:18
mlavallecool14:18
mlavalleanything else we should discuss in this section?14:18
mlavallethis week the bug deputy is ralonsoh and next it is lajoskatona's turn. you both ok with it?14:20
ralonsohyes14:20
lajoskatonaack14:20
mlavalle#topic community goals14:21
mlavalleanything new on on neutronclient deprecation ?14:21
lajoskatonayes I work on a patch for horizon fips14:22
lajoskatonaso slow progress14:22
lajoskatonahttps://review.opendev.org/c/openstack/horizon/+/93848814:22
lajoskatonathat's it for this topic14:23
mlavallethanks for the update lajoskatona ++14:23
mlavallethe other subject here is eventlet. Any updates this week?14:24
ralonsohyes, several14:24
ralonsohdocumentation for eventlet deprecation (review): https://review.opendev.org/c/openstack/neutron/+/93839014:24
ralonsohOVN agent with a new socket server implementation (see limitations): https://review.opendev.org/c/openstack/neutron/+/93754514:24
ralonsohthis server does not use eventlet14:25
ralonsohremember OVN agent is the default agent for plugin jobs14:25
ralonsohnext one14:25
ralonsohOVN metadata agent: https://review.opendev.org/c/openstack/neutron/+/938393/14:25
ralonsohsame as before, and I've refectored a bit the agent not to use oslo.services (for now)14:25
ralonsohand now it is running without eventlet14:25
ralonsohnext one14:25
ralonsohL3 thread processing reduction: https://review.opendev.org/c/openstack/neutron/+/93840614:26
mlavalleit failed CI. Is it due to the change?14:26
ralonsohmlavalle, which one?14:26
mlavallemetadata14:26
ralonsohI need to check but most probably not14:26
mlavalleack14:27
ralonsohovn job is not stable since we migrated to wsgi14:27
ralonsohthere are several bugs open14:27
ralonsohand I'm trying to address all of them14:27
ralonsohI'm checking the logs14:27
ralonsohthis is related to this bug14:27
ralonsoh--> https://review.opendev.org/c/openstack/neutron/+/93831914:28
ralonsohso continuing with the list of patches14:28
ralonsohL3 thread processing reduction: https://review.opendev.org/c/openstack/neutron/+/93840614:28
M9d0cd7d2[m]Hi people, is it possible that this configuration... (full message at <https://matrix.org/oftc/media/v1/media/download/AVMyqUurPDoOHlT2L6alQW-_y-DG89CfQy9avvE-bsgg4FyZE4OqVOZrTa-ZLL2AhqpBFC5__eTNiYiQKKruaHtCeUiUPjygAG1hdHJpeC5vcmcvWVhIS3BndmdvT3hjQ0NtcUh6b1Zzb1lQ>)14:28
ralonsohplease review and check Liu's comment14:28
ralonsohthis is the same as with the DHCP patch14:28
ralonsohmultithread does not improve the event processing performance at all14:29
mlavalleyeap14:29
ralonsohand once we move to kernel threads that are preemptive14:29
ralonsohwe can have the problem of stopped threads to start processing others14:29
ralonsohso we can have routers not fully initialized and the L3 agent processing others14:29
ralonsohand that's all for today14:29
mlavalleThanks for the update and the hard work on this topic ralonsoh ++14:30
lajoskatonaanother batch of patches from Sahid; https://review.opendev.org/q/topic:%22bug/2087939%2214:31
ralonsohyes14:31
lajoskatonamostly for os-ken14:31
ralonsohI think I had a topic for it in next section14:31
ralonsohbut let's discuss this it here14:31
ralonsohthe point is14:31
mlavalleI'll change topic14:31
ralonsohif we can't migrate os-ken to kernel threads in this cycle, I would suggest to go back again to ofctl14:31
mlavalle#topic on-demand14:32
ralonsohnow, os-ken has one single implementation, eventlet14:32
ralonsohand the effort to migrate to kernel threads is unknown14:32
ralonsohso if this is not possible in a reasonable amount of time14:32
ralonsohI would suggest restoring ocftl (that works and must do it with kernel threads)14:33
ralonsohbut this is just a heads-up14:33
sahid_ralonsoh: but is that not enough to remove the monley patching event if we keep the implemntation using eventlet?14:33
sahid_https://review.opendev.org/c/openstack/os-ken/+/93833714:33
sahid_at least for this release?14:34
ralonsohsahid_, where are you removing the monkey patching?14:34
ralonsohthis is being called from the ovs agent14:34
ralonsohand we monkey-patch before calling it14:34
ralonsohdid you check that with any neutron CI?14:34
sahid_the function hub.patch is doing the monkey patching is osken lib right?14:34
lajoskatonaralonsoh: what is ocftl?14:34
ralonsohthe CLI interface, using "ovs-ofctl" commands14:35
sahid_ralonsoh: it's basically on what i'm currently working14:35
sahid_lajoskatona: asked me to use a dep patch with neutron14:35
sahid_https://review.opendev.org/c/openstack/os-ken/+/93833714:35
ralonsohsahid_, perfect, so I'm just saying that this is an alternative to os-ken, if the migration is not possible14:35
sahid_ralonsoh: great14:36
lajoskatonaralonsoh: ahh, ok14:36
ralonsohsahid_, this patch and the CI job is using eventlet14:36
ralonsoh--> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7dd/938337/2/check/neutron-ovs-tempest-dvr/7ddc386/controller/logs/screen-q-agt.txt14:36
ralonsohso this is not a valid test14:36
sahid_i have not checked at that point14:37
sahid_but i don't see any reason that osken with its current implementation to not work without monkey patch14:37
sahid_i will elt you know14:37
ralonsohbecause the os-ken hub has one single implementation, that is based in eventlet14:38
lajoskatonayeah, os-ken has lots of modules used by neutworking projects (BGP and similar protocol files) so even if we drop hub we have to keep the rest I beleive14:38
ralonsohand you need the os-ken hub to spawn the threads: one for the agent, one for the commands and one moniting the OF table14:38
ralonsohbut it is not possible to drop the hub14:39
sahid_ralonsoh: yes but that implementation can with and without monkey patch14:39
sahid_can work14:39
sahid_that is said I don't have any issue to remove osken at all14:39
ralonsohperfect then, waiting for a CI job testing that14:39
ralonsohos-ken is much faster than osctl14:39
ralonsohofctl14:39
ralonsohthat will be a serious regression in performance14:40
sahid_interesting point, thanks14:40
sahid_btw I'm trying different king of impl to fix wait_until_true I will be glad to get any idea of which one could be the best 14:42
sahid_https://review.opendev.org/c/openstack/neutron/+/937843/914:42
sahid_ion that one I run the predicate in a separate thread instead of making it runnoing in the main thread and having a timer in a different thread14:42
sahid_but it's an other topic, sorry for the disruption14:42
mlavalleanything else to discuss today?14:43
ralonsohnot from me14:43
lajoskatonaI added one topic for on-demand14:43
mlavallego ahead, lajoskatona 14:44
lajoskatonaThere was a self +2 wf+1: https://review.opendev.org/c/openstack/neutron/+/93623514:44
lajoskatonaand I think this is a topic to discuss and see how to void such situation14:44
lajoskatonaone side as I see is that there can be frustration for slow review even no-review14:45
ralonsohlajoskatona, anyone can participate in this meeting and request for reviews14:46
lajoskatonabut in this case this patch I copied was not dying in gerrit for months, so I believe that after the vacation time there would have been review for it14:46
ralonsohor send a mail14:46
ralonsohor increase the review priority14:46
lajoskatonaralonsoh: +1 that is also true, and we have the priority board also14:46
ralonsohand, to be honest, I now have technical questions about this patch14:47
lajoskatonaso wanted to highlight this event and ask everybody to thin kabout it and when haleyb is back we can have some actions or more discussion around it14:47
mlavalleI'm of the opinion that this shouldn't be allowed14:47
lajoskatonawe can revert it and start the review again14:47
mlavalleI say let's do it14:48
mlavallehaleyb|out will be back tomorrow. he can do it14:48
lajoskatonaWe can ask Liu to participate on a meeting and discuss this topic to see why  was this the only solution He saw at that time14:48
lajoskatonamlavalle: yeah let's wait till the boss is back :-) He commented on the patch so has context for it14:49
mlavalle++14:49
lajoskatonathat's it from me14:50
mlavallethanks lajoskatona for bringing this up. really important14:50
mlavalleand I think that the proposal to let Liu voice his point of view is good14:51
mlavalleanything else for today?14:51
mlavalleok, have a great first week of 2025!14:53
mlavalle#endmeeting14:53
opendevmeetMeeting ended Tue Jan  7 14:53:26 2025 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:53
opendevmeetMinutes:        https://meetings.opendev.org/meetings/networking/2025/networking.2025-01-07-14.00.html14:53
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/networking/2025/networking.2025-01-07-14.00.txt14:53
opendevmeetLog:            https://meetings.opendev.org/meetings/networking/2025/networking.2025-01-07-14.00.log.html14:53
lajoskatonaBye14:53
ykarelo/14:53
s3rj1kthanks all14:53
ralonsohbye14:53
ralonsohfolks: https://review.opendev.org/c/openstack/neutron/+/93470115:04
slaweqralonsoh mlavalle lajoskatona please check https://review.opendev.org/c/openstack/neutron/+/937887 when You will have a minute or two, thx in advance15:06
ralonsohslaweq, sure15:06
ralonsohslaweq, one comment15:10
slaweqthx, I will change it according to your comment15:34
opendevreviewRodolfo Alonso proposed openstack/neutron master: WIP == [eventlet-deprecation] Change the state change WSGI server  https://review.opendev.org/c/openstack/neutron/+/93840416:00
greatgatsbyHello.  I'm seeing `QueuePool limit of size 1 overflow 50 reached` in neutron-server logs.  At the same time, I'm dumping the mariadb processlist in a loop and counting the neutron threads/connections and there's never above 45 from what I can tell.  This is a kolla-ansible zed deployment.17:23
greatgatsbyalso, of those 40'ish processes, the majority are in Sleep, so I'm not sure why I seem to be hitting a QueuePool limit?17:24
opendevreviewIhar Hrachyshka proposed openstack/neutron-lib master: Enable mypy for pep8 tox target  https://review.opendev.org/c/openstack/neutron-lib/+/92148419:03
opendevreviewMerged openstack/neutron master: [OVN] QoS max and min rules should be defined in LSP for phynet ports  https://review.opendev.org/c/openstack/neutron/+/93441820:26
opendevreviewMerged openstack/neutron master: [eventlet-deprecation] Add a reference document  https://review.opendev.org/c/openstack/neutron/+/93839023:58

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!