opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: dnm: just to trigger CI https://review.opendev.org/c/openstack/neutron/+/938487 | 07:13 |
---|---|---|
ralonsoh | ykarel, hello! I'm checking the n-d-r issue. So far, I see the issue is most of the times related to "delete_mac_binding_entries_by_mac" | 08:06 |
ralonsoh | https://github.com/openstack/neutron/blob/f2c3e3e68c0fa77cff5cdd22d847e3f6ea7cdd84/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L1897-L1913 | 08:07 |
ralonsoh | ykarel, no sorry, this is red herring | 08:13 |
lajoskatona | ralonsoh: Hi, ykarel has a patch to decrease the wsgi process number: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/938466 we can use that as a workaroundin the shor term | 08:13 |
ralonsoh | yes, I saw it | 08:13 |
ralonsoh | but this is just a workaround | 08:14 |
ralonsoh | the issue is that, for example with the "remove_router_interface" API call | 08:15 |
ralonsoh | the Neutron API returns OK 200 | 08:15 |
ralonsoh | for example: https://paste.opendev.org/show/bXncGrCKtq5pydxIt1iH/ | 08:16 |
ralonsoh | from https://7147d7e07a0a4e67ee4e-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf2.rackcdn.com/938399/1/check/neutron-tempest-plugin-dynamic-routing/7b5324c/controller/logs/screen-q-dr-agent.txt | 08:16 |
ralonsoh | sorry no | 08:16 |
ralonsoh | from https://7147d7e07a0a4e67ee4e-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf2.rackcdn.com/938399/1/check/neutron-tempest-plugin-dynamic-routing/7b5324c/controller/logs/screen-q-dr-agent.txt | 08:16 |
frickler | ralonsoh: that was twice the same link, this is where your paste is from https://7147d7e07a0a4e67ee4e-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf2.rackcdn.com/938399/1/check/neutron-tempest-plugin-dynamic-routing/7b5324c/controller/logs/screen-neutron-api.txt | 08:20 |
ralonsoh | right... | 08:20 |
frickler | I'm confused though as to how this is related to bgp, the failure seems to be in a completely different section of code | 08:21 |
slaweq | hi all, HNY! | 08:21 |
ralonsoh | frickler, I'm not saying this is related to bgp | 08:22 |
ralonsoh | slaweq, HNY! | 08:22 |
slaweq | ralonsoh I am looking at your patch https://review.opendev.org/c/openstack/neutron/+/934418 now and have 1 question about it | 08:22 |
slaweq | does this mean that we are effectively dropping support for ovn < 23.06 with this patch? Or it was already like that earlier and I just can't find it? | 08:22 |
ralonsoh | slaweq, it wasn't working before for physnet LSP ports | 08:23 |
ralonsoh | so with ovn<23.06 we have the API but we don't have any QOS enforcement | 08:24 |
slaweq | ahh, ok then | 08:24 |
slaweq | thx for explanation | 08:24 |
slaweq | maybe you could add some note about this in the the qos documentation maybe to make sure we have this written somewhere that this feature works only with OVN 23.06 or newer | 08:26 |
slaweq | but this can be added in the follow up patch for sure | 08:27 |
lajoskatona | Happy New Year to Everybody :-) | 08:29 |
ralonsoh | slaweq, you are right, I'll amend the release note | 08:30 |
ralonsoh | slaweq, if you have less than 1 min: https://review.opendev.org/c/openstack/neutron/+/938390/1 | 08:38 |
ralonsoh | thanks! | 08:38 |
opendevreview | Rodolfo Alonso proposed openstack/neutron master: [L3] Reduce to one single thread the event processing https://review.opendev.org/c/openstack/neutron/+/938406 | 08:46 |
opendevreview | Rodolfo Alonso proposed openstack/neutron master: [eventlet-deprecation] Replace ``eventlet.spawn_n`` usage https://review.opendev.org/c/openstack/neutron/+/938411 | 08:53 |
ykarel | ralonsoh, isn't the actual issue those DB errors that i included in the bug comment? | 09:00 |
ykarel | the success after 5 minute may be just some uwsgi fallback? | 09:00 |
frickler | lajoskatona: if you could tack an approval onto https://review.opendev.org/c/x/tap-as-a-service-dashboard/+/923819 and https://review.opendev.org/c/x/tap-as-a-service-tempest-plugin/+/923821 I would feel more confident force-merging them | 09:11 |
lajoskatona | frickler: ack, for these old unused things I feel it is ok | 09:16 |
lajoskatona | frickler: done | 09:17 |
ralonsoh | ykarel, yes, this seems to be the problem but I don't know why this is happening, why always in the router interface deletion | 09:19 |
frickler | ty, will proceed with those in a bit | 09:19 |
opendevreview | Rodolfo Alonso proposed openstack/neutron master: [eventlet-deprecation] Implement a metadata proxy without eventlet https://review.opendev.org/c/openstack/neutron/+/937545 | 09:23 |
opendevreview | Rodolfo Alonso proposed openstack/neutron master: [eventlet-deprecation] Use non-eventlet metadata proxy in OVN metadata agent https://review.opendev.org/c/openstack/neutron/+/938393 | 09:24 |
opendevreview | Rodolfo Alonso proposed openstack/neutron master: [eventlet-deprecation] Replace ``eventlet.spawn_n`` usage https://review.opendev.org/c/openstack/neutron/+/938541 | 09:31 |
slaweq | ralonsoh can you take a look at https://review.opendev.org/c/openstack/ovn-octavia-provider/+/929039 when you will have some time? | 09:31 |
ralonsoh | slaweq, sure | 09:31 |
slaweq | thx | 09:31 |
M9d0cd7d2[m] | Hi guys, I need some help while configuring OVN on a Charmed Openstack deployment. Can I address those doubts here, or there is a dedicated channel? | 09:33 |
M9d0cd7d2[m] | Ok, saw that for "support" #_oftc_#openstack:matrix.org is recommended | 09:34 |
opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: common: fix wait_until_true to support native thread https://review.opendev.org/c/openstack/neutron/+/937843 | 09:39 |
opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: remove the usage of eventlet in the OVS agent https://review.opendev.org/c/openstack/neutron/+/937765 | 09:39 |
opendevreview | Merged openstack/neutron unmaintained/yoga: [unmaintained only] Drop sdk job https://review.opendev.org/c/openstack/neutron/+/938478 | 09:41 |
opendevreview | Merged openstack/neutron unmaintained/xena: [unmaintained only] Drop sdk job https://review.opendev.org/c/openstack/neutron/+/938479 | 09:43 |
opendevreview | liuyulong proposed openstack/neutron master: Add meter bandwidth limit support https://review.opendev.org/c/openstack/neutron/+/860767 | 09:54 |
opendevreview | Merged openstack/neutron unmaintained/wallaby: [unmaintained only] Drop sdk job https://review.opendev.org/c/openstack/neutron/+/938480 | 11:27 |
opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: common: fix wait_until_true to support native thread https://review.opendev.org/c/openstack/neutron/+/937843 | 13:56 |
opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: remove the usage of eventlet in the OVS agent https://review.opendev.org/c/openstack/neutron/+/937765 | 13:56 |
mlavalle | #startmeeting networking | 14:00 |
opendevmeet | Meeting started Tue Jan 7 14:00:10 2025 UTC and is due to finish in 60 minutes. The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
opendevmeet | The meeting name has been set to 'networking' | 14:00 |
mlavalle | Ping list: bcafarel, elvira, frickler, mlavalle, mtomaska, obondarev, slaweq, tobias-urdin, ykarel, lajoskatona, jlibosva, averdagu, amotoki, haleyb, ralonsoh | 14:00 |
slaweq | o/ | 14:00 |
ralonsoh | hello | 14:00 |
frickler | \o | 14:00 |
bcafarel | o/ | 14:01 |
s3rj1k | hi all | 14:01 |
mlavalle | #announcements | 14:01 |
mlavalle | #link https://releases.openstack.org/epoxy/schedule.html | 14:02 |
lajoskatona | o/ | 14:02 |
mlavalle | We are currently in week R-12 | 14:02 |
cbuggy | o/ | 14:02 |
mlavalle | and it is the Epoxy-2 milestone | 14:03 |
ykarel | o/ | 14:03 |
opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: common: fix wait_until_true to support native thread https://review.opendev.org/c/openstack/neutron/+/937843 | 14:03 |
opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: remove the usage of eventlet in the OVS agent https://review.opendev.org/c/openstack/neutron/+/937765 | 14:03 |
mlavalle | Reminder: If you have a topic for the drivers meeting on Friday, please add it to the wiki @ https://wiki.openstack.org/wiki/Meetings/NeutronDrivers | 14:03 |
mlavalle | and haleyb|out wants us to continue using the priorities dashboard for patches in the "ready to merge" state (weekly reminder) | 14:05 |
mlavalle | Finally, have a very sucessful and happy 2025 | 14:07 |
mlavalle | Any other announcements? | 14:07 |
lajoskatona | mlavalle: +1 for 2025, and Happy New Year everybody :-) | 14:08 |
slaweq | HNY! | 14:08 |
rubasov | late o/ | 14:09 |
mlavalle | #topic Bugs | 14:09 |
bcafarel | HNY all :) | 14:09 |
mlavalle | Last week the bug deputy was obondarev and the one before it was jlibosva's turn | 14:10 |
mlavalle | but apparently nobody told them, since I didn't see any emails from them | 14:11 |
mlavalle | so last report we have is from ralonsoh: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/STPSYHHAAYJTJFD4DBWAAC2L2M2MUEM2/ | 14:11 |
ralonsoh | in any case, tomorrow morning I'll send a report for the last 2 weeks | 14:12 |
mlavalle | anything you want to highlight ralonsoh from that report? | 14:12 |
ralonsoh | we need to keep the list updated | 14:12 |
ralonsoh | no, we have 1 bug not assigned | 14:12 |
ralonsoh | https://bugs.launchpad.net/neutron/+bug/2092174 | 14:12 |
ralonsoh | but this is not a priority | 14:12 |
ralonsoh | anyone is welcome to take it | 14:13 |
ralonsoh | and, as commented, tomorrow morning I'll send an updated bug mail | 14:13 |
mlavalle | We have 9 bugs accumulated since the last report, starting with https://bugs.launchpad.net/neutron/+bug/2092407 | 14:13 |
slaweq | I can take it for sure, but this should be maybe discussed first within the team | 14:13 |
slaweq | maybe others don't want to deprecate this config option, maybe there is some use case for it which I am not aware about | 14:14 |
mlavalle | how about sending a message to the ML and then discuss it in the drivers meeting? | 14:15 |
slaweq | mlavalle sure, I will do that | 14:15 |
ralonsoh | +1 | 14:15 |
slaweq | and will add this to the Friday's meeting agenda | 14:16 |
lajoskatona | +1 for mail | 14:16 |
mlavalle | I would say give a couple of weeks between the message to the ML and discussion in the drivers meeting. That way we give opportunity to the community to give feedback | 14:17 |
slaweq | ++ | 14:18 |
mlavalle | cool | 14:18 |
mlavalle | anything else we should discuss in this section? | 14:18 |
mlavalle | this week the bug deputy is ralonsoh and next it is lajoskatona's turn. you both ok with it? | 14:20 |
ralonsoh | yes | 14:20 |
lajoskatona | ack | 14:20 |
mlavalle | #topic community goals | 14:21 |
mlavalle | anything new on on neutronclient deprecation ? | 14:21 |
lajoskatona | yes I work on a patch for horizon fips | 14:22 |
lajoskatona | so slow progress | 14:22 |
lajoskatona | https://review.opendev.org/c/openstack/horizon/+/938488 | 14:22 |
lajoskatona | that's it for this topic | 14:23 |
mlavalle | thanks for the update lajoskatona ++ | 14:23 |
mlavalle | the other subject here is eventlet. Any updates this week? | 14:24 |
ralonsoh | yes, several | 14:24 |
ralonsoh | documentation for eventlet deprecation (review): https://review.opendev.org/c/openstack/neutron/+/938390 | 14:24 |
ralonsoh | OVN agent with a new socket server implementation (see limitations): https://review.opendev.org/c/openstack/neutron/+/937545 | 14:24 |
ralonsoh | this server does not use eventlet | 14:25 |
ralonsoh | remember OVN agent is the default agent for plugin jobs | 14:25 |
ralonsoh | next one | 14:25 |
ralonsoh | OVN metadata agent: https://review.opendev.org/c/openstack/neutron/+/938393/ | 14:25 |
ralonsoh | same as before, and I've refectored a bit the agent not to use oslo.services (for now) | 14:25 |
ralonsoh | and now it is running without eventlet | 14:25 |
ralonsoh | next one | 14:25 |
ralonsoh | L3 thread processing reduction: https://review.opendev.org/c/openstack/neutron/+/938406 | 14:26 |
mlavalle | it failed CI. Is it due to the change? | 14:26 |
ralonsoh | mlavalle, which one? | 14:26 |
mlavalle | metadata | 14:26 |
ralonsoh | I need to check but most probably not | 14:26 |
mlavalle | ack | 14:27 |
ralonsoh | ovn job is not stable since we migrated to wsgi | 14:27 |
ralonsoh | there are several bugs open | 14:27 |
ralonsoh | and I'm trying to address all of them | 14:27 |
ralonsoh | I'm checking the logs | 14:27 |
ralonsoh | this is related to this bug | 14:27 |
ralonsoh | --> https://review.opendev.org/c/openstack/neutron/+/938319 | 14:28 |
ralonsoh | so continuing with the list of patches | 14:28 |
ralonsoh | L3 thread processing reduction: https://review.opendev.org/c/openstack/neutron/+/938406 | 14:28 |
M9d0cd7d2[m] | Hi people, is it possible that this configuration... (full message at <https://matrix.org/oftc/media/v1/media/download/AVMyqUurPDoOHlT2L6alQW-_y-DG89CfQy9avvE-bsgg4FyZE4OqVOZrTa-ZLL2AhqpBFC5__eTNiYiQKKruaHtCeUiUPjygAG1hdHJpeC5vcmcvWVhIS3BndmdvT3hjQ0NtcUh6b1Zzb1lQ>) | 14:28 |
ralonsoh | please review and check Liu's comment | 14:28 |
ralonsoh | this is the same as with the DHCP patch | 14:28 |
ralonsoh | multithread does not improve the event processing performance at all | 14:29 |
mlavalle | yeap | 14:29 |
ralonsoh | and once we move to kernel threads that are preemptive | 14:29 |
ralonsoh | we can have the problem of stopped threads to start processing others | 14:29 |
ralonsoh | so we can have routers not fully initialized and the L3 agent processing others | 14:29 |
ralonsoh | and that's all for today | 14:29 |
mlavalle | Thanks for the update and the hard work on this topic ralonsoh ++ | 14:30 |
lajoskatona | another batch of patches from Sahid; https://review.opendev.org/q/topic:%22bug/2087939%22 | 14:31 |
ralonsoh | yes | 14:31 |
lajoskatona | mostly for os-ken | 14:31 |
ralonsoh | I think I had a topic for it in next section | 14:31 |
ralonsoh | but let's discuss this it here | 14:31 |
ralonsoh | the point is | 14:31 |
mlavalle | I'll change topic | 14:31 |
ralonsoh | if we can't migrate os-ken to kernel threads in this cycle, I would suggest to go back again to ofctl | 14:31 |
mlavalle | #topic on-demand | 14:32 |
ralonsoh | now, os-ken has one single implementation, eventlet | 14:32 |
ralonsoh | and the effort to migrate to kernel threads is unknown | 14:32 |
ralonsoh | so if this is not possible in a reasonable amount of time | 14:32 |
ralonsoh | I would suggest restoring ocftl (that works and must do it with kernel threads) | 14:33 |
ralonsoh | but this is just a heads-up | 14:33 |
sahid_ | ralonsoh: but is that not enough to remove the monley patching event if we keep the implemntation using eventlet? | 14:33 |
sahid_ | https://review.opendev.org/c/openstack/os-ken/+/938337 | 14:33 |
sahid_ | at least for this release? | 14:34 |
ralonsoh | sahid_, where are you removing the monkey patching? | 14:34 |
ralonsoh | this is being called from the ovs agent | 14:34 |
ralonsoh | and we monkey-patch before calling it | 14:34 |
ralonsoh | did you check that with any neutron CI? | 14:34 |
sahid_ | the function hub.patch is doing the monkey patching is osken lib right? | 14:34 |
lajoskatona | ralonsoh: what is ocftl? | 14:34 |
ralonsoh | the CLI interface, using "ovs-ofctl" commands | 14:35 |
sahid_ | ralonsoh: it's basically on what i'm currently working | 14:35 |
sahid_ | lajoskatona: asked me to use a dep patch with neutron | 14:35 |
sahid_ | https://review.opendev.org/c/openstack/os-ken/+/938337 | 14:35 |
ralonsoh | sahid_, perfect, so I'm just saying that this is an alternative to os-ken, if the migration is not possible | 14:35 |
sahid_ | ralonsoh: great | 14:36 |
lajoskatona | ralonsoh: ahh, ok | 14:36 |
ralonsoh | sahid_, this patch and the CI job is using eventlet | 14:36 |
ralonsoh | --> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7dd/938337/2/check/neutron-ovs-tempest-dvr/7ddc386/controller/logs/screen-q-agt.txt | 14:36 |
ralonsoh | so this is not a valid test | 14:36 |
sahid_ | i have not checked at that point | 14:37 |
sahid_ | but i don't see any reason that osken with its current implementation to not work without monkey patch | 14:37 |
sahid_ | i will elt you know | 14:37 |
ralonsoh | because the os-ken hub has one single implementation, that is based in eventlet | 14:38 |
lajoskatona | yeah, os-ken has lots of modules used by neutworking projects (BGP and similar protocol files) so even if we drop hub we have to keep the rest I beleive | 14:38 |
ralonsoh | and you need the os-ken hub to spawn the threads: one for the agent, one for the commands and one moniting the OF table | 14:38 |
ralonsoh | but it is not possible to drop the hub | 14:39 |
sahid_ | ralonsoh: yes but that implementation can with and without monkey patch | 14:39 |
sahid_ | can work | 14:39 |
sahid_ | that is said I don't have any issue to remove osken at all | 14:39 |
ralonsoh | perfect then, waiting for a CI job testing that | 14:39 |
ralonsoh | os-ken is much faster than osctl | 14:39 |
ralonsoh | ofctl | 14:39 |
ralonsoh | that will be a serious regression in performance | 14:40 |
sahid_ | interesting point, thanks | 14:40 |
sahid_ | btw I'm trying different king of impl to fix wait_until_true I will be glad to get any idea of which one could be the best | 14:42 |
sahid_ | https://review.opendev.org/c/openstack/neutron/+/937843/9 | 14:42 |
sahid_ | ion that one I run the predicate in a separate thread instead of making it runnoing in the main thread and having a timer in a different thread | 14:42 |
sahid_ | but it's an other topic, sorry for the disruption | 14:42 |
mlavalle | anything else to discuss today? | 14:43 |
ralonsoh | not from me | 14:43 |
lajoskatona | I added one topic for on-demand | 14:43 |
mlavalle | go ahead, lajoskatona | 14:44 |
lajoskatona | There was a self +2 wf+1: https://review.opendev.org/c/openstack/neutron/+/936235 | 14:44 |
lajoskatona | and I think this is a topic to discuss and see how to void such situation | 14:44 |
lajoskatona | one side as I see is that there can be frustration for slow review even no-review | 14:45 |
ralonsoh | lajoskatona, anyone can participate in this meeting and request for reviews | 14:46 |
lajoskatona | but in this case this patch I copied was not dying in gerrit for months, so I believe that after the vacation time there would have been review for it | 14:46 |
ralonsoh | or send a mail | 14:46 |
ralonsoh | or increase the review priority | 14:46 |
lajoskatona | ralonsoh: +1 that is also true, and we have the priority board also | 14:46 |
ralonsoh | and, to be honest, I now have technical questions about this patch | 14:47 |
lajoskatona | so wanted to highlight this event and ask everybody to thin kabout it and when haleyb is back we can have some actions or more discussion around it | 14:47 |
mlavalle | I'm of the opinion that this shouldn't be allowed | 14:47 |
lajoskatona | we can revert it and start the review again | 14:47 |
mlavalle | I say let's do it | 14:48 |
mlavalle | haleyb|out will be back tomorrow. he can do it | 14:48 |
lajoskatona | We can ask Liu to participate on a meeting and discuss this topic to see why was this the only solution He saw at that time | 14:48 |
lajoskatona | mlavalle: yeah let's wait till the boss is back :-) He commented on the patch so has context for it | 14:49 |
mlavalle | ++ | 14:49 |
lajoskatona | that's it from me | 14:50 |
mlavalle | thanks lajoskatona for bringing this up. really important | 14:50 |
mlavalle | and I think that the proposal to let Liu voice his point of view is good | 14:51 |
mlavalle | anything else for today? | 14:51 |
mlavalle | ok, have a great first week of 2025! | 14:53 |
mlavalle | #endmeeting | 14:53 |
opendevmeet | Meeting ended Tue Jan 7 14:53:26 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:53 |
opendevmeet | Minutes: https://meetings.opendev.org/meetings/networking/2025/networking.2025-01-07-14.00.html | 14:53 |
opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/networking/2025/networking.2025-01-07-14.00.txt | 14:53 |
opendevmeet | Log: https://meetings.opendev.org/meetings/networking/2025/networking.2025-01-07-14.00.log.html | 14:53 |
lajoskatona | Bye | 14:53 |
ykarel | o/ | 14:53 |
s3rj1k | thanks all | 14:53 |
ralonsoh | bye | 14:53 |
ralonsoh | folks: https://review.opendev.org/c/openstack/neutron/+/934701 | 15:04 |
slaweq | ralonsoh mlavalle lajoskatona please check https://review.opendev.org/c/openstack/neutron/+/937887 when You will have a minute or two, thx in advance | 15:06 |
ralonsoh | slaweq, sure | 15:06 |
ralonsoh | slaweq, one comment | 15:10 |
slaweq | thx, I will change it according to your comment | 15:34 |
opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == [eventlet-deprecation] Change the state change WSGI server https://review.opendev.org/c/openstack/neutron/+/938404 | 16:00 |
greatgatsby | Hello. I'm seeing `QueuePool limit of size 1 overflow 50 reached` in neutron-server logs. At the same time, I'm dumping the mariadb processlist in a loop and counting the neutron threads/connections and there's never above 45 from what I can tell. This is a kolla-ansible zed deployment. | 17:23 |
greatgatsby | also, of those 40'ish processes, the majority are in Sleep, so I'm not sure why I seem to be hitting a QueuePool limit? | 17:24 |
opendevreview | Ihar Hrachyshka proposed openstack/neutron-lib master: Enable mypy for pep8 tox target https://review.opendev.org/c/openstack/neutron-lib/+/921484 | 19:03 |
opendevreview | Merged openstack/neutron master: [OVN] QoS max and min rules should be defined in LSP for phynet ports https://review.opendev.org/c/openstack/neutron/+/934418 | 20:26 |
opendevreview | Merged openstack/neutron master: [eventlet-deprecation] Add a reference document https://review.opendev.org/c/openstack/neutron/+/938390 | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!