| opendevreview | Brian Haley proposed openstack/neutron master: DNM: Test tempest port wait patch https://review.opendev.org/c/openstack/neutron/+/928472 | 00:29 |
|---|---|---|
| opendevreview | Merged openstack/neutron-tempest-plugin master: refactor: Remove unused code from utils and tests https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/928461 | 00:51 |
| opendevreview | Brian Haley proposed openstack/neutron master: Add special treatment for 'any' in SG rule API https://review.opendev.org/c/openstack/neutron/+/926498 | 00:56 |
| opendevreview | Liushy proposed openstack/neutron master: [ovn]Add one periodic task for 'agent_health_check' https://review.opendev.org/c/openstack/neutron/+/916903 | 03:22 |
| opendevreview | Brian Haley proposed openstack/neutron master: Optionally configure IPv6 metadata address https://review.opendev.org/c/openstack/neutron/+/926497 | 03:24 |
| opendevreview | Brian Haley proposed openstack/neutron master: Add special treatment for 'any' in SG rule API https://review.opendev.org/c/openstack/neutron/+/926498 | 03:31 |
| opendevreview | Liushy proposed openstack/neutron master: [ovn]Add one periodic task for 'agent_health_check' https://review.opendev.org/c/openstack/neutron/+/916903 | 04:56 |
| *** ralonsoh_ is now known as ralonsoh | 05:48 | |
| opendevreview | Lajos Katona proposed openstack/neutron master: Functional: Increase Ulimit to 3072 https://review.opendev.org/c/openstack/neutron/+/928759 | 07:40 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: Enable iptables debugging in the L3 agent functional tests https://review.opendev.org/c/openstack/neutron/+/928136 | 08:41 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron-tempest-plugin master: Add test for extension "quota-check-limit-default" https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/928770 | 09:10 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: Enable iptables debugging in the L3 agent functional tests https://review.opendev.org/c/openstack/neutron/+/928136 | 09:14 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Neutron quota engine checks the resource usage by default https://review.opendev.org/c/openstack/neutron/+/926725 | 09:19 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron-lib master: Add new API extension ``uplink-status-propagation-updatable`` https://review.opendev.org/c/openstack/neutron-lib/+/927820 | 09:31 |
| slaweq | ykarel ralonsoh lajoskatona hi, if you have a minute, please check https://review.opendev.org/c/openstack/neutron/+/928136, thx in advance | 10:21 |
| ralonsoh | sure | 10:29 |
| ralonsoh | slaweq, why this: https://review.opendev.org/c/openstack/neutron/+/928136/3/neutron/tests/functional/agent/l3/extensions/test_ndp_proxy_extension.py#44 | 10:31 |
| ralonsoh | correction: not why the change itself but why the tests are failing with this config option? | 10:38 |
| slaweq | ralonsoh TBH I don't know but it is failing due to some iptables rules not match in second run, exactly raises https://github.com/openstack/neutron/blob/master/neutron/agent/linux/iptables_manager.py#L477 | 12:02 |
| slaweq | and I don't really have time now to investigate this ndp proxy extension so that's why I added TODO there only | 12:02 |
| opendevreview | Merged openstack/neutron master: Enable iptables debugging in the L3 agent functional tests https://review.opendev.org/c/openstack/neutron/+/928136 | 12:17 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: DNM - TESTING PATCH: neutron-functional-with-uwsgi https://review.opendev.org/c/openstack/neutron/+/924335 | 13:04 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add source interface in "ping" command in ``ARPSpoofTestCase`` https://review.opendev.org/c/openstack/neutron/+/928791 | 13:04 |
| haleyb | #startmeeting networking | 14:00 |
| opendevmeet | Meeting started Tue Sep 10 14:00:31 2024 UTC and is due to finish in 60 minutes. The chair is haleyb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'networking' | 14:00 |
| haleyb | Ping list: bcafarel, elvira, frickler, mlavalle, mtomaska, obondarev, slaweq, tobias-urdin, ykarel, lajoskatona, jlibosva, averdagu, amotoki, haleyb, ralonsoh | 14:00 |
| mlavalle1 | \o | 14:00 |
| elvira | o/ | 14:00 |
| rubasov | o/ | 14:00 |
| ralonsoh | hello | 14:00 |
| mtomaska | o/ | 14:00 |
| obondarev | hi | 14:00 |
| slaweq | o/ | 14:00 |
| ihrachys | o/ | 14:01 |
| lajoskatona | o/ | 14:01 |
| bcafarel | o/ | 14:01 |
| haleyb | hi everyone, let's get started | 14:01 |
| haleyb | #topic announcements | 14:01 |
| haleyb | We are now in Dalmatian release week (R - 3) | 14:01 |
| haleyb | Focus should be on finding and fixing release-critical bugs, so that release candidates and final versions of the 2024.2 Dalmatian deliverables can be proposed, well ahead of the final 2024.2 Dalmatian release date | 14:02 |
| haleyb | All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/2024.2 branch will be cut | 14:02 |
| haleyb | This branch will track the 2024.2 Dalmatian release | 14:02 |
| haleyb | Once stable/2024.2 has been created, the master branch will be ready to switch to 2025.1 Epoxy development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing 2024.2 Dalmatian plans | 14:02 |
| frickler | o/ | 14:02 |
| haleyb | Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/2024.2 branch before a new release candidate can be proposed | 14:02 |
| haleyb | Ok, so that was a lot of pasting from the wiki | 14:03 |
| haleyb | There is an initial RC1 candidate proposed | 14:03 |
| haleyb | #link https://review.opendev.org/c/openstack/releases/+/928550 | 14:03 |
| haleyb | I'm assuming master has moved since that was proposed | 14:03 |
| haleyb | I will check and update today, but is there anything in the gate that i should be waiting for? | 14:04 |
| lajoskatona | thanks for taking care | 14:05 |
| haleyb | As once that merges, stable/2024.2 will be cut and anything critical will need to be cherry-picked | 14:05 |
| haleyb | We do have until Friday at the latest, so we can look at getting any last minute fixes in | 14:06 |
| haleyb | very quiet, so i'll move on | 14:07 |
| ralonsoh | This patch (I found today): the cycle highlights https://review.opendev.org/c/openstack/releases/+/928289 | 14:07 |
| haleyb | yes, that was next on my list | 14:07 |
| ralonsoh | sorry | 14:07 |
| haleyb | np, it does need a formatting update, but if there's something i missed, or something that doesn't need to be there please comment | 14:08 |
| haleyb | i just went through commit messages for ideas | 14:08 |
| haleyb | and regarding the RC1 deadline - there are a number of other networking-* projects that also have releases proposed | 14:09 |
| haleyb | #link https://review.opendev.org/q/project:openstack/releases | 14:10 |
| haleyb | if you contribute to any of those (too many to list), please check and +1 | 14:10 |
| haleyb | i will go through them later today and verify hashes with master branches | 14:10 |
| haleyb | So just to re-state the deadlines | 14:11 |
| haleyb | RC1 deadline: September 13th, 2024 (R-3 week) (this week) | 14:11 |
| haleyb | Final RC deadline: September 26th, 2024 (R-1 week) | 14:11 |
| haleyb | Final 2024.2 Dalmatian release: October 2nd, 2024 | 14:11 |
| haleyb | Thanks for everyone's hard work during the cycle, we're almost done :) | 14:12 |
| haleyb | that was all the announcements i had, any others or questions? | 14:13 |
| haleyb | ok, let's move onto bugs | 14:14 |
| haleyb | #topic bugs | 14:14 |
| haleyb | there were unfortunately a lot of bugs this week | 14:14 |
| haleyb | ihrachys was the bug deputy, his report is at | 14:14 |
| haleyb | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/6UHEMOMG3CM5JN4FHOETKSFGAJOBSYAK/ | 14:14 |
| ihrachys | some of these in the report are not NEW but changed their state though (e.g. more info added or requested) | 14:15 |
| haleyb | ihrachys: ack | 14:16 |
| haleyb | first one in the list is | 14:16 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2078846 | 14:16 |
| haleyb | OVN fails to transition router port to ACTIVE, claims L2 provisioning block is still present | 14:16 |
| ihrachys | i reported it; it took several hours of my time and i still could figure it fully out. i dumped all i found in the bug though. if someone is familiar with provisioning blocks / ovn router port transition to active, your help may be fitting. | 14:17 |
| haleyb | ihrachys: thanks for looking. did you see it more than once? | 14:19 |
| ihrachys | it was once in gate on my patch, hence I had to look and report :) | 14:19 |
| haleyb | https://review.opendev.org/c/openstack/neutron/+/927631 patch i guess based on the link | 14:21 |
| haleyb | well, if someone has the cycles to look that would be good, but lets get through some of the others | 14:22 |
| haleyb | next is | 14:22 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2079047 | 14:23 |
| ralonsoh | yeah, I saw this job failing all times in the periodic queue | 14:23 |
| ihrachys | oh is it in periodic after all? thought it was experimental. | 14:24 |
| ralonsoh | I have no time to review this specific configuration | 14:24 |
| ralonsoh | all periodic jobs are also experimental | 14:24 |
| ralonsoh | (not the other way around) | 14:24 |
| ihrachys | if so then I think opensearch exposes the approximate time when it started to fail | 14:24 |
| ihrachys | because it was clear on the chart (link in the bug report) when it started to spike | 14:25 |
| slaweq | maybe we should ping LiuYulong about it as he is main author of this distributed dhcp feature and those jobs | 14:25 |
| frickler | opensearch only goes back 10 days, so if it starts then, that's how much data there are, not necessarily when the issue started | 14:26 |
| ralonsoh | and is always the same test | 14:26 |
| ihrachys | frickler: ah! :( | 14:26 |
| haleyb | slaweq: yes, pinging him would be good as it could be related to those changes | 14:27 |
| ralonsoh | I though that could be related to the eventlet/wsgi change | 14:27 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/927958 | 14:27 |
| ralonsoh | but this is not the case | 14:27 |
| ralonsoh | thought* | 14:27 |
| haleyb | I will add Liu and ask about his thoughts | 14:28 |
| haleyb | ok, next one | 14:30 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2079048 | 14:30 |
| haleyb | Metadata functional tests failing due to connection timeout | 14:30 |
| ihrachys | think we merged a patch that may help to investigate the next time it hits | 14:30 |
| haleyb | yes, there was a patch to enable iptables debug | 14:31 |
| haleyb | AssertionError: metadata proxy unreachable on http://[fe80::a9fe:a9fe%port19caee]:80 before timeout | 14:31 |
| slaweq | yes, it was just merged today | 14:31 |
| slaweq | it will log iptables rules applied by the agent, maybe this will help understand why there is no connectivity there | 14:32 |
| slaweq | so far I did not found anything wrong in the logs from the job runs which I was checking | 14:32 |
| haleyb | so is it always the rate-limiting test that fails? | 14:33 |
| slaweq | as far as I saw, yes | 14:33 |
| slaweq | and it is always the same test probably | 14:33 |
| slaweq | related to ipv6 metadata | 14:34 |
| haleyb | right. i'm just thinking of ideas, but wonder if there is something different wrt ipv6 where we hit the limit too soon? or there's just another bug hiding in here | 14:35 |
| slaweq | but this is failing on the first attempt to reach to the metadata server, so there should be no limit hit yet at all | 14:36 |
| haleyb | oh, so either not listening or packets got "lost" | 14:37 |
| slaweq | IMO yes | 14:37 |
| slaweq | that's why I added this iptables debug to the tests | 14:38 |
| slaweq | as metadata in router namespace should be reachable through the iptables rules | 14:38 |
| haleyb | i will watch it and try to look | 14:38 |
| slaweq | thx haleyb | 14:38 |
| haleyb | and we have one more gate failure | 14:38 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2079831 | 14:38 |
| ralonsoh | that is related to the OVN wsgi migration patch | 14:39 |
| haleyb | this one i took last week, and added tempest as that's where the code lives | 14:39 |
| ralonsoh | thanks for taking it | 14:39 |
| haleyb | ralonsoh: oh, so did you only see the failure with that last wsgi patch? | 14:39 |
| ralonsoh | yes | 14:39 |
| haleyb | i've been trying to see the "not ACTIVE" string unsuccessfully | 14:39 |
| haleyb | ok, i might need to add another depends-on to my test patch | 14:40 |
| ralonsoh | actually the logs I provided are from this patch, some CI executions | 14:40 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/924317 | 14:40 |
| ralonsoh | actually ^^ this patch should be updated with yours | 14:40 |
| haleyb | ralonsoh: and i will update the tempest patch based on your comments, was thinking about it last night but ran out of time | 14:40 |
| ralonsoh | thanks | 14:41 |
| ihrachys | hm. do you have a theory of how does wsgi patch affects the speed of port active transition? | 14:41 |
| ralonsoh | I think (I need to confirm that) it replies faster (the API call) | 14:41 |
| ihrachys | ah. so it's vice versa - we list ports more quickly? | 14:42 |
| ralonsoh | so we receive the event (chassis binding) and we process it at the same time | 14:42 |
| ralonsoh | right | 14:42 |
| ralonsoh | but as I said, I need to confirm that | 14:42 |
| ralonsoh | (wsgi is supposed to be faster) | 14:43 |
| ihrachys | ack. but the test change seems sane regardless. we should assume a particular slowliness / snappiness of port transitions. | 14:43 |
| ihrachys | *should not | 14:43 |
| ralonsoh | the problem is that after the nova VM creation call, the ports could not be ready yet | 14:44 |
| ralonsoh | so I think this check is correct there | 14:44 |
| ralonsoh | actually we usually call https://github.com/openstack/tempest/blob/0a0e1070e573674332cb5126064b95f17099307e/tempest/scenario/test_network_basic_ops.py#L124 | 14:45 |
| ralonsoh | but not in this case because of project_networks_reachable=False | 14:45 |
| haleyb | alright, i'll move on since there's still more unassigned bugs :( | 14:49 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2078856 | 14:49 |
| haleyb | OVN invalid syntax '' in networks | 14:49 |
| ralonsoh | waiting for more info | 14:50 |
| haleyb | right, see that now | 14:50 |
| haleyb | there were also some vpnaas ones | 14:51 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2080072 | 14:51 |
| haleyb | Failed to delete vpnaas ipsec-site-connections with 502 error, ORM session: SQL execution without transaction in progress | 14:51 |
| haleyb | i will try and ping bodo, don't see him on channel | 14:53 |
| ihrachys | noticed it in gate once; seems like a clear case of sqlalchemy api not used correctly | 14:53 |
| lajoskatona | I keep an eye also on it | 14:53 |
| haleyb | alright, any other bugs to discuss, running out of time | 14:55 |
| haleyb | this weeks deputy is lucasgomes, next week is jlibosva | 14:55 |
| haleyb | i remember someone pinging lucas and he said he's good for this week | 14:56 |
| haleyb | and fwiw bug count is staying stable | 14:56 |
| haleyb | Current bug count this week: 717, up 5 from last week | 14:56 |
| mlavalle1 | haleyb: I confirmed with lucas that he will be triaging bugs this week | 14:56 |
| haleyb | mlavalle1: thanks | 14:57 |
| mlavalle1 | we even had a follow up chat about it at the end of last week | 14:57 |
| haleyb | i'll skip over community | 14:57 |
| haleyb | #topic on-demand | 14:57 |
| haleyb | anything else to discuss? | 14:57 |
| haleyb | so just as a reminder, please only merge bug fixes on master until stable/2024.2 branch created, or ask for an exception request | 14:59 |
| haleyb | as i mentioned, i'll be looking at the release patches today/tomorrow | 14:59 |
| haleyb | thanks for attending and have a good week fixing bugs :) | 15:00 |
| haleyb | #endmeeting | 15:00 |
| opendevmeet | Meeting ended Tue Sep 10 15:00:40 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/networking/2024/networking.2024-09-10-14.00.html | 15:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/networking/2024/networking.2024-09-10-14.00.txt | 15:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/networking/2024/networking.2024-09-10-14.00.log.html | 15:00 |
| ralonsoh | bye | 15:01 |
| lajoskatona | Bye | 15:01 |
| mlavalle1 | \o | 15:01 |
| slaweq | o/ | 15:01 |
| haleyb | lajoskatona: can you look at https://review.opendev.org/c/openstack/neutron/+/618208 one more time? i know you voted previously, it's an old one but one of the last on the priority dashboard - trying to get some of these closed | 15:06 |
| haleyb | ihrachys: i see you're watching there too :) | 15:06 |
| lajoskatona | haleyb: sure, I will check it | 15:11 |
| ihrachys | haleyb: the patch is simple (basically a one liner) but I wasn't sure as to the effect of this setting. I wonder why it's not enabled by default in kernel, and what we should be aware of before enabling it. | 15:20 |
| ihrachys | haleyb: the linked issue in libnetwork gives two options to handle these packets. one is the sysctl knob. another is iptables rule to drop INVALIDs. https://github.com/moby/libnetwork/issues/1090 wonder if the latter would be more isolated. (the former will affect everything on the node, even traffic that does not belong to the agent ports) | 15:24 |
| ihrachys | iptables -I INPUT -m conntrack --ctstate INVALID -j DROP | 15:24 |
| opendevreview | Eduardo Olivares proposed openstack/ovn-bgp-agent master: WIP: add retries to get_device_port_at_ovs https://review.opendev.org/c/openstack/ovn-bgp-agent/+/928822 | 15:25 |
| ihrachys | there's some concerns about the rule affecting "legit traffic" here though: https://github.com/kubernetes/kubernetes/pull/120412 the PR then switches from the iptables rule to the sysctl knob. so I dunno if one is better or the other. | 15:30 |
| ihrachys | but that's the root of my sitting on the sidelines here - I don't understand all the implications. | 15:31 |
| haleyb | ihrachys: it's been a while since i looked at the change, but the sysctl seemed the best option, and it's only in the snat namespace so shouldn't affect other things | 15:32 |
| ihrachys | ah correct, I forget about namespaces. | 15:32 |
| ihrachys | the bliss of ovn ;) | 15:33 |
| ihrachys | haleyb: ok i +2d it then; there's a very minor comment about a comment, we can ignore it or pick up on respin., | 15:37 |
| haleyb | ihrachys: thanks, i'll look and respin if necessary | 15:38 |
| opendevreview | Merged openstack/os-vif stable/2024.2: Update .gitreview for stable/2024.2 https://review.opendev.org/c/openstack/os-vif/+/928353 | 16:26 |
| opendevreview | Merged openstack/os-vif stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2 https://review.opendev.org/c/openstack/os-vif/+/928355 | 17:05 |
| samcat116 | Hi all, is there anywhere in particular I should start debugging to figure out why hitting GET /ports?fields=id takes 10s to return? We have around 5k ports on our stack here using OVN. I'm running with 20 api_workers as well | 18:02 |
| ihrachys | samcat116: how long does it take to list 100 ports? (does it scale linearly?) | 18:07 |
| samcat116 | 10 ports: 175ms... (full message at <https://matrix.org/oftc/media/v1/media/download/ASE4SBWzDqu7cUxipXvxvqTR2zAsWr9QrnN1M-TZ3sKws_YhPb_CY0Gyhi8buhqMATamJuwKpyKOntfmb7cJSstCeR3t5NUgAG1hdHJpeC5vcmcvQlNmU0ZWWG91Vk1wbWhwTXdaVkt2WlV2>) | 18:16 |
| samcat116 | So decently linearly | 18:16 |
| opendevreview | Brian Haley proposed openstack/neutron master: DNM: Test tempest port wait patch https://review.opendev.org/c/openstack/neutron/+/928472 | 21:18 |
| opendevreview | Merged openstack/tap-as-a-service master: Doc: remove sphinxcontrib-*diag from doc dependencies https://review.opendev.org/c/openstack/tap-as-a-service/+/927584 | 21:19 |
| opendevreview | Merged openstack/ovn-octavia-provider master: Fix member subnet id on a fully populated LB https://review.opendev.org/c/openstack/ovn-octavia-provider/+/928335 | 21:24 |
| opendevreview | Merged openstack/networking-bgpvpn master: Doc: remove sphinxcontrib-*diag from doc dependencies https://review.opendev.org/c/openstack/networking-bgpvpn/+/927390 | 22:24 |
| ihrachys | samcat116: fyi your message didn't go through because you seem to use matrix bridge and it's truncated. | 23:55 |
| ihrachys | but if it's linear, then... maybe it's expected? you have to serialize each port (even if it's just one id field), the work is constant per record. | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!