| opendevreview | Brian Haley proposed openstack/neutron master: Remove temporary directories created in policy tests https://review.opendev.org/c/openstack/neutron/+/936116 | 02:23 |
|---|---|---|
| *** liuxie is now known as liushy | 07:04 | |
| ralonsoh | ykarel, bcafarel for 2024.1 I'm going to temporarily disable -skip jobs. Glance is still in stable/2023.1 (not unmaintained) and the requirements are failing | 07:13 |
| ralonsoh | https://review.opendev.org/c/openstack/releases/+/934502 | 07:13 |
| ralonsoh | --> https://review.opendev.org/c/openstack/neutron/+/936087/comments/b9e88c3f_3017f76f | 07:13 |
| ralonsoh | I think this is also happening in 2023.2 | 07:13 |
| ykarel | ralonsoh, yes 2023.1 broken due to partial transition to unmaintained | 07:14 |
| ykarel | but yes ok to disable grenade jobs in the branches which are really blocked | 07:15 |
| ralonsoh | cool | 07:15 |
| ykarel | iirc we used to drop grenade when branches get to unmaintained, so let's confirm and do that | 07:16 |
| ykarel | ralonsoh, ^ | 07:16 |
| ralonsoh | so that implies -skip in 2024.1 and grenade in 2023.2 | 07:17 |
| ykarel | yes | 07:18 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2023.2: [stable-only] Disable grenade jobs in the CI https://review.opendev.org/c/openstack/neutron/+/936119 | 07:24 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2024.1: [stable-only] Disable skip-level jobs in the CI https://review.opendev.org/c/openstack/neutron/+/936120 | 07:25 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2024.1: [stable-only] Drop skip-level jobs in the CI https://review.opendev.org/c/openstack/neutron/+/936120 | 07:53 |
| ralonsoh | folks, if you can check https://bugs.launchpad.net/neutron/+bug/2088986. We would need, in this order: | 08:10 |
| ralonsoh | * https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/936036 | 08:10 |
| ralonsoh | * https://review.opendev.org/c/openstack/neutron/+/936010 | 08:11 |
| ralonsoh | * https://review.opendev.org/c/openstack/neutron/+/935883 | 08:11 |
| ralonsoh | if you have some minutes, I'll appreciate your reviews | 08:11 |
| opendevreview | Merged openstack/ovn-bgp-agent master: Bump ovsdbapp to 2.8.0 https://review.opendev.org/c/openstack/ovn-bgp-agent/+/933885 | 08:26 |
| bcafarel | I suppose for https://review.opendev.org/c/openstack/neutron/+/936119 this will also become a "drop grenade since antelope is unmaintained" patch? | 08:54 |
| ralonsoh | bcafarel, yes, I'll update this patch | 08:56 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2023.2: [stable-only] Remove grenade and skip-level jobs in the CI https://review.opendev.org/c/openstack/neutron/+/936119 | 09:46 |
| ralonsoh | ^^ this patch requires first to remove the devstack jobs in its CI | 09:46 |
| ralonsoh | there is a depends-on patch related | 09:46 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron-tempest-plugin master: Remove all eventlet Neutron API jobs https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935627 | 10:16 |
| opendevreview | Serhii Ivanov proposed openstack/neutron-specs master: Add `Agent Startup State Tracking` spec https://review.opendev.org/c/openstack/neutron-specs/+/935724 | 10:53 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [FT] ``Queue.external_ids:queue-num`` should be a number https://review.opendev.org/c/openstack/neutron/+/936141 | 11:36 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Bump psutil to 6.1.0 https://review.opendev.org/c/openstack/neutron/+/936144 | 11:59 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Bump httplib2 to 0.22.0 https://review.opendev.org/c/openstack/neutron/+/936146 | 12:05 |
| opendevreview | yatin proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with 1 worker https://review.opendev.org/c/openstack/neutron/+/936147 | 12:07 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Bump requests to 2.32.3 https://review.opendev.org/c/openstack/neutron/+/936148 | 12:08 |
| *** elodilles is now known as elodilles_afk | 13:18 | |
| danfai_ | Hi, we experienced some broadcast traffic being forwarded by the OVN tunnels between different nodes (and network segments) recently. Incoming broadcast traffic to a hypervisor running OVN was forwarded to all other connected hypervisors and there sent out to the VM and the physical interface, basically tunneling/forwarding all broadcast/multicast traffic. | 13:36 |
| danfai_ | We could reproduce this by power cycling servers (with VMs on them), and at the moment I believe, that the way ovs starts up with the tunnels configured at the time of an uncleanly shutdown might cause this. Did someone else experience this already? | 13:36 |
| danfai_ | (we are running on yoga, OVN 22.12, all hypervisors in a routed provider network) | 13:36 |
| s3rj1k | danfai_: Could that be that not all flows where recovered? | 13:42 |
| ralonsoh | danfai_, please, open a launchpad bug. The main question here is did the ovn-controller shutdown properly? Than means, did you stop the ovn-controller process gracefully? If that didn't happen, then the chassis would still be configured in the cluster | 13:45 |
| ralonsoh | you can also try setting "localnet_learn_fdb", but if you are using ports with security disabled | 13:45 |
| danfai_ | s3rj1k: What do you mean by recovered? It seems the flows were there, also the tunnel interfaces, although a restart of ovn-controller (which fixed it) did have a small diff (iirc moving some flows from table 37 to 38, or 38->39, gonna have to double check | 13:45 |
| danfai_ | ralonsoh: no, the ovn-controller died with the full hypervisor (e.g. unplugging the power and booting again) | 13:46 |
| ralonsoh | danfai_, so there you have your problem: the chassis is still registered in the cluster database | 13:46 |
| ralonsoh | including the port bindings | 13:46 |
| ralonsoh | as you said, you solved it restarting the ovn-controller | 13:46 |
| ralonsoh | in that case you can also manually delete the chassis/chassis_private registers from the SB | 13:47 |
| danfai_ | ralonsoh: ok, so this means we should add a script that not only starts ovn-controller on boot time and restarts it again? | 13:47 |
| ralonsoh | danfai_, no, what I'm saying is that if you stop an ovn-controller non gracefully (a compute node "explodes"), then you need to manually fix the DB | 13:49 |
| danfai_ | what I mean is, by default ovn-controller comes back on, but only another restart of ovn-controller fixes the tunneling of broadcast traffic | 13:49 |
| danfai_ | ralonsoh: ok, thanks, I thought the southbound would be fine by having the same rows/UUID for the chassis, but I guess the event of adding/removing needs to be triggered | 13:50 |
| ralonsoh | danfai_, if you restart the same ovn-controller but you don't remove the chassis/chassis_private registers, the controller will try to create the same | 13:51 |
| ralonsoh | this is why if you restart the compute node, it is not needed to fix the DB | 13:51 |
| ralonsoh | but if you scale down a cluster, it is needed to stop ovn-controller properly in order to let it to delete the registers | 13:51 |
| danfai_ | ralonsoh: thanks, these are some good pointers to debug/understand this further. Our main problem comes from polluting route advertisements of all VMs in the provider network when a server unexpectedly restarts. Restarting ovn-controller manually or cleaning the DBs is not an option for this since it can take a few hours until operators intervene. | 14:06 |
| ralonsoh | then you need to, somehow, to monitor the OVN agent status (OVN metadata) and have some kind of automation | 14:08 |
| luisfdez | ralonsoh: that's good to know and thanks for the information. For me to get a clear picture... are the symptoms described by danfai_ (broadcast traffic tunneled to wrong segments, etc..) the expected behaviour after a non-graceful shutdown? | 14:30 |
| luisfdez | or is the statement more like "behaviour is unpredictable if ovn-controller is not gracefully restarted". Slighlty connected to this, are you aware of any effort (RDO or similar) to provide scripts or helpers to clean things up upon abrupt restarts? | 14:30 |
| ralonsoh | luisfdez, to be honest, I have no idea of what happens when a compute is removed improperly but for sure that is something to be monitored. | 14:39 |
| ralonsoh | you can delete the agents, that will delete the OVN DB database registers | 14:39 |
| ralonsoh | no, that's not true: you need to manually delete the DB registers | 14:44 |
| ykarel | #startmeeting neutron_ci | 15:02 |
| opendevmeet | Meeting started Mon Nov 25 15:02:07 2024 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:02 |
| opendevmeet | The meeting name has been set to 'neutron_ci' | 15:02 |
| ykarel | Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira | 15:02 |
| ralonsoh | hi | 15:02 |
| ykarel | This will be video meeting this time: https://meetpad.opendev.org/neutron-ci-meetings | 15:02 |
| ykarel | #topic Actions from previous meetings | 15:04 |
| ykarel | ralonsoh to open lp for test_non_existing ft tests | 15:05 |
| ykarel | ralonsoh to check for test_update_minimum_bandwidth_queue | 15:06 |
| ykarel | #link https://bugs.launchpad.net/neutron/+bug/2089532 | 15:06 |
| ykarel | lajoskatona to send stadium fixes for test_db_base_plugin_v2 renames | 15:06 |
| ykarel | #topic Stable branches | 15:07 |
| ykarel | broken grenade with source stable/2023.1 https://launchpad.net/bugs/2089522 | 15:07 |
| ralonsoh | sorry, the first one: https://review.opendev.org/q/I3ac6364200f5124d760587612d3a9de55830f2b1 | 15:07 |
| ralonsoh | https://review.opendev.org/c/openstack/devstack/+/936126 | 15:07 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/936120 | 15:07 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/936119 | 15:08 |
| ykarel | #topic Stadium projects | 15:09 |
| ykarel | sfc, vpnaas, fwaas, bgpvpn, dynamic-routing still broken, some looks on new issues | 15:10 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2023.2: [stable-only] Remove grenade and skip-level jobs in the CI https://review.opendev.org/c/openstack/neutron/+/936119 | 15:10 |
| ykarel | #action ykarel to check/report issue for stadiums | 15:11 |
| ykarel | #topic Rechecks | 15:11 |
| ykarel | 1/35 bare recehecks | 15:12 |
| ykarel | couple of rechecks due to os-ken, docker rate limits and other known random issues | 15:12 |
| ykarel | #topic Tempest/Scenario | 15:12 |
| ykarel | Still we have some random issues with the wsgi switch, like | 15:13 |
| ykarel | - reopened https://bugs.launchpad.net/neutron/+bug/2087824 | 15:13 |
| ykarel | - https://bugs.launchpad.net/neutron/+bug/2083570 | 15:13 |
| ykarel | - https://bugs.launchpad.net/neutron/+bug/2085421 | 15:13 |
| ykarel | test_add_subport https://bugs.launchpad.net/neutron/+bug/2086602 | 15:13 |
| ykarel | https://review.opendev.org/c/openstack/neutron/+/936147 | 15:14 |
| ykarel | #topic grenade | 15:19 |
| ykarel | - https://launchpad.net/bugs/2089522 | 15:20 |
| ykarel | #topic Periodic | 15:20 |
| ykarel | - unmaintained/2023.1 jobs broken on partial transition to unmaintained | 15:20 |
| ykarel | #topic Grafana | 15:20 |
| ykarel | https://grafana.opendev.org/d/f913631585/neutron-failure-rate | 15:20 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935144 | 15:23 |
| ralonsoh | https://review.opendev.org/q/I3ac6364200f5124d760587612d3a9de55830f2b1 | 15:23 |
| ykarel | #topic On Demand | 15:23 |
| ralonsoh | https://paste.opendev.org/show/bJS1bpjw09UaoOZpyxpo/ | 15:25 |
| ykarel | #endmeeting | 15:27 |
| opendevmeet | Meeting ended Mon Nov 25 15:27:52 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:27 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/neutron_ci/2024/neutron_ci.2024-11-25-15.02.html | 15:27 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/neutron_ci/2024/neutron_ci.2024-11-25-15.02.txt | 15:27 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/neutron_ci/2024/neutron_ci.2024-11-25-15.02.log.html | 15:27 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == Add extension "uplink-status-propagation-updatable" https://review.opendev.org/c/openstack/neutron/+/931641 | 15:33 |
| *** elodilles_afk is now known as elodilles | 15:36 | |
| opendevreview | Merged openstack/neutron-tempest-plugin master: Remove transition logic for old tempest option renaming https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/936048 | 15:44 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add a detailed debug message in case of segment allocation fail https://review.opendev.org/c/openstack/neutron/+/936171 | 16:21 |
| opendevreview | Merged openstack/neutron unmaintained/2023.1: Update .gitreview for unmaintained/2023.1 https://review.opendev.org/c/openstack/neutron/+/935103 | 17:32 |
| opendevreview | Merged openstack/neutron-tempest-plugin master: Install docker-buildx for Ubuntu https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935144 | 18:18 |
| opendevreview | Amir Nikpour proposed openstack/neutron-specs master: Add spec for NFS-Ganesha extension in OVN Agent https://review.opendev.org/c/openstack/neutron-specs/+/936063 | 19:23 |
| opendevreview | Amir Nikpour proposed openstack/neutron-specs master: Add spec for NFS-Ganesha extension in OVN Agent https://review.opendev.org/c/openstack/neutron-specs/+/936063 | 19:47 |
| opendevreview | Brian Haley proposed openstack/neutron master: Optionally configure IPv6 metadata address https://review.opendev.org/c/openstack/neutron/+/926497 | 23:34 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!