| *** liuyulong has joined #openstack-neutron | 01:25 | |
| opendevreview | liuyulong proposed openstack/neutron master: Add the DHCPReponder for IPv6 https://review.opendev.org/c/openstack/neutron/+/773283 | 01:27 |
|---|---|---|
| opendevreview | liuyulong proposed openstack/neutron master: Add agent extension 'dhcp' for ovs agent https://review.opendev.org/c/openstack/neutron/+/776567 | 01:27 |
| opendevreview | liuyulong proposed openstack/neutron master: Add fullstack test case for OVS DHCP extension https://review.opendev.org/c/openstack/neutron/+/776568 | 01:27 |
| *** spatel has joined #openstack-neutron | 02:05 | |
| *** manpreetk has quit IRC | 02:08 | |
| *** spatel has quit IRC | 02:51 | |
| *** spatel has joined #openstack-neutron | 02:54 | |
| *** spatel has quit IRC | 03:22 | |
| *** gthiemon1e has joined #openstack-neutron | 04:31 | |
| *** gthiemonge has quit IRC | 04:31 | |
| *** gthiemon1e has quit IRC | 04:32 | |
| *** gthiemonge has joined #openstack-neutron | 04:32 | |
| *** manub has joined #openstack-neutron | 04:47 | |
| *** manpreetk has joined #openstack-neutron | 05:02 | |
| *** digitalsimboja has joined #openstack-neutron | 05:16 | |
| *** opendevreview has quit IRC | 05:26 | |
| *** lajoskatona has joined #openstack-neutron | 05:53 | |
| *** opendevreview has joined #openstack-neutron | 06:05 | |
| opendevreview | Merged openstack/neutron master: Fix "config-routed-networks" document errors https://review.opendev.org/c/openstack/neutron/+/795581 | 06:05 |
| *** ltomasbo has joined #openstack-neutron | 06:15 | |
| opendevreview | Slawek Kaplonski proposed openstack/neutron-lib master: Revert "Add network parameter to some of the ML2 API methods" https://review.opendev.org/c/openstack/neutron-lib/+/795617 | 06:17 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron-lib master: Revert "Add network parameter to some of the ML2 API methods" https://review.opendev.org/c/openstack/neutron-lib/+/795617 | 06:18 |
| *** jlibosva has joined #openstack-neutron | 06:51 | |
| *** ralonsoh has joined #openstack-neutron | 06:54 | |
| *** rpittau|afk is now known as rpittau | 06:54 | |
| lajoskatona | ralonsoh, obondarev: https://review.opendev.org/c/openstack/oslo.privsep/+/794993 big chunks are from ralonsoh's https://review.opendev.org/c/openstack/oslo.privsep/+/782981 | 06:59 |
| lajoskatona | ralonsoh, obondarev: I see though assertionerror (https://opendev.org/openstack/oslo.privsep/src/branch/master/oslo_privsep/comm.py#L167 ) like here: https://a0d9b83e558c2b5b4833-56ee05c3af7ad9e598ae576b02ec5a60.ssl.cf2.rackcdn.com/794993/2/check/openstack-tox-py38/aedb81c/job-output.txt | 07:01 |
| lajoskatona | ralonsoh, obondarev: I have to check that | 07:02 |
| *** lucasagomes has joined #openstack-neutron | 07:07 | |
| frickler | slaweq, others: has it ever been considered for RBAC to have an access_as_owner action or similar that would also allow actions like creating ports with allowed-address-pairs set? or is that a nonsensical idea? | 07:07 |
| slaweq | frickler it wasn't considered, but I think that You can probably play with policy rules to allow creation of ports with allowed address pairs in shared network, no? | 07:09 |
| frickler | slaweq: well I would prefer it to be able to be set by users and not require admin access. I also need it only on specific networks, can one do policy rules that have a network filter? | 07:12 |
| slaweq | nope, policy rules are for all networks, not per specific one | 07:12 |
| slaweq | there is opened BP to add access-as-readonly to the RBAC | 07:13 |
| slaweq | You can propose another RFE for access-as-owner action probably | 07:13 |
| frickler | do you have a link for that handy? otherwise I can do a search myself | 07:13 |
| frickler | https://bugs.launchpad.net/neutron/+bug/1875516 , but that looks almost abandoned :-( | 07:15 |
| opendevmeet | Launchpad bug 1875516 in neutron "[RFE] Allow sharing security groups as read-only" [Wishlist,New] | 07:15 |
| slaweq | frickler yes, that's the one | 07:16 |
| slaweq | and it's opened for long time | 07:16 |
| slaweq | :/ | 07:16 |
| frickler | so that likely means that if I propose the RFE, I should also be ready to implement it myself. guess I need to think about that a bit, thx so far | 07:18 |
| *** luksky has joined #openstack-neutron | 07:29 | |
| *** elvira has joined #openstack-neutron | 07:32 | |
| slaweq | I can try to help but only as low-prio thing for me as I have other things on my plate | 07:34 |
| ralonsoh | lajoskatona, let me check this patch | 07:38 |
| *** digitalsimboja has quit IRC | 07:38 | |
| *** isabek has joined #openstack-neutron | 07:51 | |
| *** elvira1 has joined #openstack-neutron | 07:54 | |
| *** elvira has quit IRC | 07:56 | |
| *** elvira1 is now known as elvira | 07:56 | |
| opendevreview | liuyulong proposed openstack/neutron master: Add agent extension 'dhcp' for ovs agent https://review.opendev.org/c/openstack/neutron/+/776567 | 08:04 |
| opendevreview | liuyulong proposed openstack/neutron master: Add fullstack test case for OVS DHCP extension https://review.opendev.org/c/openstack/neutron/+/776568 | 08:04 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: [Doc] Remove one of the known DVR HA limitations https://review.opendev.org/c/openstack/neutron/+/795531 | 08:05 |
| *** isabek_ has joined #openstack-neutron | 08:16 | |
| *** isabek has quit IRC | 08:17 | |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/wallaby: [OVN] Add "filter-validation" extension to supported list https://review.opendev.org/c/openstack/neutron/+/795623 | 08:28 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/wallaby: Add "filter-validation" support for "OVNL3RouterPlugin" https://review.opendev.org/c/openstack/neutron/+/795455 | 08:30 |
| *** digitalsimboja has joined #openstack-neutron | 08:38 | |
| *** digitalsimboja has quit IRC | 08:47 | |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Use "multiprocessing.Queue" for "TestNeutronServer" related tests https://review.opendev.org/c/openstack/neutron/+/793899 | 08:54 |
| *** lajoskatona has quit IRC | 08:57 | |
| *** lajoskatona has joined #openstack-neutron | 09:06 | |
| *** lajoskatona1 has joined #openstack-neutron | 09:08 | |
| *** lajoskatona has quit IRC | 09:14 | |
| *** digitalsimboja has joined #openstack-neutron | 09:23 | |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/rocky: Exclude fallback tunnel devices from netns cleanup https://review.opendev.org/c/openstack/neutron/+/795624 | 09:33 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/queens: Exclude fallback tunnel devices from netns cleanup https://review.opendev.org/c/openstack/neutron/+/795625 | 09:33 |
| *** jlibosva has quit IRC | 09:53 | |
| *** jlibosva has joined #openstack-neutron | 09:53 | |
| *** TMM has quit IRC | 10:36 | |
| *** TMM has joined #openstack-neutron | 10:36 | |
| ralonsoh | bcafarel, https://review.opendev.org/c/openstack/neutron/+/790702/2/zuul.d/tempest-multinode.yaml#152 | 10:39 |
| ralonsoh | one qq | 10:39 |
| ralonsoh | the neutron config in the subnode will be the same as in the primary one, right? | 10:39 |
| ralonsoh | except if we add something different there | 10:40 |
| ralonsoh | (btw, I think this duplication is a bad cherry-pick) | 10:40 |
| ralonsoh | it is NOT in the master branch patch | 10:41 |
| bcafarel | :) yes it probably comes from cherry-pick, though the question is still valid | 10:41 |
| bcafarel | my hunch is it should use the primary vars if there is no override, but not sure | 10:42 |
| ralonsoh | bcafarel, yeah, this parameter is the same in both nodes | 10:42 |
| ralonsoh | so I don't need to duplicate it | 10:42 |
| ralonsoh | subnode will have the same generated config file | 10:42 |
| ralonsoh | right? | 10:42 |
| bcafarel | I think so yes (and if that's not the case, we have an issue in master) | 10:43 |
| ralonsoh | perfect, I'll remove this duplication | 10:43 |
| bcafarel | hmm that though we can check in a recent master job run see if it has the param properly set! | 10:43 |
| ralonsoh | right, I'll do right now | 10:44 |
| ralonsoh | bcafarel, nope, https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_66c/795059/2/check/neutron-tempest-multinode-full-py3/66cb24e/compute1/logs/etc/neutron/neutron_conf.txt | 10:45 |
| ralonsoh | the configs are independent | 10:45 |
| ralonsoh | I need to amend the master zuul job definition | 10:46 |
| bcafarel | ralonsoh: I was almost as fast, was looking at https://7408824defcee8201456-a3def1e703d420249f59d3783ec441e9.ssl.cf5.rackcdn.com/318542/30/check/neutron-tempest-multinode-full-py3/c344d27/compute1/logs/etc/neutron/neutron_conf.txt | 10:46 |
| ralonsoh | and the cherry-pick, lucky strike, is correct | 10:46 |
| bcafarel | ralonsoh: fixing bugs without even realizing it :) | 10:47 |
| *** digitalsimboja has quit IRC | 10:47 | |
| ralonsoh | hahahahah | 10:47 |
| ralonsoh | that was a lucky strike for sure | 10:47 |
| opendevreview | Merged openstack/neutron master: Using 31-Bit and 32-Bit prefixes for IPv4 reasonably https://review.opendev.org/c/openstack/neutron/+/318542 | 10:48 |
| bcafarel | :) (and indeed neutron.conf in victoria backport is OK) | 10:49 |
| opendevreview | Merged openstack/neutron master: Added dnsmasq for opensuse installation https://review.opendev.org/c/openstack/neutron/+/795170 | 10:49 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add "nova:live_migration_events" flag to subnode in multinode CI job https://review.opendev.org/c/openstack/neutron/+/795761 | 11:15 |
| ralonsoh | ^^^ once this patch is merged, I'll push a backport for W | 11:16 |
| bcafarel | ralonsoh++ | 11:29 |
| *** hjensas|afk has quit IRC | 11:49 | |
| *** hjensas has joined #openstack-neutron | 11:49 | |
| *** whoami-rajat has quit IRC | 11:57 | |
| *** manub has quit IRC | 12:15 | |
| *** manub has joined #openstack-neutron | 12:15 | |
| *** spatel has joined #openstack-neutron | 12:19 | |
| opendevreview | sean mooney proposed openstack/os-vif master: [WIP] mock ovs.poller.Poller https://review.opendev.org/c/openstack/os-vif/+/795770 | 12:24 |
| *** yoctozepto is now known as Guest1570 | 12:36 | |
| *** Guest1570 is now known as yoctozepto | 12:36 | |
| ralonsoh | folks, just a heads-up | 12:43 |
| ralonsoh | https://review.opendev.org/c/openstack/requirements/+/788339 | 12:43 |
| ralonsoh | SQLAlchemy has been bumped to 1.4.15 | 12:44 |
| ralonsoh | just to stay alert to the CI if you see weird things related to the DB | 12:44 |
| ralonsoh | thanks! | 12:44 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add "nova:live_migration_events" flag to subnode in multinode CI job https://review.opendev.org/c/openstack/neutron/+/795761 | 12:48 |
| *** sean-k-mooney has joined #openstack-neutron | 13:01 | |
| sean-k-mooney | o/ | 13:01 |
| sean-k-mooney | ralonsoh: tl;dr regarding https://review.opendev.org/c/openstack/os-vif/+/795770 is that currently the ovsdbapp is starting an instance ot the ovs poller internallly on what it really wants to be a real thread but its monky patched. | 13:02 |
| sean-k-mooney | ralonsoh: due to how the ovs poller works it cna deadlock the nova compute agent | 13:03 |
| ralonsoh | sean-k-mooney, but how do you plan to use this connection then? | 13:03 |
| ralonsoh | I must be missing something there | 13:04 |
| sean-k-mooney | well it passes the os-vif functional tests so its actully creating port in ovs with this | 13:04 |
| sean-k-mooney | os-vif never reads form the ovs-db | 13:04 |
| sean-k-mooney | it only ever writes to it so we dont need to pool for state changes | 13:05 |
| sean-k-mooney | ovsdbapp does this for neturon to monitor ovs in the l2 agent | 13:05 |
| sean-k-mooney | that is just a temp patch to see if this will work | 13:05 |
| ralonsoh | I still don't understand | 13:05 |
| ralonsoh | this poller is needed to run commands | 13:05 |
| ralonsoh | https://github.com/openstack/ovsdbapp/blob/5d24facd36ca06b1f5f793c834da2efa4fa3a508/ovsdbapp/backend/ovs_idl/connection.py#L95-L137 | 13:05 |
| sean-k-mooney | no its not | 13:05 |
| sean-k-mooney | the pooler is not what is runnign the command | 13:06 |
| ralonsoh | ahhh no | 13:06 |
| sean-k-mooney | its the idl object that is | 13:06 |
| ralonsoh | you are right | 13:06 |
| ralonsoh | this "run" is not the command.run | 13:06 |
| sean-k-mooney | im just making the fd_wait and block noops | 13:06 |
| ralonsoh | this is just for the local cache of the DB | 13:06 |
| sean-k-mooney | yep | 13:06 |
| sean-k-mooney | so with this hack because that is what that patch is the cache will be stale | 13:06 |
| sean-k-mooney | but for os-vif that is fine since we never read anything | 13:07 |
| ralonsoh | ok ok, maybe I can propose a patch for ovsdbapp | 13:07 |
| sean-k-mooney | i was going to do that after | 13:07 |
| ralonsoh | making this poller something optional | 13:07 |
| ralonsoh | perfect! | 13:07 |
| sean-k-mooney | to add a poll=True option ot the constuctor | 13:07 |
| ralonsoh | well, not the poller but the "run" thread | 13:07 |
| sean-k-mooney | ah right | 13:07 |
| sean-k-mooney | so if this does not work my alternitve plan | 13:08 |
| sean-k-mooney | was to chagne this https://github.com/openstack/ovsdbapp/blob/5d24facd36ca06b1f5f793c834da2efa4fa3a508/ovsdbapp/backend/ovs_idl/connection.py#L91 | 13:08 |
| sean-k-mooney | to always create a real pthread | 13:08 |
| sean-k-mooney | even when its being used form a monky patched context | 13:08 |
| sean-k-mooney | currenlty its a green thread | 13:08 |
| ralonsoh | we also use greenthreads | 13:09 |
| ralonsoh | and is working | 13:09 |
| sean-k-mooney | it works in nova but ocationally it blocks | 13:09 |
| sean-k-mooney | https://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py#L59-L63 | 13:09 |
| sean-k-mooney | this i think is why ^ | 13:09 |
| ralonsoh | (well, this is also a problem in Neutron) | 13:09 |
| ralonsoh | we don't properly handle user threads | 13:09 |
| sean-k-mooney | yes it would affect neutron too | 13:09 |
| sean-k-mooney | causing the l2 agent to perodically hang | 13:10 |
| ralonsoh | this is a topic to be discussed with Terry | 13:10 |
| ralonsoh | thanks a lot | 13:10 |
| ralonsoh | because this is causing problems too | 13:10 |
| ralonsoh | in Neutron | 13:10 |
| sean-k-mooney | let me see if i can show you what it look like in the log | 13:10 |
| ralonsoh | and, btw, is it legit to mix user and OS threds | 13:10 |
| ralonsoh | there is no problem (if done correctly) | 13:10 |
| sean-k-mooney | yes if done correctly its fine and this already has all the locking required | 13:11 |
| opendevreview | Merged openstack/neutron-lib master: API "subnet-segmentid-writable" should inherit field definition https://review.opendev.org/c/openstack/neutron-lib/+/795340 | 13:11 |
| sean-k-mooney | we do it for oslo.messaging already | 13:11 |
| sean-k-mooney | the heartbeat thread is a real pthread | 13:11 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/bb6fd21b5d8c471a89f4f6598aa84e5d/log/controller/logs/screen-n-cpu.txt#12235-12262 | 13:14 |
| sean-k-mooney | there you can see it blocks the nova agent for ~5 seconds | 13:14 |
| sean-k-mooney | and it does it a few times before other things get a change to run | 13:14 |
| ralonsoh | Terry told me this is wrong | 13:15 |
| ralonsoh | "Without polling, we can't respond to the echo requests that the server sends and it can" | 13:15 |
| ralonsoh | ^^ | 13:15 |
| ralonsoh | disconnect us. | 13:15 |
| sean-k-mooney | yep i know its a hack | 13:15 |
| ralonsoh | this is why you can't do this | 13:15 |
| ralonsoh | ok ok hehehe | 13:15 |
| sean-k-mooney | we would reconect but i want to see if that log message goes away | 13:16 |
| ralonsoh | checking the logs now | 13:16 |
| sean-k-mooney | the correct way as i said i probly to put the poller in a real pthread | 13:16 |
| ralonsoh | right | 13:16 |
| *** otherwiseguy has joined #openstack-neutron | 13:16 | |
| ralonsoh | there he is | 13:17 |
| *** obondarev has joined #openstack-neutron | 13:17 | |
| sean-k-mooney | opendevreview: o/ | 13:17 |
| sean-k-mooney | ... | 13:17 |
| sean-k-mooney | otherwiseguy: o/ | 13:17 |
| otherwiseguy | sean-k-mooney: \o | 13:17 |
| ralonsoh | otherwiseguy, sean-k-mooney is talking about making this | 13:17 |
| ralonsoh | https://github.com/openstack/ovsdbapp/blob/5d24facd36ca06b1f5f793c834da2efa4fa3a508/ovsdbapp/backend/ovs_idl/connection.py#L91 | 13:17 |
| ralonsoh | a real thread | 13:17 |
| sean-k-mooney | to workaround https://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py#L59-L63 | 13:18 |
| ralonsoh | that could solve some problems we have with greenthreads too | 13:18 |
| otherwiseguy | ralonsoh: is it not as real a thread as Python can provide? | 13:18 |
| sean-k-mooney | otherwiseguy: no | 13:18 |
| ralonsoh | no if this is called from a monkeypatch program | 13:18 |
| sean-k-mooney | otherwiseguy: eventlet has monky patched it | 13:18 |
| ralonsoh | right | 13:18 |
| sean-k-mooney | otherwiseguy: it still wont escape the gil | 13:18 |
| otherwiseguy | but ovsdbapp doesn't monkeypatch it. | 13:19 |
| ralonsoh | Neutron does | 13:19 |
| sean-k-mooney | nova and neutorn does | 13:19 |
| ralonsoh | and neutron calls ovsdbapp | 13:19 |
| otherwiseguy | right, but that is a choice of the user of the library is all I'm saying. :) | 13:19 |
| sean-k-mooney | in this case nova mokeypatches the world, it call os-vif, os-vif then calls ovsdbapp | 13:19 |
| sean-k-mooney | otherwiseguy: ovs un monkeypatches https://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py#L270-L290 | 13:20 |
| ralonsoh | otherwiseguy, yeah... you are right, but we must be aware of this | 13:20 |
| ralonsoh | https://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py#L59-L63 | 13:20 |
| ralonsoh | so I think this is a good idea that ovsdbapp offers this option | 13:20 |
| ralonsoh | to use a OS thread there, instead of eventlet | 13:21 |
| sean-k-mooney | you really only see this issue in the ci because some fo the nodes are slow | 13:22 |
| *** elvira has quit IRC | 13:22 | |
| sean-k-mooney | but i suspect it woudl also happen when a node is recaching capasity | 13:22 |
| ralonsoh | or any race condition in the code not returning the GIL | 13:23 |
| sean-k-mooney | ill submit a second patch to ovsdbapp to ty and moge that to a real pthread and we can see if that helps | 13:24 |
| ralonsoh | cool for me! | 13:24 |
| sean-k-mooney | we can see what the ci says in terem of it if break anything | 13:24 |
| otherwiseguy | sean-k-mooney: When you say OS thread, are you talking about some kind of C-extension thing so we can actually use multiple cores? I'm not following how there is going to be a performance benefit since python threads suck. | 13:25 |
| sean-k-mooney | otherwiseguy: we dont care about performace we just dont want the pooling to deadlock the eventlet event loop | 13:26 |
| otherwiseguy | gotcha | 13:26 |
| sean-k-mooney | right now when the poller blocks like here https://github.com/openstack/ovsdbapp/blob/5d24facd36ca06b1f5f793c834da2efa4fa3a508/ovsdbapp/backend/ovs_idl/connection.py#L106 | 13:26 |
| sean-k-mooney | the entire agent blocks until the timeout expires or the file desciptor gets a kick form ovs | 13:27 |
| sean-k-mooney | im totally fine with it doing that as long as tha tis not the tread on which the eventlet event loop is running on | 13:27 |
| otherwiseguy | sean-k-mooney: ralonsoh: but it looks like we are using the _SelectSelect class and not select.poll? https://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py#L153. The only place I see that would actually use the system poll is in check_connection_completion() which is only used when open/connect on a stream is called. Am I misreading that? | 13:34 |
| ralonsoh | otherwiseguy, the problem is the thread using the poller | 13:34 |
| ralonsoh | _SelectSelect | 13:35 |
| ralonsoh | sorry | 13:35 |
| ralonsoh | https://github.com/openstack/ovsdbapp/blob/5d24facd36ca06b1f5f793c834da2efa4fa3a508/ovsdbapp/backend/ovs_idl/connection.py#L91 | 13:35 |
| ralonsoh | this thread that executes "run" is a greenthread | 13:35 |
| ralonsoh | (as commented because Neutron and Nova are monkey patched) | 13:35 |
| ralonsoh | that should be a OS thread (preemptive thread, the context is given by the OS, not the user in the code, stopping the running eventlet thread) | 13:36 |
| otherwiseguy | right, and I thought greenthreads should be fine with _SelectSelect because it doesn't use poll and was made to work with greenthreads? | 13:36 |
| * otherwiseguy tries to catch up | 13:36 | |
| otherwiseguy | The comment makes it sound like that is literally its reason for existing. | 13:37 |
| opendevreview | Merged openstack/neutron stable/train: Enable ovsdb debug messages in functional and fullstack https://review.opendev.org/c/openstack/neutron/+/794625 | 13:38 |
| sean-k-mooney | otherwiseguy: i think they had inteded it to work that way but as you can see form the nova logs https://zuul.opendev.org/t/openstack/build/bb6fd21b5d8c471a89f4f6598aa84e5d/log/controller/logs/screen-n-cpu.txt#12235-12262 | 13:39 |
| *** digitalsimboja has joined #openstack-neutron | 13:39 | |
| sean-k-mooney | there are direct 5 second gaps in output while we wait for the time out to expire | 13:39 |
| sean-k-mooney | no other processing seams to be happening on the agent | 13:39 |
| sean-k-mooney | this results in some rpc calls timing out during the live migration tests | 13:40 |
| sean-k-mooney | most of the time this pooling returns instantly so its not an issue | 13:40 |
| sean-k-mooney | these long pause seam to also be assocaited with reconnects | 13:42 |
| otherwiseguy | sean-k-mooney: weird, because I think each of those state changes requires a call to idl.run() to happen, which only happens if we pass a block() call. And nothing happens in between them. | 13:42 |
| otherwiseguy | Yeah, reconnects are definitely super expensive. If there was an active transaction, we'd be waiting on on it to complete (we queue txns and block popping the results with commit_block()) | 13:44 |
| ralonsoh | but there we have the problem, in the block() call | 13:44 |
| ralonsoh | https://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py#L137-L140 | 13:44 |
| ralonsoh | that will wait | 13:44 |
| ralonsoh | at this point, if this is a user thread, the GIL will be given to the next one | 13:45 |
| ralonsoh | if the next one never returns, this wait method will never continue | 13:45 |
| sean-k-mooney | well in theory that select should be monkeypatched and yeild | 13:46 |
| sean-k-mooney | but it does not look like that is what is happening here | 13:47 |
| sean-k-mooney | ralonsoh: otherwiseguy lee just noted that his might be the same issue https://bugs.launchpad.net/nova/+bug/1863889 | 13:47 |
| opendevmeet | Launchpad bug 1863889 in OpenStack Compute (nova) "Revert resize problem in neutron-tempest-dvr-ha-multinode-full" [Medium,Confirmed] | 13:47 |
| *** pjakuszew has joined #openstack-neutron | 13:54 | |
| opendevreview | Bodo Petermann proposed openstack/neutron master: Config option to enable OVN IDL on other workers https://review.opendev.org/c/openstack/neutron/+/795781 | 13:55 |
| *** elvira has joined #openstack-neutron | 13:56 | |
| *** digitalsimboja has quit IRC | 13:56 | |
| sean-k-mooney | otherwiseguy: am i correct in saying ovsdbapp does not use oslo.utils? | 13:57 |
| sean-k-mooney | ya no oslo libs in requirements | 13:58 |
| sean-k-mooney | ok ill just port the relevent logic | 13:58 |
| otherwiseguy | sean-k-mooney: correct. we try to avoid any openstack requirements since there are non-openstack users. (except we use oslo testing stuff) | 13:59 |
| otherwiseguy | I'm still going to be shocked if that select() call is blocking. | 13:59 |
| otherwiseguy | :) | 13:59 |
| sean-k-mooney | otherwiseguy: ok by the way we need to fix the packaging of the ovs python binding in centos/rhel | 13:59 |
| sean-k-mooney | otherwiseguy: currently ovsdbapp depend on the ovs python bidnings which depend on libopenvswitch | 14:00 |
| otherwiseguy | sean-k-mooney: yeah, the C extension for not-stupidly-slow json parsing. | 14:00 |
| sean-k-mooney | which is fine if it was not for the fact that in cenots libopenvswich is part of the main openvswithc package so we end up pulling in all of openvswitch and dpdk | 14:00 |
| otherwiseguy | (it's literally 60x slower) | 14:01 |
| sean-k-mooney | right but my point is libopenvswtich shoudl be its own package | 14:01 |
| sean-k-mooney | since that is all the python bindings needs | 14:01 |
| sean-k-mooney | its one of the reason our continer image are so big | 14:02 |
| opendevreview | Mamatisa Nurmatov proposed openstack/neutron master: use payloads for PORT AFTER_UPDATE events https://review.opendev.org/c/openstack/neutron/+/795117 | 14:02 |
| *** pjakuszew has quit IRC | 14:02 | |
| sean-k-mooney | installing nova-api on centos pulls in all of ovs and dpdk even though it does not use os-vif but that is in the requirement for python-nova, and since os-vif pulls in ovsdbapp ... | 14:03 |
| *** pjakuszew has joined #openstack-neutron | 14:04 | |
| *** pjakuszew has quit IRC | 14:05 | |
| *** pjakuszew has joined #openstack-neutron | 14:05 | |
| otherwiseguy | sean-k-mooney: oh my. yeah, I assumed it was just requiring openvswitch-devel (which has the libs/headers). Requiring openvswitch itself is silly. | 14:08 |
| sean-k-mooney | well it required libopenvswitch but its provided by the main openvswitch package | 14:09 |
| *** pjakuszew is now known as cz3|work | 14:09 | |
| *** cz3|work is now known as pjakuszew | 14:10 | |
| sean-k-mooney | the only thing that really hurts is container size | 14:10 |
| *** pjakuszew has left #openstack-neutron | 14:10 | |
| sean-k-mooney | but its still not ideal | 14:10 |
| otherwiseguy | in rhel openvswitch2.13 it looks like python3-openvswitch directly requires openvswitch: Requires: %{pkgname} = %{?epoch:%{epoch}:}%{version}-%{release} | 14:11 |
| sean-k-mooney | ew | 14:11 |
| otherwiseguy | sean-k-mooney: so changes everywhere :D | 14:11 |
| *** cz3|work has joined #openstack-neutron | 14:17 | |
| opendevreview | sean mooney proposed openstack/ovsdbapp master: always use real threads when polling ovs. https://review.opendev.org/c/openstack/ovsdbapp/+/795789 | 14:17 |
| sean-k-mooney | ok cool there is a tempest job on ovsdbapp ovsdbapp-neutron-ovn-tempest-ovs-release | 14:18 |
| otherwiseguy | sean-k-mooney: if we un-patch threading, then all of the locking we do will block eventlet, right? | 14:19 |
| sean-k-mooney | it would yes but tha that is inside the run method that would be on the new thread i think | 14:20 |
| sean-k-mooney | ah we lock in start and stop | 14:20 |
| otherwiseguy | We basically lock around all run() calls. | 14:20 |
| sean-k-mooney | well run should always be called in a sperate trhead | 14:21 |
| sean-k-mooney | i could revert the loc to use the patched version but i dont think that woudl be correct | 14:21 |
| sean-k-mooney | well | 14:22 |
| sean-k-mooney | see its tricky | 14:22 |
| sean-k-mooney | when monkeypatched we want it to yeild when it trys to take the lock and cang with out yeilding the thread | 14:22 |
| sean-k-mooney | just the greentread | 14:23 |
| sean-k-mooney | maybe we should use a filesystem lock | 14:23 |
| otherwiseguy | the fact that nothing happens even between those multiple state changes/run calls, makes me really wonder about what the problem is. | 14:23 |
| otherwiseguy | The problem happening during reconnects makes sense, since that is the only time that it uses system poll. | 14:26 |
| sean-k-mooney | well mocking the poller had the opicite effect i wanted but it is what you were expecting | 14:27 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/39c7eed869ce4035be359cb1f7a5b8f6/log/controller/logs/screen-n-cpu.txt#7212 | 14:27 |
| otherwiseguy | For the normal poller we use in ovsdbapp, we have the safe _SelectSelect. | 14:27 |
| sean-k-mooney | its cause it to have to reconnect every 5 seconds | 14:27 |
| otherwiseguy | But in python-ovs, it uses get_system_poll() in the connection open/connect calls. | 14:27 |
| otherwiseguy | Yeah, mocking poller would break everything. We'd not respond to the echo requests from the server and it would disconnect us. | 14:28 |
| otherwiseguy | Having python-ovs's stream.py not use the system poller seems like it would be the fix for the issue at reconnect. | 14:28 |
| sean-k-mooney | yes perhaps | 14:29 |
| sean-k-mooney | althoguh it raises the question as to why it need to reconnect | 14:29 |
| otherwiseguy | timeout too low, ovsdb-server is singlethreaded and dumb. probe intervals default to 5s which is too small. I have a patch that removes probe intervals in favor of tcp keepalives for the *client generated* echo requests. | 14:30 |
| otherwiseguy | sean-k-mooney: https://review.opendev.org/c/openstack/neutron/+/794892 | 14:30 |
| otherwiseguy | but there are also the server generated requests, which are configured in NB/SB Connection table as inactivity_probe | 14:31 |
| sean-k-mooney | ah i see we could port that to os-vif i guess but could we do this in ovsdbapp instead | 14:31 |
| otherwiseguy | My thinking being that tcp keepalives at least won't have ovsdb-server as a bottleneck. | 14:32 |
| otherwiseguy | sean-k-mooney: the biggest issue is that ovsdb-server was designed around the OVS switch usecase, which means small dbs and few connections. OVN on the other hand, large DBs with 100s of connections...a single threaded ovsdb server is ... not ideal. | 14:33 |
| otherwiseguy | Ideally that neutron patch would be in python-ovs and configurable. I didn't do it in ovsdbapp because I try to behave like a normal library and not change behavior in the library when I don't have to, but if it seems after running this for a while that it is definitely far superior, I'd be open to forcing it on people. ovsdbapp doesn't have config files, but I could make it a non-default option in releases. | 14:37 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [DNM] Test ovsdbapp patch https://review.opendev.org/c/openstack/neutron/+/795820 | 14:39 |
| sean-k-mooney | otherwiseguy: well even if we could pass in a flag to the constotor of the connection for example to alther the behaivor we could opt into it | 14:44 |
| sean-k-mooney | otherwiseguy: for now im not against porting the change you did in neutron to os-vif but i need to digest what you did first | 14:45 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Bump os-ken to 2.0.0 https://review.opendev.org/c/openstack/neutron/+/793735 | 14:46 |
| sean-k-mooney | otherwiseguy: am i correct in saying you are usign the no probes mixin to disable the echo and redefining the tcp and ssl stream calles by inheritng form the mixin and registering the new classes to be used by ovs stream module instead | 14:48 |
| otherwiseguy | sean-k-mooney: basically a) set SO_KEEPALIVE to enable tcp keepalives (relying on system sysctl settings for the TCP_KEEPIDLE/KEEP_CNT/KEEPINTVL which in OSP we default to 5/1/1, but is 2 hours default in linux) b) disable sending probes via echo request (from the client) | 14:48 |
| otherwiseguy | sean-k-mooney: Correct. | 14:48 |
| sean-k-mooney | ya ok we could try that in os-vif also although i am not sure that will help with the reconnect issue | 14:49 |
| otherwiseguy | It might be a good idea to override with configurable TCP_KEEPIDLE, TCP_KEEPCNT, TCP_KEEPINTVL. | 14:49 |
| sean-k-mooney | os-vif does have config options so we can at least make some of this configurable if we need too | 14:50 |
| otherwiseguy | sean-k-mooney: it *could* help, if it's the client-side echo request at 5s default is what is causing it to disconnect and reconnect. | 14:50 |
| otherwiseguy | It could also be the server-side echo requests, but neutron has those configured to much longer times than 5s. | 14:51 |
| sean-k-mooney | well its at least worth a try | 14:51 |
| sean-k-mooney | do you mind if i port it or would you like to do it | 14:52 |
| otherwiseguy | Since we're seeing 5s timeouts, I'd put my money on it. With ovsdbapp you can also when you create the Idl, you can pass probe_interval={milliseconds} to increase the probe interval as well. | 14:53 |
| otherwiseguy | s/ovsdbapp/ovs/ | 14:55 |
| otherwiseguy | Here is how neutron does it: https://github.com/openstack/neutron/blob/a0cd7e76390c1f35aff5f95612cc0b7bcf0eaa9e/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py#L481 | 14:55 |
| sean-k-mooney | ok | 14:55 |
| sean-k-mooney | so the thing is though in os-vif we dont ever need to monitor the ovsdb | 14:56 |
| sean-k-mooney | i was going to say we never read from the ovsdb and that is almost true | 14:56 |
| otherwiseguy | sean-k-mooney: Feel free to port it. I'm happy to do it if you need me to as well. Yeah, the python-ovs Idl, last I checked, didn't support a write-only method, though the C idl does, I think. | 14:57 |
| sean-k-mooney | i belive we have one check to see if the ovsdb support setting the mtu on a port | 14:57 |
| sean-k-mooney | the main reason we keep the connection open today id to not need to generate teh client and do the schema verions stuff evey time we add or remvoe a port | 14:58 |
| otherwiseguy | But the way OVS IDL stuff works mostly assumes that you have stuff in memory. The Command objects that ship with ovsdbapp do lookups by name, which would require in-memory copies of the rows, etc. So even if you think you aren't reading, you are. | 14:58 |
| *** manub has quit IRC | 15:01 | |
| otherwiseguy | sean-k-mooney: In general, the IDL method of doing things kind of crappy (but better than the old way of calling the -ctl commands, which basically create a new IDL for every call). Maintaining an in-memory copy of the DB rows you need *per connection/worker* is gross. You can at least limit the tables/rows to only the ones you care about at least with register_table: https://github.com/openstack/neutron/blob/a0cd7e76390c1f35aff5f95612c | 15:04 |
| otherwiseguy | c0b7bcf0eaa9e/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py#L505 and cond_change(): https://github.com/openstack/neutron/blob/a0cd7e76390c1f35aff5f95612cc0b7bcf0eaa9e/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py#L711 | 15:04 |
| otherwiseguy | which can also cut down on how long reconnects take. | 15:05 |
| sean-k-mooney | i see we porably shoudl do that so | 15:07 |
| sean-k-mooney | i think we just need either the interface table or ports table | 15:08 |
| *** lucasagomes has quit IRC | 15:08 | |
| sean-k-mooney | i think ports have interfaces i always mix those up | 15:08 |
| *** liuyulong has quit IRC | 15:12 | |
| *** isabek_ has quit IRC | 15:12 | |
| *** liuyulong has joined #openstack-neutron | 15:12 | |
| opendevreview | Merged openstack/neutron-lib master: Revert "Add network parameter to some of the ML2 API methods" https://review.opendev.org/c/openstack/neutron-lib/+/795617 | 15:15 |
| otherwiseguy | sean-k-mooney: yeah, you just need to make sure that if you grab a table that has a refTable column that references a different table, to get that table too (at least if you ever reference that column--it *should* just display the uuid instead of a row when accessing the column) | 15:15 |
| opendevreview | Kamil Sambor proposed openstack/neutron master: Set trunk sub-port when bind profile is created https://review.opendev.org/c/openstack/neutron/+/795334 | 15:18 |
| *** tkajinam has quit IRC | 15:19 | |
| otherwiseguy | sean-k-mooney: yeah, a quick fix would be https://github.com/openstack/os-vif/blob/d8af3568b8b92748f61029a96c46fd513b6795c2/vif_plug_ovs/ovsdb/impl_idl.py#L26 to remove the register_all() and do register_table() for what you need, and then in the next line pass probe_interval=60000 to idl.Idl and see if it helps. | 15:22 |
| sean-k-mooney | ok ill work that into may patches to os-vif | 15:23 |
| sean-k-mooney | im on an internal team call but ill try an poc this later today | 15:24 |
| otherwiseguy | I'd bet it'll prevent some reconnects, though of course fixing the blocking during a reconnect would be the python-ovs socket_util.get_system_poll() call. | 15:24 |
| otherwiseguy | Great! Let me know if there's anything you need. | 15:25 |
| sean-k-mooney | are you going to follow up with ovs on the python-ovs change | 15:26 |
| sean-k-mooney | one thing i coudl do in os-vif is monkey patch socket_util.get_system_poll but thats not great | 15:26 |
| sean-k-mooney | e.g. make it just return the safe select implemenation | 15:27 |
| sean-k-mooney | i have a number of ways to proceed in anycase | 15:27 |
| *** digitalsimboja has joined #openstack-neutron | 15:28 | |
| otherwiseguy | sean-k-mooney: yeah, it's something I should at least raise on the mailing list. | 15:36 |
| otherwiseguy | I was happy that they basically had a method to monkey-patch the TCP/SSLStream classes, so I didn't feel *quite* as bad. | 15:37 |
| otherwiseguy | "It's not my fault that you had a global dict and then a method that wrote to it that you didn't prefix with an underscore." | 15:37 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: DVR: Populate ARP entries of the allowed_address_pairs to the routers https://review.opendev.org/c/openstack/neutron/+/601336 | 15:38 |
| *** digitalsimboja has quit IRC | 15:46 | |
| *** obondarev has quit IRC | 15:57 | |
| *** elvira has quit IRC | 15:57 | |
| *** ltomasbo has quit IRC | 16:01 | |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Populate self.floating_ips_dict using "ip rule" information https://review.opendev.org/c/openstack/neutron/+/794604 | 16:11 |
| *** rpittau is now known as rpittau|afk | 16:24 | |
| opendevreview | Merged openstack/neutron master: ovn: Don't use dict.remove() for filtering dhcp ports in db-sync https://review.opendev.org/c/openstack/neutron/+/789085 | 16:29 |
| opendevreview | Merged openstack/neutron master: Payloads for PORT: BEFORE_CREATE and PRECOMMIT_CREATE https://review.opendev.org/c/openstack/neutron/+/794579 | 16:31 |
| opendevreview | Bodo Petermann proposed openstack/neutron master: Config option to enable OVN IDL on other workers https://review.opendev.org/c/openstack/neutron/+/795781 | 16:41 |
| opendevreview | sean mooney proposed openstack/ovsdbapp master: always use real threads when polling ovs. https://review.opendev.org/c/openstack/ovsdbapp/+/795789 | 16:47 |
| opendevreview | Merged openstack/networking-ovn stable/train: Make networking-ovn-tripleo-ci-centos-7-containers-multinode non-voting https://review.opendev.org/c/openstack/networking-ovn/+/795500 | 16:51 |
| *** jlibosva has quit IRC | 16:52 | |
| *** raildo has joined #openstack-neutron | 17:18 | |
| *** sean-k-mooney has quit IRC | 17:41 | |
| *** cz3|work is now known as pjakuszew | 18:08 | |
| opendevreview | Merged openstack/neutron master: [DHCP] Fix cleanup_deleted_ports method https://review.opendev.org/c/openstack/neutron/+/794755 | 18:13 |
| opendevreview | Merged openstack/neutron master: Remove "mitogen" library installation from "docs" job https://review.opendev.org/c/openstack/neutron/+/793377 | 18:13 |
| *** digitalsimboja has joined #openstack-neutron | 18:24 | |
| *** digitalsimboja has quit IRC | 18:52 | |
| *** sean-k-mooney has joined #openstack-neutron | 19:04 | |
| sean-k-mooney | otherwiseguy: ralonsoh its just one ci run but with https://review.opendev.org/c/openstack/ovsdbapp/+/795789 there are not reconnects | 19:05 |
| sean-k-mooney | otherwiseguy: ralonsoh but the next most recent ci run for ovsdbapp has reconnects | 19:07 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/9b751acd11664482a681705fa2c7047c/log/controller/logs/screen-n-cpu.txt | 19:07 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/9b751acd11664482a681705fa2c7047c/log/controller/logs/screen-n-cpu.txt#7252 | 19:07 |
| sean-k-mooney | otherwiseguy: ralonsoh so https://review.opendev.org/c/openstack/ovsdbapp/+/795789 does seam to fix that issue | 19:08 |
| sean-k-mooney | now that i have udpated it to also cater for the fact that neutron-api runs under wsgi and while eventlest is avaiable it not patchs | 19:09 |
| sean-k-mooney | which is why it failed in the first version | 19:09 |
| sean-k-mooney | im still going to explore the os-vif optimisations but this seams to prevent the reconnects | 19:09 |
| *** digitalsimboja has joined #openstack-neutron | 19:10 | |
| sean-k-mooney | i would guess previously te echos were not replied to in time because nova was bussy doing things and eventlet and the ovs poller was sharing a single os thread now they are not | 19:11 |
| *** digitalsimboja has quit IRC | 19:38 | |
| *** supamatt has joined #openstack-neutron | 20:02 | |
| *** spatel has quit IRC | 20:06 | |
| *** lajoskatona1 has quit IRC | 20:07 | |
| *** TMM has quit IRC | 20:25 | |
| *** TMM has joined #openstack-neutron | 20:25 | |
| *** spatel has joined #openstack-neutron | 20:35 | |
| opendevreview | Brian Haley proposed openstack/ovn-octavia-provider master: Support creating members without a subnet ID https://review.opendev.org/c/openstack/ovn-octavia-provider/+/795896 | 20:45 |
| *** spatel has quit IRC | 20:52 | |
| *** ralonsoh has quit IRC | 20:56 | |
| opendevreview | Brian Haley proposed openstack/ovn-octavia-provider master: Add Health Monitor support https://review.opendev.org/c/openstack/ovn-octavia-provider/+/713253 | 21:35 |
| opendevreview | Brian Haley proposed openstack/ovn-octavia-provider master: Support creating members without a subnet ID https://review.opendev.org/c/openstack/ovn-octavia-provider/+/795896 | 21:45 |
| *** spatel has joined #openstack-neutron | 21:47 | |
| opendevreview | Merged openstack/neutron master: Mark fullstack test_l2_agent_restart as unstable https://review.opendev.org/c/openstack/neutron/+/794228 | 21:53 |
| *** spatel has quit IRC | 22:06 | |
| *** yamamoto has quit IRC | 22:22 | |
| *** yamamoto has joined #openstack-neutron | 22:33 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!