| hberaud[m] | @itamarst ongoing release https://github.com/eventlet/eventlet/pull/1008 | 08:57 |
|---|---|---|
| ralonsoh | hi hberaud[m], I've identified the issue with RPC (oslo.messaging), Neutron and eventlet | 09:45 |
| ralonsoh | this is the last patch pushed: https://github.com/eventlet/eventlet/commit/06ec82896ebb9a26edaf6e1ad4d63393990f15b7#diff-029df1ae9b7431e9cdd6fa95ae7a2eed832727e6054f7f4b050f3d2c754e8816R367 | 09:45 |
| ralonsoh | and makes sense because the error we are receiving is related to | 09:45 |
| ralonsoh | self._cond = threading.Condition() | 09:46 |
| ralonsoh | https://paste.opendev.org/show/bwb1L0WAhKAODIWyiep1/ | 09:46 |
| hberaud[m] | I'd argue that normally it should be fixed by https://github.com/eventlet/eventlet/pull/1002 | 09:46 |
| hberaud[m] | sorry wrong link | 09:46 |
| hberaud[m] | https://github.com/eventlet/eventlet/issues/1001 | 09:47 |
| hberaud[m] | which python version are you using? | 09:48 |
| ralonsoh | 3.10 | 09:48 |
| ralonsoh | So the last patch in the repo should fix that? | 09:49 |
| ralonsoh | I can try it | 09:49 |
| hberaud[m] | ah if you are using python 3.10 then that could be something else | 09:51 |
| hberaud[m] | but, yeah, please try it with the latest version, just to be sure | 09:52 |
| hberaud[m] | there is also another ongoing release | 09:52 |
| ralonsoh | I'm still seeing this error with eventlet master | 09:58 |
| hberaud[m] | ack | 09:59 |
| ralonsoh | the call "_green_existing_locks(original_rlock_type)" is breaking somehow the existing rlocks | 09:59 |
| hberaud[m] | ralonsoh: please can you fill a github issue with your observations, and, if possible a reproducer? | 09:59 |
| ralonsoh | but the rlock is created before, when "self._cond = threading.Condition()" is called | 10:00 |
| ralonsoh | hberaud[m], right now | 10:00 |
| hberaud[m] | thanks | 10:00 |
| ralonsoh | hberaud[m], ohhh my, I think this is **our** error (Neutron error) | 10:05 |
| ralonsoh | we are multi monkey_patching!! | 10:05 |
| hberaud[m] | indeed, could be the problem | 10:05 |
| ralonsoh | if I add a condition to check if monkey_patched has been called, everything works | 10:05 |
| ralonsoh | pfffff | 10:05 |
| ralonsoh | sorry for the noise! | 10:05 |
| hberaud[m] | no problem | 10:05 |
| ralonsoh | (1 day because of this) | 10:05 |
| hberaud[m] | ralonsoh: but | 10:05 |
| hberaud[m] | you could interested by https://github.com/eventlet/eventlet/pull/1005 | 10:06 |
| hberaud[m] | (not yet released) | 10:06 |
| ralonsoh | ahhhhh exactly this! | 10:07 |
| ralonsoh | yes | 10:07 |
| ralonsoh | we need (1) monkey patch ASAP and (2) do it only once | 10:07 |
| hberaud[m] | Eventlet is like an escape game... | 10:09 |
| frickler | so running "EVENTLET_MONKEYPATCH=1 tox -e py3" seems to explode pretty early for multiple repos where I tried that (with 0.39.0) | 12:09 |
| frickler | I'm also seeing some weird issue with heat unit tests, with eventlet >= 0.37.0 some of the individual tests take about 3s each instead of like 30ms, causing job timeouts. the tests are still passing though | 12:11 |
| frickler | if you want to reproduce, take a subset like "tox -e py3 heat.tests.clients.test_clients", that gives me 20s vs. 5m | 12:12 |
| frickler | (also please complain if eventlet-bumping is considered offtopic here, as that's not actually removing it) | 12:13 |
| hberaud[m] | frickler: thanks for the heads up | 12:35 |
| hberaud[m] | frickler: no problem, having this kind of discussion help us to triagge things and to observe problem as soon as possible | 12:37 |
| hberaud[m] | s/triage/ | 12:38 |
| amorin | one question about EVENTLET_MONKEYPATCH=1. If we do that, do we need to remove all eventlet.monkey_patch() from code? I am asking because of ralonsoh message stating that we should do it only once? | 13:52 |
| ralonsoh | amorin, but there will be some kind of variable/method to know that | 13:54 |
| ralonsoh | I mean, to know if eventlet did the monkey_patch by its own | 13:54 |
| amorin | from your patch here: https://review.opendev.org/c/openstack/neutron/+/937527 | 13:56 |
| amorin | if we do EVENTLET_MONKEYPATCH=1 tox -e py3 | 13:56 |
| amorin | it will monkey_patch twice, isn't it? | 13:56 |
| ralonsoh | yes, and without this patch it will call monkey_patch 3 times | 13:57 |
| ralonsoh | so this method in Neutron should also consider the env variable EVENTLET_MONKEYPATCH | 13:57 |
| ralonsoh | is that correct? | 13:57 |
| amorin | ok, that's what I understand also, correct for me | 13:58 |
| ralonsoh | perfect, I'll propose another patch saying that this will required with eventlet 0.39.0 | 13:58 |
| itamarst | Does OpenStack still support Python 3.8? | 15:32 |
| itamarst | wondering if we can drop support from eventlet or if I should try to make it work | 15:32 |
| hberaud[m] | itamarst: https://governance.openstack.org/tc/reference/runtimes/2025.1.html | 15:53 |
| hberaud[m] | technically, it is not anymore supported | 15:53 |
| hberaud[m] | but I suppose that lot of deliverables still test it in some ways | 15:54 |
| hberaud[m] | (openstack deliverables) | 15:54 |
| hberaud[m] | amorin: concerning the EVENTLET_MONKEYPATCH=1, IIUC it will avoid to monkey patche our projects here and there thousand of times. itamarst correct me if I'm wrong | 15:56 |
| itamarst | ok, I'm working on fix | 16:00 |
| itamarst | re EVENTLET_MONKEYPATCH, the issue is you want to monkey patch before _anything_ gets imported | 16:00 |
| itamarst | and this can be very difficult | 16:00 |
| itamarst | e.g. if you're running tests, the stdlib unittest module may import asyncio, which we don't want | 16:01 |
| itamarst | so this way the patching happens even before e.g. a test runner starts up | 16:01 |
| itamarst | it's probably also helpful for applications, yes, although it's a little trickier there because if you don't set that env variable your application is now broken | 16:02 |
| itamarst | and also maybe less necessary because you have control of the application (vs e.g. pytest or stestr or whatever where you don't) | 16:02 |
| hberaud[m] | ralonsoh: FYI ^ | 16:03 |
| ralonsoh | hberaud[m], I've proposed a patch | 16:03 |
| amorin | I did try to use EVENTLET_MONKEYPATCH and removed all call of monkey_patch from mistral code before running py3. I am ending in some sort of infinite loop | 16:03 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/937535 | 16:03 |
| amorin | super(SSLContext, SSLContext).verify_mode.__set__(self, value) | 16:03 |
| amorin | [Previous line repeated 481 more times] | 16:03 |
| amorin | have you hit that somewhere maybe on neutron side? | 16:04 |
| amorin | (I am using python3.10) | 16:04 |
| ralonsoh | amorin, what library is that? | 16:04 |
| ralonsoh | ok, yes but I dont' remember now where... | 16:04 |
| amorin | here is the full stacktrace: https://plik.ovh/file/W6OJ1yYpFgcEfrnY/YU1Bz0piLyXjFghx/t | 16:06 |
| ralonsoh | yes yes one sec | 16:06 |
| ralonsoh | amorin, https://review.opendev.org/c/openstack/neutron/+/925376 | 16:07 |
| ralonsoh | https://bugs.launchpad.net/neutron/+bug/2075147 | 16:07 |
| ralonsoh | this is your problem | 16:07 |
| ralonsoh | I need to go now, please ping me tomorrow if you still have problems | 16:07 |
| amorin | thanks! | 16:08 |
| itamarst | ah so | 16:08 |
| itamarst | I guess that's a problem with EVENTLET_MONKEYPATCH which is maybe worth documenting | 16:08 |
| itamarst | it will patch _all_ Python programs | 16:08 |
| itamarst | or it could be solved by removing the env variable on startup, which is maybe a better idea | 16:09 |
| itamarst | no that wouldn't solve it | 16:09 |
| itamarst | so yeah maybe better just to use it for testing | 16:10 |
| itamarst | and not recommend people use it | 16:10 |
| itamarst | hberaud[m]: gonna file an issue for that I guess :( | 16:10 |
| itamarst | I really hate eventlet :( | 16:10 |
| hberaud[m] | yeah eventlet is a nightmare | 16:11 |
| hberaud[m] | ack | 16:11 |
| hberaud[m] | thanks for your clarifications | 16:11 |
| itamarst | I filed an issue and will update docs etc | 16:13 |
| itamarst | 😢 | 16:13 |
| hberaud[m] | thanks | 16:26 |
| JayF | there's something mildly humorous about the fact that eventlet is the best maintained it's been in years ... because we're trying to get rid of it | 16:36 |
| itamarst | it's possible EVENTLET_MONKEYPATCH isn't actually what I need for oslo.log either | 16:43 |
| itamarst | and that I misdiagnosed problem, and you need to just create the hub early, rather than monkey patch early | 16:47 |
| frickler | amorin: are you running tox with that env var set or what command produces your stacktrace? | 16:58 |
| itamarst | note that EVENTLET_MONKEYPATCH is not in any released version and might get ripped out depending how my experiments go | 17:04 |
| itamarst | yep, gonna rip it out | 17:24 |
| itamarst | this whole thing is soooo brittle | 18:09 |
| JayF | I'm honestly more than a little excited about the state of things once we've excised eventlet from everything. Wondering how many little random breakages/edge cases are going to melt away. | 18:14 |
| itamarst | ono | 18:19 |
| itamarst | asyncio hub's thing means there might be _two_ original unpatched threading modules in existence 😭 | 18:20 |
| itamarst | so maybe I'm introducing subtle bizarre bugs | 18:20 |
| itamarst | I am also actively trying ton ot think about all the new brokenness I keep encountering | 18:51 |
| itamarst | the idea that you have multiple copies of a stdlib module is sooo broken | 19:17 |
| itamarst | it's possible a rewrite of the monkey patching system might need to happen, but not going to try that unless it becomes clearly necessary | 19:20 |
| -opendevstatus- NOTICE: Gerrit will undergo a short restart to pick up some bugfixes for the 3.10 release that we upgraded to. | 19:25 | |
| itamarst | or maybe much more fine grained monkeypatching for asyncio modules | 19:29 |
| itamarst | or rather, unmonkeypatching | 19:29 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!