| *** EmilienM has joined #openstack-mistral | 00:41 | |
| EmilienM | hello | 00:41 |
|---|---|---|
| EmilienM | apetrich, rakhmerov, d0ugal: can you folks take a look at https://bugs.launchpad.net/mistral/+bug/1818757 ? | 00:42 |
| openstack | Launchpad bug 1818757 in tripleo "Mistral's eventlet monkeypatching breaks git-python" [Critical,Triaged] | 00:42 |
| *** mmethot has joined #openstack-mistral | 01:15 | |
| rakhmerov | EmilienM: ok | 04:29 |
| *** gkadam has quit IRC | 06:13 | |
| *** akovi has joined #openstack-mistral | 06:48 | |
| *** quiquell|off is now known as quiquell | 06:53 | |
| *** pgaxatte has joined #openstack-mistral | 07:21 | |
| apetrich | EmilienM, looking | 07:33 |
| apetrich | oh dear eventlet | 07:33 |
| d0ugal | That will likely be hard to solve | 07:39 |
| d0ugal | so I guess we need a workaround | 07:40 |
| d0ugal | apetrich: How did that code ever work tho? | 07:40 |
| apetrich | d0ugal, I'm still trying to understand | 07:42 |
| *** quiquell is now known as quiquell|brb | 07:50 | |
| apetrich | d0ugal, where is that select.pool coming from? | 07:57 |
| *** gkadam has joined #openstack-mistral | 08:14 | |
| apetrich | d0ugal, in ipython that snipplet is even more annoying | 08:23 |
| *** quiquell|brb is now known as quiquell | 08:28 | |
| d0ugal | apetrich: It is a result of eventlet monkeypatching | 08:46 |
| d0ugal | essentially it seems the git library isn't compatable with the patch | 08:46 |
| d0ugal | So either it never worked or git/ventlet updated? | 08:46 |
| d0ugal | apetrich: subprocess doesn't work with eventlet | 08:47 |
| d0ugal | and the git lib uses subprocess | 08:47 |
| apetrich | eventlet == 0.19.0 works | 08:51 |
| apetrich | eventlet 0.20.0 onwards breaks | 08:51 |
| d0ugal | Interesting | 08:52 |
| d0ugal | https://github.com/openstack/requirements/blob/master/global-requirements.txt#L58-L59 | 08:52 |
| d0ugal | :-D | 08:52 |
| d0ugal | 0.24.1 is the latest, damn we are far behind then | 08:52 |
| apetrich | found this bug https://github.com/eventlet/eventlet/issues/290 that pointed to that version | 08:52 |
| apetrich | that's a very healthy way of future proofing versions | 08:53 |
| apetrich | so what do we do? add 0.24 to the list? | 08:56 |
| apetrich | that is a problem https://github.com/openstack/mistral/blob/master/requirements.txt#L11 | 08:58 |
| d0ugal | apetrich: add 0.24 to the list of versions not allowed? | 09:05 |
| d0ugal | apetrich: sorry was just having some breakfast | 09:05 |
| d0ugal | apetrich: Remember for tripleo the rpm versions are what count anyway | 09:06 |
| rakhmerov | woow.. | 09:35 |
| rakhmerov | sorry, I missed some of this conversation | 09:36 |
| rakhmerov | so we had an old version of eventlet for a long time and now it is updated and causes issues? | 09:37 |
| rakhmerov | d0ugal, apetrich: ^ | 09:37 |
| apetrich | rakhmerov, kind of | 09:37 |
| rakhmerov | may be if it's only about certain versions we could really exclude them | 09:37 |
| rakhmerov | not sure though if it's OK from TripleO perspective | 09:37 |
| apetrich | so in brief mistral is using in the requirements 0.209 which is old (uptodate is 0.24.1) | 09:38 |
| rakhmerov | yeah | 09:38 |
| apetrich | but that causes problems with gitpython that uses subprocess comunicate | 09:38 |
| rakhmerov | hah... | 09:38 |
| rakhmerov | I definitely miss that requirements update bot.. | 09:38 |
| apetrich | I got it to work with python 3.6.7 with eventlet 0.19.0 | 09:38 |
| rakhmerov | I see | 09:39 |
| apetrich | it seems that it is version specific. d0ugal tried with 3.7 and it works | 09:39 |
| apetrich | I saw a comment of something similar that it is not happening on mac. but that is a non issue to us | 09:39 |
| rakhmerov | so does it depend on version of Python also? | 09:39 |
| rakhmerov | don't get it.. | 09:39 |
| rakhmerov | I thought the issue was only in the new version of eventlet, no? | 09:40 |
| apetrich | eventlet and python | 09:42 |
| apetrich | I'm still trying to understand what is happening | 09:43 |
| rakhmerov | ok | 09:52 |
| rakhmerov | btw, we need to make a release this week | 09:53 |
| rakhmerov | if we find a solution it'd be good to include it | 09:53 |
| apetrich | aye | 09:54 |
| *** jlejeune_ has joined #openstack-mistral | 10:03 | |
| d0ugal | rakhmerov: Yeah, it is a confusing issue. I'm not really sure I understand it or even understand where the fault it :) | 10:12 |
| rakhmerov | ok | 10:13 |
| d0ugal | Meaning I don't know who to blame, but I can't think of any changes we can make it mistral | 10:13 |
| rakhmerov | yeah.. me neither | 10:14 |
| d0ugal | I would be tempted to mark it is invalid on mistral, because I don't think it is a mistral bug | 10:14 |
| rakhmerov | if it's so tricky the only thing that comes to mind is to make some workaround in tripleO (like pinning a version) | 10:15 |
| rakhmerov | yeah | 10:15 |
| d0ugal | Yeah | 10:15 |
| rakhmerov | we can't remove monkey patching obviously | 10:15 |
| d0ugal | and we can't change eventlet or gitpython :) | 10:15 |
| rakhmerov | yep | 10:16 |
| rakhmerov | right | 10:16 |
| apetrich | d0ugal, rakhmerov this works http://paste.openstack.org/show/747330/ | 10:21 |
| apetrich | it is not a solution | 10:21 |
| apetrich | but | 10:21 |
| apetrich | I don't know but | 10:21 |
| d0ugal | apetrich: it might be a solution :) | 10:21 |
| rakhmerov | can it be made on the TripleO side? | 10:22 |
| apetrich | Sure | 10:22 |
| rakhmerov | it'd be kind of weird to see this code in Mistral | 10:22 |
| d0ugal | Agreed, that code would go in tripleo-common | 10:22 |
| apetrich | http://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/utils/config.py#n15 | 10:23 |
| apetrich | do we do a try catch? | 10:23 |
| apetrich | on that import? | 10:23 |
| d0ugal | apetrich: Any idea why that helps? | 10:23 |
| d0ugal | apetrich: The import doesn't fail | 10:23 |
| apetrich | yeah. true | 10:23 |
| d0ugal | I hate monkey patching so much :) | 10:24 |
| apetrich | I don't know why but the subprocess on __init__ from git is not quite working | 10:24 |
| apetrich | subprocess patching | 10:24 |
| rakhmerov | :) | 10:24 |
| apetrich | it is not using green subprocess as it should due to the massive monkeypatch that we do on the executor | 10:25 |
| apetrich | no idea why though | 10:25 |
| rakhmerov | (smiling on Dougal's feelings about monkey patching) | 10:25 |
| apetrich | so say we all about monkey patching | 10:25 |
| rakhmerov | yeah | 10:25 |
| rakhmerov | it is what it is | 10:26 |
| apetrich | going to submit a patch and see how it goes | 10:26 |
| apetrich | d0ugal, ^^ | 10:26 |
| d0ugal | Sounds good | 10:26 |
| d0ugal | apetrich: any idea why we are not seeing this problem in CI? or is it not covered? | 10:26 |
| d0ugal | We can probably move to #tripleo now I guess. | 10:26 |
| apetrich | sure | 10:30 |
| *** smrcascao has quit IRC | 11:41 | |
| EmilienM | apetrich d0ugal rakhmerov: thank you very much | 11:57 |
| apetrich | EmilienM, just pushed a patch up | 12:09 |
| EmilienM | apetrich: thanks a ton | 12:58 |
| apetrich | EmilienM, no worries :) | 12:58 |
| *** pgaxatte has quit IRC | 13:05 | |
| *** pgaxatte has joined #openstack-mistral | 13:07 | |
| *** quiquell is now known as quiquell|lunch | 13:42 | |
| *** quiquell|lunch is now known as quiquell | 13:55 | |
| *** akovi has quit IRC | 14:07 | |
| *** gkadam_ has joined #openstack-mistral | 14:16 | |
| *** smrcascao has joined #openstack-mistral | 14:17 | |
| *** gkadam has quit IRC | 14:19 | |
| *** gkadam__ has joined #openstack-mistral | 15:00 | |
| *** pgaxatte has quit IRC | 15:02 | |
| *** gkadam_ has quit IRC | 15:03 | |
| *** pgaxatte has joined #openstack-mistral | 15:04 | |
| *** quiquell is now known as quiquell|off | 15:21 | |
| apetrich | EmilienM, d0ugal still not solved. the error changed. just above this line http://logs.openstack.org/38/641338/2/check/tripleo-ci-centos-7-containers-multinode/63dc73f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-03-06_14_33_08 | 15:29 |
| apetrich | good news is that I can reproduce locally | 15:47 |
| *** gkadam__ has quit IRC | 16:42 | |
| *** pgaxatte has quit IRC | 16:51 | |
| *** mmethot has quit IRC | 17:44 | |
| *** mmethot has joined #openstack-mistral | 17:45 | |
| *** jtomasek has quit IRC | 18:06 | |
| d0ugal | apetrich: Let me know if I can help tomorrow | 19:46 |
| apetrich | d0ugal, I've just posted a change that works | 19:46 |
| apetrich | d0ugal, but it is pretty ugly. I want to see it go through CI and tomorrow we think about it | 19:47 |
| *** irclogbot_1 has joined #openstack-mistral | 21:11 | |
| *** irclogbot_1 has quit IRC | 21:28 | |
| *** apetrich has quit IRC | 22:08 | |
| *** apetrich has joined #openstack-mistral | 22:11 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!