| *** chlong has quit IRC | 00:13 | |
| *** chlong has joined #openstack-mistral | 00:26 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-mistralclient: Updated from global requirements https://review.openstack.org/360154 | 01:11 |
|---|---|---|
| *** Ephur has quit IRC | 02:37 | |
| *** bobh has joined #openstack-mistral | 03:00 | |
| *** tonytan4ever has quit IRC | 03:13 | |
| *** Ephur has joined #openstack-mistral | 03:16 | |
| *** chlong has quit IRC | 03:23 | |
| *** vishwanathj has joined #openstack-mistral | 03:25 | |
| *** Ephur has quit IRC | 03:30 | |
| *** vishwanathj is now known as vishwanathj_zzz | 03:31 | |
| *** chlong has joined #openstack-mistral | 03:35 | |
| *** chlong has quit IRC | 03:38 | |
| *** chlong has joined #openstack-mistral | 03:38 | |
| *** tonytan4ever has joined #openstack-mistral | 04:13 | |
| *** tonytan4ever has quit IRC | 04:18 | |
| *** Ravikiran_K has joined #openstack-mistral | 04:29 | |
| *** bobh has quit IRC | 04:30 | |
| *** jaosorior has joined #openstack-mistral | 05:10 | |
| *** janki has joined #openstack-mistral | 05:23 | |
| *** vishwanathj_zzz has quit IRC | 06:10 | |
| *** vishwanathj_zzz has joined #openstack-mistral | 06:10 | |
| *** vishwanathj_zzz has quit IRC | 06:32 | |
| *** vishwanathj_zzz has joined #openstack-mistral | 06:32 | |
| rakhmerov | kong: done | 06:43 |
| openstackgerrit | Merged openstack/mistral: Updated from global requirements https://review.openstack.org/360236 | 06:52 |
| openstackgerrit | Renat Akhmerov proposed openstack/mistral: Change execution mechanism for 'join' tasks https://review.openstack.org/356343 | 06:57 |
| *** sharatss has joined #openstack-mistral | 07:01 | |
| openstackgerrit | zhangyanxian proposed openstack/mistral: Some minior code optimization in post_test_hook https://review.openstack.org/360286 | 07:08 |
| openstackgerrit | zhangyanxian proposed openstack/mistral: Some minior code optimization in post_test_hook https://review.openstack.org/360286 | 07:09 |
| openstackgerrit | zhangyanxian proposed openstack/mistral: Some minior code optimization in post_test_hook https://review.openstack.org/360286 | 07:09 |
| *** Ravikiran_K has quit IRC | 07:14 | |
| openstackgerrit | zhangyanxian proposed openstack/mistral: Some minior code optimization in post_test_hook.sh https://review.openstack.org/360286 | 07:15 |
| *** tonytan4ever has joined #openstack-mistral | 07:15 | |
| *** vishwanathj_zzz has quit IRC | 07:19 | |
| *** vishwanathj_zzz has joined #openstack-mistral | 07:20 | |
| *** tonytan4ever has quit IRC | 07:20 | |
| brunograz_ | kong: Thx! | 07:26 |
| *** jpich has joined #openstack-mistral | 07:41 | |
| *** shardy has joined #openstack-mistral | 07:47 | |
| *** mflobo has joined #openstack-mistral | 07:51 | |
| openstackgerrit | Merged openstack/python-mistralclient: Updated from global requirements https://review.openstack.org/360154 | 07:58 |
| *** vishwanathj_zzz has quit IRC | 08:10 | |
| *** vishwanathj_zzz has joined #openstack-mistral | 08:10 | |
| openstackgerrit | zhangyanxian proposed openstack/mistral: Some minor code optimization in post_test_hook.sh https://review.openstack.org/360286 | 08:25 |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** openstackgerrit has joined #openstack-mistral | 08:48 | |
| rakhmerov | ddeja: hi, here? | 09:00 |
| rakhmerov | I have some questions on kombu based RPC | 09:01 |
| rakhmerov | well, first of all, what is the status of your patches? Is https://review.openstack.org/#/c/354091 considered done? I saw that you were discussing something with kong | 09:02 |
| *** tonytan4ever has joined #openstack-mistral | 09:16 | |
| *** tonytan4ever has quit IRC | 09:20 | |
| rakhmerov | ddeja: ^ | 09:27 |
| *** vishwanathj_zzz has quit IRC | 09:52 | |
| *** vishwanathj_zzz has joined #openstack-mistral | 09:53 | |
| ddeja | rakhmerov: hi | 09:54 |
| rakhmerov | hi | 09:54 |
| ddeja | rakhmerov: yup, this one is done | 09:54 |
| openstackgerrit | Renat Akhmerov proposed openstack/python-mistralclient: Send access info to server https://review.openstack.org/354803 | 09:55 |
| openstackgerrit | Renat Akhmerov proposed openstack/mistral: Update documentation for multi-vim-support https://review.openstack.org/356567 | 09:58 |
| ddeja | rakhmerov: I'm going for lunch, please write your question, I'll anwser when I got back | 09:58 |
| rakhmerov | ok | 09:58 |
| rakhmerov | yeah, I wanted to ask for one more thing in RPC actually | 09:59 |
| rakhmerov | we need something like 'executor' parameter in o.m | 09:59 |
| rakhmerov | so that we specifically could do something like executor='blocking' | 09:59 |
| rakhmerov | I found some interesting limitations with executor='eventlet' that I don't know how to overcome yet | 10:00 |
| *** sharatss has quit IRC | 10:02 | |
| openstackgerrit | Merged openstack/mistral: Fix for not working 'run-action' on kombu driver https://review.openstack.org/354091 | 10:19 |
| openstackgerrit | Merged openstack/mistral: Some minor code optimization in post_test_hook.sh https://review.openstack.org/360286 | 10:20 |
| openstackgerrit | Andras Kovi proposed openstack/mistral: Accept service catalog from client side https://review.openstack.org/355368 | 10:25 |
| openstackgerrit | Andras Kovi proposed openstack/mistral: Accept service catalog from client side https://review.openstack.org/355368 | 10:30 |
| *** vishwanathj_zzz has quit IRC | 10:55 | |
| *** vishwanathj_zzz has joined #openstack-mistral | 10:56 | |
| openstackgerrit | Andras Kovi proposed openstack/mistral: Update documentation for multi-vim-support https://review.openstack.org/356567 | 11:00 |
| ddeja | rakhmerov: OK, I see | 11:10 |
| ddeja | as far as I can tell in kombu driver we are using something like blocking executor | 11:11 |
| rakhmerov | ddeja: what do you think? Is it too hard to do? | 11:11 |
| rakhmerov | really? | 11:11 |
| rakhmerov | can you check that somehow? | 11:11 |
| rakhmerov | if it's blocking then it's something that we probably need :) | 11:11 |
| ddeja | le me test it to be sure, but i think all the calls would be done 'one-by-one' | 11:12 |
| *** tonytan4ever has joined #openstack-mistral | 11:17 | |
| *** tonytan4ever has quit IRC | 11:22 | |
| *** nmakhotkin has joined #openstack-mistral | 11:27 | |
| openstackgerrit | Andras Kovi proposed openstack/mistral: Accept service catalog from client side https://review.openstack.org/355368 | 11:27 |
| ddeja | rakhmerov: I've just checked - kombu driver would process all the calls one-by-one | 11:44 |
| ddeja | so it is using 'blocking' | 11:45 |
| rakhmerov | ook.. | 11:45 |
| rakhmerov | let me think.. | 11:45 |
| rakhmerov | ok with me for now, later on we can think about using something else if kombu can somehow allow it | 11:46 |
| rakhmerov | maybe not | 11:46 |
| ddeja | i guess it could be done in Ocata cycle | 11:46 |
| rakhmerov | basically, I've been struggling with a strange issue for the last week and finally understood why it happened | 11:46 |
| rakhmerov | because of executor='eventlet' | 11:46 |
| rakhmerov | ddeja: yes | 11:46 |
| rakhmerov | generally, what I found is that green threads (used by eventlet) cannot be used if we make blocking calls in them | 11:47 |
| rakhmerov | like I do in order to synchronize some 'join' related stuff | 11:47 |
| rakhmerov | in this case green threads may get into deadlock | 11:48 |
| rakhmerov | :) | 11:48 |
| ddeja | oh, I see | 11:48 |
| rakhmerov | for some reason, green threads dispatching doesn't work as I expected | 11:48 |
| ddeja | and you would like to make oslo use blocking? | 11:48 |
| rakhmerov | I don't like this idea but yes, for now yes | 11:49 |
| rakhmerov | until we find a different way | 11:49 |
| rakhmerov | I'll give you an example, I reproduced it in a simple form | 11:49 |
| ddeja | but it will have performance impact? | 11:49 |
| rakhmerov | sec | 11:49 |
| ddeja | ok\ | 11:49 |
| rakhmerov | yes, it's slightly slower (not critically though) | 11:49 |
| rakhmerov | but it works w/o deadlocks in green threads :) | 11:50 |
| ddeja | ok | 11:50 |
| *** jaosorior has quit IRC | 11:52 | |
| rakhmerov | ddeja: http://paste.openstack.org/show/563368/ | 11:52 |
| rakhmerov | this deadlocks | 11:52 |
| *** jaosorior has joined #openstack-mistral | 11:52 | |
| rakhmerov | I just run multiple green threads where each of them tries to do something with this named lock (via DB) that I added recently | 11:52 |
| rakhmerov | - the 1st one acquires the lock | 11:53 |
| rakhmerov | - the 1st one (I mean TX) calls sleep() and passes execution to the next green thread | 11:53 |
| rakhmerov | - the 2nd TX starts and tries to acquire the lock and blocks | 11:54 |
| rakhmerov | that's it | 11:54 |
| ddeja | oh, I see | 11:54 |
| rakhmerov | and it hangs forever because execution is never given back to the 1st TX | 11:54 |
| rakhmerov | which needs to release the lock | 11:54 |
| ddeja | yup | 11:55 |
| *** dprince has joined #openstack-mistral | 11:55 | |
| rakhmerov | I actually thought that when we do an external IO call it should switch green thread | 11:55 |
| rakhmerov | btw, it would never happen with real parallel threads | 11:55 |
| rakhmerov | with real parallelism TX 1 would just continue in parallel and release the lock | 11:56 |
| rakhmerov | that's why I switched to 'blocking' in https://review.openstack.org/#/c/356343/ | 11:57 |
| rakhmerov | and it now works in 100% cases for me, I do some ridiculous tests on my machine like 500 tasks joined by 501st task | 11:58 |
| rakhmerov | so basically this makes transactions run 1-by-1 on the same instance of engine | 11:58 |
| ddeja | can we use 'threading' instead of eventlet? | 11:59 |
| ddeja | http://docs.openstack.org/developer/oslo.messaging/executors.html | 11:59 |
| rakhmerov | I tried, the same problem | 11:59 |
| ddeja | oh, OK | 11:59 |
| rakhmerov | yeah | 11:59 |
| rakhmerov | the fundamental reason is that there's no parallelism here | 11:59 |
| rakhmerov | I mean real parallelism | 11:59 |
| rakhmerov | we can potentially use something like multiprocessing module probably | 12:00 |
| rakhmerov | but it's not a near future | 12:00 |
| ddeja | agree | 12:00 |
| rakhmerov | the thing that I don't understand actually yet is why it doesn't switch threads when I call DB | 12:01 |
| rakhmerov | I thought it should | 12:01 |
| rakhmerov | maybe we need to monkeypatch some additional modules or something | 12:01 |
| rakhmerov | I don't know | 12:01 |
| rakhmerov | so I'm now reading about green threads to understand more about them | 12:03 |
| rakhmerov | :) | 12:03 |
| *** tonytan4ever has joined #openstack-mistral | 12:10 | |
| ddeja | OK. let me know if you need some help with it :) | 12:14 |
| *** tonytan_brb has joined #openstack-mistral | 12:17 | |
| rakhmerov | ok | 12:17 |
| rakhmerov | I just really want to understand whether it's our misconfiguration or it's a real limitation | 12:18 |
| *** bobh has joined #openstack-mistral | 12:18 | |
| *** vishwanathj_zzz has quit IRC | 12:18 | |
| *** vishwanathj_zzz has joined #openstack-mistral | 12:18 | |
| *** tonytan4ever has quit IRC | 12:19 | |
| *** shardy has quit IRC | 12:21 | |
| *** bradjones has quit IRC | 12:21 | |
| *** shardy has joined #openstack-mistral | 12:22 | |
| *** jaosorior is now known as jaosorior_away | 12:25 | |
| *** bradjones has joined #openstack-mistral | 12:27 | |
| *** bradjones has quit IRC | 12:27 | |
| *** bradjones has joined #openstack-mistral | 12:27 | |
| *** bobh has quit IRC | 12:42 | |
| brunograz_ | hi all, I filed a bug describing the issue we had a couple of days ago - here is the link: https://bugs.launchpad.net/mistral/+bug/1615993. | 12:59 |
| openstack | Launchpad bug 1615993 in Mistral "Cron trigger does not execute Openstack actions - Cannot Authorize API client" [Undecided,New] | 12:59 |
| *** vishwanathj_zzz is now known as vishwanathj | 13:00 | |
| brunograz_ | I'm currently trying to solve this bug but I'm a bit new to Mistral and tbh this is a non trivial error. | 13:00 |
| brunograz_ | What I can see is that the trust_id is correct and the context created out of it is also correct. | 13:01 |
| brunograz_ | Now what I cannot understand is how the action is executed and how the authentication out of the context is done. | 13:02 |
| *** janki has quit IRC | 13:03 | |
| *** janki has joined #openstack-mistral | 13:05 | |
| *** vishwanathj has quit IRC | 13:15 | |
| *** vishwanathj has joined #openstack-mistral | 13:16 | |
| *** Ephur has joined #openstack-mistral | 13:22 | |
| ddeja | rakhmerov: still there? | 13:23 |
| ddeja | brunograz_: Hi, I'm looking on your bug | 13:26 |
| ddeja | good news, I can see the cause, but didn't found the root yet | 13:27 |
| brunograz_ | ddeja: great! what is causing this issue btw? | 13:28 |
| brunograz_ | We are looking into the code as well trying to understand the mechaniscs of cron triggers | 13:29 |
| ddeja | brunograz_: from what I see now | 13:32 |
| ddeja | wrong context information is passed from API layer | 13:32 |
| ddeja | it just have auth token, project id and user id | 13:33 |
| ddeja | wheras when using API, there is auth url and much much more | 13:33 |
| ddeja | when using direct call to API* | 13:34 |
| brunograz_ | ddeja - we did try adding an auth_url to the context passed by the API. iirc this info is in services/security.py. Not sure though if there are more information missing | 13:35 |
| brunograz_ | we are getting a 403 error in keystone which means in keystone terminology - The identity was successfully authenticated but it is not authorized to perform the requested action. | 13:36 |
| brunograz_ | the same request has the trust id and the authorization is delegated | 13:40 |
| ddeja | brunograz_: I might have a fix, let me try and test it... | 13:43 |
| brunograz_ | ddeja: V cool! thx | 13:44 |
| *** janki has quit IRC | 13:45 | |
| *** jaosorior_away is now known as jaosorior | 13:48 | |
| ddeja | brunograz_: nope, that doesn't help it :/ | 13:50 |
| *** vishwanathj has quit IRC | 13:55 | |
| *** vishwanathj has joined #openstack-mistral | 13:55 | |
| brunograz_ | ddeja: ooh :O, what did you try? | 14:03 |
| ddeja | I found out that periodic service is calling our rpc layer direclty rather that using engine client | 14:05 |
| ddeja | I hope that that's the problem, but no | 14:05 |
| brunograz_ | ddeja: were you able to reproduce the error? | 14:07 |
| *** jaosorior has quit IRC | 14:08 | |
| *** mflobo has quit IRC | 14:08 | |
| *** janki has joined #openstack-mistral | 14:08 | |
| brunograz_ | ddeja: I'm not sure what you mean abt engine client. could you point us what this is located? | 14:09 |
| ddeja | brunograz_: yes, I reproduced it | 14:11 |
| ddeja | changing the engine_client does not help, so it doeasn't matter | 14:11 |
| brunograz_ | cool, just wanted to make sure it wasn't only us :) | 14:11 |
| brunograz_ | found the root cause of the problem - https://github.com/openstack/mistral/blob/master/mistral/actions/openstack/actions.py#L60 | 14:28 |
| brunograz_ | the execution of the method _get_client stops at this line | 14:28 |
| ddeja | brunograz_: well, this is not the root problem | 14:29 |
| *** janki has quit IRC | 14:29 | |
| brunograz_ | which goes here | 14:29 |
| brunograz_ | https://github.com/openstack/mistral/blob/master/mistral/utils/openstack/keystone.py#L67 | 14:29 |
| ddeja | it starts way before it goes to NovaActions | 14:29 |
| *** nmakhotkin has quit IRC | 14:30 | |
| brunograz_ | sorry, it might not be the root of the problem. but we identified an issue with this method | 14:30 |
| ddeja | brunograz_: yup, here is where the exception is throw | 14:31 |
| ddeja | brunograz_: there is a problem with generating auth_ctx at the beggining on the API side | 14:33 |
| ddeja | I think I fixed it, give me a minut | 14:33 |
| ddeja | minute* | 14:33 |
| brunograz_ | sure :) | 14:34 |
| jtomasek | rakhmerov: Hi, In recent mistral the workflow executions don't include the 'output' is that a known problem? | 14:40 |
| *** vishwanathj has quit IRC | 14:44 | |
| *** vishwanathj has joined #openstack-mistral | 14:44 | |
| *** vishwanathj has quit IRC | 14:46 | |
| *** vishwanathj has joined #openstack-mistral | 14:46 | |
| *** openstackgerrit has quit IRC | 14:48 | |
| *** openstackgerrit has joined #openstack-mistral | 14:49 | |
| ddeja | brunograz_: OK. I've checked that user_id and project_id used by both cron triggers and directly calling 'execute' are the same | 15:03 |
| *** tonytan4ever has joined #openstack-mistral | 15:03 | |
| ddeja | the only thing that is different is token_id, which is usuall situation | 15:04 |
| ddeja | I'm looking how the token creation difference beetwen these two to find the root cause | 15:04 |
| *** jpich has quit IRC | 15:04 | |
| brunograz_ | ddeja: cool ok, will let you know if I find something interesting regarding this error | 15:06 |
| *** tonytan_brb has quit IRC | 15:06 | |
| *** dprince has quit IRC | 15:09 | |
| *** brian_price has joined #openstack-mistral | 15:13 | |
| *** janki has joined #openstack-mistral | 15:14 | |
| *** vishwanathj has quit IRC | 15:15 | |
| *** brian_price has quit IRC | 15:15 | |
| *** vishwanathj has joined #openstack-mistral | 15:16 | |
| *** mflobo has joined #openstack-mistral | 15:17 | |
| *** brian_price has joined #openstack-mistral | 15:17 | |
| *** vishwanathj has quit IRC | 15:18 | |
| *** vishwanathj has joined #openstack-mistral | 15:19 | |
| *** brian_price has quit IRC | 15:25 | |
| *** brian_price has joined #openstack-mistral | 15:25 | |
| *** brian_price has quit IRC | 15:37 | |
| *** vishwanathj has quit IRC | 15:44 | |
| *** vishwanathj has joined #openstack-mistral | 15:45 | |
| *** jchhatbar has joined #openstack-mistral | 15:58 | |
| *** janki has quit IRC | 15:59 | |
| *** jchhatbar is now known as janki | 16:00 | |
| *** vishwanathj has quit IRC | 16:04 | |
| *** vishwanathj has joined #openstack-mistral | 16:05 | |
| *** tonytan_brb has joined #openstack-mistral | 16:15 | |
| *** tonytan4ever has quit IRC | 16:17 | |
| *** dprince has joined #openstack-mistral | 16:19 | |
| *** vishwanathj has quit IRC | 16:27 | |
| *** vishwanathj has joined #openstack-mistral | 16:28 | |
| *** dprince has quit IRC | 16:37 | |
| *** jpich has joined #openstack-mistral | 16:40 | |
| *** rrecio has joined #openstack-mistral | 16:50 | |
| *** Kiall has quit IRC | 16:51 | |
| *** dprince has joined #openstack-mistral | 16:53 | |
| *** janki has quit IRC | 17:09 | |
| *** mflobo has quit IRC | 17:14 | |
| *** Kiall has joined #openstack-mistral | 17:36 | |
| *** Ephur has quit IRC | 17:49 | |
| *** vishwanathj has quit IRC | 18:47 | |
| *** vishwanathj has joined #openstack-mistral | 18:48 | |
| *** krotscheck has left #openstack-mistral | 18:51 | |
| *** brian_price has joined #openstack-mistral | 19:00 | |
| *** Ephur has joined #openstack-mistral | 19:23 | |
| *** jpich has quit IRC | 20:00 | |
| *** tonytan_brb is now known as tonytan4ever | 20:02 | |
| *** dprince has quit IRC | 20:07 | |
| *** vishwanathj has quit IRC | 20:19 | |
| *** vishwanathj has joined #openstack-mistral | 20:20 | |
| *** vishwanathj has quit IRC | 20:28 | |
| *** vishwanathj has joined #openstack-mistral | 20:29 | |
| *** Ephur has quit IRC | 20:32 | |
| *** gyee has joined #openstack-mistral | 20:42 | |
| *** vishwanathj has quit IRC | 21:16 | |
| *** vishwanathj has joined #openstack-mistral | 21:16 | |
| *** vishwanathj has quit IRC | 21:26 | |
| *** vishwanathj has joined #openstack-mistral | 21:27 | |
| *** shardy has quit IRC | 21:57 | |
| *** vishwanathj has quit IRC | 22:01 | |
| *** vishwanathj has joined #openstack-mistral | 22:02 | |
| *** vishwanathj has quit IRC | 22:17 | |
| *** vishwanathj has joined #openstack-mistral | 22:18 | |
| *** vishwanathj has quit IRC | 22:23 | |
| *** chlong has quit IRC | 22:58 | |
| *** chlong has joined #openstack-mistral | 23:11 | |
| *** chlong has quit IRC | 23:35 | |
| *** rrecio has quit IRC | 23:39 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!