| *** bobh has joined #openstack-mistral | 00:00 | |
| *** bobh has quit IRC | 00:05 | |
| *** bobh has joined #openstack-mistral | 00:33 | |
| *** jamielennox is now known as jamielennox|away | 01:01 | |
| *** jamielennox|away is now known as jamielennox | 01:08 | |
| *** evgenyl has quit IRC | 01:51 | |
| *** evgenyl has joined #openstack-mistral | 01:52 | |
| *** bobh has quit IRC | 02:38 | |
| openstackgerrit | Lingxian Kong proposed openstack/mistral-specs: Move spec files to correct location https://review.openstack.org/387766 | 03:00 |
|---|---|---|
| *** jamielennox is now known as jamielennox|away | 03:01 | |
| openstackgerrit | Lingxian Kong proposed openstack/mistral-specs: Move spec files to correct location https://review.openstack.org/387766 | 03:15 |
| *** jamielennox|away is now known as jamielennox | 03:33 | |
| *** rbrady is now known as rbrady-afk | 03:35 | |
| kong | d0ugal: hi, i remember you added the tripleo-ci, could you please take a look at why jenkins failed for this patch in stable/newton: https://review.openstack.org/#/c/387757/? | 03:45 |
| kong | i'm not familiar with how tripleo works with mistral | 03:46 |
| rakhmerov | d0ugal: hi, sorry, missed your last message yesterday | 04:02 |
| rakhmerov | ping me anytime when you're ready | 04:03 |
| *** dkushwaha has joined #openstack-mistral | 05:08 | |
| *** jaosorior has joined #openstack-mistral | 05:18 | |
| *** chlong has quit IRC | 05:47 | |
| *** chlong has joined #openstack-mistral | 06:01 | |
| *** shardy has joined #openstack-mistral | 07:08 | |
| *** shardy has quit IRC | 07:22 | |
| *** jpich has joined #openstack-mistral | 07:39 | |
| d0ugal | rakhmerov: Thanks, today is a particularly busy one for me, so I might try tomorrow :) | 07:52 |
| d0ugal | rakhmerov: but the tl;dr is that we have lots of repetition in our workflows. How would you feel about the idea of workflow inheritance? | 07:52 |
| d0ugal | :) | 07:52 |
| rakhmerov | ok | 07:52 |
| d0ugal | I guess it could be a bit like ad-hoc actions, with a base defined etc. | 07:53 |
| d0ugal | For example, we have this in a large number of places: https://github.com/openstack/tripleo-common/blob/master/workbooks/plan_management.yaml#L110-L121 | 07:53 |
| d0ugal | That we could probably reduce by using a ad-hoc workflow, but there are others too | 07:54 |
| d0ugal | brb | 07:55 |
| d0ugal | rakhmerov: ^ | 07:55 |
| rakhmerov | hm.. it generally feels good to me, need to discuss details | 07:56 |
| rakhmerov | reading | 07:56 |
| rakhmerov | yes, that makes sense to me | 07:56 |
| *** igormarnat has quit IRC | 07:56 | |
| rakhmerov | my only concern is that we should not overcomplicate the language | 07:56 |
| *** akuznetsova has quit IRC | 07:56 | |
| rakhmerov | people like the fact that it's pretty simple now | 07:57 |
| d0ugal | rakhmerov: Yeah, I would consider it a more advanced feature | 07:59 |
| d0ugal | but I think if you have a large number of workflows it should make it easier to maintain them | 08:00 |
| rakhmerov | yeah | 08:00 |
| rakhmerov | ok | 08:00 |
| rakhmerov | can you include it into our summit topics pls with some basic description? | 08:01 |
| *** akuznetsova has joined #openstack-mistral | 08:01 | |
| d0ugal | rakhmerov: Sure, I'll try and start a draft spec to describe it a bit more | 08:02 |
| rakhmerov | ok | 08:03 |
| *** igormarnat has joined #openstack-mistral | 08:03 | |
| rakhmerov | generally, I think it's a good idea | 08:03 |
| d0ugal | Great, I wanted to check this before I spent time writing a spec :) | 08:04 |
| *** openstackgerrit has quit IRC | 08:04 | |
| *** openstackgerrit has joined #openstack-mistral | 08:05 | |
| *** flwang1 has joined #openstack-mistral | 08:09 | |
| flwang1 | rakhmerov: ping | 08:10 |
| rakhmerov | flwang1: hi | 08:10 |
| flwang1 | need your help | 08:10 |
| rakhmerov | yes | 08:10 |
| flwang1 | see bug https://bugs.launchpad.net/mistral/+bug/1634090 | 08:10 |
| openstack | Launchpad bug 1634090 in Mistral "Failed to create keystone client" [Undecided,New] - Assigned to Fei Long Wang (flwang) | 08:10 |
| flwang1 | and i also test a very simple workflow, which just do nova.servers_list | 08:10 |
| rakhmerov | ok, I see | 08:12 |
| rakhmerov | can you show me your Mistral config? | 08:12 |
| rakhmerov | specifically how you configure auth | 08:12 |
| flwang1 | in mistral executor log, i also got http://paste.openstack.org/show/586127/ | 08:12 |
| flwang1 | sure, wait a sec | 08:12 |
| rakhmerov | yeah, the error in the executor is a consequence | 08:13 |
| flwang1 | http://paste.openstack.org/show/586128/ | 08:13 |
| rakhmerov | looks fine to me | 08:16 |
| rakhmerov | what env vars do you source on the client side? | 08:16 |
| rakhmerov | I mean, with what vars do you initialize Mistral client? | 08:17 |
| flwang1 | wait a sec | 08:18 |
| flwang1 | http://paste.openstack.org/show/586129/ | 08:19 |
| flwang1 | i installed mistral by devstack | 08:22 |
| rakhmerov | export OS_PASSWORD=passw0rdshow/586128/ | 08:22 |
| rakhmerov | is this ok? | 08:23 |
| flwang1 | why do i need a different password? | 08:23 |
| flwang1 | i don't think it's related | 08:23 |
| rakhmerov | I mean it looks strange | 08:23 |
| rakhmerov | but ok | 08:23 |
| therve | I don't think mistral is equipped to handle trust tokens | 08:24 |
| therve | It doesn't keystone sessions or keystoneauth to do the job | 08:24 |
| flwang1 | rakhmerov: seems the password is changed by paste, my original password is passw0rd | 08:24 |
| rakhmerov | flwang1: ok | 08:24 |
| rakhmerov | flwang1: this is what I use on the client usually: http://paste.openstack.org/show/586130/ | 08:25 |
| rakhmerov | it works | 08:25 |
| flwang1 | rakhmerov: hmm... i don't think it's related to the rc file, tbh | 08:26 |
| rakhmerov | not sure if it's the reason but, for example, I use OS_AUTH_URI, not OS_AUTH_URL | 08:26 |
| flwang1 | since i can user it for all the other services | 08:27 |
| rakhmerov | ok | 08:27 |
| flwang1 | to be more sepcifci | 08:27 |
| flwang1 | we're using a trust token to access mistral | 08:27 |
| flwang1 | from zaqar | 08:27 |
| rakhmerov | yes | 08:27 |
| flwang1 | and i can see the context is this | 08:28 |
| flwang1 | MistralContext {u'project_name': None, u'user_id': u'af6f84aaf58442cb85fc34e9ba6d58a4', u'roles': [u''], u'auth_uri': u'http://127.0.0.1:5000/v3', u'auth_cacert': None, u'auth_token': u'e86a0754f9e04fc79ca58b6d833a6f8a', u'expires_at': u'2016-10-18T08:11:25.000000Z', u'is_trust_scoped': False, u'project_id': None, u'user_name': u'admin', u'target_service_catalog': None} | 08:28 |
| flwang1 | is_trust_scoped is False | 08:28 |
| flwang1 | so https://github.com/openstack/mistral/blob/master/mistral/utils/openstack/keystone.py#L125 line is false, can then go to line 136 | 08:29 |
| therve | flwang1, is_trust_scoped is an internal flag to know if mistral issued the trust, I think | 08:29 |
| rakhmerov | hm.. | 08:30 |
| rakhmerov | this is, btw, relatively a new change | 08:30 |
| flwang1 | therve: yep, but it's causing mistral goes to line 136 | 08:30 |
| flwang1 | technically, it should go to line 126 | 08:30 |
| therve | I don't think so, because it's using service credentials there | 08:30 |
| rakhmerov | so, just to clarify: you pass a token to Mistral obtained from a trust, right? | 08:31 |
| flwang1 | rakhmerov: yep | 08:32 |
| flwang1 | and in my latest tests, even i use a normal token, i still failed to do any openstack service actions | 08:32 |
| flwang1 | i hope it's my env issue, but it's a new devstack env | 08:32 |
| flwang1 | before re-installed this env, i got the same error before | 08:33 |
| rakhmerov | flwang1: ok, let me try to get the author of these changes involved | 08:35 |
| flwang1 | therve: probably you're right, since there is no trust id in ctx | 08:35 |
| rakhmerov | therve: my understanding is that "is_trust_scoped" is to determine the type of token | 08:35 |
| rakhmerov | whether it was obtained from a trust or not | 08:36 |
| flwang1 | therve: rakhmerov: is it related the project_id is None? | 08:36 |
| therve | rakhmerov, Right, but only if the trust was created by mistral | 08:36 |
| therve | It's not meant to handle trust passed from the outside, AFAICT | 08:36 |
| flwang1 | therve: +1 | 08:37 |
| rakhmerov | hm.. maybe | 08:37 |
| rakhmerov | flwang1: yeah, I think project_id must not be None | 08:37 |
| rakhmerov | try to pass it too | 08:37 |
| flwang1 | it's based on my latest simple mistral test | 08:38 |
| flwang1 | let me re run my script to see the context | 08:39 |
| flwang1 | rakhmerov: if you have a local env, could you pls try run a simple workflow to trigger a heat or even nova action? | 08:42 |
| therve | flwang1, http://paste.openstack.org/show/586134/ works locally for me | 08:44 |
| flwang1 | MistralContext {u'project_name': u'admin', u'user_id': u'af6f84aaf58442cb85fc34e9ba6d58a4', u'roles': [u'admin'], u'auth_uri': u'http://127.0.0.1:5000/v3', u'auth_cacert': None, u'auth_token': u'ac172d0eeb154ec3ab3416fecafcb202', u'expires_at': u'2016-10-18T09:43:14.000000Z', u'is_trust_scoped': False, u'project_id': u'13401d12e2b44e9893c744868687d5b2', u'user_name': u'admin', u'target_service_catalog': None} | 08:44 |
| therve | It works around the issue, but it may be good enough for now | 08:44 |
| flwang1 | therve: awesome, let me try | 08:46 |
| flwang1 | therve: except mistral executor, do i need to restart any other processes? | 08:49 |
| rakhmerov | I'd restart all of them | 08:52 |
| therve | flwang1, You only need to restart api actually | 08:53 |
| flwang1 | i got same error after only restarted executor, now i'm going to restart api as well | 08:54 |
| rakhmerov | therve: do you think it's only a workaround? | 08:59 |
| flwang1 | therve: it works :) | 08:59 |
| rakhmerov | :) | 08:59 |
| rakhmerov | good | 08:59 |
| therve | rakhmerov, Yeah because if the catalog is not there we're still not going to be able to fetch it | 08:59 |
| flwang1 | rakhmerov: therve: do you mind me propose it as the initial patch? | 08:59 |
| rakhmerov | therve: did you check if we have a bug filed for that? | 08:59 |
| therve | rakhmerov, No | 09:00 |
| rakhmerov | flwang1: yeah, that's what I thought too | 09:00 |
| therve | flwang1, No | 09:00 |
| flwang1 | therve: oh, do you have any better idea? | 09:00 |
| flwang1 | i'm happy to follow up | 09:00 |
| flwang1 | after summit :D | 09:00 |
| therve | flwang1, I mean I don't mind, no | 09:00 |
| rakhmerov | at least let's file a bug pls | 09:00 |
| therve | Well the bug is already filled, no? | 09:01 |
| rakhmerov | therve: can you file a bug please? | 09:01 |
| flwang1 | rakhmerov: i have filed a bug | 09:01 |
| rakhmerov | therve: I don't know, honestly | 09:01 |
| flwang1 | https://bugs.launchpad.net/mistral/+bug/1634090 | 09:01 |
| openstack | Launchpad bug 1634090 in Mistral "Failed to create keystone client" [Undecided,New] - Assigned to Fei Long Wang (flwang) | 09:01 |
| rakhmerov | ooh, shoot.... | 09:01 |
| rakhmerov | sorry, seems like my head is somewhere else | 09:01 |
| rakhmerov | ok, yes | 09:01 |
| flwang1 | cool, i will do it and add you guys as reviewer, then we can shape it there | 09:02 |
| flwang1 | thank you so much for your help, you make my day | 09:02 |
| flwang1 | i can finally trigger a heat action in mistral | 09:02 |
| rakhmerov | flwang1: thanks to therve | 09:02 |
| rakhmerov | flwang1: I'll make the slides either today or tomorrow | 09:03 |
| rakhmerov | and get into the demo you prepared | 09:03 |
| flwang1 | rakhmerov: awesome, thanks | 09:04 |
| rakhmerov | np, thanks for delays, hot time | 09:04 |
| *** openstackgerrit has quit IRC | 09:04 | |
| flwang1 | therve: any idea for this http://paste.openstack.org/show/586141/ | 09:04 |
| rakhmerov | ...sorry for delayes.. | 09:04 |
| *** akovi has joined #openstack-mistral | 09:04 | |
| flwang1 | i can call mark unhealthy, but failed when calling stack update | 09:04 |
| flwang1 | rakhmerov: no worries | 09:04 |
| *** openstackgerrit has joined #openstack-mistral | 09:04 | |
| rakhmerov | akovi: hi Andras | 09:04 |
| therve | flwang1, Did you pass --existing? | 09:04 |
| rakhmerov | we just discussed a bug we have with trust tokens | 09:05 |
| flwang1 | therve: stacks_update: {action: heat.stacks_update stack_id=<% $.root_stack_id %> existing=True} | 09:05 |
| rakhmerov | you can read our conversation about and look at the workaround proposed by therve | 09:05 |
| flwang1 | rakhmerov: is it correct ? if i want to pass the existing param | 09:06 |
| rakhmerov | akovi: I thought you could probably propose a non-workaround solution for this | 09:06 |
| therve | flwang1, It's calling PUT, so it didn't take into account existing | 09:07 |
| akovi | I don't know if I can help but I'll try my best | 09:07 |
| rakhmerov | flwang1: honestly, don't know, you need to look at how stacks_update works | 09:07 |
| rakhmerov | akovi: ok, thanks | 09:07 |
| rakhmerov | akovi: the bug is here: https://bugs.launchpad.net/mistral/+bug/1634090 | 09:08 |
| openstack | Launchpad bug 1634090 in Mistral "Failed to create keystone client" [High,New] - Assigned to Fei Long Wang (flwang) | 09:08 |
| flwang1 | therve: https://github.com/openstack/python-heatclient/blob/master/heatclient/v1/stacks.py#L179 | 09:09 |
| flwang1 | rakhmerov: for a param like above, how should i pass the existing? | 09:09 |
| akovi | can you please clarify what is the heat service version? | 09:10 |
| rakhmerov | flwang1: it looks OK to me | 09:11 |
| flwang1 | rakhmerov: ok, thanks | 09:12 |
| rakhmerov | akovi: as far as I understand, flwang1 uses devstack (latest or close to latest) | 09:13 |
| akovi | we have some heat tests in the pipeline that do not fail, right? | 09:14 |
| akovi | we may need to adjust those too after this bug is solved | 09:14 |
| flwang1 | rakhmerov: pls see line https://github.com/openstack/python-heatclient/blob/master/heatclient/v1/stacks.py#L176 | 09:14 |
| flwang1 | after stack_id, it's a kwargs | 09:14 |
| flwang1 | so if i want to pass 'existing' | 09:15 |
| flwang1 | should i write the action like this stacks_update: { action: heat.stacks_update stack_id=<% $.root_stack_id %> kwargs={existing:True}} ? | 09:15 |
| flwang1 | instead of stacks_update: {action: heat.stacks_update stack_id=<% $.root_stack_id %> existing=True} | 09:15 |
| *** shardy has joined #openstack-mistral | 09:16 | |
| *** dkushwaha has quit IRC | 09:17 | |
| openstackgerrit | Fei Long Wang proposed openstack/mistral: Get service catalog from token info https://review.openstack.org/387883 | 09:28 |
| rakhmerov | akovi: I think this is not specifically related to Heat, it's a major issue when Mistral needs to use a token made of trust which is just given to Mistral (Mistral itself didn't create that trust) | 09:29 |
| flwang1 | rakhmerov: can you answer my above question? | 09:33 |
| flwang1 | should i write the action like this stacks_update: { action: heat.stacks_update stack_id=<% $.root_stack_id %> kwargs={existing:True}} ? | 09:33 |
| flwang1 | instead of stacks_update: {action: heat.stacks_update stack_id=<% $.root_stack_id %> existing=True} | 09:33 |
| rakhmerov | flwang1: give me a minute.. | 09:33 |
| flwang1 | rakhmerov: thanks | 09:33 |
| akovi | rakhmerov: we haven't played around with trusts yet | 09:36 |
| rakhmerov | flwang1: try existing=true | 09:38 |
| rakhmerov | akovi: ok | 09:38 |
| rakhmerov | flwang1: generally, just existing=true should work because params are passed as kwargs to methods | 09:39 |
| rakhmerov | the only problem is that Mistral syntax is slightly different when we use it inside a Heat template (very inconvenient) and I don't know all these differences | 09:40 |
| rakhmerov | mostly I use Mistral language directly | 09:40 |
| rakhmerov | so it might make a difference | 09:40 |
| flwang1 | rakhmerov: ok, so the case matters here? | 09:40 |
| flwang1 | True vs. true | 09:40 |
| rakhmerov | we would like to change that in the near perspective so that we could use same syntax in both cases | 09:41 |
| rakhmerov | flwang1: it's just a YAML's true | 09:41 |
| rakhmerov | yes | 09:41 |
| *** jaosorior has quit IRC | 09:42 | |
| *** jaosorior has joined #openstack-mistral | 09:42 | |
| flwang1 | rakhmerov: good, it works now. thank you very much | 09:47 |
| rakhmerov | np | 09:47 |
| *** akovi has quit IRC | 09:55 | |
| *** jtomasek is now known as jtomasek|biab | 10:40 | |
| *** chlong has quit IRC | 10:46 | |
| *** dprince has joined #openstack-mistral | 11:14 | |
| *** dprince has quit IRC | 11:14 | |
| *** shardy is now known as shardy_lunch | 11:19 | |
| openstackgerrit | pawnesh kumar proposed openstack/python-mistralclient: Fix warning when running tox -e docs https://review.openstack.org/387957 | 11:27 |
| openstackgerrit | pawnesh kumar proposed openstack/python-mistralclient: Fix warning when running tox -e docs https://review.openstack.org/387957 | 11:30 |
| openstackgerrit | pawnesh kumar proposed openstack/python-mistralclient: Fix the warning while executing "tox -e pylint". https://review.openstack.org/387963 | 11:38 |
| *** rbrady-afk is now known as rbrady | 11:51 | |
| *** jtomasek|biab is now known as jtomasek | 11:54 | |
| *** dprince has joined #openstack-mistral | 11:55 | |
| *** janki has joined #openstack-mistral | 12:17 | |
| *** shardy_lunch is now known as shardy | 12:21 | |
| *** Qiming_ has joined #openstack-mistral | 12:56 | |
| *** Qiming_ has quit IRC | 12:56 | |
| *** jaosorior is now known as jaosorior_mtg | 13:04 | |
| *** jtomasek is now known as jtomasek|afk | 13:33 | |
| *** bobh has joined #openstack-mistral | 13:34 | |
| *** bobh has quit IRC | 13:36 | |
| *** bobh has joined #openstack-mistral | 13:36 | |
| *** bobh has quit IRC | 13:36 | |
| *** jaosorior_mtg is now known as jaosorior | 13:46 | |
| *** jaosorior has quit IRC | 14:16 | |
| d0ugal | rakhmerov: Are you around? | 14:24 |
| rakhmerov | d0ugal: here | 14:34 |
| rakhmerov | d0ugal: btw, I replied to you in ML | 14:34 |
| rakhmerov | d0ugal: let's try to find a better slot | 14:35 |
| d0ugal | rakhmerov: oh, thanks - I'll take a look at that again. | 14:35 |
| rakhmerov | d0ugal: excuse me if I was a little bit offensive | 14:35 |
| rakhmerov | d0ugal: do you have something to discuss? Some questions? | 14:36 |
| d0ugal | rakhmerov: haha, no problem - I was probably a bit terse in my reply - sorry. | 14:37 |
| *** bobh has joined #openstack-mistral | 14:37 | |
| d0ugal | rakhmerov: Yeah, we have a small usability issue I wanted to discuss. | 14:37 |
| *** vishwanathj has joined #openstack-mistral | 14:37 | |
| d0ugal | rakhmerov: We use this pattern: https://github.com/openstack/tripleo-common/blob/master/workbooks/plan_management.yaml#L137-L143 | 14:37 |
| rakhmerov | sure | 14:37 |
| d0ugal | rakhmerov: Basically, if there isn't an error we continue. If there is an error we create it. | 14:38 |
| d0ugal | rakhmerov: however, that means we end up with tracebacks in our log, and people debugging often find them and think it is a problem | 14:38 |
| d0ugal | rakhmerov: so, I wonder if there is a way we could have tasks that we expect to fail, but not look scary in the logs :) | 14:38 |
| rakhmerov | :)) | 14:39 |
| rakhmerov | where exactly do you create an error? | 14:39 |
| d0ugal | rakhmerov: so the error is in the executor log | 14:40 |
| rakhmerov | yes, I mean what's the reason for this? | 14:40 |
| rakhmerov | what line in this file should I look at? | 14:40 |
| d0ugal | rakhmerov: okay, I think I explained it badly. | 14:40 |
| rakhmerov | np | 14:41 |
| d0ugal | rakhmerov: Our task uses the standard openstack actions - we do that to check if a swift container and mistral environment exist | 14:41 |
| d0ugal | rakhmerov: because if they do, we want to stop the workflow, since it will try to recreate them otherwise | 14:41 |
| *** bobh has quit IRC | 14:41 | |
| d0ugal | rakhmerov: so, we use the on-error and on-success to control the workflow. | 14:42 |
| rakhmerov | ok | 14:42 |
| rakhmerov | I see | 14:42 |
| d0ugal | rakhmerov: so the first time the container doesn't exist, but the workflow creates it. | 14:42 |
| d0ugal | Then the log contains this: http://paste.openstack.org/show/586207/ | 14:42 |
| rakhmerov | wait a sec | 14:42 |
| d0ugal | ok :) | 14:42 |
| rakhmerov | just want to clarify | 14:42 |
| rakhmerov | mistral.environments_get name=<% $.container %> | 14:42 |
| rakhmerov | this action leads to 'on-error' to trigger if env in Mistral doesn't exist? | 14:43 |
| rakhmerov | right? | 14:43 |
| d0ugal | rakhmerov: correct | 14:44 |
| rakhmerov | yeah, I see, same for swifth | 14:44 |
| d0ugal | rakhmerov: and we want that to happen sometimes, so it is fine | 14:44 |
| rakhmerov | got it | 14:44 |
| d0ugal | but it just means the logs look scary | 14:44 |
| rakhmerov | yeah, I see the problem now | 14:44 |
| d0ugal | It's only a small problem, but everyone debugging issues asks me "is this the problem?" | 14:44 |
| d0ugal | We could solve it by writing a custom action, but I don't want to do that :) | 14:45 |
| d0ugal | I wonder if there could be a flag or a way to suppress errors or specific errors. | 14:45 |
| rakhmerov | yeah, the obvious solution is just to write a wrapper around these actions | 14:45 |
| rakhmerov | it doesn't require any changes in Mistral code base | 14:45 |
| d0ugal | True | 14:45 |
| d0ugal | I guess that is the easy option | 14:45 |
| rakhmerov | no, nothing like this flag actually exists | 14:46 |
| rakhmerov | unfortunately | 14:46 |
| rakhmerov | hm... let me think | 14:46 |
| rakhmerov | the misunderstanding comes from the fact that we have two types of errors | 14:47 |
| rakhmerov | 1) Errors showing that something is wrong with Mistral itself | 14:47 |
| rakhmerov | 2) Errors coming from actions | 14:47 |
| rakhmerov | in case #1 it's totally ok to see these errors in log | 14:48 |
| rakhmerov | in case #2 it's probably ok too, BUT.. They could be probably displayed in a different way | 14:48 |
| d0ugal | Yeah, it is a tricky problem I think | 14:49 |
| rakhmerov | so that we clearly see that it's an error not related to Mistral setup | 14:49 |
| d0ugal | I don't know if Mistral should be changed for this, I am just starting to think about it | 14:49 |
| ddeja | rakhmerov: maybe in case #2 they should be seen as debug log, not error? | 14:49 |
| d0ugal | ddeja: but sometimes errors from actions are real errors | 14:49 |
| d0ugal | lol, if that makes sense | 14:49 |
| rakhmerov | ddeja: that's not a bad idea I think | 14:50 |
| ddeja | d0ugal: yes, but they are not related with how mistral operates | 14:50 |
| d0ugal | ddeja: True | 14:50 |
| d0ugal | ddeja: We already solve that by using multiple log files | 14:50 |
| ddeja | I see | 14:50 |
| rakhmerov | it's not bad in a sense that we should be using a different logging for it | 14:50 |
| d0ugal | when you last looked it all went into one, now we have an executor, engine and api | 14:50 |
| ddeja | yes | 14:51 |
| d0ugal | example: http://logs.openstack.org/44/375544/15/check/gate-tripleo-ci-centos-7-undercloud/9b50ad5/logs/var/log/mistral/ | 14:51 |
| d0ugal | and that is actually a good example | 14:51 |
| d0ugal | executor has two errors I expect, that are fine. The third one is the real issue. | 14:51 |
| rakhmerov | aha | 14:51 |
| rakhmerov | d0ugal: another topic for the design summit? :) | 14:52 |
| d0ugal | Yeah, I'll add it :) | 14:52 |
| rakhmerov | ok | 14:52 |
| rakhmerov | I don't believe we will solve it this week | 14:52 |
| d0ugal | haha, no | 14:52 |
| d0ugal | I don't have any urgency here, it would just make my life easier :-D | 14:52 |
| rakhmerov | at the summit I'll tell more thoughts on debugging and usability that are related to this too | 14:52 |
| rakhmerov | it's a part of a bigger picture | 14:53 |
| rakhmerov | yeah, me too | 14:53 |
| rakhmerov | we have lots of ideas | 14:53 |
| d0ugal | Great, thanks for the chat about it | 14:53 |
| rakhmerov | in the last month or two, I discussed a lot of them with my colleagues who use Mistral every day | 14:53 |
| rakhmerov | for example, we have an idea to add a function on API that allows to easily find all objects in ERROR state | 14:54 |
| rakhmerov | for example, your WF failed | 14:54 |
| rakhmerov | and you don't know how to investigate it | 14:54 |
| d0ugal | that would be very useful | 14:55 |
| rakhmerov | you do something like: mistral find-errors <execution_id> | 14:55 |
| d0ugal | Yeah, so Heat has a similar command | 14:55 |
| rakhmerov | and it gives all actions and tasks in ERROR in a hierarchical form | 14:55 |
| d0ugal | `openstack stack failures list $STACKNAME` | 14:55 |
| rakhmerov | yes | 14:55 |
| rakhmerov | btw, one of my colleagues is now working on a YAQL function to do that | 14:56 |
| rakhmerov | as far as I know, close to get it done | 14:56 |
| rakhmerov | but that's for a slightly different purpose | 14:57 |
| rakhmerov | to be able to see something in a WF itself | 14:57 |
| *** vishwanathj has quit IRC | 14:57 | |
| rakhmerov | ddeja: I know you're not going to the summit but I'd like to ask you to add topics that are important for you too | 14:58 |
| ddeja | rakhmerov: yeah, sure | 14:58 |
| rakhmerov | into the etherpad I created | 14:58 |
| rakhmerov | d0ugal, ddeja: this time we have only two work sessions but we will have a contributors meetup on Friday. Hopefully, we can continue there | 14:59 |
| d0ugal | rakhmerov: yeah, that would be good. I need to check what time I leave on Friday, I might not be able to stay for it all | 14:59 |
| rakhmerov | yeah, that's what I'm afraid of :) | 15:00 |
| d0ugal | rakhmerov: it would be good to meetup before the mistral sessions if you have time. | 15:00 |
| rakhmerov | people usually leave on Friday | 15:00 |
| rakhmerov | or simply too lazy :) | 15:00 |
| d0ugal | lol | 15:00 |
| rakhmerov | d0ugal: absolutely, sir | 15:00 |
| d0ugal | I don't know if you are involved in any other projects? | 15:00 |
| rakhmerov | I would be glad to meet before our official sessions | 15:01 |
| d0ugal | I will be attending some other stuff too | 15:01 |
| rakhmerov | me too, but I'm sure we'll have time | 15:01 |
| rakhmerov | my list of mandatory sessions/talks is not that huge | 15:01 |
| d0ugal | I'll probably be at Mistral/TripleO/Zaqar sessions | 15:02 |
| rakhmerov | I'll be there probably too | 15:02 |
| rakhmerov | I mean TripleO and Zaqar :) | 15:02 |
| d0ugal | oh, cool! | 15:03 |
| d0ugal | rakhmerov: I'm sure the mistral work sessions were not on friday last time I looked | 15:12 |
| d0ugal | hmm | 15:12 |
| d0ugal | That causes more conflicts for me I think, damn | 15:12 |
| *** vishwanathj has joined #openstack-mistral | 15:12 | |
| rakhmerov | d0ugal: they are not | 15:13 |
| rakhmerov | wait a sec... | 15:13 |
| rakhmerov | you scare me | 15:13 |
| rakhmerov | ooh, summit schedule web site is down.. | 15:14 |
| d0ugal | rakhmerov: http://i.imgur.com/eOUEx05.png | 15:14 |
| d0ugal | I think they were on thursday when I last looked. | 15:15 |
| rakhmerov | ooh, yeah | 15:15 |
| rakhmerov | I forgot myself | 15:15 |
| rakhmerov | is it too bad for you? | 15:15 |
| d0ugal | rakhmerov: It's not good :( | 15:16 |
| d0ugal | I'll have to see what I can do | 15:16 |
| d0ugal | because I guess we are stuck with it now | 15:16 |
| d0ugal | I had planned everything based on the previous version | 15:16 |
| rakhmerov | f...k | 15:16 |
| rakhmerov | what do you mean by previous version? | 15:16 |
| rakhmerov | there was another version of schedule? | 15:16 |
| d0ugal | rakhmerov: Yeah, I guess I looked when they were still updating it | 15:17 |
| d0ugal | so it is probably my fault | 15:17 |
| rakhmerov | no worries | 15:18 |
| rakhmerov | please see if you can do something | 15:18 |
| rakhmerov | I would love to have you on those meetings | 15:18 |
| rakhmerov | if you can't be there then we need to meet separately | 15:18 |
| rakhmerov | in any case | 15:18 |
| *** vinod_ has joined #openstack-mistral | 15:24 | |
| d0ugal | rakhmerov: Yup, we should do that anyway | 15:25 |
| d0ugal | rakhmerov: I am looking into it now. | 15:25 |
| rakhmerov | ok | 15:25 |
| vinod_ | @rakhmerov Hi | 15:26 |
| vinod_ | last week I was here asking about Mistral's ability to scale ... I was told you had done some tests on the same ... | 15:26 |
| vinod_ | so if you have any notes to share, config options to tweak, gotchas to avoid kind of stuff that would be very helpful | 15:27 |
| rakhmerov | vinod_: give a min, I'll be with you.. | 15:31 |
| vinod_ | @rakhmerov Sure. Thanks. Shall wait | 15:34 |
| *** bobh has joined #openstack-mistral | 15:38 | |
| rakhmerov | vinod_: sorry, soon :) | 15:40 |
| *** bobh has quit IRC | 15:42 | |
| rakhmerov | vinod_: ok, here | 15:44 |
| rakhmerov | so, the biggest limitation that exists now as far as scaling is that you can't use multiple engines if you run workflows that have "join" tasks | 15:44 |
| rakhmerov | that's gonna be fixed I believe in about a month | 15:44 |
| rakhmerov | executors and apis are totally ok to scale as you need | 15:45 |
| rakhmerov | all components are stateless so scaling is pretty easy | 15:46 |
| rakhmerov | what's missing though now monitoring and auto-scaling | 15:46 |
| rakhmerov | if some component dies then Mistral can't rebalance itself and, for example, spin up fresh components | 15:47 |
| rakhmerov | that's all ahead on a Roadmap | 15:47 |
| rakhmerov | but honestly, I don't believe it'll be done in Ocata cycle | 15:47 |
| rakhmerov | it's a far future (> 6 months) | 15:47 |
| rakhmerov | as far as config tweaks it's pretty simple: if you use different hosts for different components your config will be probably the same except maybe IPs if you configure them explicitly | 15:48 |
| rakhmerov | like IP for a web server | 15:49 |
| rakhmerov | otherwise you might want to split your logs | 15:49 |
| rakhmerov | in this case you need to direct logs into different files and you'll have to use either different config files or pass log config properties in your mistral startup commands | 15:50 |
| rakhmerov | as far as numbers, it's kind of hard to provide something certain because it makes sense to talk only about workflows of a certain configuration | 15:51 |
| vinod_ | yes, without a benchmark and standard environment numbers wont mean much indeed | 15:51 |
| rakhmerov | for example, even if we have 2 workflows with 100 tasks their runtime maybe very different depending on their topology | 15:51 |
| rakhmerov | but | 15:52 |
| vinod_ | can u explain what u mean by 'join' tasks? | 15:52 |
| rakhmerov | I can still provide some info | 15:52 |
| vinod_ | or point me to some url which explains it? | 15:52 |
| rakhmerov | for example, I've been experimenting with parallel tasks (simple WF where we just have a number parallel not related tasks) and in the summer such a WF with 500 parallel tasks took minutes to complete on my laptop | 15:53 |
| rakhmerov | now it takes seconds | 15:53 |
| rakhmerov | some other types of workflows too | 15:53 |
| vinod_ | ok | 15:54 |
| rakhmerov | which means that in the last few months we improved performance by ~ 2 orders of magnitude | 15:54 |
| rakhmerov | as far as how, that's a long different story | 15:54 |
| vinod_ | great improvement indeed! cheers to your team :-) | 15:54 |
| rakhmerov | trying hard | 15:54 |
| rakhmerov | ok, now on your last question | 15:54 |
| rakhmerov | 'join' task is a task that 1) merges multiple parallel routes in a WF 2) works as a distributed mutex and syncs all the incoming routes | 15:55 |
| rakhmerov | #2 means that a 'join' task will only start when all incoming routes will reach it | 15:56 |
| rakhmerov | docs: http://docs.openstack.org/developer/mistral/dsl/dsl_v2.html#join | 15:57 |
| vinod_ | ok, got that ... we might be having only a few such scenarios ... | 15:57 |
| vinod_ | At peak loads, Im expecting to have about 50 mistral template executions a minute ... how many tasks that would translate into isnt something I cannot guess right now | 15:57 |
| rakhmerov | the implementation of 'join' is pretty tricky and it causes a lot of pain with scaling | 15:57 |
| rakhmerov | vinod_: a number of executions (workflows) don't matter too much, yes | 15:58 |
| rakhmerov | more importantly, how many tasks that will be and what their topologies (configurations) are | 15:59 |
| rakhmerov | with joins, w/o joins, etc. | 15:59 |
| rakhmerov | parallelism | 15:59 |
| rakhmerov | if you don't have many joins then the Mistral overhead will be pretty small | 16:00 |
| rakhmerov | new tasks quickly gets scheduled with just a few DB queries required for that | 16:00 |
| vinod_ | ok ... would you believe that the code is optimized to use all the cores available? Im just wondering if additional cores can get me better throughput | 16:03 |
| vinod_ | im a noob to python and not so comfortable reading a lot of code yet, otherwise I would dig into the codebase than ask this question | 16:04 |
| rakhmerov | hm.. good question | 16:04 |
| rakhmerov | I see, np | 16:04 |
| rakhmerov | so | 16:04 |
| rakhmerov | Mistral API component definitely yes | 16:05 |
| rakhmerov | it can use multiple cores | 16:05 |
| rakhmerov | because it runs separate processes underneath the surface | 16:05 |
| rakhmerov | Mistral Engine and Mistral Executor themselves no | 16:06 |
| rakhmerov | but you can run many of them on same host | 16:06 |
| rakhmerov | scale them kind of locally | 16:06 |
| rakhmerov | in this case they will be using multiple cores as just being separate processes | 16:06 |
| rakhmerov | vinod_: excuse me, I have to leave now | 16:08 |
| vinod_ | i c, so even if I have the API server on a stronger vm with more cores, I will still be constrained by the engine's and executor's throughput ... but atleast I can keep the users happy with a quick response on the UI that way | 16:08 |
| rakhmerov | if you have any other questions, you're very welcome | 16:08 |
| vinod_ | thanks a lot @rakhmerov! | 16:08 |
| rakhmerov | welcome | 16:08 |
| vinod_ | we are planning to use Mistral and wanted to know it better before we decide ..ur inputs give me more confidence :-) | 16:09 |
| vinod_ | im not sure what time zone you are in. Sorry if its too late in the night already for u | 16:11 |
| vinod_ | wish u a pleasant day ahead! | 16:11 |
| *** vinod_ has quit IRC | 16:12 | |
| *** clenimar has joined #openstack-mistral | 16:13 | |
| *** jtomasek|afk is now known as jtomasek | 16:22 | |
| d0ugal | rakhmerov: BTW, I should be fine for both work sessions. Just. :) | 16:33 |
| *** jpich has quit IRC | 16:44 | |
| *** janki has quit IRC | 16:55 | |
| *** Kiall_ has quit IRC | 17:33 | |
| *** pkoniszewski has quit IRC | 17:33 | |
| *** jistr has quit IRC | 17:33 | |
| *** enykeev_ has quit IRC | 17:33 | |
| *** cargonza has quit IRC | 17:33 | |
| *** EmilienM has quit IRC | 17:33 | |
| *** Kiall has joined #openstack-mistral | 17:33 | |
| *** pkoniszewski has joined #openstack-mistral | 17:33 | |
| *** enykeev has joined #openstack-mistral | 17:34 | |
| *** jistr has joined #openstack-mistral | 17:34 | |
| *** EmilienM has joined #openstack-mistral | 17:36 | |
| *** bobh has joined #openstack-mistral | 17:39 | |
| *** cargonza has joined #openstack-mistral | 17:44 | |
| *** bobh has quit IRC | 17:44 | |
| *** dprince has quit IRC | 17:52 | |
| *** dprince has joined #openstack-mistral | 18:07 | |
| *** bobh has joined #openstack-mistral | 18:40 | |
| *** bobh has quit IRC | 18:45 | |
| *** bobh has joined #openstack-mistral | 19:57 | |
| *** flwang1 has quit IRC | 19:59 | |
| *** flwang1 has joined #openstack-mistral | 21:01 | |
| *** dprince has quit IRC | 21:28 | |
| *** flwang1 has quit IRC | 21:52 | |
| *** flwang1 has joined #openstack-mistral | 21:52 | |
| *** bobh has quit IRC | 22:15 | |
| *** bobh has joined #openstack-mistral | 23:00 | |
| kong | rakhmerov, ddeja, please review https://review.openstack.org/#/c/387757, which i think is critical for stable/newton branch | 23:01 |
| *** bobh has quit IRC | 23:19 | |
| *** bobh has joined #openstack-mistral | 23:33 | |
| *** bobh has quit IRC | 23:39 | |
| *** vishwanathj has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
