| *** jistr has quit IRC | 00:00 | |
| *** jistr has joined #openstack-mistral | 00:01 | |
| *** toddjohn has quit IRC | 00:05 | |
| *** tonytan4ever has quit IRC | 00:22 | |
| *** jkup has joined #openstack-mistral | 00:41 | |
| *** jkup has quit IRC | 00:42 | |
| *** tonytan4ever has joined #openstack-mistral | 00:48 | |
| *** cheneydc has joined #openstack-mistral | 00:56 | |
| *** cheneydc has quit IRC | 00:56 | |
| *** bobh has joined #openstack-mistral | 01:38 | |
| *** bobh has quit IRC | 01:40 | |
| *** bobh has joined #openstack-mistral | 01:43 | |
| *** bobh has quit IRC | 02:28 | |
| *** bobh has joined #openstack-mistral | 02:31 | |
| *** bobh has quit IRC | 02:43 | |
| *** toddjohn has joined #openstack-mistral | 02:47 | |
| *** toddjohn has quit IRC | 02:49 | |
| *** toddjohn has joined #openstack-mistral | 02:49 | |
| *** toddjohn has quit IRC | 02:54 | |
| *** gyee has quit IRC | 02:57 | |
| *** brian_price has quit IRC | 03:13 | |
| *** tonytan4ever has quit IRC | 03:14 | |
| *** tonytan4ever has joined #openstack-mistral | 04:15 | |
| *** tonytan4ever has quit IRC | 04:20 | |
| *** jaosorior has joined #openstack-mistral | 04:55 | |
| *** janki has joined #openstack-mistral | 05:29 | |
| *** jpeeler has quit IRC | 05:45 | |
| openstackgerrit | Renat Akhmerov proposed openstack/mistral: Fix workflow and join completion logic https://review.openstack.org/353063 | 05:45 |
|---|---|---|
| *** jpeeler has joined #openstack-mistral | 05:49 | |
| *** vishwanathj has quit IRC | 05:56 | |
| *** vishwanathj has joined #openstack-mistral | 06:28 | |
| openstackgerrit | Renat Akhmerov proposed openstack/mistral: Fix workflow and join completion logic https://review.openstack.org/353063 | 06:48 |
| *** Ravikiran_K has joined #openstack-mistral | 06:52 | |
| *** sshnaidm has quit IRC | 06:56 | |
| *** jistr has quit IRC | 07:18 | |
| *** jistr has joined #openstack-mistral | 07:25 | |
| openstackgerrit | Merged openstack/mistral: Towards non-locking model: make 'with-items' work w/o locks https://review.openstack.org/352386 | 07:26 |
| openstackgerrit | Renat Akhmerov proposed openstack/mistral: Fix workflow and join completion logic https://review.openstack.org/353063 | 07:29 |
| *** shardy has joined #openstack-mistral | 07:35 | |
| rakhmerov | ddeja, hparekh: guys, please review https://review.openstack.org/#/c/353063/ as soon as you can. It fixes the problem with 'join' and workflow completion | 07:39 |
| rakhmerov | it's a critical thing because now 'join' tests fail pretty often and this blocks the development | 07:40 |
| jaosorior | rakhmerov: hey, how's it going? Do you know if master is already usable? | 07:41 |
| *** jpich has joined #openstack-mistral | 07:43 | |
| *** kong has quit IRC | 07:50 | |
| *** nmakhotkin has joined #openstack-mistral | 08:16 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** openstackgerrit has joined #openstack-mistral | 08:18 | |
| ddeja | rakhmerov: looking | 08:20 |
| rakhmerov | thanks | 08:20 |
| ddeja | rakhmerov: from the code base it looks OK | 08:42 |
| ddeja | I can +2 it | 08:44 |
| ddeja | But I would like to have a question to your changes as a whole | 08:44 |
| rakhmerov | ddeja: yeah | 08:44 |
| *** shardy has quit IRC | 08:44 | |
| rakhmerov | sure | 08:44 |
| ddeja | becouse from reviewing particular patchsets it's not clear to me, why you are moving many things to scheduler | 08:45 |
| *** shardy has joined #openstack-mistral | 08:45 | |
| ddeja | like launching next task | 08:45 |
| ddeja | or it was there before, just not so explicitly? | 08:45 |
| rakhmerov | no, it was not there | 08:51 |
| rakhmerov | yeah, I'll explain but I need to step out for ~2 hours urgently | 08:52 |
| rakhmerov | I'll explain when I get back | 08:52 |
| ddeja | rakhmerov: OK, no problem, take your time | 08:52 |
| rakhmerov | ddeja: please also review my other patches, they are superimportant too | 09:00 |
| *** shardy is now known as shardy_mtg | 09:00 | |
| rakhmerov | thanks in advance | 09:00 |
| rakhmerov | ddeja: we already discussed one of them in details (caching thing) | 09:00 |
| ddeja | yup | 09:00 |
| *** pbelamge has joined #openstack-mistral | 09:29 | |
| *** mflobo has joined #openstack-mistral | 10:13 | |
| openstackgerrit | Merged openstack/mistral: Fix workflow and join completion logic https://review.openstack.org/353063 | 10:24 |
| *** shardy_mtg is now known as shardy | 10:25 | |
| *** mflobo has quit IRC | 10:29 | |
| *** nmakhotkin has quit IRC | 10:32 | |
| *** mflobo has joined #openstack-mistral | 10:54 | |
| *** dprince has joined #openstack-mistral | 10:59 | |
| *** mflobo has quit IRC | 11:01 | |
| *** Ravikiran_K has quit IRC | 11:06 | |
| rakhmerov | ddeja: I'm here | 11:14 |
| rakhmerov | so | 11:15 |
| ddeja | rakhmerov: o/ | 11:15 |
| rakhmerov | the reason why we're moving things to scheduler is in non-locking model | 11:15 |
| rakhmerov | previously, we just locked the whole graph and made sure that one transaction is modifying it exclusively | 11:15 |
| *** kong has joined #openstack-mistral | 11:16 | |
| rakhmerov | it didn't work well in terms of performance | 11:16 |
| ddeja | yup | 11:16 |
| rakhmerov | so we needed to remove locks | 11:16 |
| rakhmerov | question: how do we still support consistent changes changes and checks? | 11:16 |
| rakhmerov | so the approach is as follows | 11:16 |
| rakhmerov | we let actions and tasks update themselves w/o locking, which is fine | 11:17 |
| rakhmerov | example | 11:17 |
| rakhmerov | action completion | 11:17 |
| rakhmerov | action result arrives: | 11:17 |
| rakhmerov | 1) store action result and change its state | 11:18 |
| rakhmerov | 2) complete task (change it's state) | 11:18 |
| rakhmerov | 3) calculate next tasks | 11:18 |
| rakhmerov | 4) schedule next tasks | 11:18 |
| rakhmerov | so here 1) and 2) are ok to do w/o locks, right? | 11:18 |
| rakhmerov | because there's no parallel access to these objects | 11:19 |
| ddeja | yes, they are totaly independent | 11:19 |
| rakhmerov | yes | 11:19 |
| rakhmerov | I'm now talking about a simple task with one action | 11:19 |
| rakhmerov | 'with-items' is more tricky thing :) | 11:19 |
| ddeja | sure | 11:19 |
| rakhmerov | but 3) requires consistent read of several DB objects | 11:19 |
| rakhmerov | ok, even not 3), sorry | 11:20 |
| rakhmerov | 5) check workflow completion | 11:20 |
| rakhmerov | #5 requires consistent read | 11:20 |
| rakhmerov | because we need to see states of multiple tasks in order to understand that WF is done | 11:21 |
| ddeja | yup | 11:21 |
| rakhmerov | if we do this w/o locks it'll be failing for obvious reason | 11:21 |
| rakhmerov | phantom reads are possible | 11:21 |
| rakhmerov | so the solution is just to schedule WF completion check to a later time | 11:21 |
| rakhmerov | so that it runs in a separate transaction | 11:22 |
| rakhmerov | and we have a guarantee that there will be at least one TX that will see a consistent state of those multiple tasks | 11:22 |
| ddeja | yes | 11:23 |
| rakhmerov | and in order to reduce the number of such delated calls (for scheduler) we use that 'unique_key' column | 11:23 |
| rakhmerov | so that if 5 tasks scheduled this kind of check only the last one will be in DB | 11:23 |
| ddeja | cool | 11:23 |
| rakhmerov | but, as you remember it needs to be in combination with 'processing' flag not to interfere with a check that's already being processed | 11:24 |
| rakhmerov | yep | 11:24 |
| rakhmerov | the same thing for determining if 'join' task is allowed to start | 11:24 |
| rakhmerov | it requires consistent reading of multiple other tasks | 11:24 |
| rakhmerov | and the same thing with 'with-items', it requires consistent reading of multiple action states | 11:25 |
| ddeja | oh, NOW i fully understand why processing is part of a key | 11:25 |
| rakhmerov | yeah :) | 11:25 |
| rakhmerov | because we can't 'steal' tasks that Scheduler is already processing | 11:25 |
| rakhmerov | and there's one more tricky thing with 'join' tasks | 11:26 |
| rakhmerov | imagine tasks A and B joined at task C | 11:26 |
| rakhmerov | * A calculated next tasks -> C | 11:26 |
| rakhmerov | * B calculated next tasks -> C | 11:27 |
| rakhmerov | these are two separate transactions | 11:27 |
| ddeja | yes | 11:27 |
| rakhmerov | if we just allow them to schedule their next tasks we may get two instances of C | 11:27 |
| rakhmerov | even if we check in each of them that C already exists | 11:27 |
| rakhmerov | but we don't have a guarantee that after such check (but before the end of TX) C is not inserted already in parallel | 11:28 |
| *** jaosorior has quit IRC | 11:28 | |
| ddeja | exacly | 11:28 |
| rakhmerov | solution: use unique constraint for inserting such tasks (and generally objects) and use that method 'insert_or_ignore' | 11:29 |
| *** jaosorior has joined #openstack-mistral | 11:29 | |
| rakhmerov | which works atomically and prevents from duplication | 11:29 |
| ddeja | yup | 11:29 |
| rakhmerov | that's it ) | 11:29 |
| rakhmerov | so we now put more load onto Scheduler, of course, which potentially may be a problem | 11:30 |
| ddeja | ok thanks for the explenation | 11:30 |
| rakhmerov | but its functionality is much simpler and can't be tuned independently etc. | 11:30 |
| rakhmerov | and overall much less contention | 11:30 |
| ddeja | one question - how do we make sure that workflow do not complete twice? | 11:30 |
| rakhmerov | good question | 11:31 |
| rakhmerov | I'm planning to make some more changes to do it more safely | 11:31 |
| ddeja | OK | 11:31 |
| rakhmerov | but even now nothing really bad can happen as far as I can tell | 11:32 |
| rakhmerov | because | 11:32 |
| rakhmerov | if two tasks move WF to SUCCESS it's not a problem, it'll just rewrite the same result | 11:32 |
| rakhmerov | and probability is low because we have limited number of scheduler jobs (unique key again) | 11:33 |
| ddeja | sure, but what if such workflow is a subworkflow | 11:33 |
| *** jaosorior has quit IRC | 11:33 | |
| *** jaosorior has joined #openstack-mistral | 11:33 | |
| ddeja | we complete it twice and then we can have twice 'on success' in parent workflow | 11:33 |
| rakhmerov | if one task is errored that it's impossible to rewrite ERROR with SUCCESS because there will never be a state of WF graph in DB with all tasks in SUCCESS state | 11:33 |
| rakhmerov | subworkflow doesn't have any difference comparing to a regular WF | 11:34 |
| rakhmerov | it's the same kind of workflow | 11:34 |
| ddeja | yes, but I'm wondering if setting it state to success wouldn't start some other logic like calculating next tasks etc. | 11:35 |
| rakhmerov | you mean you're concerned with its result delivery to a parent WF? | 11:35 |
| ddeja | yes | 11:36 |
| ddeja | if they would be delivered twice | 11:36 |
| rakhmerov | it's OK because changing a task state in our model is idempotent | 11:36 |
| rakhmerov | it will be just ignored for the second time | 11:37 |
| ddeja | oh yes, you are right | 11:37 |
| ddeja | there is always if state is success: return | 11:37 |
| rakhmerov | yep | 11:37 |
| ddeja | OK, thanks for the explenation | 11:37 |
| rakhmerov | sure, I'm glad you're interested | 11:38 |
| ddeja | rakhmerov: well, I like to understand what I'm reviewing and going to review ;) | 11:42 |
| ddeja | thanks a lot! | 11:42 |
| rakhmerov | ddeja: ok ) | 11:42 |
| *** mflobo has joined #openstack-mistral | 12:02 | |
| *** mflobo has quit IRC | 12:07 | |
| *** dprince has quit IRC | 12:10 | |
| *** dprince has joined #openstack-mistral | 12:11 | |
| *** mflobo has joined #openstack-mistral | 12:12 | |
| openstackgerrit | Merged openstack/mistral: Towards non-locking model: remove pessimistic locks https://review.openstack.org/352895 | 12:32 |
| openstackgerrit | Renat Akhmerov proposed openstack/mistral: Slightly improve workflow trace logging https://review.openstack.org/353451 | 12:48 |
| *** rbrady has joined #openstack-mistral | 12:48 | |
| *** Ephur has joined #openstack-mistral | 13:00 | |
| *** pbelamge has quit IRC | 13:01 | |
| *** pbelamge has joined #openstack-mistral | 13:01 | |
| *** mflobo has quit IRC | 13:02 | |
| *** tonytan4ever has joined #openstack-mistral | 13:21 | |
| *** harlowja has quit IRC | 13:23 | |
| *** vishwanathj has joined #openstack-mistral | 13:30 | |
| *** bobh has joined #openstack-mistral | 13:34 | |
| *** bobh has quit IRC | 13:35 | |
| *** bobh has joined #openstack-mistral | 13:52 | |
| *** toddjohn has joined #openstack-mistral | 13:55 | |
| *** Ephur has quit IRC | 13:57 | |
| jaosorior | Hey guys | 14:04 |
| jaosorior | one question regarding mistral's keystone endpoint | 14:04 |
| jaosorior | how are you guys configuring it? | 14:04 |
| jaosorior | versioned or versionless? | 14:04 |
| *** catintheroof has joined #openstack-mistral | 14:09 | |
| *** kong has quit IRC | 14:10 | |
| *** bobh has quit IRC | 14:15 | |
| *** brian_price has joined #openstack-mistral | 14:23 | |
| *** toddjohn_ has joined #openstack-mistral | 14:25 | |
| *** toddjohn has quit IRC | 14:27 | |
| *** toddjohn has joined #openstack-mistral | 14:29 | |
| *** janki has quit IRC | 14:31 | |
| *** toddjohn_ has quit IRC | 14:33 | |
| *** rrecio has joined #openstack-mistral | 14:58 | |
| *** d0ugal has quit IRC | 15:01 | |
| *** pbelamge has quit IRC | 15:32 | |
| *** pbelamge has joined #openstack-mistral | 16:16 | |
| *** pbelamge has quit IRC | 16:18 | |
| *** tonytan_brb has joined #openstack-mistral | 16:21 | |
| *** tonytan4ever has quit IRC | 16:24 | |
| *** jaosorior has quit IRC | 16:29 | |
| *** Ephur has joined #openstack-mistral | 16:35 | |
| *** tonytan_brb is now known as tonytan4ever | 16:36 | |
| *** pbelamge has joined #openstack-mistral | 16:40 | |
| *** pbelamge has quit IRC | 16:48 | |
| *** pbelamge has joined #openstack-mistral | 16:48 | |
| *** jpich has quit IRC | 16:53 | |
| *** pbelamge has quit IRC | 17:11 | |
| *** harlowja has joined #openstack-mistral | 17:27 | |
| *** rbrady is now known as rbrady-food | 17:33 | |
| *** tonytan4ever has quit IRC | 17:35 | |
| *** pbelamge has joined #openstack-mistral | 17:47 | |
| *** toddjohn has quit IRC | 18:13 | |
| *** toddjohn has joined #openstack-mistral | 18:14 | |
| *** Ravikiran_K has joined #openstack-mistral | 18:18 | |
| *** rbrady-food is now known as rbrady | 18:32 | |
| *** catintheroof has quit IRC | 18:34 | |
| *** shardy is now known as shardy_afk | 18:35 | |
| *** tonytan4ever has joined #openstack-mistral | 18:35 | |
| *** tonytan4ever has quit IRC | 18:40 | |
| *** rbrady has quit IRC | 18:47 | |
| *** toddjohn has quit IRC | 18:58 | |
| *** toddjohn has joined #openstack-mistral | 18:59 | |
| *** rbrady has joined #openstack-mistral | 19:02 | |
| *** tonytan4ever has joined #openstack-mistral | 19:05 | |
| *** d0ugal has joined #openstack-mistral | 20:02 | |
| *** dprince has quit IRC | 20:16 | |
| *** shardy_afk has quit IRC | 20:31 | |
| *** rbrady has quit IRC | 20:42 | |
| *** shardy_afk has joined #openstack-mistral | 20:48 | |
| *** shardy_afk has quit IRC | 20:58 | |
| *** toddjohn has quit IRC | 21:23 | |
| *** Ravikiran_K has quit IRC | 21:38 | |
| *** pbelamge has quit IRC | 21:39 | |
| *** shardy has joined #openstack-mistral | 21:40 | |
| *** Ephur has quit IRC | 21:49 | |
| *** toddjohn has joined #openstack-mistral | 21:58 | |
| *** tonytan4ever has quit IRC | 21:58 | |
| *** toddjohn has quit IRC | 22:02 | |
| *** kong has joined #openstack-mistral | 22:04 | |
| *** shardy has quit IRC | 22:07 | |
| *** bobh has joined #openstack-mistral | 22:26 | |
| *** bobh has quit IRC | 22:38 | |
| *** tonytan4ever has joined #openstack-mistral | 22:59 | |
| *** rrecio has quit IRC | 23:01 | |
| *** tonytan4ever has quit IRC | 23:03 | |
| *** tonytan4ever has joined #openstack-mistral | 23:07 | |
| openstackgerrit | Merged openstack/mistral: Slightly improve workflow trace logging https://review.openstack.org/353451 | 23:21 |
| *** Ephur has joined #openstack-mistral | 23:25 | |
| *** toddjohn has joined #openstack-mistral | 23:26 | |
| *** chlong|mtg has quit IRC | 23:31 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!