Wednesday, 2016-08-10

*** jistr has quit IRC00:00
*** jistr has joined #openstack-mistral00:01
*** toddjohn has quit IRC00:05
*** tonytan4ever has quit IRC00:22
*** jkup has joined #openstack-mistral00:41
*** jkup has quit IRC00:42
*** tonytan4ever has joined #openstack-mistral00:48
*** cheneydc has joined #openstack-mistral00:56
*** cheneydc has quit IRC00:56
*** bobh has joined #openstack-mistral01:38
*** bobh has quit IRC01:40
*** bobh has joined #openstack-mistral01:43
*** bobh has quit IRC02:28
*** bobh has joined #openstack-mistral02:31
*** bobh has quit IRC02:43
*** toddjohn has joined #openstack-mistral02:47
*** toddjohn has quit IRC02:49
*** toddjohn has joined #openstack-mistral02:49
*** toddjohn has quit IRC02:54
*** gyee has quit IRC02:57
*** brian_price has quit IRC03:13
*** tonytan4ever has quit IRC03:14
*** tonytan4ever has joined #openstack-mistral04:15
*** tonytan4ever has quit IRC04:20
*** jaosorior has joined #openstack-mistral04:55
*** janki has joined #openstack-mistral05:29
*** jpeeler has quit IRC05:45
openstackgerritRenat Akhmerov proposed openstack/mistral: Fix workflow and join completion logic  https://review.openstack.org/35306305:45
*** jpeeler has joined #openstack-mistral05:49
*** vishwanathj has quit IRC05:56
*** vishwanathj has joined #openstack-mistral06:28
openstackgerritRenat Akhmerov proposed openstack/mistral: Fix workflow and join completion logic  https://review.openstack.org/35306306:48
*** Ravikiran_K has joined #openstack-mistral06:52
*** sshnaidm has quit IRC06:56
*** jistr has quit IRC07:18
*** jistr has joined #openstack-mistral07:25
openstackgerritMerged openstack/mistral: Towards non-locking model: make 'with-items' work w/o locks  https://review.openstack.org/35238607:26
openstackgerritRenat Akhmerov proposed openstack/mistral: Fix workflow and join completion logic  https://review.openstack.org/35306307:29
*** shardy has joined #openstack-mistral07:35
rakhmerovddeja, hparekh: guys, please review https://review.openstack.org/#/c/353063/ as soon as you can. It fixes the problem with 'join' and workflow completion07:39
rakhmerovit's a critical thing because now 'join' tests fail pretty often and this blocks the development07:40
jaosoriorrakhmerov: hey, how's it going? Do you know if master is already usable?07:41
*** jpich has joined #openstack-mistral07:43
*** kong has quit IRC07:50
*** nmakhotkin has joined #openstack-mistral08:16
*** openstackgerrit has quit IRC08:18
*** openstackgerrit has joined #openstack-mistral08:18
ddejarakhmerov: looking08:20
rakhmerovthanks08:20
ddejarakhmerov: from the code base it looks OK08:42
ddejaI can +2 it08:44
ddejaBut I would like to have a question to your changes as a whole08:44
rakhmerovddeja: yeah08:44
*** shardy has quit IRC08:44
rakhmerovsure08:44
ddejabecouse from reviewing particular patchsets it's not clear to me, why you are moving many things to scheduler08:45
*** shardy has joined #openstack-mistral08:45
ddejalike launching next task08:45
ddejaor it was there before, just not so explicitly?08:45
rakhmerovno, it was not there08:51
rakhmerovyeah, I'll explain but I need to step out for ~2 hours urgently08:52
rakhmerovI'll explain when I get back08:52
ddejarakhmerov: OK, no problem, take your time08:52
rakhmerovddeja: please also review my other patches, they are superimportant too09:00
*** shardy is now known as shardy_mtg09:00
rakhmerovthanks in advance09:00
rakhmerovddeja: we already discussed one of them in details (caching thing)09:00
ddejayup09:00
*** pbelamge has joined #openstack-mistral09:29
*** mflobo has joined #openstack-mistral10:13
openstackgerritMerged openstack/mistral: Fix workflow and join completion logic  https://review.openstack.org/35306310:24
*** shardy_mtg is now known as shardy10:25
*** mflobo has quit IRC10:29
*** nmakhotkin has quit IRC10:32
*** mflobo has joined #openstack-mistral10:54
*** dprince has joined #openstack-mistral10:59
*** mflobo has quit IRC11:01
*** Ravikiran_K has quit IRC11:06
rakhmerovddeja: I'm here11:14
rakhmerovso11:15
ddejarakhmerov: o/11:15
rakhmerovthe reason why we're moving things to scheduler is in non-locking model11:15
rakhmerovpreviously, we just locked the whole graph and made sure that one transaction is modifying it exclusively11:15
*** kong has joined #openstack-mistral11:16
rakhmerovit didn't work well in terms of performance11:16
ddejayup11:16
rakhmerovso we needed to remove locks11:16
rakhmerovquestion: how do we still support consistent changes changes and checks?11:16
rakhmerovso the approach is as follows11:16
rakhmerovwe let actions and tasks update themselves w/o locking, which is fine11:17
rakhmerovexample11:17
rakhmerovaction completion11:17
rakhmerovaction result arrives:11:17
rakhmerov1) store action result and change its state11:18
rakhmerov2) complete task (change it's state)11:18
rakhmerov3) calculate next tasks11:18
rakhmerov4) schedule next tasks11:18
rakhmerovso here 1) and 2) are ok to do w/o locks, right?11:18
rakhmerovbecause there's no parallel access to these objects11:19
ddejayes, they are totaly independent11:19
rakhmerovyes11:19
rakhmerovI'm now talking about a simple task with one action11:19
rakhmerov'with-items' is more tricky thing :)11:19
ddejasure11:19
rakhmerovbut 3) requires consistent read of several DB objects11:19
rakhmerovok, even not 3), sorry11:20
rakhmerov5) check workflow completion11:20
rakhmerov#5 requires consistent read11:20
rakhmerovbecause we need to see states of multiple tasks in order to understand that WF is done11:21
ddejayup11:21
rakhmerovif we do this w/o locks it'll be failing for obvious reason11:21
rakhmerovphantom reads are possible11:21
rakhmerovso the solution is just to schedule WF completion check to a later time11:21
rakhmerovso that it runs in a separate transaction11:22
rakhmerovand we have a guarantee that there will be at least one TX that will see a consistent state of those multiple tasks11:22
ddejayes11:23
rakhmerovand in order to reduce the number of such delated calls (for scheduler) we use that 'unique_key' column11:23
rakhmerovso that if 5 tasks scheduled this kind of check only the last one will be in DB11:23
ddejacool11:23
rakhmerovbut, as you remember it needs to be in combination with 'processing' flag not to interfere with a check that's already being processed11:24
rakhmerovyep11:24
rakhmerovthe same thing for determining if 'join' task is allowed to start11:24
rakhmerovit requires consistent reading of multiple other tasks11:24
rakhmerovand the same thing with 'with-items', it requires consistent reading of multiple action states11:25
ddejaoh, NOW i fully understand why processing is part of a key11:25
rakhmerovyeah :)11:25
rakhmerovbecause we can't 'steal' tasks that Scheduler is already processing11:25
rakhmerovand there's one more tricky thing with 'join' tasks11:26
rakhmerovimagine tasks A and B joined at task C11:26
rakhmerov* A calculated next tasks -> C11:26
rakhmerov* B calculated next tasks -> C11:27
rakhmerovthese are two separate transactions11:27
ddejayes11:27
rakhmerovif we just allow them to schedule their next tasks we may get two instances of C11:27
rakhmeroveven if we check in each of them that C already exists11:27
rakhmerovbut we don't have a guarantee that after such check (but before the end of TX) C is not inserted already in parallel11:28
*** jaosorior has quit IRC11:28
ddejaexacly11:28
rakhmerovsolution: use unique constraint for inserting such tasks (and generally objects) and use that method 'insert_or_ignore'11:29
*** jaosorior has joined #openstack-mistral11:29
rakhmerovwhich works atomically and prevents from duplication11:29
ddejayup11:29
rakhmerovthat's it )11:29
rakhmerovso we now put more load onto Scheduler, of course, which potentially may be a problem11:30
ddejaok thanks for the explenation11:30
rakhmerovbut its functionality is much simpler and can't be tuned independently etc.11:30
rakhmerovand overall much less contention11:30
ddejaone question - how do we make sure that workflow do not complete twice?11:30
rakhmerovgood question11:31
rakhmerovI'm planning to make some more changes to do it more safely11:31
ddejaOK11:31
rakhmerovbut even now nothing really bad can happen as far as I can tell11:32
rakhmerovbecause11:32
rakhmerovif two tasks move WF to SUCCESS it's not a problem, it'll just rewrite the same result11:32
rakhmerovand probability is low because we have limited number of scheduler jobs (unique key again)11:33
ddejasure, but what if such workflow is a subworkflow11:33
*** jaosorior has quit IRC11:33
*** jaosorior has joined #openstack-mistral11:33
ddejawe complete it twice and then we can have twice 'on success' in parent workflow11:33
rakhmerovif one task is errored that it's impossible to rewrite ERROR with SUCCESS because there will never be a state of WF graph in DB with all tasks in SUCCESS state11:33
rakhmerovsubworkflow doesn't have any difference comparing to a regular WF11:34
rakhmerovit's the same kind of workflow11:34
ddejayes, but I'm wondering if setting it state to success wouldn't start some other logic like calculating next tasks etc.11:35
rakhmerovyou mean you're concerned with its result delivery to a parent WF?11:35
ddejayes11:36
ddejaif they would be delivered twice11:36
rakhmerovit's OK because changing a task state in our model is idempotent11:36
rakhmerovit will be just ignored for the second time11:37
ddejaoh yes, you are right11:37
ddejathere is always if state is success: return11:37
rakhmerovyep11:37
ddejaOK, thanks for the explenation11:37
rakhmerovsure, I'm glad you're interested11:38
ddejarakhmerov: well, I like to understand what I'm reviewing and going to review ;)11:42
ddejathanks a lot!11:42
rakhmerovddeja: ok )11:42
*** mflobo has joined #openstack-mistral12:02
*** mflobo has quit IRC12:07
*** dprince has quit IRC12:10
*** dprince has joined #openstack-mistral12:11
*** mflobo has joined #openstack-mistral12:12
openstackgerritMerged openstack/mistral: Towards non-locking model: remove pessimistic locks  https://review.openstack.org/35289512:32
openstackgerritRenat Akhmerov proposed openstack/mistral: Slightly improve workflow trace logging  https://review.openstack.org/35345112:48
*** rbrady has joined #openstack-mistral12:48
*** Ephur has joined #openstack-mistral13:00
*** pbelamge has quit IRC13:01
*** pbelamge has joined #openstack-mistral13:01
*** mflobo has quit IRC13:02
*** tonytan4ever has joined #openstack-mistral13:21
*** harlowja has quit IRC13:23
*** vishwanathj has joined #openstack-mistral13:30
*** bobh has joined #openstack-mistral13:34
*** bobh has quit IRC13:35
*** bobh has joined #openstack-mistral13:52
*** toddjohn has joined #openstack-mistral13:55
*** Ephur has quit IRC13:57
jaosoriorHey guys14:04
jaosoriorone question regarding mistral's keystone endpoint14:04
jaosoriorhow are you guys configuring it?14:04
jaosoriorversioned or versionless?14:04
*** catintheroof has joined #openstack-mistral14:09
*** kong has quit IRC14:10
*** bobh has quit IRC14:15
*** brian_price has joined #openstack-mistral14:23
*** toddjohn_ has joined #openstack-mistral14:25
*** toddjohn has quit IRC14:27
*** toddjohn has joined #openstack-mistral14:29
*** janki has quit IRC14:31
*** toddjohn_ has quit IRC14:33
*** rrecio has joined #openstack-mistral14:58
*** d0ugal has quit IRC15:01
*** pbelamge has quit IRC15:32
*** pbelamge has joined #openstack-mistral16:16
*** pbelamge has quit IRC16:18
*** tonytan_brb has joined #openstack-mistral16:21
*** tonytan4ever has quit IRC16:24
*** jaosorior has quit IRC16:29
*** Ephur has joined #openstack-mistral16:35
*** tonytan_brb is now known as tonytan4ever16:36
*** pbelamge has joined #openstack-mistral16:40
*** pbelamge has quit IRC16:48
*** pbelamge has joined #openstack-mistral16:48
*** jpich has quit IRC16:53
*** pbelamge has quit IRC17:11
*** harlowja has joined #openstack-mistral17:27
*** rbrady is now known as rbrady-food17:33
*** tonytan4ever has quit IRC17:35
*** pbelamge has joined #openstack-mistral17:47
*** toddjohn has quit IRC18:13
*** toddjohn has joined #openstack-mistral18:14
*** Ravikiran_K has joined #openstack-mistral18:18
*** rbrady-food is now known as rbrady18:32
*** catintheroof has quit IRC18:34
*** shardy is now known as shardy_afk18:35
*** tonytan4ever has joined #openstack-mistral18:35
*** tonytan4ever has quit IRC18:40
*** rbrady has quit IRC18:47
*** toddjohn has quit IRC18:58
*** toddjohn has joined #openstack-mistral18:59
*** rbrady has joined #openstack-mistral19:02
*** tonytan4ever has joined #openstack-mistral19:05
*** d0ugal has joined #openstack-mistral20:02
*** dprince has quit IRC20:16
*** shardy_afk has quit IRC20:31
*** rbrady has quit IRC20:42
*** shardy_afk has joined #openstack-mistral20:48
*** shardy_afk has quit IRC20:58
*** toddjohn has quit IRC21:23
*** Ravikiran_K has quit IRC21:38
*** pbelamge has quit IRC21:39
*** shardy has joined #openstack-mistral21:40
*** Ephur has quit IRC21:49
*** toddjohn has joined #openstack-mistral21:58
*** tonytan4ever has quit IRC21:58
*** toddjohn has quit IRC22:02
*** kong has joined #openstack-mistral22:04
*** shardy has quit IRC22:07
*** bobh has joined #openstack-mistral22:26
*** bobh has quit IRC22:38
*** tonytan4ever has joined #openstack-mistral22:59
*** rrecio has quit IRC23:01
*** tonytan4ever has quit IRC23:03
*** tonytan4ever has joined #openstack-mistral23:07
openstackgerritMerged openstack/mistral: Slightly improve workflow trace logging  https://review.openstack.org/35345123:21
*** Ephur has joined #openstack-mistral23:25
*** toddjohn has joined #openstack-mistral23:26
*** chlong|mtg has quit IRC23:31

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!