Friday, 2016-07-22

*** toddjohn has quit IRC01:02
*** cheneydc has joined #openstack-mistral01:03
*** cheneydc has quit IRC01:04
*** bobh has joined #openstack-mistral01:11
*** FL1SK has quit IRC01:27
*** bobh has quit IRC01:28
*** bobh has joined #openstack-mistral02:50
*** rakhmero_ has joined #openstack-mistral02:57
*** bobh has quit IRC03:06
*** rakhmero_ has quit IRC03:11
openstackgerritLingxian Kong proposed openstack/mistral: Add event engine service  https://review.openstack.org/32050003:47
openstackgerritLingxian Kong proposed openstack/mistral: Add functional tests for event engine functions  https://review.openstack.org/33646303:47
openstackgerritLingxian Kong proposed openstack/mistral: Add event trigger REST API  https://review.openstack.org/32050903:47
*** tonytan_brb has joined #openstack-mistral03:57
*** tonytan4ever has quit IRC03:59
*** FL1SK has joined #openstack-mistral04:05
*** rakhmero_ has joined #openstack-mistral04:24
*** d0ugal has quit IRC04:34
*** d0ugal has joined #openstack-mistral04:36
*** harlowja has joined #openstack-mistral04:57
hparekhrakhmerov, hi05:24
*** rakhmero_ has quit IRC05:32
*** harlowja has quit IRC05:38
openstackgerrithardik proposed openstack/mistral: Added 'pip install -r requirements.txt' instruction  https://review.openstack.org/34534705:41
openstackgerrithardik proposed openstack/python-mistralclient: Added 'pip install -r requirements.txt' instruction  https://review.openstack.org/34535105:45
*** apsarshaik has quit IRC06:10
*** apsarshaik has joined #openstack-mistral06:11
*** cheneydc has joined #openstack-mistral06:17
*** cheneydc has quit IRC06:25
*** d0ugal has quit IRC07:20
*** _achatterjee_ has joined #openstack-mistral07:21
*** _achatterjee_ has left #openstack-mistral07:22
*** apsarshaik has quit IRC07:31
*** rakhmero_ has joined #openstack-mistral07:34
*** Ravikiran_K has joined #openstack-mistral07:40
*** jpich has joined #openstack-mistral07:49
openstackgerritMerged openstack/mistral: Added 'pip install -r requirements.txt' instruction  https://review.openstack.org/34534707:49
*** rakhmero_ has quit IRC07:49
rakhmerovhparekh: hi07:50
rakhmerovhparekh, akuznetsova, nmakhotkin, kong: guys, please review https://review.openstack.org/#/c/344167/, https://review.openstack.org/#/c/344228, https://review.openstack.org/#/c/344569 and https://review.openstack.org/#/c/345295/07:52
*** d0ugal has joined #openstack-mistral07:52
*** vgnbkr has quit IRC08:11
*** vgnbkr has joined #openstack-mistral08:11
*** openstackgerrit has quit IRC08:11
*** shardy has joined #openstack-mistral08:13
*** openstackgerrit has joined #openstack-mistral08:14
*** d0ugal has quit IRC08:30
openstackgerritMerged openstack/python-mistralclient: Added 'pip install -r requirements.txt' instruction  https://review.openstack.org/34535108:32
rakhmerovddeja: hi, here?08:34
ddejarakhmerov: hi08:34
rakhmerovddeja: remember you sent a patch last about the last fall that proposed a non-locking architecture for Mistral engine?08:35
rakhmerovI can't find that patch for some reason, do have a link by any chance?08:35
rakhmerovdo you..08:35
ddejarakhmerov: you mean this one? https://review.openstack.org/#/c/343791/08:35
rakhmerovI'd like to examine this this idea again08:35
ddejawait, the last fall?08:36
rakhmerovno, it was long time ago08:36
rakhmerovyes08:36
ddejayou mean, like a 8 months ago?08:36
rakhmerovit was Nov or Dec I think08:36
rakhmerovyes08:36
rakhmerov:)08:36
rakhmerovyou proposed to get rid of locking in engine at all08:36
ddejaI don't remember to send any patches for this08:37
ddejait was rather discussed idea08:37
ddejabut without any implementation08:37
rakhmerovhm.. I though you did08:37
rakhmerovor maybe it was Roman's patch08:37
ddejalet me double check08:37
rakhmerovyeah, please08:37
ddeja_gryf: ^^08:37
rakhmerovI'm now working on improving Mistral performance and made sure that our heavy locking model doesn't perform well enough08:38
rakhmerovin fact, it's very very slow08:38
ddejarakhmerov: My only patch from last fall to mistral is this: https://review.openstack.org/#/c/343791/08:39
ddejasorry, wrong link08:39
ddejathis https://review.openstack.org/#/c/251370/08:39
rakhmerovI run 50 parallel tasks then joined into one branch and my cluster gets broken because of DB deadlocks08:39
ddejanot good08:39
rakhmerovnot good at all!08:40
rakhmerovI'm observing a huge contention on executions table08:40
rakhmerovsome queries take > 1 min to complete08:40
_gryfI don't remeber any patch by me on that area08:42
_gryfI remeber there was a discussion08:43
rakhmerovddeja: yeah, it seems that patch )08:43
ddejaOK, good :)08:43
rakhmerovbut ok, it's a little different from what settled in my mind )08:44
rakhmerovok, thanks anyway08:44
ddejano problem08:45
rakhmerovddeja: btw, it's still interesting to me, could you please comment on something?08:45
rakhmerovhttps://review.openstack.org/#/c/251370/3/mistral/engine/task_handler.py08:45
rakhmerovwhat did you try to do with these db_api.update_task_where() calls?08:46
rakhmerovkind of optimistic locking?08:46
rakhmerovbecause in default_engine.py you actually removed locking (that's what settled in my mind :) )08:46
ddejarakhmerov: I removed locking becouse it was not working08:47
ddejaI don't know why it wasn't working at that moment, but it was not blocking one transaction from another08:48
rakhmerovooh, yes08:48
rakhmerovit was because of the wrong transaction isolation08:48
rakhmerovbut what about update_task_where()? What was the idea?08:48
ddejaThe engine as a whole was still working thanks to this https://review.openstack.org/#/c/251370/3/mistral/db/v2/sqlalchemy/api.py08:48
*** dulek has joined #openstack-mistral08:49
* dulek to the rescue!08:49
ddejaIt was an idea that dulek proposed08:49
ddeja(he works in cinder and they have similar problem)08:50
dulekOkay, let my try to find quite cool tests done by gegulieo…08:51
dulekHere you go: https://github.com/Akrog/test-cinder-atomic-states08:52
rakhmerovdulek: hi08:52
rakhmerovok08:52
rakhmerovddeja, dulek: problem similar to what?08:53
rakhmerovheavy locking?08:53
dulekYup, we were looking into making state changes of volumes atomic.,08:53
dulekFor that we've needed some kind of locking to make sure only one instance of a service can modify the row.08:53
rakhmerovyes08:54
dulekLink show the results - we've evaluated SELECT FOR UPDATE, UPDATE WHERE and different Tooz locks with different backends.08:54
dulekUPDATE WHERE was based on something like:08:55
rakhmerovin case of Mistral it's pretty hairy (dunno about Cinder details) to have optimistic locks because the whole graph of DB objects (workflow and its tasks and actions) should be accessed exclusively by one thread08:55
rakhmerovotherwise data consistency is lost08:55
dulekUPDATE volumes SET status="deleting" WHERE id=<id> AND status!="DELETING";08:55
rakhmerovyeah, it's an optimistic locking08:56
ddejarakhmerov: and it was similar in my patch ^08:56
rakhmerovone subtype of it08:56
rakhmerovddeja: yeah, I see now, ok08:56
rakhmerovit's helpful08:56
rakhmerovalthough for Mistral it wouldn't have worked08:56
rakhmerovin general cas08:56
dulekrakhmerov: Ah, so the problem is that you would need to lock multiple tables, and this locks only one?08:56
rakhmerovcase08:57
rakhmerovdulek: yes, exactly08:57
rakhmerovimagine the graph: any change to the graph requires reading the whole graph in general case08:57
rakhmerovconsistenly08:57
ddejarakhmerov, dulek: It's rather multiple rows in one table, i think08:57
rakhmerovstrong consistency is required08:58
dulekSo maybe Tooz? You may be interested in tests done by Neutron guys: http://lists.openstack.org/pipermail/openstack-dev/2016-July/099752.html08:58
rakhmerovddeja: in fact, yes, but it's not so important08:58
rakhmerovthe key thing here: we need to lock graph of objects08:58
rakhmerovas long as indexes are set properly DB won't lock the whole table so having these objects in one table is the same as having them in different tables08:59
rakhmerovdulek: thanks, I'll look at it08:59
rakhmerovddeja: btw, moving different kinds of executions into different tables will probably help but theoretically not too much09:01
rakhmerovwe already thought about making this change09:01
rakhmerovI'll need to check this option too, anyway09:02
rakhmerovI may be easily wrong09:02
*** d0ugal has joined #openstack-mistral09:22
*** toddjohn has joined #openstack-mistral09:39
*** toddjohn has quit IRC09:43
*** fan has joined #openstack-mistral09:50
fanHi, I09:51
fan I use std.http to access one website, but met following issue:09:51
fan2016-07-21 22:38:28.135 2264 INFO mistral.engine.rpc [-] Received RPC request 'run_action'[rpc_ctx=MistralContext {u'project_name': None, u'user_id': None, u'roles': [u''], u'auth_token': None, u'is_trust_scoped': False, u'service_catalog': None, u'project_id': None, u'user_name': None}, action_ex_id=d091df28-5092-4e2b-aaca-061ffc3a9eb0, action_class=mistral.actions.std_actions.HTTPAction, attributes={}, params={u'body': Non09:51
fan  File "/usr/lib/python2.7/dist-packages/oslo_utils/encodeutils.py", line 49, in safe_decode     return text.decode(incoming, errors)   File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode     return codecs.utf_8_decode(input, errors, True) TypeError: don't know how to handle UnicodeDecodeError in error callback09:51
fanWho knows how to fix it?09:52
ddejaHi fan09:53
fanyes.09:54
ddeja1) next time please update traceback somewhere else, like https://etherpad.openstack.org/, it would be easier to read. 2) please try std.mistral_http instead and see if it helps09:54
ddejawrong link, it should be http://paste.openstack.org/ :)09:56
openstackgerritLingxian Kong proposed openstack/mistral: Add event engine service  https://review.openstack.org/32050010:07
openstackgerritLingxian Kong proposed openstack/mistral: Add functional tests for event engine functions  https://review.openstack.org/33646310:07
openstackgerritLingxian Kong proposed openstack/mistral: Add event trigger REST API  https://review.openstack.org/32050910:07
konghparekh: seems something wrong in server side https://review.openstack.org/#/c/344052/10:09
konghah, indeed, http://logs.openstack.org/95/345295/1/check/gate-mistral-devstack-dsvm/7106935/console.html10:10
kongrakhmerov seems related to your patches10:10
kongpagination query is affected10:10
hparekhkong, yes I think so10:10
rakhmerovwhat's up?10:11
kongrakhmerov: http://logs.openstack.org/95/345295/1/check/gate-mistral-devstack-dsvm/7106935/console.html10:11
rakhmerovwhat should I look at?10:12
rakhmerovyes, I see but my patches are not merged yet into master10:13
konghttp://paste.openstack.org/show/539682/10:13
*** fan has quit IRC10:13
kongrakhmerov: ooh, yeah10:13
rakhmerovand they don't change the behaviour, they just remove a warning10:13
rakhmerovit clearly says "Client-side error" :)10:14
* kong is looking10:14
rakhmerovso lengths of two arrays are different, I don't know why, needs to be investigated10:15
kongbecause I also see this error in your patch: https://review.openstack.org/#/c/345295/10:15
rakhmerovkong, this means that it's not related with Hardik's and my patches10:16
rakhmerovit's something else10:17
kongyes10:17
rakhmerovthat's already merged10:17
rakhmerovI guess there were some changes in server side in API after which proper client side changes were not made10:17
rakhmerovyou need to look at recently merged patches and find the first one where it crashed10:18
rakhmerovbut why am I telling you.. you know yourself..10:18
rakhmerov:)10:18
kong:-)10:19
* kong is trying to digest every patch merged these days, to keep up with all you guys.10:20
rakhmerov:)))10:22
rakhmerovkong: how do you do this? How do you set status in IRC?10:23
kongslash+me10:23
kong'/me is going to do what....'10:23
kongget rid of tje quote10:24
kongrakhmerov: have a try10:24
* rakhmerov wishes kong good luck with catching up and is hoping that kong wants to take more work..10:25
rakhmerov:)10:25
* kong teaches renat something, so renat owes kong a beer10:25
rakhmerovhaha :)) Sure10:27
* rakhmerov if kong decides to show up in Barcelona )10:27
kongrakhmerov: I can't go for summit neither in Barcelona, unfortunately10:27
rakhmerovooh, that's bad10:27
kongconflict with my plan back to china10:28
rakhmerovwhaaat?10:29
rakhmerovyou're moving back to China?10:29
kongi mean, my vacation10:29
kongmy parents-in-law will go back China that itme10:30
rakhmerovooh, just for vacation10:30
rakhmerovI see10:30
rakhmerovok10:30
kongI have to go with them10:30
rakhmerovyeah10:30
kongrakhmerov: I left a comment in your patch https://review.openstack.org/#/c/34529510:38
kongrakhmerov: for more detail: http://paste.openstack.org/show/539725/10:39
rakhmerovooh! I see10:39
rakhmerovyeah, I understand10:39
rakhmerovno need to explain10:39
kong:-)10:39
rakhmerovwill fix it soon10:39
rakhmerovthanks!10:39
kongI believe you know that10:39
rakhmerovbut why does Hardik's patch fail?10:39
rakhmerovit should take mistral server from master10:39
kongrakhmerov: no idea, maybe another reason10:39
rakhmerovkong: wait a minute..10:40
rakhmerovwhere exactly does this error happen?10:40
rakhmerovin oslo_db?10:40
rakhmerovcan you show me exact place?10:41
kong File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/utils.py", line 119, in paginate_query10:41
kongrakhmerov: btw, I have tested the event trigger function in my real environment, yes, real rabbitmq10:41
kongit's a little bit tricky to add some complex functional tests because it need access to rabbitmq10:42
rakhmerovyeah, I see10:43
rakhmerovok10:43
kongexcept one day, we can send some notifications from inside mistral10:44
rakhmerovone day?10:47
rakhmerovkong: btw, can you please review https://review.openstack.org/#/c/344167/ ?10:47
rakhmerovit's the only patch left in this series..10:48
kongrakhmerov: i mean, some day in future :-)10:48
rakhmerovok )10:48
kongrakhmerov: the patch is good to me. do you plan to add some functional tests10:49
rakhmerovkong: yes, this sequence of patches is not finished yet10:49
rakhmerovI'm not closing BP10:49
kongrakhmerov: ok10:49
rakhmerovI just had to switch to something else for a couple of weeks I think10:49
kongvacation?10:50
*** d0ugal has quit IRC10:54
rakhmerovno, I'm now committed to improve Mistral performance10:55
rakhmerovit works really slowly on some cases10:55
kongrakhmerov: hah, I know10:55
rakhmerovsimple cases actually )10:55
openstackgerritMerged openstack/mistral: Add the new endpoint /v2/tasks/<id>/workflow_executions  https://review.openstack.org/34416710:57
kongrakhmerov: all my patches are green, could you please have a look?10:57
openstackgerritMerged openstack/mistral: Move REST resources action, action execution and task to resources.py  https://review.openstack.org/34422810:57
kongooh, shit, maybe I have to rebase again.10:58
kongfor resources are all leave in a single file, including event_trigger I added10:58
rakhmerovkong: sure10:59
rakhmerovI will look at them, don't worry10:59
kongrakhmerov: just want to forget them, and switch to something else in mistral10:59
openstackgerritMerged openstack/mistral: Move the remainder of REST resources to resources.py  https://review.openstack.org/34456911:00
rakhmerovkong: you can switch now while we're reviewing11:01
rakhmerovnot a problem11:01
kongrakhmerov: I don't like rebasing over and over again :-)11:02
rakhmerovyeah, I know11:02
rakhmerovok, I promise to review soon11:02
kongi know these patches indeed are huge. add additional process, additional service api, etc.11:02
* kong have to look after the baby for a while11:03
rakhmerovkong, akuznetsova: just replied on https://review.openstack.org/#/c/344748/111:06
openstackgerritRenat Akhmerov proposed openstack/mistral: Error handling test: error in 'publish' for a task with 'on-error'  https://review.openstack.org/34474811:07
openstackgerritRenat Akhmerov proposed openstack/mistral: Get rid of oslo_db warning about "id" not being in "sort_keys"  https://review.openstack.org/34529511:07
openstackgerritLingxian Kong proposed openstack/mistral: Add event engine service  https://review.openstack.org/32050011:28
openstackgerritLingxian Kong proposed openstack/mistral: Add functional tests for event engine functions  https://review.openstack.org/33646311:28
openstackgerritLingxian Kong proposed openstack/mistral: Add event trigger REST API  https://review.openstack.org/32050911:28
openstackgerritRenat Akhmerov proposed openstack/mistral: Remove task result for collection REST requests  https://review.openstack.org/34598511:37
rakhmerovakuznetsova, kong, hparekh, ddeja: very nasty bug causing a lot of pain - https://review.openstack.org/#/c/345985/11:40
*** clenimar has quit IRC11:44
*** clenimar has joined #openstack-mistral11:45
openstackgerritRenat Akhmerov proposed openstack/mistral: Remove task result for collection REST requests  https://review.openstack.org/34598511:57
openstackgerritRenat Akhmerov proposed openstack/mistral: Get rid of oslo_db warning about "id" not being in "sort_keys"  https://review.openstack.org/34529511:57
*** dprince has joined #openstack-mistral11:58
rakhmerovkong: https://review.openstack.org/#/c/345295/ is fixed now, functional tests pass12:32
rakhmerovI had to fix them a little bit12:33
openstackgerritDawid Deja proposed openstack/mistral: Use 'rpc_response_timeout' in kombu driver  https://review.openstack.org/34600512:35
*** bobh has joined #openstack-mistral12:41
*** toddjohn has joined #openstack-mistral12:41
*** d0ugal has joined #openstack-mistral12:48
*** bobh has quit IRC13:00
*** shardy is now known as shardy_mtg13:01
*** krotscheck is now known as lolscheck13:12
*** lolscheck is now known as krotscheck13:31
*** Ravikiran_K has quit IRC13:33
*** d0ugal has quit IRC13:33
*** shardy_mtg has quit IRC13:34
*** shardy_mtg has joined #openstack-mistral13:36
*** shardy_mtg is now known as shardy13:36
*** ramishra has quit IRC13:51
*** ramishra has joined #openstack-mistral13:52
*** jtomasek has quit IRC14:14
*** clenimar has quit IRC14:21
*** Kiall has joined #openstack-mistral14:30
*** jtomasek has joined #openstack-mistral14:35
*** rrecio has joined #openstack-mistral14:52
*** rrecio_ has joined #openstack-mistral14:53
*** rrecio has quit IRC14:56
*** clenimar has joined #openstack-mistral15:06
*** jistr is now known as jistr|mtg15:09
*** dprince has quit IRC15:15
*** jistr|mtg is now known as jistr15:17
*** stevebaker has quit IRC15:39
*** krotscheck has quit IRC15:39
*** apsarshaik has joined #openstack-mistral15:40
*** apsarshaik has quit IRC15:40
*** apsarshaik has joined #openstack-mistral15:41
*** krotscheck has joined #openstack-mistral15:47
*** stevebaker has joined #openstack-mistral15:47
*** dprince has joined #openstack-mistral15:47
*** tonytan_brb is now known as tonytan4ever16:13
*** dprince has quit IRC16:15
*** jistr is now known as jistr|off16:16
*** jistr|off is now known as jistr16:17
*** apsarshaik has quit IRC16:25
*** apsarshaik has joined #openstack-mistral16:27
*** apsarshaik has quit IRC16:30
*** apsarshaik has joined #openstack-mistral16:30
*** dprince has joined #openstack-mistral16:49
*** toddjohn has quit IRC16:54
*** jpich has quit IRC16:55
*** apsarshaik has quit IRC17:09
*** apsarshaik has joined #openstack-mistral17:16
*** toddjohn has joined #openstack-mistral17:54
*** toddjohn has quit IRC17:59
*** toddjohn has joined #openstack-mistral18:08
*** apsarshaik has quit IRC18:09
*** toddjohn has quit IRC18:29
*** toddjohn has joined #openstack-mistral18:29
*** toddjohn has quit IRC18:30
*** toddjohn has joined #openstack-mistral18:31
*** toddjohn has quit IRC18:35
*** toddjohn has joined #openstack-mistral18:37
*** toddjohn has quit IRC18:41
*** gyee has joined #openstack-mistral18:46
*** gyee has quit IRC18:50
*** gyee has joined #openstack-mistral19:01
*** gyee has quit IRC19:25
*** toddjohn has joined #openstack-mistral19:30
*** toddjohn has quit IRC19:45
*** toddjohn has joined #openstack-mistral19:46
*** toddjohn has quit IRC19:54
*** toddjohn has joined #openstack-mistral20:18
*** toddjohn has quit IRC20:20
*** clenimar has quit IRC20:25
*** bobh has joined #openstack-mistral20:28
*** bobh has quit IRC20:31
*** dprince has quit IRC21:26
*** shardy has quit IRC21:50
*** rrecio_ has quit IRC22:51
*** rrecio has joined #openstack-mistral22:53
*** rrecio has quit IRC23:05
*** vishwanathj has quit IRC23:09
*** bobh has joined #openstack-mistral23:46

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!