Thursday, 2017-02-09

*** jaosorior has joined #openstack-mistral00:20
*** jaosorior has quit IRC00:31
*** rbrady has quit IRC00:46
*** bobh has joined #openstack-mistral00:52
*** bobh has quit IRC00:52
*** bobh has joined #openstack-mistral00:52
*** zhurong has joined #openstack-mistral01:29
*** zhurong has quit IRC01:35
*** bobh has quit IRC02:11
*** gongysh has quit IRC02:40
*** gongysh has joined #openstack-mistral02:40
*** rbrady has joined #openstack-mistral02:42
*** rbrady has quit IRC02:42
*** rbrady has joined #openstack-mistral02:42
*** gongysh has quit IRC02:45
*** thrash is now known as thrash|g0ne03:29
*** sharatss has joined #openstack-mistral04:48
*** jpich has joined #openstack-mistral06:55
*** gongysh has joined #openstack-mistral07:30
*** sharatss has quit IRC08:29
*** sharatss has joined #openstack-mistral08:30
zhenguorakhmerov: do you have some concerns if I move mistral dashboard py35 gate to voting?, seems it works well for a long time08:37
rakhmerovzhenguo: yes, go ahead08:38
rakhmerovthanks08:38
zhenguorakhmerov: ok thanks08:38
*** AJaeger has joined #openstack-mistral08:56
AJaegermistral team, could I get a second +2 and +A on https://review.openstack.org/431143 , please?08:57
*** openstackgerrit has quit IRC09:03
*** openstackgerrit has joined #openstack-mistral09:24
openstackgerritSharat Sharma proposed openstack/mistral master: Support i18n for LOG.warning  https://review.openstack.org/43086609:24
ddejarakhmerov: Hi, I've spend a lot of time yesterday trying to investigate it09:34
ddejabut with no luck09:34
rakhmerovddeja: hi Dawid09:34
rakhmerovI see09:35
rakhmerovddeja: let me know if you're out of ideas09:35
rakhmerovI'd like to merge it before the end of RC-209:35
rakhmerovAJaeger: done (w/o second +2 though)09:35
ddejarakhmerov: I've tried to recheck it, but it still fails...09:36
rakhmerovok09:36
ddejaI have one idea why it is happening, but I don't understand why it shows after we change the serialization method09:36
ddejabut it looks like we a) open rpc connection using kombu and use it for communication beetwen API<->EvenEngineService09:37
rakhmerovyes, so?09:37
ddejab) EventEngine starts new rpc connection to listen to notification using standard oslo mechanism09:38
ddejaand in the same moment we got 'connection refused' on API side09:38
ddejathat's what it looks like from logs09:38
rakhmerovhm..09:38
rakhmerovwerid09:38
rakhmerovweird09:39
ddejaI'll try to run it locally and see if I can somehow debug it09:39
ddejait's weird, since it has nothing to do with serialization09:39
ddejaif there is a problem with using both kombu driver and oslo in the same time, it should pop up erlier09:39
rakhmerovddeja: ok, Dawid, thanks for looking at it. The important thing for me: if you feel you're stuck or don't have time to continue etc. please let me know asap09:39
ddejaI'll run those tests locally today09:40
ddejaif it doesn't resolve it, then I have no other ideas09:40
rakhmerovmay it be related with, e.g. thread safety?09:40
rakhmerovnow we use the same serializer instance09:41
rakhmerovthis polymorphic serializer09:41
rakhmerovit's a singleton09:41
ddejahm, maybe...09:41
rakhmerovbut on the other hand, it's stateless09:41
AJaegerrakhmerov: thanks09:41
rakhmerovthe instance doesn't have any state that could affect threads09:41
rakhmerovAJaeger: np :)09:42
ddejarakhmerov: it's something to start with09:42
rakhmerovddeja: and also, encoder and decoder functions in Kombu used to be assigned with static methods09:42
rakhmerovnow they are bound to an instance09:42
rakhmerovI wasn't sure it was 100% OK when I was changing it09:43
rakhmerovtheoretically this should not make any difference but who knows09:43
rakhmerovooh, so all three tests are related to event triggers09:44
ddejaok, thanks for letting me now09:44
ddejayes09:44
rakhmerovthat's interesting09:44
ddejathey basically fails on creating the event trigger09:44
ddejarakhmerov: can we create the event trigger using mistral client09:45
ddejaI can't see such option after running 'mistral help'09:45
rakhmerovlet me check09:45
rakhmerovwe should be able to09:45
rakhmerovooh, no09:46
rakhmerovwe can't09:46
rakhmerovit's neither in Python API nor in CLI09:46
rakhmerovonly REST API09:46
rakhmerovddeja: you can also experiment with different serializers. Say what happens if you return the previous serializer (with static methods) and run these tests again?09:48
rakhmerovmaybe it's not a serialization problem at all but I just changed something else09:48
rakhmerovdirectly or indirectly09:48
rakhmerovif you get the old serializer back then these tests I guess need to pass, at least creating event trigger09:49
rakhmeroveverything should work except deserializing Result object09:49
rakhmerovI mean, try to vary something and experiment09:50
rakhmerovanother assumption: something related with event triggers is not serialized/deserialized properly09:51
rakhmerovit may fail somewhere09:52
ddejarakhmerov: OK, I'll try09:52
rakhmerovddeja: I also found this in the log http://paste.openstack.org/show/598192/09:55
rakhmerovnot sure it's related though09:55
rakhmerovin engine log09:55
ddejarakhmerov: I don't think it is09:57
rakhmerovyeah..09:57
ddejayesterday I went trough the code09:58
ddejait's only API - EventEngine communication09:58
rakhmerovok09:58
rakhmerovddeja: Dawid, I would really start with checking how event_engine.create_event_trigger() works via RPC10:02
rakhmerovon the API side we have:10:02
rakhmerov        rpc.get_event_engine_client().create_event_trigger(10:03
rakhmerov            trig.to_dict(),10:03
rakhmerov            events10:03
rakhmerov        )10:03
rakhmerovwe need to make sure that this works10:03
rakhmerovw/o even looking at oslo notifications etc.10:03
rakhmerovevent_engine itself doesn't return anything (None)10:03
rakhmerovmaybe None doesn't get serialized properly10:04
rakhmerovor trigger represented as dict10:04
rakhmerovwhen we pass it to the RPC server10:04
rakhmerovit seems a lot like some exception is swallowed somewhere10:07
*** sharatss has quit IRC10:12
*** sharatss has joined #openstack-mistral10:13
ddejaok10:14
ddejaI'll look on it10:14
*** sharatss has quit IRC10:19
*** sharatss has joined #openstack-mistral10:19
openstackgerritSharat Sharma proposed openstack/mistral master: Added style enfore checks for assert statements  https://review.openstack.org/43142710:36
openstackgerritMerged openstack/mistral master: Prepare for using standard python tests  https://review.openstack.org/43114310:36
*** AJaeger has left #openstack-mistral10:46
openstackgerritSharat Sharma proposed openstack/mistral master: Support i18n for LOG.warning  https://review.openstack.org/43086610:53
*** tuan_ has joined #openstack-mistral11:16
tuan_Hi, could you guys please review that patch #5 of this commit11:19
tuan_https://review.openstack.org/#/c/424621/11:19
*** gongysh has quit IRC11:41
*** jkilpatr has quit IRC11:44
*** thrash|g0ne is now known as thrash11:47
*** jkilpatr has joined #openstack-mistral11:59
*** dprince has joined #openstack-mistral12:13
*** bobh has joined #openstack-mistral12:29
*** bobh has quit IRC12:33
ddejarakhmerov: still there?12:38
rakhmerovyes sir12:39
ddejahttp://paste.openstack.org/show/598221/12:39
ddejalines 124-125 and lines 151-16012:39
ddejathis works12:39
rakhmerov:)))12:40
rakhmerovthat was the issue?12:40
ddejait seems so12:41
rakhmerovsecond...12:41
rakhmerovnot sure I understand everything yet12:41
rakhmerovlooking..12:41
ddejathe only change is that I've written my own serializer/deserializer and used them in kombu12:41
rakhmerovok, can you explain pls?12:41
ddejato be honest, I have no idea why it works12:42
rakhmerovin 124-125 you're still using self._serialize12:42
rakhmerovwhich is the old way of serializing it12:42
ddejayes12:42
rakhmerovsame for deserializing12:42
ddejayes12:42
rakhmerov_serialize_message() is the new way12:42
rakhmerovbut you don't use it here12:43
rakhmerovwhat's the point?12:43
ddejaand I use new way for serialize/deserialize inpput parameters12:43
rakhmerovwhere?12:43
ddejain kombu_client/server12:43
ddejait is already in patch12:43
rakhmerovwhere do you use these two methods?12:43
rakhmerovok12:43
ddejathe only change from the current patch12:44
ddejais this file and lines I gave you12:44
ddejarest is the same12:44
ddejait looks like there is a problem with using same serializer for kombu and for oslo12:44
rakhmerovooh, gosh...12:45
rakhmerovI really don't understand12:45
ddejafrom my limited knowledge12:45
rakhmerovwait a second..12:45
ddejaof how kombu and o.m works12:45
rakhmerovaah, I see now12:45
rakhmerovI just didn't carefully look at your last patchset12:46
ddejaOK12:46
rakhmerovso, then I don't understand what this encoder/decoder is for?12:46
rakhmerov:)12:46
ddejaok, we have like 2 layers of serialization12:46
ddejalet's say we want to send object Result via RPC12:47
rakhmerovI think I see12:47
rakhmerovgo on pls..12:47
ddejafirstly we would use the new serializers written by you12:47
rakhmerovyes12:47
ddejaand serialize it to dictionary12:47
ddejaand we would put this dictionary in the second dictionary called 'body'12:47
ddejaso this would be something like body = {'params' : { *result dict* }}12:48
rakhmerovpolymorphic serializer always returns a string12:48
ddejaok, so it would be a string ;)12:48
rakhmerovit can't be a dict12:48
rakhmerovit's a json string in general case12:48
ddejanevertheless, it would be put inside a dict under the params key12:48
rakhmerovok12:48
rakhmerovhow about return values?12:49
ddejathis body dict have some other info, like reply_id, and soon12:49
rakhmerovaaah12:49
ddejaso, we need to serilize it, to send it over RPC12:49
ddejathat is why we have 2 serializers12:49
ddejaand, the main point12:49
ddejawhen we declare a serialize into kombu12:49
rakhmerovddeja: we could probably use some noop methods for encoder/decoder, it may just work :)12:50
ddejausing serialization.register12:50
rakhmerovok12:50
ddejathen this serializer is passed to kombu Connection object12:50
ddeja(from my limited knowledge_12:50
ddejaor maybe not to this object, but it is passed to kombu library12:50
ddejaso, here's what happens on the EvenEngine side12:51
rakhmerovok12:51
ddejawe create kombu connection using polymorphic serializer - polymorfic serializer methods are passed to kombu12:51
rakhmerovyep12:51
ddejathen, we want to listen to event, so we create the oslo_event_listener12:51
rakhmerovaha12:51
ddejawhich also uses polymorfic serializer12:51
ddejaand it also passes it to kombu12:52
rakhmerovsame instance, yes12:52
ddejawe passes same instance twice12:52
ddejawhich results in broken connection12:52
rakhmerovok12:52
ddejaI have no idea why12:52
rakhmerov:)12:52
ddejabut the good news it12:52
rakhmerovhm... why is it a problem for Kombu? )12:52
rakhmerovweird12:52
ddejathat there is no reason to pass this serializer directly to kombu in kombu driver12:52
rakhmerovwe can actually make it non singleton12:52
rakhmerovnp12:52
rakhmerovthat's true12:53
rakhmerovI agree12:53
ddejasince we are sure that at this point we have regular dict to serialize, we can just use the json.dumps12:53
rakhmerovbut does it actually have to be a string?12:53
ddejathat is what json.dumps would construct from dict12:53
rakhmerovwhat is the type of the encoder function?12:53
rakhmerovI mean its return value12:53
ddejaencoder should return string12:54
rakhmerovdo we even need to use json.dumps?12:54
ddejawe need12:54
rakhmerovyou sure? I has to return a string?12:54
rakhmerovno dict, no list?12:54
rakhmerovno ints?12:54
ddejahttps://github.com/openstack/mistral/blob/master/mistral/engine/rpc_backend/kombu/kombu_client.py#L125-L12912:54
ddejathis is send over the rpc12:54
rakhmerovbecause with o.m the only requirement for serializer is to return something primitive12:55
rakhmerovthat could be put in json12:55
ddejawell, it is even simpler12:55
rakhmerovddeja: yes, I understand this12:55
ddejawe can just drop the mistral_serialization12:55
rakhmerovI mean do we even need to serialize it with json.dumps?12:55
ddejaand use 'json'12:55
ddejano, we don't12:55
ddejahttps://github.com/openstack/mistral/blob/master/mistral/engine/rpc_backend/kombu/kombu_client.py#L14612:55
rakhmerovyes, that's what I mean12:55
ddejawe can change this line to 'json'12:55
rakhmerovbecause this body dict will always contain only primitive values12:56
ddejaand it should work fine12:56
rakhmerovwe know that on 100%12:56
ddejayes12:56
ddejasince we are parsing the kwargs manually12:56
rakhmerovok, this is clear now12:56
rakhmerovanother question I'm worried about12:56
ddejahm?12:56
tuan_https://review.openstack.org/#/c/424621/12:57
rakhmerovwhere do we deserialize an RPC method return value?12:57
tuan_could you guys please review it12:57
tuan_sorry for interrupting12:57
tuan_:(12:57
rakhmerovtuan_: yes12:57
rakhmerovwe will12:57
ddejarakhmerov: second, let me find a line...12:57
rakhmerovddeja: I see that we serialize/deserialize arguments, but don't see the same for return value12:57
tuan_thanks, may i join you guys for kombu problem12:57
tuan_:D12:57
ddejarakhmerov: which return value?12:58
rakhmerovtuan_: if you're here you already joined ;)12:58
ddejarakhmerov: you mean - we made a call12:58
ddejaserver got it, process it and now sends the results back?12:59
rakhmerovddeja: for example, we called rpc.executor_client.run_action()12:59
rakhmerovwhich returns Result object12:59
rakhmerovyes12:59
rakhmerovwe need to deserialize this Result object properly12:59
ddejarakhmerov: in this particular case, it uses cast to send it back, not the resend mechanism ;)12:59
rakhmerovand serialize too12:59
ddejathat's why it works12:59
ddejahttps://github.com/openstack/mistral/blob/master/mistral/engine/default_executor.py#L10913:00
rakhmerovooh, you mean here it goes as a parameter13:00
rakhmerovnot a return value13:00
rakhmerovI see13:00
ddejayup, it is send individually13:01
rakhmerovok13:01
ddejabut I can add mechanism for serializing/deserializng return values13:01
rakhmerovbut we also need to make it work for synchronous calls13:01
ddejait works now, but it may stop work in the future13:01
rakhmerovwhy?13:01
ddejait works now since all synchornous calls return some primitives13:02
ddejabut if some synchornous call stars to return something non-primitive, it would stop working13:02
rakhmerovyes, if we don't add proper serialization13:02
rakhmerovof course13:02
*** bobh has joined #openstack-mistral13:02
ddejaok, I'll work on appriopriete fix13:03
ddejaand I would send it in a few hours13:03
ddeja:)13:03
rakhmerovok13:03
rakhmerovthanks!13:03
rakhmerov:) interesting stuff13:03
ddejayup13:03
rakhmerovI'm almost sure it will work even if we drop this 'mistral_serialization' completely13:04
rakhmerovcan make a bet :)13:04
ddejayes, we will13:04
ddejait will*13:05
rakhmerovDawid, will you be available next week?13:05
ddejamaybe a little less the usuall13:05
rakhmerovok13:05
ddejabut yes13:05
rakhmerovI need to discuss some other things with you13:05
ddejano problem13:05
rakhmerovok, I'm out13:05
ddejaOK13:06
*** bobh has quit IRC13:27
*** brunograz1 has joined #openstack-mistral13:48
*** gongysh has joined #openstack-mistral13:52
*** brunograz has joined #openstack-mistral13:53
*** brunograz has quit IRC14:14
*** brunograz1 has quit IRC14:22
*** catintheroof has joined #openstack-mistral14:31
*** brunograz has joined #openstack-mistral14:37
*** ChatSharp has joined #openstack-mistral14:59
*** ChatSharp has left #openstack-mistral14:59
openstackgerritDawid Deja proposed openstack/mistral master: Refactor RPC serialization: add polymophic serializer  https://review.openstack.org/42964315:11
ddejarakhmerov: I hope this will do a job https://review.openstack.org/42964315:34
rakhmerovok15:49
openstackgerritluong tuan proposed openstack/mistral master: Deleting the expired execution in chunk  https://review.openstack.org/42462115:52
*** tuan_ has quit IRC16:01
*** brunograz has quit IRC16:03
*** tuan has joined #openstack-mistral16:03
*** brunograz has joined #openstack-mistral16:09
*** yohana has joined #openstack-mistral16:10
*** gongysh has quit IRC16:13
*** yohana has left #openstack-mistral16:15
*** sharatss has quit IRC16:30
*** jaosorior has joined #openstack-mistral16:31
*** thrash is now known as thrash|f00dz16:38
*** vishwanathj has joined #openstack-mistral16:50
*** vishwanathj has quit IRC16:57
*** tuan has quit IRC16:59
*** jaosorior is now known as jaosorior_mtg17:03
*** tuan_ has joined #openstack-mistral17:04
*** catintheroof has quit IRC17:06
*** tuan_ has quit IRC17:07
*** catintheroof has joined #openstack-mistral17:07
openstackgerritluong tuan proposed openstack/mistral master: Deleting the expired execution in chunk  https://review.openstack.org/42462117:10
*** catintheroof has quit IRC17:11
*** jpich has quit IRC17:30
*** dprince has quit IRC17:44
*** dprince has joined #openstack-mistral17:45
rakhmerovddeja: something is still failing17:55
rakhmerova bunch of tests :)17:55
*** thrash|f00dz is now known as thrash18:13
*** catintheroof has joined #openstack-mistral18:26
*** jkilpatr_ has joined #openstack-mistral18:28
*** jkilpatr has quit IRC18:31
*** jaosorior_mtg has quit IRC18:35
*** jaosorior has joined #openstack-mistral18:35
*** hrybacki____ is now known as hrybacki19:08
*** jaosorior has quit IRC19:20
*** rbrady is now known as rbrady-afk19:39
*** EmilienM has joined #openstack-mistral19:51
*** mwhahaha has joined #openstack-mistral19:51
EmilienMhello19:51
EmilienMhttps://review.openstack.org/#/c/425581/3/mistral/actions/openstack/action_generator/base.py@3819:51
EmilienMthis is breaking deployments that run Mistral in WSGI with apache19:51
EmilienMbefore we take the revert way, is there any thought?19:51
*** jaosorior has joined #openstack-mistral20:11
EmilienMI filed a bug https://bugs.launchpad.net/mistral/+bug/166336820:12
openstackLaunchpad bug 1663368 in Mistral "mistral-api in WSGI fails with "cannot register CLI option" error" [Undecided,New]20:12
openstackgerritEmilien Macchi proposed openstack/mistral master: Revert "External OpenStack action mapping file support"  https://review.openstack.org/43173520:14
*** pcastets has quit IRC20:15
*** jkilpatr_ has quit IRC20:18
*** jkilpatr has joined #openstack-mistral20:19
kongEmilienM: hi, i wonder how it breaks deployment with wsgi?20:24
kongooh, i saw the log link in bug report20:25
EmilienM:)20:25
kongwill take a look20:25
EmilienMthanks!20:25
*** harlowja_ has joined #openstack-mistral20:35
*** harlowja has quit IRC20:37
konghi, EmilienM, i think there is no need to have the whole patch reverted, just change the cli option back to normal config option20:49
konghmm.. think again, it's ok we revert the whole patch, let's see other reviewers option20:53
*** rbrady-afk is now known as rbrady20:56
EmilienMkong: I agree21:13
EmilienMkong: I'm ok to not revert21:13
EmilienMif we have a fix21:13
*** jkilpatr has quit IRC21:34
*** jkilpatr has joined #openstack-mistral21:52
*** harlowja_ has quit IRC21:58
*** harlowja has joined #openstack-mistral22:05
*** jaosorior has quit IRC22:37
*** jaosorior has joined #openstack-mistral22:38
*** jaosorior has quit IRC23:10
*** jaosorior has joined #openstack-mistral23:11
*** dprince has quit IRC23:18
*** jaosorior has quit IRC23:29
*** dprince has joined #openstack-mistral23:50
*** dprince has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!