| *** catintheroof has joined #openstack-mistral | 00:04 | |
| *** Ephur has quit IRC | 00:10 | |
| *** catintheroof has quit IRC | 00:30 | |
| *** rbrady-afk has quit IRC | 00:42 | |
| *** jaosorior has quit IRC | 00:46 | |
| *** Kevin_Zheng has joined #openstack-mistral | 01:32 | |
| *** rbrady-afk has joined #openstack-mistral | 01:54 | |
| *** gongysh has joined #openstack-mistral | 02:11 | |
| *** ist has quit IRC | 02:13 | |
| *** Kevin_Zheng has quit IRC | 03:37 | |
| rakhmerov | ddeja: awesome job! | 04:41 |
|---|---|---|
| rakhmerov | thanks | 04:41 |
| *** sharatss has joined #openstack-mistral | 04:43 | |
| openstackgerrit | Merged openstack/mistral-dashboard master: Update reno for stable/ocata https://review.openstack.org/428492 | 05:20 |
| *** sharatss has quit IRC | 05:31 | |
| *** sharatss has joined #openstack-mistral | 05:49 | |
| *** sharat has joined #openstack-mistral | 05:49 | |
| openstackgerrit | Merged openstack/python-mistralclient master: Read project_id and project_name in env https://review.openstack.org/409647 | 05:55 |
| openstackgerrit | Renat Akhmerov proposed openstack/mistral master: WIP: Refactor Mistral RPC serialization https://review.openstack.org/428591 | 06:18 |
| *** sharat has quit IRC | 06:27 | |
| *** sharatss has quit IRC | 06:27 | |
| *** sharat has joined #openstack-mistral | 06:28 | |
| *** sharatss has joined #openstack-mistral | 06:29 | |
| *** Kevin_Zheng has joined #openstack-mistral | 06:33 | |
| openstackgerrit | Merged openstack/mistral master: Update reno for stable/ocata https://review.openstack.org/428489 | 06:38 |
| *** sharatss has quit IRC | 07:02 | |
| openstackgerrit | Renat Akhmerov proposed openstack/mistral master: Refactor RPC serialization: remove JsonPayloadSerializer class https://review.openstack.org/428591 | 07:19 |
| *** sharatss has joined #openstack-mistral | 07:58 | |
| ddeja | rakhmerov: no problem | 08:02 |
| rakhmerov | ddeja: how did you find the reason of this? | 08:03 |
| gongysh | kong, hi | 08:08 |
| gongysh | who knows where to find the arch doc for mistral? | 08:09 |
| gongysh | I want to know how the mistral scales | 08:09 |
| ddeja | rakhmerov: I can tell you in a next hour, I've just started a meeting ;) | 08:10 |
| rakhmerov | ok | 08:10 |
| rakhmerov | gongysh: http://docs.openstack.org/developer/mistral/ is the official source | 08:10 |
| rakhmerov | and https://wiki.openstack.org/wiki/Mistral - but this is a little bit obsolete | 08:11 |
| rakhmerov | gongysh: you can also ask here, some tiny technical details may not be in the docs | 08:11 |
| rakhmerov | and we keep working on docs actively | 08:12 |
| *** sharatss has quit IRC | 08:26 | |
| *** jpich has joined #openstack-mistral | 08:32 | |
| *** FL1SK has quit IRC | 08:32 | |
| openstackgerrit | Jeremy Liu proposed openstack/python-mistralclient master: Add project_id and project_name options https://review.openstack.org/428640 | 08:38 |
| *** shardy has joined #openstack-mistral | 08:54 | |
| ddeja | rakhmerov: ok, I'm free | 09:00 |
| ddeja | So, basically I've started to dug through the traceback from https://bugs.launchpad.net/mistral/+bug/1659819 | 09:02 |
| openstack | Launchpad bug 1659819 in Mistral "Tox mysql and postgres gates are failing" [High,Fix released] - Assigned to Dawid Deja (dawid-deja-0) | 09:02 |
| ddeja | and realised that a lot of paths has '.venv' in them instead of '.tox' | 09:04 |
| ddeja | moreover, I've run the tests on my devstack VM | 09:04 |
| ddeja | and they passed | 09:04 |
| ddeja | but they did run in the .tox direcotry, not the virtualenv | 09:04 |
| ddeja | so I've started to compare what packages are used on both my local env and on virtualmachine | 09:05 |
| ddeja | and realised that it is really messed up | 09:05 |
| ddeja | so I started to compare what is a difference in launching tests beetwen regular tox and mysql tox | 09:06 |
| ddeja | and found out we are using run_test.sh | 09:06 |
| ddeja | After I understand how this script works | 09:06 |
| ddeja | I have a though 'hm, it seems like we are making venv inside the venv, let's try to use no-venv flag' and it worked | 09:07 |
| openstackgerrit | Jeremy Liu proposed openstack/python-mistralclient master: Add project_id and project_name options https://review.openstack.org/428640 | 09:14 |
| *** chlong has quit IRC | 09:38 | |
| rakhmerov | ddeja: ok, go ti | 10:07 |
| rakhmerov | got it | 10:07 |
| rakhmerov | thanks | 10:07 |
| ddeja | rakhmerov: no problem :) | 10:10 |
| rakhmerov | ddeja: I have a question about serialization in RPC | 10:10 |
| rakhmerov | I saw you added KombuSerializer | 10:10 |
| rakhmerov | which uses jsonutils.to_primitive() to serialize an object and jsonutils.loads() to deserialize it | 10:11 |
| rakhmerov | how does it work when you use it for a custom class? | 10:12 |
| rakhmerov | serialization is more or less clear, it'll just represent it as a dict etc. | 10:12 |
| rakhmerov | but will it restore the same object when it deserializes it? | 10:12 |
| rakhmerov | seems like it won't | 10:12 |
| rakhmerov | so, in fact, our serialization doesn't work with any custom types really | 10:14 |
| rakhmerov | I'm talking about both o.m and kombu | 10:14 |
| rakhmerov | I just wanna fix a number of things in serialization and trying to understand how it works | 10:15 |
| *** gongysh has quit IRC | 10:19 | |
| ddeja | rakhmerov: I need to look on a code... | 10:36 |
| *** FL1SK has joined #openstack-mistral | 10:36 | |
| rakhmerov | ddeja: I think I found the answer | 10:36 |
| rakhmerov | experimenting now.. | 10:37 |
| ddeja | rakhmerov: OK | 10:38 |
| rakhmerov | it's just looking weird | 10:38 |
| rakhmerov | serialization happens via jsonutils.to_primitive() | 10:38 |
| rakhmerov | but deserialization via json.loads() | 10:38 |
| rakhmerov | I don't mean to say it's your fault, it's rather a question to folks who created oslo_serialization | 10:39 |
| rakhmerov | no symmetry | 10:39 |
| ddeja | well, it is not json.loads() but jsonutils.loads | 10:41 |
| ddeja | which is the same, but we are using consistent library after all | 10:41 |
| rakhmerov | ooh, yes, sorry | 10:41 |
| rakhmerov | overlooked it | 10:41 |
| ddeja | but yes, it is werid | 10:42 |
| rakhmerov | ddeja: ok, the main thing is that it doesn't work for objects :) | 10:42 |
| ddeja | rakhmerov: yes, you need to serialize them to dict/json | 10:43 |
| rakhmerov | jsonutils.to_primitive() simply converts an object into a dict | 10:43 |
| ddeja | yes | 10:43 |
| rakhmerov | which doesn't leave any info about how to instantiate it | 10:44 |
| rakhmerov | what class to use | 10:44 |
| rakhmerov | hm... ok | 10:44 |
| ddeja | we can use OVO for this | 10:44 |
| rakhmerov | does it allow something like this? | 10:44 |
| rakhmerov | not too much familiar with it | 10:44 |
| rakhmerov | know the general idea though | 10:45 |
| ddeja | what OVO does it simply converts the object to dict on the sender side | 10:45 |
| ddeja | and on the receiver side it re-creates the object from dict | 10:45 |
| rakhmerov | ok, how does it know what class to instantiate on the receiver side? | 10:45 |
| rakhmerov | is it solved somehow? | 10:46 |
| rakhmerov | what if we want to pass different types? | 10:46 |
| rakhmerov | I'll look at OVO more closely | 10:47 |
| rakhmerov | basically what I'd like to do is to create some sort of polymorphic serializer | 10:47 |
| rakhmerov | which also inserts some additional key into a object representation | 10:48 |
| rakhmerov | which should help deserialize the object correctly | 10:48 |
| rakhmerov | that should be simple even w/o OVO | 10:48 |
| ddeja | OVO simply passes the class name throught the RPC | 10:48 |
| rakhmerov | yes, ok | 10:49 |
| rakhmerov | btw, ddeja, one more question | 10:49 |
| ddeja | ? | 10:49 |
| rakhmerov | does o.m allow to work with binary data? | 10:50 |
| ddeja | what do you mean? | 10:50 |
| ddeja | by 'binary data'? | 10:50 |
| rakhmerov | ok, sec, I'll explain | 10:50 |
| rakhmerov | what I'd like to do is compress the data on the sender side and decompress it on the receiver side | 10:50 |
| rakhmerov | using gzip or bz2 | 10:51 |
| rakhmerov | of course, MQ itself supports it | 10:51 |
| ddeja | ok, it is doable | 10:51 |
| rakhmerov | so that we don't need to compress on our own | 10:51 |
| ddeja | we just needs to convert it to string | 10:51 |
| ddeja | rabbitMQ supports it? | 10:51 |
| rakhmerov | wait a sec | 10:51 |
| rakhmerov | yeah, I believe I've seen this option in a message properties somewhere | 10:52 |
| rakhmerov | you just tell it "compress the data with gzip" and it does it | 10:52 |
| rakhmerov | what I'm thinking though is that o.m is a wrapper | 10:53 |
| rakhmerov | which again hides lots of stuff available in Rabbit for example | 10:53 |
| rakhmerov | and I didn't find anything in o.m to do this automatically except kombu_compression property (something like this) | 10:53 |
| rakhmerov | which is deprecated | 10:53 |
| ddeja | rakhmerov: hm, ok | 10:54 |
| rakhmerov | so, I thought we could just do it on our own using zlib | 10:54 |
| ddeja | so yes, there may be some options that are hidden by o.m | 10:54 |
| rakhmerov | but zlib generates bytes | 10:54 |
| ddeja | we can always convert it to string | 10:54 |
| rakhmerov | so I wonder if it makes any difference for o.m if we pass a byte string | 10:55 |
| rakhmerov | yeah | 10:55 |
| rakhmerov | (just thinking loudly) | 10:55 |
| rakhmerov | ok | 10:55 |
| ddeja | byte string should be OK | 10:55 |
| rakhmerov | I'll try | 10:55 |
| rakhmerov | the thing is that compression is not so heavy in terms of CPU | 10:55 |
| rakhmerov | but RPC itself is very heavy | 10:56 |
| rakhmerov | it could make a big difference if we use compression | 10:56 |
| rakhmerov | ddeja: do you think sounds reasonable? Or may be I'm missing something | 10:57 |
| ddeja | rakhmerov: I'm not sure | 11:12 |
| rakhmerov | :) ok | 11:12 |
| ddeja | we should experiment to confirm it | 11:12 |
| rakhmerov | yes, I'll do it | 11:12 |
| ddeja | but in some cases it may be resonable | 11:13 |
| rakhmerov | I'm looking at OVO | 11:13 |
| rakhmerov | yes | 11:13 |
| ddeja | but IMO RPC is not this bad | 11:13 |
| ddeja | I've even created a project in which i was sending screen from my pc to my mobile | 11:13 |
| ddeja | over RPC | 11:13 |
| rakhmerov | so? | 11:14 |
| rakhmerov | you mean RabbitMQ? :) | 11:14 |
| ddeja | yes | 11:14 |
| rakhmerov | :)) | 11:14 |
| ddeja | and it was working fine | 11:14 |
| ddeja | around 15-20 FPS | 11:14 |
| rakhmerov | ok, what I found recently is: | 11:14 |
| rakhmerov | 0. I used dict with 20000 entries as a data sent over RPC | 11:15 |
| rakhmerov | 1. RPC call with o.m takes ~0.7 sec | 11:15 |
| rakhmerov | 2. Building a string representation for this dict takes ~0.03 sec | 11:16 |
| rakhmerov | that's why I'm saying that RPC itself takes much more than in-memory manipulations with data of this size | 11:16 |
| ddeja | ok I see | 11:16 |
| rakhmerov | and that's why I'm saying that hopefully our kombu RPC works faster | 11:17 |
| rakhmerov | ok, anyway, seems like OVO is a good fit for doing serialization in flexibly | 11:17 |
| rakhmerov | will try to use it | 11:17 |
| ddeja | rakhmerov: but how many times faster is to send this string instead of dict? | 11:18 |
| rakhmerov | and experiment | 11:18 |
| ddeja | becouse buliding string from dict is fast, but we still have to send it | 11:18 |
| rakhmerov | yes, it doesn't matter actually | 11:18 |
| rakhmerov | it will convert into a string anyway | 11:18 |
| rakhmerov | I mentioned building a string because this shows that data conversion itself is fast | 11:19 |
| ddeja | OK, I see | 11:19 |
| rakhmerov | the most heavy part is behind the scenes | 11:19 |
| rakhmerov | in socket/rabbit/socket | 11:19 |
| ddeja | rabbitmq is slow, since it had it guaratnees | 11:19 |
| rakhmerov | yes | 11:19 |
| rakhmerov | that's one of the reasons | 11:20 |
| rakhmerov | too | 11:20 |
| rakhmerov | but if I do it with small dicts RPC takes under 0.05 sec | 11:20 |
| rakhmerov | on my machine | 11:20 |
| rakhmerov | so size of the data matters much | 11:21 |
| *** thrash|g0ne is now known as thrash | 11:22 | |
| ddeja | yes, after all it is all splitted to chunks | 11:22 |
| rakhmerov | btw, one interesting thing is that DB queries with the same amount of date take under 0.04 sec | 11:22 |
| rakhmerov | :) | 11:22 |
| rakhmerov | I mean with dicts with 20000 entries | 11:22 |
| rakhmerov | so RPC is the slowest part in the whole system | 11:22 |
| ddeja | with DB we only have server-client communicate | 11:22 |
| rakhmerov | yes, true | 11:22 |
| rakhmerov | of course | 11:22 |
| ddeja | with RPC we have sender->broker->client->broker->sender | 11:22 |
| rakhmerov | yep | 11:23 |
| ddeja | for call | 11:23 |
| ddeja | and, if we have durable queue, broker needs to store the message on disk | 11:23 |
| rakhmerov | which leads me to one more interesting thought: we don't need to use RPC in many cases :) | 11:23 |
| rakhmerov | yes | 11:23 |
| ddeja | well, we need to somehow communicate beetwen services... | 11:23 |
| rakhmerov | :) | 11:24 |
| *** gongysh has joined #openstack-mistral | 11:24 | |
| rakhmerov | I mean that maybe it doesn't always make sense to use executors | 11:24 |
| rakhmerov | in our architecture | 11:24 |
| ddeja | that would be huuuge | 11:24 |
| rakhmerov | I looked at some other workflow engines, i.e. Camunda (BPMN based stuff) doesn't seem to have such a concept | 11:25 |
| rakhmerov | they just have engine that scales | 11:25 |
| rakhmerov | that's it | 11:25 |
| ddeja | but we can potientialy have a system with one engine thread | 11:25 |
| rakhmerov | they have a concept of worker but it's a completely different thing | 11:25 |
| ddeja | and a lot of workers threads | 11:25 |
| ddeja | and only have rpc beetwen API-Engine | 11:25 |
| rakhmerov | yes | 11:25 |
| rakhmerov | well, in Python this may be tricky because of its threading system | 11:26 |
| rakhmerov | remember what I told about using blocking external calls and eventlet green threads | 11:26 |
| rakhmerov | but this is something to think about | 11:26 |
| ddeja | ok, so we can have different process but on the same machine | 11:27 |
| ddeja | or different processes | 11:27 |
| ddeja | and use linux pipes for communication | 11:28 |
| rakhmerov | yeah | 11:28 |
| rakhmerov | we could probably use something like multiprocessing lib | 11:28 |
| ddeja | but this would be really huge change | 11:28 |
| rakhmerov | something that our API server does | 11:28 |
| rakhmerov | it spawns multiple processes via oslo.service | 11:29 |
| ddeja | yes | 11:29 |
| rakhmerov | yes, it is a huge change | 11:29 |
| rakhmerov | agree | 11:29 |
| ddeja | and there is a question - how many of customers (if any?) uses standalone executors | 11:29 |
| rakhmerov | but think about this: seems like for many many Mistral actions the RPC overhead is much bigger than runtime of those actions | 11:29 |
| ddeja | standalone - only executor on a server | 11:29 |
| rakhmerov | ddeja: according to my knowledge, not many at this point | 11:30 |
| rakhmerov | of course, if we decide to make this change it needs to be configurable | 11:30 |
| ddeja | rakhmerov: if so, then we can potentially change the architecture | 11:30 |
| rakhmerov | because executors still have a number of pros | 11:30 |
| rakhmerov | like: you can run them on machines w/o access to DB etc. | 11:31 |
| rakhmerov | yeah | 11:31 |
| rakhmerov | the thing is that I saw numbers and it made me think in this direction | 11:31 |
| rakhmerov | previously I didn't have these numbers | 11:32 |
| ddeja | or | 11:32 |
| ddeja | we can let the executors to have access to the DB | 11:32 |
| rakhmerov | ddeja: it all actually comes from some of the performance requirements we have internally at Nokia | 11:32 |
| rakhmerov | that's why I'm digging this again | 11:32 |
| ddeja | and only use RPC for control | 11:32 |
| rakhmerov | hm... I don't know. It's a separate discussion | 11:33 |
| ddeja | the we will be only sending small amount of data | 11:33 |
| rakhmerov | there were users who had a string requirement not to allow executors to have DB access | 11:33 |
| ddeja | like 'execute_action: 12345677' | 11:33 |
| rakhmerov | I think it's a good topic for the PTG | 11:33 |
| ddeja | rakhmerov: oh | 11:33 |
| * ddeja wish to be there... | 11:34 | |
| rakhmerov | for example, StackStorm had this requirement, we can discuss it again with Winson | 11:34 |
| ddeja | yeah | 11:34 |
| rakhmerov | ddeja: any chance to use Travel Support Program? :) | 11:34 |
| ddeja | I think it's too late | 11:35 |
| rakhmerov | ok | 11:35 |
| rakhmerov | ddeja: ok, thanks for discussing it with me. I needed to discuss it with someone, I needed to speak out | 11:35 |
| ddeja | rakhmerov: no problem, that was a pleasure | 11:36 |
| rakhmerov | :) | 11:36 |
| rakhmerov | same to me | 11:36 |
| *** jkilpatr has quit IRC | 11:38 | |
| *** jkilpatr has joined #openstack-mistral | 11:57 | |
| *** gongysh has quit IRC | 12:03 | |
| *** jkilpatr has quit IRC | 12:04 | |
| *** jkilpatr has joined #openstack-mistral | 12:17 | |
| *** catintheroof has joined #openstack-mistral | 12:20 | |
| *** catintheroof has quit IRC | 12:20 | |
| *** catintheroof has joined #openstack-mistral | 12:21 | |
| *** bobh has joined #openstack-mistral | 13:19 | |
| *** dprince has joined #openstack-mistral | 13:23 | |
| *** bobh has quit IRC | 13:29 | |
| *** shardy is now known as shardy_mtg | 13:34 | |
| *** sharatss has joined #openstack-mistral | 13:39 | |
| *** jpich has quit IRC | 13:55 | |
| *** rbrady-afk is now known as mock-hates-me | 14:04 | |
| *** mock-hates-me is now known as rbrady | 14:11 | |
| *** shardy_mtg is now known as shardy | 14:11 | |
| *** sharatss has quit IRC | 14:23 | |
| *** jtomasek has quit IRC | 14:25 | |
| *** jtomasek has joined #openstack-mistral | 14:26 | |
| *** thrash is now known as thrash|biab | 14:54 | |
| *** sharatss has joined #openstack-mistral | 14:55 | |
| *** chlong has joined #openstack-mistral | 15:23 | |
| *** jaosorior has joined #openstack-mistral | 15:40 | |
| *** tuan_ has joined #openstack-mistral | 15:43 | |
| tuan_ | hi guys | 15:44 |
| tuan_ | now what tool openstack foundation use for generating docs | 15:45 |
| tuan_ | sphinx or swagger? | 15:45 |
| *** thrash|biab is now known as thrash | 16:17 | |
| d0ugal | tuan_: sphinx | 16:26 |
| d0ugal | I don't know what swagger is :) | 16:26 |
| *** mgershen has joined #openstack-mistral | 16:26 | |
| d0ugal | Maybe it is also used in places? | 16:26 |
| tuan_ | d0ugal: oh, here you are | 16:27 |
| tuan_ | :D | 16:27 |
| tuan_ | thanks | 16:27 |
| *** mgershen has quit IRC | 16:27 | |
| d0ugal | np | 16:27 |
| * d0ugal wont be here for long | 16:27 | |
| tuan_ | i saw some links they are talking about usung swagger for api docs | 16:27 |
| d0ugal | Interesting, maybe I am out of touch | 16:28 |
| d0ugal | but so far, I have only used Sphinx in OpenStack. | 16:28 |
| tuan_ | ahha, thank you | 16:29 |
| tuan_ | we just now have to redesign our API | 16:29 |
| d0ugal | We do? | 16:29 |
| tuan_ | oh sorry | 16:29 |
| tuan_ | Nokia | 16:29 |
| tuan_ | :D | 16:29 |
| d0ugal | oh :) | 16:29 |
| tuan_ | btw, thanks again and have a nice weekend | 16:30 |
| tuan_ | :) | 16:30 |
| d0ugal | np | 16:30 |
| tuan_ | and if you have time | 16:30 |
| tuan_ | could you please review my patch | 16:30 |
| d0ugal | Link? | 16:30 |
| tuan_ | https://review.openstack.org/#/c/424621/ | 16:30 |
| d0ugal | Looking now | 16:30 |
| *** chlong has quit IRC | 16:36 | |
| *** chlong has joined #openstack-mistral | 16:52 | |
| *** d0ugal is now known as d0ugal|skiing | 16:54 | |
| *** tuan_ has quit IRC | 17:08 | |
| *** chlong has quit IRC | 17:16 | |
| *** jkilpatr_ has joined #openstack-mistral | 17:31 | |
| *** jkilpatr has quit IRC | 17:32 | |
| *** sharat has quit IRC | 17:37 | |
| *** shardy has quit IRC | 17:38 | |
| *** dprince has quit IRC | 17:50 | |
| *** vgnbkr has quit IRC | 17:56 | |
| *** bobh has joined #openstack-mistral | 17:57 | |
| *** vgnbkr has joined #openstack-mistral | 17:57 | |
| *** Ephur has joined #openstack-mistral | 18:17 | |
| *** dprince has joined #openstack-mistral | 18:20 | |
| *** bobh has quit IRC | 18:33 | |
| *** chlong has joined #openstack-mistral | 18:38 | |
| *** chlong has quit IRC | 19:11 | |
| *** jkilpatr_ has quit IRC | 19:18 | |
| *** jkilpatr has joined #openstack-mistral | 19:20 | |
| *** chlong has joined #openstack-mistral | 19:21 | |
| *** chlong has quit IRC | 19:27 | |
| *** bobh has joined #openstack-mistral | 19:30 | |
| *** chlong has joined #openstack-mistral | 19:40 | |
| *** bobh has quit IRC | 20:36 | |
| *** bobh has joined #openstack-mistral | 20:51 | |
| *** catintheroof has quit IRC | 21:11 | |
| *** chlong has quit IRC | 21:44 | |
| *** dprince has quit IRC | 22:24 | |
| *** chlong has joined #openstack-mistral | 22:42 | |
| *** jkilpatr has quit IRC | 22:51 | |
| *** jaosorior has quit IRC | 22:52 | |
| *** jkilpatr has joined #openstack-mistral | 23:25 | |
| *** Ephur has quit IRC | 23:36 | |
| *** bobh has quit IRC | 23:42 | |
| *** Ephur has joined #openstack-mistral | 23:43 | |
| *** rbrady has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!