| *** peopleme1ge has joined #openstack-marconi | 00:11 | |
| *** peoplemerge has quit IRC | 00:12 | |
| *** amitgandhi has joined #openstack-marconi | 00:33 | |
| *** amitgandhi has quit IRC | 00:37 | |
| wpf | peoplemerge_: py33 works after tox --recreate | 00:48 |
|---|---|---|
| *** oz_akan has joined #openstack-marconi | 00:52 | |
| *** oz_akan has quit IRC | 00:57 | |
| *** amitgandhi has joined #openstack-marconi | 01:34 | |
| *** amitgandhi has quit IRC | 01:38 | |
| *** renlt has joined #openstack-marconi | 01:49 | |
| *** oz_akan has joined #openstack-marconi | 02:07 | |
| *** renlt has quit IRC | 02:11 | |
| *** jeffreycoho has joined #openstack-marconi | 02:21 | |
| *** amitgandhi has joined #openstack-marconi | 02:34 | |
| *** amitgandhi has quit IRC | 02:39 | |
| *** wirehead_ has joined #openstack-marconi | 03:27 | |
| *** amitgandhi has joined #openstack-marconi | 03:35 | |
| *** oz_akan has quit IRC | 03:36 | |
| *** amitgandhi has quit IRC | 03:40 | |
| *** amitgandhi has joined #openstack-marconi | 04:36 | |
| *** amitgandhi has quit IRC | 04:40 | |
| *** chandankumar has joined #openstack-marconi | 04:50 | |
| *** chandankumar has quit IRC | 05:12 | |
| *** prashanthr_ has joined #openstack-marconi | 05:15 | |
| *** prashanthr_ has quit IRC | 05:19 | |
| *** prashanthr_ has joined #openstack-marconi | 05:21 | |
| *** amitgandhi has joined #openstack-marconi | 05:37 | |
| *** amitgandhi has quit IRC | 05:41 | |
| *** flaper87|afk is now known as flaper87 | 05:46 | |
| *** flaper87 is now known as flaper87|afk | 06:12 | |
| *** k4n0 has joined #openstack-marconi | 06:16 | |
| *** chandankumar has joined #openstack-marconi | 06:34 | |
| *** amitgandhi has joined #openstack-marconi | 06:37 | |
| *** amitgandhi has quit IRC | 06:42 | |
| *** Catherine has joined #openstack-marconi | 06:54 | |
| *** Catherine has quit IRC | 06:59 | |
| *** chandankumar has quit IRC | 07:33 | |
| *** amitgandhi has joined #openstack-marconi | 07:38 | |
| *** amitgandhi has quit IRC | 07:43 | |
| *** prashanthr_ has quit IRC | 08:27 | |
| *** prashanthr_ has joined #openstack-marconi | 08:31 | |
| *** chandankumar has joined #openstack-marconi | 08:32 | |
| *** amitgandhi has joined #openstack-marconi | 08:39 | |
| *** amitgandhi has quit IRC | 08:43 | |
| *** amitgandhi has joined #openstack-marconi | 09:40 | |
| *** amitgandhi has quit IRC | 09:44 | |
| *** chandankumar has quit IRC | 09:46 | |
| *** chandankumar has joined #openstack-marconi | 10:10 | |
| *** chandankumar has quit IRC | 10:11 | |
| *** chandankumar has joined #openstack-marconi | 10:11 | |
| *** amitgandhi has joined #openstack-marconi | 10:41 | |
| *** amitgandhi has quit IRC | 10:45 | |
| *** mkoderer has joined #openstack-marconi | 10:47 | |
| *** malini has joined #openstack-marconi | 11:27 | |
| *** malini has quit IRC | 11:27 | |
| *** malini has joined #openstack-marconi | 11:28 | |
| *** mwagner_lap has quit IRC | 11:29 | |
| *** tongli has joined #openstack-marconi | 11:29 | |
| *** denis_makogon has joined #openstack-marconi | 11:32 | |
| *** amitgandhi has joined #openstack-marconi | 11:41 | |
| *** amitgandhi has quit IRC | 11:45 | |
| *** vkmc has joined #openstack-marconi | 11:49 | |
| *** vkmc has quit IRC | 11:49 | |
| *** vkmc has joined #openstack-marconi | 11:49 | |
| vkmc | morning all | 12:09 |
| *** abettadapur has joined #openstack-marconi | 12:13 | |
| *** k4n0 has quit IRC | 12:28 | |
| prashanthr_ | vkmc: Good morning! | 12:34 |
| prashanthr_ | thanks for the review | 12:34 |
| vkmc | prashanthr_, hey :) | 12:34 |
| vkmc | good evening for you! | 12:35 |
| vkmc | np, those are just style nits, no need to block the queue for that | 12:36 |
| vkmc | it looks great! | 12:36 |
| prashanthr_ | anyways wil do it and submit it :) | 12:38 |
| prashanthr_ | let all the reviews be in | 12:38 |
| prashanthr_ | vkmc: Did the __init__ realignment | 12:41 |
| prashanthr_ | great catch actually | 12:41 |
| prashanthr_ | somehow i forget such small things | 12:41 |
| *** amitgandhi has joined #openstack-marconi | 12:42 | |
| vkmc | thanks prashanthr_! | 12:45 |
| *** sriram has joined #openstack-marconi | 12:46 | |
| *** amitgandhi has quit IRC | 12:46 | |
| *** Catherin_ has joined #openstack-marconi | 12:59 | |
| *** mpanetta has joined #openstack-marconi | 13:00 | |
| *** Catherin_ has quit IRC | 13:00 | |
| *** Catherin_ has joined #openstack-marconi | 13:01 | |
| *** jmckind has joined #openstack-marconi | 13:06 | |
| *** mpanetta has quit IRC | 13:10 | |
| *** mpanetta_ has joined #openstack-marconi | 13:10 | |
| *** mpanetta has joined #openstack-marconi | 13:11 | |
| *** mpanetta_ has quit IRC | 13:11 | |
| *** mpanetta_ has joined #openstack-marconi | 13:13 | |
| *** mpanetta has quit IRC | 13:13 | |
| vkmc | darn python26 gate is messing with us | 13:14 |
| *** keith_newstadt has joined #openstack-marconi | 13:22 | |
| *** mpanetta_ has quit IRC | 13:25 | |
| *** mpanetta has joined #openstack-marconi | 13:25 | |
| *** mpanetta has quit IRC | 13:26 | |
| *** mpanetta has joined #openstack-marconi | 13:27 | |
| *** oz_akan has joined #openstack-marconi | 13:28 | |
| *** mpanetta_ has joined #openstack-marconi | 13:31 | |
| vkmc | brb | 13:32 |
| *** mwagner_lap has joined #openstack-marconi | 13:33 | |
| *** Obulpathi has joined #openstack-marconi | 13:34 | |
| *** mpanetta has quit IRC | 13:34 | |
| abettadapur | vkmc: I responded to some of your comments | 13:38 |
| *** Catherin_ has left #openstack-marconi | 13:39 | |
| *** amitgandhi has joined #openstack-marconi | 13:43 | |
| *** ametts has joined #openstack-marconi | 13:43 | |
| *** jay-atl has joined #openstack-marconi | 13:43 | |
| *** amitgandhi has quit IRC | 13:57 | |
| *** itisit has joined #openstack-marconi | 13:59 | |
| peoplemerge_ | g'morning all! | 14:02 |
| sriram | good morning :) | 14:07 |
| *** amitgandhi has joined #openstack-marconi | 14:18 | |
| *** abettadapur has quit IRC | 14:19 | |
| mpanetta_ | morning :) | 14:30 |
| *** mpanetta_ is now known as mpanetta | 14:30 | |
| malini | morning !! | 14:31 |
| malini | looks like our gate is broken - all patches have an import error from falcon | 14:32 |
| *** rwsu has joined #openstack-marconi | 14:32 | |
| *** kgriffs|afk is now known as kgriffs | 14:33 | |
| kgriffs | malini: yeah, I just saw that | 14:34 |
| malini | it is just the 2.6 | 14:34 |
| kgriffs | there is some bug in the way packages get installed, to make falcon think it is being installed with python 2.7 and not 2.6 | 14:34 |
| kgriffs | so it doesn't require ordered dict | 14:35 |
| kgriffs | let me see what I can do | 14:35 |
| malini | cool..thx!! | 14:35 |
| *** cpallares has joined #openstack-marconi | 14:40 | |
| *** prashanthr_1 has joined #openstack-marconi | 14:42 | |
| *** prashanthr_ has quit IRC | 14:43 | |
| prashanthr_1 | good morning all. Can you please review https://review.openstack.org/#/c/97178/ when free ? | 14:43 |
| sriram | was digging around, Entropy does kinda look like notifications. https://wiki.openstack.org/wiki/Entropy | 14:44 |
| sriram | sure prashanthr_1, will give it a go soon :) | 14:44 |
| malini | sure prashanthr_1 | 14:44 |
| prashanthr_1 | thanks malini and sriram :) | 14:45 |
| sriram | Obulpathi, malini : ^ | 14:45 |
| Obulpathi | let me look into it | 14:45 |
| Obulpathi | thanks sriram :) | 14:46 |
| *** prashanthr_1 has quit IRC | 14:46 | |
| malini | We should redirect the entropy Join Us to Marconi Join Us :-P | 14:46 |
| sriram | haha lol :P | 14:46 |
| *** AAzza has joined #openstack-marconi | 14:48 | |
| *** tonytan4ever has joined #openstack-marconi | 14:48 | |
| Obulpathi | wow .. on the top level .. what they are doing and what we did for notifications is very similar .. | 14:52 |
| Obulpathi | except that use cases are different | 14:52 |
| sriram | yeah, thats why it peaked my interest. | 14:53 |
| sriram | wrong peaked.. | 14:53 |
| * sriram facepalm | 14:53 | |
| sriram | piqued | 14:53 |
| Obulpathi | let me talk to them and see if we can have them collaborate with us | 14:53 |
| Obulpathi | hahah .. lol sriram .. | 14:54 |
| Obulpathi | thans buddy tough ... :) | 14:54 |
| sriram | np | 14:54 |
| *** AAzza has quit IRC | 14:58 | |
| *** chandankumar has quit IRC | 15:00 | |
| *** openstackgerrit has joined #openstack-marconi | 15:03 | |
| *** tonytan4ever has quit IRC | 15:06 | |
| *** tonytan4ever has joined #openstack-marconi | 15:16 | |
| peoplemerge_ | malini: glad to hear it's not just my falcon | 15:39 |
| peoplemerge_ | kgriffs: I made progress this weekend on msgpack | 15:39 |
| kgriffs | cool | 15:39 |
| peoplemerge_ | https://gist.github.com/peoplemerge/39a6babea5bc4741f2cd | 15:39 |
| peoplemerge_ | I want to get all of the inherited tests passing and hope to get POST method into review today | 15:40 |
| *** shakamunyi has joined #openstack-marconi | 15:54 | |
| sriram | bbl lunch | 15:58 |
| *** Obulpathi has quit IRC | 16:01 | |
| *** chandankumar has joined #openstack-marconi | 16:07 | |
| *** chandankumar has quit IRC | 16:19 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/marconi: Updated from global requirements https://review.openstack.org/106506 | 16:24 |
| *** tonytan4ever has quit IRC | 16:24 | |
| * peoplemerge_ is leaving for the office | 16:25 | |
| *** chandankumar has joined #openstack-marconi | 16:33 | |
| *** celttechie has joined #openstack-marconi | 16:35 | |
| *** chandankumar has quit IRC | 16:39 | |
| *** kgriffs is now known as kgriffs|afk | 16:41 | |
| *** chandankumar has joined #openstack-marconi | 16:53 | |
| *** mkoderer has quit IRC | 17:02 | |
| *** abettadapur has joined #openstack-marconi | 17:08 | |
| *** cpallares has quit IRC | 17:14 | |
| *** Obulpathi has joined #openstack-marconi | 17:16 | |
| *** tonytan4ever has joined #openstack-marconi | 17:26 | |
| *** tonytan4ever has quit IRC | 17:31 | |
| *** kgriffs|afk is now known as kgriffs | 17:32 | |
| peopleme1ge | kgriffs: Remember our discussion on msgpack instance optimization? msgpack.packb()x5 is slower than packer = msgpack.Packer() + packer.pack()xn (obvious object creation/GC) | 17:36 |
| peopleme1ge | I have something I'm not sure how to implement regarding the unpacker | 17:36 |
| *** peopleme1ge is now known as peoplemerge | 17:40 | |
| peoplemerge | https://github.com/peoplemerge/marconi/blob/4d600dae5bdf47e0b72921de3ec3963e31dd8d47/marconi/queues/transport/wsgi/utils.py#L84 | 17:40 |
| *** kgriffs is now known as kgriffs|afk | 17:41 | |
| peoplemerge | it can wait for review I guess | 17:42 |
| peoplemerge | I'm just not sure how to avoid instantiating the unpacker each time if I make it a singleton. It might be possible to make a pool or attach an instance per wsgi thread. | 17:44 |
| peoplemerge | That's a little beyond my abilities to produce, certainly in time for j2 close/j3 start. | 17:45 |
| *** shakamunyi_ has joined #openstack-marconi | 17:45 | |
| peoplemerge | (in the singleton case, we'd face race conditions with multiple threads feeding a single instance) | 17:46 |
| *** shakamunyi has quit IRC | 17:49 | |
| *** Catherin_ has joined #openstack-marconi | 17:51 | |
| *** Catherin_ has quit IRC | 17:51 | |
| *** Catherin_ has joined #openstack-marconi | 17:52 | |
| *** Catherin_ is now known as Catherine_ | 18:08 | |
| *** kgriffs|afk is now known as kgriffs | 18:18 | |
| *** vkmc has quit IRC | 18:24 | |
| *** vkmc has joined #openstack-marconi | 18:24 | |
| kgriffs | peoplemerge: hey, just wanted to follow up on instantiating the unpacker | 18:29 |
| kgriffs | we don't have to worry about thread-safety IMHO, since people typically deploy python web services using multiple separate worker procs, and use green threads to multiplex I/O | 18:31 |
| kgriffs | peoplemerge: anyway, I would say, first make it work, then we will worry about making it fast. :) | 18:33 |
| kgriffs | just use msgpack.unpackb for now, and put a TODO(peoplemerge) comment on there | 18:33 |
| kgriffs | that's my $0.02 anyway | 18:34 |
| *** Catherine_ has left #openstack-marconi | 18:36 | |
| *** tonytan4ever has joined #openstack-marconi | 18:38 | |
| vkmc | kgriffs, hey :) how are you? if you have a moment I would like your opinion about the some implementation details of the AMQP transport and the client | 18:45 |
| vkmc | s/the some/some | 18:46 |
| kgriffs | sure. what's up? | 18:47 |
| vkmc | well, in the case of the AMQP transport | 18:48 |
| vkmc | without entering in details, AMQP has a special type of message that allows it to be super flexible between implementations | 18:49 |
| vkmc | so... we should respect it as much as possible | 18:49 |
| *** itisit has quit IRC | 18:49 | |
| vkmc | thing is, sounds like a good idea to store it marshalled in the storage backend? | 18:49 |
| vkmc | or do you think on a better way to save it so it's compatible with our API and also keep the AMQP message format? | 18:50 |
| *** chandankumar has quit IRC | 18:52 | |
| *** chandankumar has joined #openstack-marconi | 18:53 | |
| kgriffs | hmm | 18:54 |
| kgriffs | ok, so let me make sure I understand | 18:54 |
| kgriffs | are you saying every message that comes in has the potential of including a bunch of metadata or something that we need to treat a opaque in terms of our backend message store? | 18:56 |
| vkmc | (sorry if I'm not giving you enough context, I don't want to flood you with details about AMQP) | 18:56 |
| vkmc | currently we are handling messages in the storage backend considering a couple of parameters, the body and the ttl | 18:58 |
| *** denis_makogon has quit IRC | 18:59 | |
| vkmc | the transport pass an iterable list of messages to the storage with those params so the storage process them accordingly | 18:59 |
| vkmc | in AMQP we receive an AMQP message, with its own attributes | 18:59 |
| kgriffs | ok. BTW, how are your dealing with TTL | 19:00 |
| *** alcabrera|afk is now known as alcabrera | 19:00 | |
| vkmc | right now... I just setted a default ttl | 19:01 |
| kgriffs | OK. | 19:01 |
| vkmc | but I could use AMQP message ttl directly | 19:01 |
| vkmc | (although it could be not initialized) | 19:02 |
| kgriffs | ok, so the question is, how to preserve those AMQP-specific message attributes? | 19:02 |
| vkmc | exactly | 19:02 |
| kgriffs | so, supposing we were to just msgpack.packb all that and put it in the body field | 19:04 |
| kgriffs | so, client A submits a message via AMQP | 19:04 |
| kgriffs | then client H retrieves the message using HTTP | 19:04 |
| kgriffs | what would happen then? | 19:05 |
| vkmc | good point | 19:05 |
| vkmc | it wouldn't work | 19:05 |
| kgriffs | vkmc: I'm assuming an AMQP message has a well-defined "payload" part, separate from the other attributes? | 19:06 |
| vkmc | my first approach was to 'parse them' | 19:06 |
| vkmc | in order to avoid that situation | 19:06 |
| vkmc | yes it does | 19:06 |
| *** vkmc has left #openstack-marconi | 19:09 | |
| *** vkmc has joined #openstack-marconi | 19:09 | |
| vkmc | but if we parse them we don't know for sure which attributes has an assigned value... so we can store them in the backend | 19:11 |
| *** alcabrera is now known as alcabrera|afk | 19:14 | |
| kgriffs | sorry, just a sec | 19:16 |
| vkmc | sure :) | 19:17 |
| *** mwagner_lap has quit IRC | 19:18 | |
| kgriffs | http://www.educreations.com/lesson/view/marconi-and-amqp/22973755/?s=p2eDo3&ref=app | 19:20 |
| kgriffs | trying something new | 19:20 |
| vkmc | :D | 19:21 |
| kgriffs | so, is this picture kinda sorta what we are after? | 19:21 |
| kgriffs | basically, store all attributes that were inserted | 19:22 |
| kgriffs | only give back basic attributes for HTTP requests | 19:22 |
| kgriffs | but return "extended" attrs as well for AMQP requests | 19:22 |
| vkmc | that's exactly the situation yeah | 19:24 |
| kgriffs | ok... | 19:24 |
| kgriffs | what do you think about adding an "extended attributes" field to our message store schema | 19:24 |
| kgriffs | then you can msgpack whatever, slam it into that field | 19:24 |
| kgriffs | probably should namespace it | 19:25 |
| kgriffs | like | 19:25 |
| vkmc | that could work | 19:25 |
| kgriffs | {"amqp": { ... } } ===> extended_attrs or meta or something | 19:25 |
| vkmc | I'm only worried about what will happen when an AMQP retrieves that message | 19:25 |
| vkmc | it should get an AMQP message | 19:25 |
| kgriffs | what else do you need in order to reconstruct the original AMQP message? | 19:26 |
| vkmc | well, basically... this are the properties we should preserve in order to reconstruct it for a consumer http://qpid.apache.org/releases/qpid-proton-0.7/protocol-engine/python/api/index.html | 19:27 |
| vkmc | in the Message class, id, user_id, subject, reply_to, ... | 19:29 |
| kgriffs | oic | 19:29 |
| kgriffs | http://qpid.apache.org/releases/qpid-proton-0.7/protocol-engine/python/api/index.html | 19:29 |
| kgriffs | hmmm | 19:29 |
| kgriffs | so, if we just marshal everything to the body field in the messag estore | 19:30 |
| kgriffs | the HTTP API would return a body that is like | 19:30 |
| vkmc | it would return an instance of the Message object | 19:31 |
| kgriffs | {"subject": "cats vs. dogs", "body": ...} | 19:32 |
| kgriffs | whereas a message POSTed with HTTP would be | 19:32 |
| kgriffs | {...} | 19:32 |
| kgriffs | (the body directly) | 19:33 |
| kgriffs | I feel like we should normalize things in the message store | 19:33 |
| kgriffs | so that body is body, regardless of the transport | 19:33 |
| kgriffs | I wonder if it would be interesting to expose those other attributes to HTTP clients too | 19:34 |
| vkmc | hmm | 19:35 |
| vkmc | well, the message has a content-type | 19:36 |
| kgriffs | https://gist.github.com/anonymous/f42e64d748c2625ff77a | 19:37 |
| vkmc | it looks good | 19:37 |
| kgriffs | so, assume that we add a field called extattrs or something | 19:39 |
| * kgriffs is channelling file system design | 19:39 | |
| kgriffs | or just | 19:39 |
| kgriffs | xattrs | 19:39 |
| kgriffs | whatever | 19:39 |
| kgriffs | then, as I said before, you can drop namespaced extended attributes in there | 19:40 |
| kgriffs | different transports can expose those differently | 19:40 |
| vkmc | yeah... some extra field to storage everything that is just for AMQP (in this case) | 19:40 |
| kgriffs | for HTTP, we may not allow you to POST them, but we will return them in a listing using a well-known algorithm | 19:40 |
| kgriffs | basically, by taking each top-level key in xattrs and putting that in as a top-level key in the message document that is returned to the client | 19:41 |
| kgriffs | so if you have this for xattrs: | 19:41 |
| kgriffs | {"amqp | 19:41 |
| kgriffs | oops | 19:41 |
| kgriffs | try again | 19:41 |
| kgriffs | {"proton": { ... }, "susans_cool_plugin_thing": {...}} | 19:42 |
| kgriffs | you would get a message doc like | 19:42 |
| kgriffs | {"id": "50b68a50d6f5b8c8a7c62b01", "ttl": 120, "age": 53, "proton": {...}, "susans_cool_plugin_thing": {...}, "body": {...}} | 19:43 |
| vkmc | it makes sense | 19:44 |
| vkmc | that if you perform a GET with an HTTP client | 19:44 |
| vkmc | but if you do the same with an AMQP client, we should reconstruct the Proton message with the attributes in proton: { ... } | 19:44 |
| kgriffs | yep | 19:45 |
| vkmc | I like it :) | 19:46 |
| kgriffs | I wonder if we should use some dotted convention or something for the key names | 19:46 |
| kgriffs | hmmm | 19:46 |
| kgriffs | probably YAGNI | 19:46 |
| kgriffs | are those attributes standardized in AMQP 1.0 ? | 19:48 |
| vkmc | yes, Proton follows the AMQP 1.0 standard strictly | 19:50 |
| vkmc | so, we could use those attribute names for Marconi | 19:50 |
| kgriffs | I'm just trying to figure out whether we need to define a schema for the xattr key names | 19:53 |
| vkmc | sure | 19:54 |
| kgriffs | {"amqp": {...}} | 19:54 |
| kgriffs | or... | 19:54 |
| *** chandankumar has quit IRC | 19:56 | |
| kgriffs | {"vnd.amqp-v1.0": {...}} | 19:57 |
| *** whenry has joined #openstack-marconi | 19:57 | |
| kgriffs | {"amqp-v1.0": {...}} | 19:57 |
| kgriffs | hmm | 19:58 |
| kgriffs | I think we should for sure include a version | 19:58 |
| vkmc | I prefer the latter | 19:59 |
| kgriffs | ok, let's run with this | 20:00 |
| kgriffs | (the latter) | 20:00 |
| vkmc | I don't know if it's better if we leave the creation of new attributes for third-party drivers | 20:00 |
| vkmc | or if we provide a general, unique one | 20:00 |
| kgriffs | well, by definition, we can't anticipate what 3rd-party drivers will want/need | 20:01 |
| vkmc | yes... that's true | 20:01 |
| kgriffs | so, best to provide a naming guideline and then some kind of "registry" of types to avoid name collisions | 20:01 |
| kgriffs | kind of like the way internet media types work | 20:01 |
| vkmc | makes sense | 20:02 |
| kgriffs | vkmc: we can start a wiki page for this right now, but we will need a documentation blueprint for creating a "3rd-party driver guide" that also mentioned this | 20:02 |
| kgriffs | in other words | 20:02 |
| vkmc | I started a wiki for AMQP https://wiki.openstack.org/wiki/Marconi/specs/amqp/api/v1 | 20:03 |
| vkmc | I could mention about this and link it to a future 3rd party driver guide | 20:03 |
| kgriffs | will you start a wiki page that is a general 3rd-party driver dev guide, and create a blueprint for moving that over to the user docs at some point? | 20:03 |
| kgriffs | you can link to that from the home page | 20:03 |
| vkmc | sure, I could do that :) | 20:03 |
| kgriffs | excellent, thanks! | 20:03 |
| vkmc | np | 20:04 |
| kgriffs | we should get our Redis d00d to contribute to that page too | 20:04 |
| vkmc | thanks for guided brainstorming, as always | 20:04 |
| vkmc | sure, I'll ping prashanthr | 20:04 |
| kgriffs | thanks! | 20:06 |
| vkmc | and a one extra question kgriffs | 20:07 |
| kgriffs | shoot | 20:07 |
| vkmc | regarding the client | 20:07 |
| vkmc | right now we have CRUD for queues | 20:07 |
| vkmc | and I added CRUD for messages | 20:07 |
| vkmc | does it make sense to add CRUD for claims? | 20:08 |
| kgriffs | oh, you mean command line client, not the library | 20:08 |
| vkmc | in the spec you didn't mentioned it so I wasn't sure https://blueprints.launchpad.net/python-marconiclient/+spec/cli | 20:08 |
| kgriffs | yeah, I'd say anything the API supports, you should be able to do from the CLI | 20:09 |
| vkmc | awesome | 20:09 |
| vkmc | :) | 20:09 |
| kgriffs | thanks for working on the client, BTW! | 20:09 |
| vkmc | np! it's fun | 20:09 |
| kgriffs | I'd love to get a queue "tail" command, by the way. :D | 20:09 |
| vkmc | noted :) | 20:10 |
| kgriffs | with regex highlighting as a bonus | 20:10 |
| kgriffs | console colors FTW! | 20:10 |
| vkmc | +1 | 20:10 |
| vkmc | haha | 20:10 |
| kgriffs | srsly, though, that is super helpful when diagnosing a problem in a distributed system | 20:11 |
| kgriffs | we have a tool like that that we use for Marconi's predecessor, used in Rackspace Cloud Backup | 20:11 |
| kgriffs | I probably submitted a bp on it... aaaaanyway | 20:11 |
| kgriffs | keep on rockin' | 20:12 |
| kgriffs | oh... | 20:12 |
| kgriffs | one more thing | 20:12 |
| vkmc | yeah I can't imagine that... I haven't worked with such huge deployments but going through logs can be tedious | 20:13 |
| vkmc | in any scale | 20:13 |
| kgriffs | so, on the subject of marshaling | 20:14 |
| kgriffs | I have been wanted for a long time to migrate the mongodb driver to store bodies as snappy-compressed msgpacked blobs | 20:15 |
| kgriffs | in the sqlalchemy driver, this isn't such a concern since that driver is just for development | 20:15 |
| kgriffs | so, JSON is fine | 20:15 |
| kgriffs | now, we are going to add another opaque field, xattrs | 20:15 |
| kgriffs | I guess there are a few questions to "unpack" here. ;) | 20:16 |
| kgriffs | actually... hmmm | 20:16 |
| kgriffs | first thing to do is design the interface | 20:17 |
| kgriffs | how will the storage base class be modified to support arbitrary xattrs | 20:17 |
| vkmc | hmm, so by storing blobs we don't care too much about the actual content of the message | 20:17 |
| kgriffs | where, a single xattr is defined as name: value pair | 20:17 |
| vkmc | right now we only support strings, right? | 20:17 |
| kgriffs | let's see. | 20:18 |
| kgriffs | right now everything is JSON | 20:18 |
| kgriffs | so we have only coded with JSON types in mind | 20:18 |
| kgriffs | however, we are adding msgpack support to the HTTP driver | 20:18 |
| kgriffs | so storage drivers have to allow for all msgpack types | 20:18 |
| vkmc | oic | 20:19 |
| kgriffs | well, actually, the driver has to allow for whatever msgpack types translate to in Python when they are deserialized | 20:19 |
| kgriffs | in SQL we JSON-encode the body before storing it | 20:20 |
| kgriffs | peoplemerge may have to change that as part of his msgpack work | 20:20 |
| kgriffs | in the mongo driver we just try to write the dict, and as long as BSON supports it, we are cool | 20:21 |
| kgriffs | so, I suspect msgpack --> dict --> bson will work OK | 20:21 |
| kgriffs | so that is the situation with how we store the body today | 20:22 |
| kgriffs | now we need to store xattrs | 20:22 |
| kgriffs | how shall we go about that? | 20:22 |
| *** alcabrera|afk is now known as alcabrera | 20:22 | |
| vkmc | if we add attributes on demand, like an amqp-v1.0 attr, then we should be able to manage them for different clients | 20:24 |
| vkmc | when a HTTP client request a message with that attribute, it should ignore it and get only what it requires... the body, ttl and age | 20:25 |
| vkmc | but if an AMQP client request a message with that attribute, Marconi should reconstruct a Proton message using those attributes accordingly and the client should get that message | 20:26 |
| kgriffs | we could also just dump out all the message attributes as I mentioned before, to HTTP. Not sure how useful that is for clients to see | 20:26 |
| kgriffs | may be helpful for diagnostics? | 20:26 |
| kgriffs | are any of those xattrs something that can't be expressed as JSON? | 20:27 |
| kgriffs | I guess that is the rub | 20:28 |
| kgriffs | oh crumbs | 20:28 |
| kgriffs | just thought of something | 20:28 |
| vkmc | in this case... yeah, could be useful as extra information but HTTP cannot do anything with it | 20:28 |
| kgriffs | if someone posts binary payload via msgpack or amqp, and you want to retrieve using JSON... we will have to detect that and base64-encode I guess | 20:28 |
| kgriffs | peoplemerge: ^^^ | 20:29 |
| *** mpanetta has quit IRC | 20:29 | |
| vkmc | that's why I thought about marshalling | 20:32 |
| vkmc | the body at least | 20:32 |
| kgriffs | AMQP body can be binary, right? | 20:38 |
| *** abettadapur has quit IRC | 20:38 | |
| *** sriram has quit IRC | 20:40 | |
| vkmc | kgriffs, yeah it can | 20:41 |
| vkmc | here is the type system spec if you wanna check it out http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-types-v1.0-os.html#section-types | 20:42 |
| kgriffs | ok, I am playing with pymongo to see how it handles binary strings | 20:45 |
| kgriffs | ack | 20:51 |
| kgriffs | http://api.mongodb.org/python/current/tutorial.html#a-note-on-unicode-strings | 20:51 |
| kgriffs | so, this fails (just tried it) | 20:51 |
| kgriffs | db.foo.insert({'thing': b'\x81\x82\x83\x84\x85'}) | 20:51 |
| vkmc | darn | 20:52 |
| kgriffs | this worked | 20:52 |
| kgriffs | db.foo.insert({'thing': bson.Binary(b'\x81\x82\x83\x84\x85')}) | 20:52 |
| vkmc | so we have to check if the payload is binary | 20:53 |
| kgriffs | but then when you do a find | 20:53 |
| kgriffs | you get | 20:53 |
| kgriffs | {u'_id': ObjectId('53cd7d7911cfa134e670dde9'), | 20:53 |
| kgriffs | u'thing': Binary('\x81\x82\x83\x84\x85', 0)} | 20:53 |
| kgriffs | assume that is assigned to a var called "doc" | 20:54 |
| kgriffs | json.dumps(doc) crashes and burns | 20:54 |
| kgriffs | looks like Binary class implements __str__ | 20:55 |
| kgriffs | so json calls str on this thing | 20:55 |
| kgriffs | which yields '\x81\x82\x83\x84\x85' | 20:55 |
| kgriffs | and then json chokes because it can't convert to utf-8 | 20:55 |
| kgriffs | bleh | 20:55 |
| kgriffs | I think we will have to mention this in the driver dev guide | 20:57 |
| vkmc | yeah | 20:57 |
| vkmc | or make mongo store msgpack | 20:58 |
| vkmc | or that's not an option? | 20:58 |
| * vkmc got confused | 20:58 | |
| kgriffs | yeah, there are several related things we need to sort out | 20:59 |
| vkmc | I wonder if Redis can store binary | 21:00 |
| vkmc | if so, we can enforce choosing Redis if AMQP is enabled | 21:00 |
| kgriffs | I'd rather not couple the transport with the backend if we can avoid it | 21:01 |
| vkmc | not the best option but it could work just until we migrate to msgpack or we find another solution | 21:01 |
| kgriffs | let's walk through this | 21:01 |
| kgriffs | first, http | 21:01 |
| kgriffs | actually, http+json | 21:01 |
| kgriffs | someone submits a JSON document. We decode it, assuming all the strings are UTF-8 encoded. | 21:02 |
| kgriffs | now we have a python dict | 21:03 |
| kgriffs | does that dict contain unicode strings? | 21:03 |
| kgriffs | looks like we do | 21:04 |
| kgriffs | strutils.safe_decode(stream.read(len), 'utf-8') | 21:04 |
| kgriffs | ok, that gives a six.text_type string | 21:05 |
| vkmc | good | 21:05 |
| kgriffs | then we do json.loads | 21:05 |
| kgriffs | looks like that decodes to a dict which contains u'' stirngs | 21:06 |
| kgriffs | OK, so everything that is submitted using JSON via HTTP will be persisted as u'' strings | 21:07 |
| kgriffs | so then pymongo will UTF-8 encode those and store as BSON strings | 21:07 |
| kgriffs | sqlalchemy driver will dump to a JSON string, utf-8 encoded | 21:08 |
| kgriffs | ~~~ | 21:08 |
| kgriffs | now, let's walk through what happens with http+msgpack | 21:08 |
| kgriffs | msgpack document comes in | 21:09 |
| kgriffs | we deserialize that to a dict | 21:09 |
| kgriffs | that dict may contain a mixture of b'' and u'' | 21:09 |
| kgriffs | let me verify | 21:09 |
| kgriffs | yeah, so it will contain ttl and body fields. the value of body may be binary, or one of it's sub-fields may be. | 21:12 |
| vkmc | binary or unicode | 21:12 |
| kgriffs | well, technically they could both be unicode - unicode has multiple encodings, utf-16, utf-8, etc. | 21:12 |
| kgriffs | but let's say they use the binary msgpack field type | 21:13 |
| vkmc | cool | 21:13 |
| kgriffs | then it should unpack to str in py2 and binary in py3 | 21:13 |
| kgriffs | specifying the binary msgpack type when the client encodes is important | 21:14 |
| kgriffs | m = msgpack.packb(value, use_bin_type=True) | 21:14 |
| kgriffs | otherwise it is stored using a byte array "string" | 21:14 |
| *** cpallares has joined #openstack-marconi | 21:14 | |
| kgriffs | and when decoding, you don't know whether to decode as a binary string or decode from utf-8 to utf-16 | 21:15 |
| kgriffs | in any case, assuming the client did flag bin fields as such | 21:15 |
| kgriffs | then we decode to a b'' string in py2 | 21:15 |
| kgriffs | but then pymongo is going to barf unless we wrap with Binary | 21:15 |
| kgriffs | boooh | 21:15 |
| vkmc | :/ | 21:16 |
| kgriffs | I don't want to have to crawl the dict looking for strings to wrap | 21:16 |
| kgriffs | (I'm assuming this is only necessary on py2, but still boooh) | 21:16 |
| vkmc | no... it's too expensive | 21:16 |
| kgriffs | man, peoplemerge really should be here | 21:16 |
| kgriffs | we will have to have him read today's log. :) | 21:17 |
| kgriffs | BUT, if we just tell msgpack | 21:18 |
| kgriffs | msgpack.packb(value, use_bin_type=True) | 21:18 |
| vkmc | peoplemerge, peoplemerge_ ^^ (all the pings!) | 21:18 |
| kgriffs | then it will store b'' as-is, and utf-8 encode six.text_type | 21:18 |
| kgriffs | so that would solve storing binary strings | 21:18 |
| kgriffs | but I think we have a problem getting those out the other end | 21:19 |
| kgriffs | let's walk through that next | 21:19 |
| vkmc | sure | 21:19 |
| kgriffs | no, suppose we store the body using msgpack | 21:19 |
| kgriffs | along comes a client | 21:20 |
| kgriffs | says "gimme some 'a that there JSON' | 21:20 |
| kgriffs | so we read the mspack blob from storage | 21:20 |
| kgriffs | deserialize to JSON | 21:20 |
| kgriffs | actually, sorry | 21:21 |
| kgriffs | not JSON | 21:21 |
| kgriffs | first, a python dict | 21:21 |
| kgriffs | that dict will have some b'this-is-not-utf-8' | 21:21 |
| kgriffs | so, we can't simply encode that to JSON, because our JSON must only contain valid UTF-8 bytes | 21:22 |
| kgriffs | at this point, we don't have any information that tells us that the original string was binary | 21:23 |
| kgriffs | so... what do we do? | 21:23 |
| *** shakamunyi_ has quit IRC | 21:24 | |
| vkmc | we should add some kind of flag | 21:24 |
| vkmc | when storing that message | 21:24 |
| vkmc | so when the retrieval happens we know how to reconstruct it | 21:24 |
| vkmc | that could be extra information in the xattrs field | 21:25 |
| kgriffs | in the case of AMQP, that would work. The body is a blob | 21:26 |
| vkmc | each driver has to make sure that adds the required information in xattrs in order to be able to retrieve that information later | 21:26 |
| kgriffs | however, if the message was submitted with msgpack, we could end up with a mixed scenario | 21:26 |
| kgriffs | msgpack comes in, we deserialize, and ideally we have utf-8 encoded strings turn into six.text_type | 21:27 |
| kgriffs | and client specified binary fields turn into six.binary_type | 21:28 |
| *** Obulpathi has quit IRC | 21:29 | |
| kgriffs | then when we serialize back to msgpack for storing, those two field types (both of which may be present in the body) will be flagged as binary and string types, respectively, in the msgpack format | 21:29 |
| kgriffs | but here is what has be concerned | 21:29 |
| kgriffs | actually... | 21:29 |
| kgriffs | we may be ok | 21:29 |
| kgriffs | a unicode string we be output back as six.text_type I think | 21:30 |
| kgriffs | and a binary will be six.binary_type | 21:30 |
| kgriffs | now if we received these via JSON | 21:30 |
| kgriffs | then serialize to msgpack, they will be str type | 21:30 |
| kgriffs | and when we get them back out they will only have six.text_type strings | 21:30 |
| kgriffs | (need to verify) | 21:31 |
| flwang | hey guys, sorry for the rush in | 21:31 |
| kgriffs | so... we could say, any field that is six.binary_type will need to be base64-encoded if the client only accepts JSON | 21:31 |
| flwang | so what's the meeting time of this week? | 21:31 |
| *** amitgandhi has quit IRC | 21:32 | |
| vkmc | flwang, it looks like we will stick one more time to the regular time | 21:32 |
| flwang | yep, I saw kgriffs's mail, just wanna confirm | 21:33 |
| vkmc | oh cool | 21:33 |
| flwang | vkmc: thanks :) | 21:33 |
| vkmc | flwang, np! | 21:33 |
| flwang | kgriffs: the deadline of j-2 is 24 Jul, right? | 21:33 |
| kgriffs | yeah | 21:34 |
| kgriffs | I'd like to get as much as possible merged a day early | 21:34 |
| flwang | kgriffs: I assume my /health bp is on your list :D | 21:34 |
| kgriffs | yeah... we are going to have to slip a few things, but let's get as much done as we can | 21:35 |
| flwang | btw, I have addressed all your comments | 21:35 |
| kgriffs | cool, I'll take another look | 21:35 |
| flwang | pls revisit it when you're most convenience, cheers | 21:35 |
| kgriffs | sure thing | 21:35 |
| kgriffs | vkmc: re unicode madness | 21:35 |
| kgriffs | note for posterity | 21:36 |
| flwang | cool | 21:36 |
| vkmc | D: | 21:36 |
| kgriffs | In [62]: msgpack.unpackb(m, encoding='utf-8') | 21:36 |
| kgriffs | Out[62]: {u'thing': u'thing2', 'value': '\x81\x82\x83\x84\x85'} | 21:36 |
| kgriffs | setting the encoding is important | 21:36 |
| kgriffs | otherwise you get | 21:36 |
| kgriffs | In [60]: msgpack.unpackb(m) | 21:36 |
| kgriffs | Out[60]: {'thing': 'thing2', 'value': '\x81\x82\x83\x84\x85'} | 21:36 |
| kgriffs | that was assuming m came from: | 21:37 |
| kgriffs | In [58]: m = msgpack.packb({u'thing':u'thing2', 'value': value}, use_bin_type=True) | 21:37 |
| kgriffs | where | 21:37 |
| kgriffs | value = b'\x81\x82\x83\x84\x85' | 21:37 |
| kgriffs | ok, so here is the thing that really stinks | 21:37 |
| kgriffs | how do we efficiently base64-encode stuff on the response? | 21:37 |
| kgriffs | can we register our own serializer for six.binary_type? | 21:38 |
| *** celttechie has quit IRC | 21:39 | |
| *** jmckind has quit IRC | 21:39 | |
| vkmc | doesn't look like :/ | 21:39 |
| vkmc | base64.b64encode()? | 21:46 |
| vkmc | it should work for py2 y py3 | 21:47 |
| kgriffs | yes, but we need a way to b64 encode instances of six.binary_type when serializing to JSON | 21:47 |
| kgriffs | hmmm. | 21:49 |
| kgriffs | in the case of AMQP we can have any of a number of content types for the body | 21:49 |
| kgriffs | let's see | 21:50 |
| vkmc | yes | 21:50 |
| kgriffs | suppose we add a content-type field to messages | 21:50 |
| vkmc | we can have primitive types or more complex structures in the payload | 21:50 |
| vkmc | AMQP messages has a content-type fields we can use | 21:51 |
| * kgriffs is thinking | 21:52 | |
| *** oz_akan has quit IRC | 21:55 | |
| *** tonytan4ever has quit IRC | 21:55 | |
| kgriffs | vkmc: are those AMQP content types unique to AMQP or are they internet media types? | 21:55 |
| vkmc | kgriffs, internet media types | 21:57 |
| kgriffs | can you include encoding | 21:58 |
| kgriffs | like | 21:58 |
| vkmc | I'm checking the RFC | 21:58 |
| kgriffs | "application/xml; encoding=utf-16" | 21:58 |
| vkmc | this one http://www.ietf.org/rfc/rfc2046.txt | 21:58 |
| vkmc | it only mentions text/plain or text/enriched | 21:58 |
| kgriffs | oops, I mean | 21:59 |
| kgriffs | "application/xml; charset=utf-16" | 21:59 |
| vkmc | the content type of AMQP is a MIME content type | 21:59 |
| vkmc | and yeah, there is another field | 21:59 |
| vkmc | content-encoding | 22:00 |
| vkmc | :) | 22:00 |
| kgriffs | lots of RFCs | 22:00 |
| kgriffs | https://tools.ietf.org/html/rfc6838 | 22:00 |
| kgriffs | vkmc: oic.... | 22:00 |
| peoplemerge | kgriffs: vkmc: just checked scrollback, I think we're good on all those cases | 22:01 |
| kgriffs | so, it has been proposed in the past by a couple different people to not use JSON at all and move stuff like TTL into X-headers | 22:01 |
| kgriffs | but then it becomes a pain when you want to get more than one message at a time | 22:01 |
| peoplemerge | We unpack msgpack on reciept so it's just a dict to store to mongo etc after | 22:02 |
| peoplemerge | everything is set to msgpck binary | 22:02 |
| *** rwsu has quit IRC | 22:02 | |
| *** keith_newstadt has quit IRC | 22:02 | |
| * peoplemerge is in meetings, be back this eve | 22:03 | |
| kgriffs | peoplemerge: the issue is that after you unpack and send to pymongo, pymongo will barf on six.binary_type if they contain non-unicode chars | 22:03 |
| kgriffs | so I think we need to change the mongo driver to store as msgpack rather than BSON | 22:04 |
| kgriffs | there isn't any reason for mongo to "see" the body fields in any case (they can be opaque) | 22:04 |
| peoplemerge | kgriffs: as part of this story | 22:04 |
| peoplemerge | ? | 22:04 |
| kgriffs | peoplemerge: it will have to be, otherwise you can't get tests that post binary field types in msgpack to marconi | 22:05 |
| kgriffs | peoplemerge: I've actually been wanting to do this for a while anyway | 22:05 |
| peoplemerge | kgriffs: ok | 22:05 |
| peoplemerge | can't mongo intuit json structs for querying tho? | 22:05 |
| kgriffs | because it lets us also use lz4 or snappy | 22:05 |
| peoplemerge | interesting, hadn't thought of that | 22:06 |
| kgriffs | peoplemerge: not automatically | 22:06 |
| vkmc | brb | 22:06 |
| kgriffs | you have to wrap the string in a bson.Binary object | 22:06 |
| peoplemerge | ah nice | 22:06 |
| kgriffs | but... then you have to take the pain to detect what fields to wrap | 22:06 |
| kgriffs | and it seems like we might as well just persist back to msgpack | 22:07 |
| kgriffs | then wrap that whole mspack in a single bson.Binary() | 22:07 |
| peoplemerge | I must got to this meeting but if anyone wants to play with msgpack POST and my ugly unrefactored code https://github.com/peoplemerge/marconi/tree/v1.1msgpack instructions https://gist.github.com/peoplemerge/39a6babea5bc4741f2cd | 22:07 |
| peoplemerge | it works tho :) | 22:07 |
| kgriffs | ah, thanks for sharing | 22:08 |
| vkmc | thanks peoplemerge! | 22:10 |
| vkmc | kgriffs, would you like to start a discussion about this in the ml? | 22:11 |
| vkmc | oops phone, brb again | 22:14 |
| *** mwagner_lap has joined #openstack-marconi | 22:31 | |
| *** keith_newstadt has joined #openstack-marconi | 22:34 | |
| *** Obulpathi has joined #openstack-marconi | 22:47 | |
| *** Obulpathi has quit IRC | 22:52 | |
| *** Obulpathi has joined #openstack-marconi | 22:53 | |
| *** shakamunyi_ has joined #openstack-marconi | 22:53 | |
| *** Obulpathi has quit IRC | 22:55 | |
| *** ametts has quit IRC | 22:56 | |
| *** Catherine has joined #openstack-marconi | 22:56 | |
| peoplemerge | kgriffs: So sorry about that! | 22:57 |
| * peoplemerge is bac | 22:57 | |
| *** Catherine has quit IRC | 23:01 | |
| kgriffs | yeah, we need to discuss this with the team. I will try to write up some specs and put them out for discussion. | 23:06 |
| kgriffs | two things I am pretty sure about though | 23:07 |
| kgriffs | 1. we need to store stuff as msgpack rather than JSON/BSON | 23:07 |
| kgriffs | 2. we should store the content type with the message (regardless of whether we expose that to the clients in message listings) | 23:08 |
| kgriffs | by content type, I mean the type that the body was received as (json, msgpack, or in the case of amqp, whatever the content_type field is) | 23:08 |
| kgriffs | it's getting late for me... | 23:09 |
| peoplemerge | nice | 23:09 |
| kgriffs | so I will have to work on this more tomorrow | 23:09 |
| kgriffs | I started to write up this bp: | 23:09 |
| kgriffs | https://blueprints.launchpad.net/marconi/+spec/store-messages-as-binary | 23:09 |
| vkmc | thanks kgriffs | 23:10 |
| peoplemerge | kgriffs: deadline for j2 is here, should I attempt to finish what's on the translation layer today / tomorrow? | 23:10 |
| kgriffs | I think we will need another bp or something for storing content_type with the message as well - or that may be part of an existing bp, not sure yet | 23:10 |
| peoplemerge | j3 starts wed, no? | 23:10 |
| vkmc | yeah peoplemerge! | 23:10 |
| kgriffs | peoplemerge: mmm, I think we will need to slip this to j3 to get it done right | 23:10 |
| kgriffs | (to be honest) | 23:10 |
| peoplemerge | k | 23:10 |
| kgriffs | because I think it will depend on the bp I linked above | 23:11 |
| kgriffs | but we need to try and get all v1.1 stuff that slips done at the front end of j3 | 23:11 |
| peoplemerge | kgriffs: reading store-message-as-binary bp | 23:11 |
| peoplemerge | there's lots of work to do on python-marconiclient | 23:12 |
| peoplemerge | I didn't know where to start becasuse there is no 1.1 there | 23:12 |
| kgriffs | yep, there sure is a lot of work to do | 23:12 |
| peoplemerge | flaper87 commented on that last night | 23:13 |
| kgriffs | that's another reason we need to get 1.1 done ASAP on the server side, so we can update the client and test against the new API | 23:13 |
| peoplemerge | oic | 23:13 |
| peoplemerge | kgriffs: the bp makes sense to me. I assume we'll want compression be something admin-tunable, right? So if they're already sending compressed data, anything we try to do will make it worse | 23:16 |
| kgriffs | that is a good point. | 23:21 |
| kgriffs | question is, what is the overhead for lz4 or snappy | 23:22 |
| kgriffs | in two cases | 23:22 |
| kgriffs | a base64-encoded jpeg | 23:22 |
| kgriffs | and a raw binary string with the same jpeg | 23:23 |
| kgriffs | ideally, it would just be "always on" | 23:23 |
| kgriffs | or, if we know the content type, we could be smart about it... we could check magic bytes at the beginning of the string | 23:23 |
| kgriffs | hmmm | 23:23 |
| kgriffs | maybe this is something that should wait for K | 23:23 |
| peoplemerge | compression is just a small part of the bp | 23:24 |
| peoplemerge | Do people really use messaging for large binaries like images and vid? | 23:26 |
| peoplemerge | When we architected Verifi's batch platform, I argued against gi-normous binaries in a mq | 23:27 |
| peoplemerge | successfully | 23:27 |
| peoplemerge | kgriffs: always-on might be a better idea, or at least a tunable queue-level thing you can turn off | 23:29 |
| kgriffs | peoplemerge: well, what do you know, I lz4'd a jpg | 23:29 |
| peoplemerge | kgriffs: did it get smaller or larger? | 23:29 |
| kgriffs | it actually got smaller | 23:29 |
| peoplemerge | try doing it with an h.265 vid :) | 23:29 |
| kgriffs | and for the record I agree with you re embedding binary objects | 23:29 |
| kgriffs | generally to be avoided | 23:30 |
| kgriffs | and I don't think it is the most common use case for messaging | 23:30 |
| kgriffs | before size: 19126 | 23:30 |
| kgriffs | after: 18128 | 23:30 |
| peoplemerge | it's not nothing :) | 23:31 |
| kgriffs | took 15 microseconds to do it | 23:31 |
| kgriffs | (on my MBP) | 23:31 |
| peoplemerge | IMO a domain specific *good* algo will outsmart a general purpose one | 23:31 |
| peoplemerge | jp2k? | 23:32 |
| kgriffs | yep, for sure | 23:32 |
| *** cpallares has quit IRC | 23:32 | |
| peoplemerge | but in messaging do we care? | 23:32 |
| *** shakamunyi_ has quit IRC | 23:33 | |
| kgriffs | idk, maybe. We'd have to do some benchmarks, also insert tons of messages see how much we can fit in a reasonable amount of RAM w/o compression | 23:33 |
| peoplemerge | A wise man once said "thou shalt not build hopelessly general apps for thine unknown intended market" | 23:33 |
| kgriffs | that's why I'm starting to think this should go in it's own bp... needs some experimentation | 23:33 |
| peoplemerge | agreed | 23:34 |
| kgriffs | OK, I updated the description on the bp to note that we won't do compression at this time, but the bp does pave the way for it if we want to do it later | 23:36 |
| peoplemerge | version field++ | 23:37 |
| kgriffs | flwang: ping | 23:45 |
| flwang | kgriffs: pong | 23:49 |
| kgriffs | flwang: hey, I just wanted to let you know I'm reviewing the health patch | 23:50 |
| flwang | my baby is waiting for bless | 23:50 |
| kgriffs | oh, no problem | 23:50 |
| kgriffs | on the case | 23:50 |
| flwang | kgriffs: I mean my baby /health patch :D | 23:50 |
| kgriffs | lol | 23:50 |
| kgriffs | at first you had me there, then I realized what you meant | 23:50 |
| flwang | haha | 23:51 |
| flwang | hope we can make it in j-2 | 23:51 |
| kgriffs | I was at church yesterday and we were talking about baby blessings for real. So I guess i had it on my mind. :p | 23:51 |
| flwang | then in j-3, we can fix bugs and adding more KPI | 23:51 |
| kgriffs | aaaaanyway | 23:51 |
| kgriffs | ok | 23:51 |
| flwang | kgriffs: haha, wow | 23:51 |
| kgriffs | so far I just found a little IMO, but I don't think it should block the patch | 23:51 |
| flwang | kgriffs: sure, I can fix it right now and give you a new patch set, sir | 23:52 |
| peoplemerge | haha love it! | 23:52 |
| kgriffs | flwang: I like the _handle_status | 23:54 |
| kgriffs | DRY | 23:54 |
| flwang | it is what it looks based on the valuable comments from you guys :) | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!