| *** reed_ has quit IRC | 00:00 | |
| *** kgriffs is now known as kgriffs_afk | 00:10 | |
| openstackgerrit | A change was merged to openstack/marconi: chore: Expose commit hash in version module https://review.openstack.org/47459 | 00:19 |
|---|---|---|
| *** malini_afk is now known as malini | 00:26 | |
| *** nosnos has joined #openstack-marconi | 00:50 | |
| *** kgriffs_afk is now known as kgriffs | 01:11 | |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: feat(mongodb): Partition data across multiple databases https://review.openstack.org/45952 | 01:32 |
| *** kgriffs is now known as kgriffs_afk | 01:42 | |
| *** malini is now known as malini_afk | 02:05 | |
| *** ayoung has quit IRC | 03:59 | |
| *** tvb has joined #openstack-marconi | 04:46 | |
| *** tvb has quit IRC | 04:58 | |
| *** tvb|afk has joined #openstack-marconi | 05:54 | |
| *** tvb|afk has quit IRC | 05:54 | |
| *** tvb|afk has joined #openstack-marconi | 05:54 | |
| *** flaper87|afk is now known as flaper87 | 06:43 | |
| *** ykaplan has joined #openstack-marconi | 07:59 | |
| *** yassine has joined #openstack-marconi | 08:11 | |
| *** tvb|afk has quit IRC | 08:14 | |
| *** tvb|afk has joined #openstack-marconi | 08:14 | |
| *** tvb|afk has quit IRC | 08:14 | |
| *** tvb|afk has joined #openstack-marconi | 08:14 | |
| *** flaper87 is now known as flaper87|afk | 08:55 | |
| *** flaper87|afk is now known as flaper87 | 08:56 | |
| *** ykaplan has quit IRC | 10:13 | |
| *** flaper87 is now known as flaper87|afk | 10:17 | |
| *** flaper87|afk is now known as flaper87 | 10:29 | |
| *** ykaplan has joined #openstack-marconi | 10:39 | |
| *** flaper87 is now known as flaper87|afk | 11:16 | |
| *** flaper87|afk is now known as flaper87 | 11:21 | |
| *** nosnos has quit IRC | 11:33 | |
| *** tedross has joined #openstack-marconi | 11:45 | |
| *** yassine_ has joined #openstack-marconi | 11:52 | |
| *** yassine has quit IRC | 11:56 | |
| *** ykaplan has quit IRC | 12:16 | |
| *** flaper87 is now known as flaper87|afk | 12:20 | |
| *** flaper87|afk is now known as flaper87 | 12:33 | |
| *** kgriffs_afk is now known as kgriffs | 12:44 | |
| *** mpanetta has joined #openstack-marconi | 12:59 | |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: feat(mongodb): Partition data across multiple databases https://review.openstack.org/45952 | 13:04 |
| *** ykaplan has joined #openstack-marconi | 13:08 | |
| mpanetta | Mornin all. | 13:11 |
| flaper87 | mpanetta: good morning | 13:11 |
| mpanetta | flaper87: How goes? | 13:12 |
| flaper87 | mpanetta: good, what about you? | 13:12 |
| mpanetta | Not too bad, stayed up way too late last night heh. | 13:12 |
| *** jcru has joined #openstack-marconi | 13:14 | |
| *** yassine has joined #openstack-marconi | 13:15 | |
| flaper87 | mpanetta: hahahaha, \_/? here, take some coffee | 13:15 |
| mpanetta | flaper87: Thanks hehe | 13:16 |
| *** yassine_ has quit IRC | 13:19 | |
| * flaper87 is used to stay up too late | 13:20 | |
| flaper87 | dunno if it's a good thing but I'm not used to sleep much | 13:21 |
| mpanetta | I used to be. | 13:21 |
| * flaper87 gets bored while sleeping | 13:21 | |
| flaper87 | seriously | 13:21 |
| flaper87 | :D | 13:21 |
| mpanetta | Was working on PCB design, and listening to music. | 13:21 |
| mpanetta | haha | 13:21 |
| *** oz_akan_ has joined #openstack-marconi | 13:22 | |
| mpanetta | flaper87: Do you know if all the patches for proxy have been merged yet? | 13:24 |
| flaper87 | mpanetta: not all of them | 13:24 |
| flaper87 | there are some pending | 13:25 |
| flaper87 | I'll review them in a bit | 13:25 |
| mpanetta | Ok, cool. | 13:25 |
| *** acabrera has joined #openstack-marconi | 13:33 | |
| acabrera | Morning! | 13:33 |
| *** acabrera is now known as alcabrera | 13:34 | |
| flaper87 | alcabrera: good morning | 13:34 |
| flaper87 | :D | 13:34 |
| alcabrera | flaper87: :) | 13:34 |
| kgriffs | anybody know the trick to passing multiple params to a test function with ddt? | 13:38 |
| kgriffs | or do you just pass in a tuple to a single param and explode it yourself? | 13:39 |
| alcabrera | kgriffs: I've seen the technique in malini's tests before, hmm... | 13:40 |
| flaper87 | mmh, not me | 13:40 |
| flaper87 | :/ | 13:41 |
| alcabrera | I'd need to look it up. | 13:41 |
| alcabrera | kgriffs: found it! | 13:41 |
| flaper87 | kgriffs: when you get a chance, could you take a look here: https://review.openstack.org/#/q/status:open+project:openstack/python-marconiclient,n,z | 13:42 |
| kgriffs | sure | 13:42 |
| flaper87 | those patches need some love :( | 13:42 |
| alcabrera | Check out unit/queues/transport/wsgi/test_claims:76 | 13:42 |
| flaper87 | :D | 13:42 |
| kgriffs | I got a dentist appt soon, but will be back and can review stuff after that | 13:42 |
| flaper87 | kgriffs: thank you! | 13:42 |
| flaper87 | and good luck with that! | 13:42 |
| alcabrera | flaper87: I'm looking - wait, I've already reviewed these. :P | 13:42 |
| kgriffs | alcabrera: ah, I see. must be args, kwargs | 13:42 |
| alcabrera | kgriffs: dentist, yikes! Hope that goes well. :) | 13:42 |
| flaper87 | alcabrera: yup :P | 13:43 |
| kgriffs | me too! | 13:43 |
| flaper87 | I was very direct! | 13:43 |
| flaper87 | huahuhaua | 13:43 |
| *** malini_afk is now known as malini | 13:44 | |
| alcabrera | flaper87, kgriffs: Whenever you can, I'm hoping to have these three proxy patches get a sanity check and a merge (https://review.openstack.org/#/q/status:open+marconi+branch:master+topic:bp/placement-service,n,z). :D | 13:48 |
| alcabrera | I'm still ironing out wrinkles on the latest proxy patch, but those three have been through the review gauntlet. :) | 13:49 |
| kgriffs | ok | 13:50 |
| kgriffs | alcabrera: replied to your question on https://review.openstack.org/#/c/45952/ | 13:50 |
| kgriffs | now i *really* gotta run | 13:50 |
| kgriffs | ttfn | 13:50 |
| alcabrera | kgriffs: take care! | 13:50 |
| *** kgriffs is now known as kgriffs_afk | 13:50 | |
| mpanetta | bye | 13:51 |
| mpanetta | alcabrera: Is there a patchset ready to test yet? Or am I just too excited to test? | 13:51 |
| *** amitgandhi has joined #openstack-marconi | 13:52 | |
| alcabrera | mpanetta: this one (https://review.openstack.org/#/c/48737/) has been ready since yesterday. :D | 13:54 |
| alcabrera | mpanetta: It will get a few modifications API-wise (moving to PATCH from PUT for changing hosts and weight on an existing partition), but the core of it is solid. | 13:54 |
| mpanetta | Hmm, we do not have a test for the admin interface yet AFAIK. | 13:54 |
| mpanetta | I think we are only testing passthrough. | 13:55 |
| mpanetta | I assume we will manually be creating the partitions for test. | 13:55 |
| alcabrera | mpanetta: That's correct on both counts: no tests for admin interface (except unit tests), and partitions will be manually created. | 13:55 |
| alcabrera | I'd be interested in the results of passthrough testing, in any case. | 13:56 |
| mpanetta | Cool | 13:56 |
| mpanetta | I can't wait until I get a chance to start working on automating these tests. | 13:56 |
| mpanetta | Which reminds me... Is there a document that describes how to hook in to the system, like the garritbot does here so I can get commit notfications programmatically? | 13:57 |
| alcabrera | hmmm... | 13:57 |
| alcabrera | Not as far as I know. | 13:58 |
| alcabrera | unless... | 13:58 |
| alcabrera | Starring the github repo for patches merged to master? | 13:58 |
| mpanetta | That just sends emails tho doesn't it? | 13:58 |
| alcabrera | That doesn't give you news on patches submitted for review, though. | 13:58 |
| mpanetta | Yeah | 13:58 |
| mpanetta | That is all done through launchpad? | 13:59 |
| alcabrera | mpanetta: Github exposes an RSS feed for repos you're watching. That's independent of launchpad. | 13:59 |
| flaper87 | you can also watch projects on gerrit | 14:00 |
| flaper87 | it'll send you emails | 14:00 |
| flaper87 | but it's a bit spammy | 14:00 |
| alcabrera | hmmm, I've yet to try that. | 14:00 |
| mpanetta | Hmm | 14:00 |
| alcabrera | I don't need more spam, haha. >.> | 14:00 |
| flaper87 | damn, #@!$#@%! connection | 14:02 |
| mpanetta | ? | 14:04 |
| mpanetta | I didn't see ya drop | 14:04 |
| mpanetta | Actually I am surprsed I have not dropped yet today... | 14:04 |
| flaper87 | mpanetta: I'm just like that! :P | 14:04 |
| mpanetta | flaper87: Or you use an IRC proxy, or have it in a screen somewhere :P | 14:05 |
| flaper87 | mpanetta: damn, you figured it out! :P | 14:05 |
| mpanetta | haha | 14:05 |
| flaper87 | I think my former ISP is taking revenge on me in these last days of contract | 14:05 |
| flaper87 | I can picture a guy unplugging my wire every 5 mins | 14:06 |
| * mpanetta loves tuesdays at rackspace | 14:06 | |
| mpanetta | lol | 14:06 |
| mpanetta | Mmm frut tuesdays! | 14:06 |
| *** Alex_Gaynor has quit IRC | 14:15 | |
| *** gleicon___ has quit IRC | 14:17 | |
| *** tvb|afk has quit IRC | 14:18 | |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: fix(proxy): forward marconi headers https://review.openstack.org/47628 | 14:24 |
| zyuan | kgriffs_afk: i failed to find a parameter name better than `d` :( | 14:24 |
| malini | mpanetta, alcabrera: Sorry missed the discussion on proxy tests | 14:26 |
| malini | I have somewhat of a skeleton for the proxy functional tests | 14:27 |
| malini | + ideas on enhancement, make them runnable with tox | 14:27 |
| malini | the second part of it is going to take some time & wont be ready in the next week or so | 14:27 |
| malini | But the basic API tests (tht assume we have a partition to play around with on the tests) are almost there | 14:28 |
| malini | & what do you call a pass through testing ? | 14:29 |
| alcabrera | malini: When I hear "pass thorugh testing", I imagine the functional tests for the queues API being run so that they're pointed at a proxy. | 14:32 |
| alcabrera | Then, it just checks that the proxy is able to behave like a marconi-server (given that the proxy has marconi-server partitions registered prior to the test runnning). | 14:33 |
| malini | alcabrera: assuming the proxy is already set up (or) the tests spin up the proxy ? | 14:33 |
| malini | alcabrera: in tht case, I have most of pass through testing already in the works | 14:34 |
| malini | I am almost 75% through it | 14:34 |
| alcabrera | malini: assuming the proxy is already set up would probably be easiest in the short term, but having the tests launch two marconi-servers, an admin proxy, a proxy forwarder, registering the partitions, and then executing would be better for automation in the long run. | 14:34 |
| alcabrera | awesome, malini! | 14:34 |
| * flaper87 is working on the config thing | 14:34 | |
| flaper87 | wish me luck | 14:34 |
| malini | alcabrera: exactly..you read my mind :) | 14:34 |
| flaper87 | if you don't hear from me every 5mins, CALL 911 | 14:35 |
| flaper87 | no wait, it'll take them way to much to get here | 14:35 |
| malini | flaper87: do I need the area code ? ;) | 14:35 |
| flaper87 | CALL 118 | 14:35 |
| malini | good luck! | 14:35 |
| alcabrera | flaper87: lol, good luck. :) | 14:37 |
| flaper87 | btw, do you guys follow OSReactions on twitter? | 14:37 |
| flaper87 | we're not publishing all our posts on the planet | 14:37 |
| flaper87 | you know, we don't want to spam OS Planet | 14:37 |
| flaper87 | but twitter seems like a spamable stream | 14:38 |
| flaper87 | :P | 14:38 |
| alcabrera | flaper87: not following yet. :P | 14:38 |
| alcabrera | flaper87: now I'm following, haha. | 14:39 |
| flaper87 | alcabrera: not that you should, just wanted to let you know | 14:39 |
| flaper87 | :D | 14:39 |
| flaper87 | alcabrera: awesome | 14:39 |
| alcabrera | flaper87: haha, of course. A little openstack amusement won't hurt my twitter stream. :D | 14:41 |
| *** Alex_Gaynor has joined #openstack-marconi | 14:41 | |
| *** ayoung has joined #openstack-marconi | 14:44 | |
| zyuan | https://review.openstack.org/#/c/45952/ ready to merge, afaics | 14:45 |
| *** mpanetta_ has joined #openstack-marconi | 14:58 | |
| *** mpanetta has quit IRC | 14:58 | |
| *** mpanetta_ is now known as mpanetta | 15:24 | |
| *** kgriffs_afk is now known as kgriffs | 15:29 | |
| alcabrera | kgriffs: o/ | 15:30 |
| kgriffs | hi | 15:30 |
| flaper87 | HOLY*********************************** | 15:31 |
| * flaper87 just saw something | 15:31 | |
| alcabrera | flaper87: ? | 15:31 |
| mpanetta | Uhoh | 15:32 |
| malini | it better be UFO | 15:32 |
| malini | The aliens got him :( | 15:33 |
| alcabrera | malini: we | 15:34 |
| alcabrera | malini: no more flaper87. With aliens, he's in a whole new world now. :P | 15:34 |
| malini | was a good guy.. | 15:35 |
| mpanetta | Ah crap, how am I gonna test now? :P | 15:35 |
| flaper87 | soooooo: What's wrong here? https://github.com/openstack/marconi/blob/master/marconi/queues/bootstrap.py#L48 | 15:35 |
| flaper87 | and line 60 | 15:35 |
| alcabrera | ooohhh | 15:35 |
| alcabrera | I see. | 15:35 |
| alcabrera | queues | 15:35 |
| flaper87 | :D | 15:35 |
| * flaper87 fixes that NOW! | 15:36 | |
| alcabrera | that things that get missed when porting projects to new directory structures~ | 15:36 |
| flaper87 | we were lucky because we didn't fix that here neither https://github.com/openstack/marconi/blob/master/setup.cfg#L29 | 15:36 |
| alcabrera | lol I just checked | 15:36 |
| alcabrera | Things continued to work silently, deviously. | 15:36 |
| kgriffs | mpanetta: you see the new marconi.version.verify_sha function? | 15:37 |
| mpanetta | kgriffs: Yes, TYV! | 15:37 |
| mpanetta | er TYVM! | 15:37 |
| kgriffs | kool | 15:37 |
| kgriffs | hope it helps. :D | 15:38 |
| mpanetta | Oh crap, I just checked out and installed master on test env heh | 15:38 |
| flaper87 | malini: btw, LOL @ UFO | 15:38 |
| flaper87 | (rofl) | 15:38 |
| malini | glad to have you back flaper87 ;) | 15:39 |
| mpanetta | He could have at least brought a UFO back with him... | 15:41 |
| flaper87 | you realize you wiped your laptop when git review gives you "Command not found" | 15:42 |
| mpanetta | oops | 15:42 |
| openstackgerrit | Flavio Percoco proposed a change to openstack/marconi: Fix queue's packages namespaces https://review.openstack.org/49174 | 15:42 |
| flaper87 | very quick review ^ | 15:42 |
| flaper87 | alcabrera: kgriffs ^ | 15:42 |
| zyuan | kgriffs: ping | 15:43 |
| kgriffs | mpanetta: I'd love to get a perf test on this sucker - https://review.openstack.org/#/c/45952/ | 15:43 |
| kgriffs | (before it is merged) | 15:43 |
| zyuan | kgriffs: i said in comments, the name 'capitalized' is like 'sorted' | 15:43 |
| mpanetta | Er, um. Will I have time before y'all are done mergng for main test? | 15:44 |
| zyuan | it can't be a verb | 15:44 |
| zyuan | kgriffs: and i can't find a name better than 'd'; can you give me a suggestion? | 15:44 |
| kgriffs | no, you can leave as-is | 15:44 |
| kgriffs | I forgot that python uses that convention | 15:45 |
| zyuan | (dict -- name collision; 'dic'? oh no....) | 15:45 |
| kgriffs | source? | 15:45 |
| kgriffs | idk | 15:45 |
| alcabrera | flaper87: +2'd | 15:45 |
| zyuan | ...... | 15:45 |
| kgriffs | headers? | 15:45 |
| alcabrera | zyuan: how about headers? | 15:46 |
| zyuan | you had headers_dict somewhere | 15:46 |
| zyuan | it's ok... i mean 'headers' | 15:46 |
| kgriffs | flaper87: https://review.openstack.org/#/c/48287/7/marconiclient/errors.py | 15:47 |
| kgriffs | what is the advantage of that iml vs. just calling super and passing in the msg | 15:47 |
| kgriffs | ? | 15:47 |
| zyuan | err, not quite... | 15:47 |
| zyuan | this function is generic, it does not know whether the input is headers... | 15:48 |
| zyuan | maybe i just say 'dictionary' | 15:48 |
| zyuan | :( | 15:48 |
| flaper87 | kgriffs: I guess message formatting for subclasses and more specific errors | 15:48 |
| flaper87 | notice the *maybe* | 15:49 |
| alcabrera | zyuan: given that it is generic, it does make assumptions about the format of the input strings. They are of the format "a-b". | 15:49 |
| flaper87 | I'm not very happy with that, I think I pulled that in when porting some things from oslo.messaging and then never used it. | 15:49 |
| kgriffs | if you like, you could make it generic by allowing the caller to pass a sep string | 15:49 |
| kgriffs | and it could default to "-" | 15:49 |
| zyuan | alcabrera: ah, yea | 15:50 |
| kgriffs | but, not sure if it's worth it (YAGNI) | 15:50 |
| flaper87 | kgriffs: I prefer the caller to format the message itself | 15:50 |
| zyuan | stop... lets just call it headers... | 15:50 |
| flaper87 | kgriffs: I don't think it is | 15:50 |
| alcabrera | zyuan: +1 | 15:50 |
| flaper87 | it's a left over from a very nasty copy / paste | 15:50 |
| kgriffs | flaper87: heh. paste happens. | 15:50 |
| flaper87 | kgriffs: yeaaah :P | 15:51 |
| flaper87 | kgriffs: let me do that now that you're reviewing | 15:51 |
| kgriffs | flaper87: how accurate is this? http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/ | 15:51 |
| kgriffs | it's a few years old, so I am just wondering if RabbitMQ has fixed some of those warts in the meantime | 15:52 |
| *** yassine has quit IRC | 15:52 | |
| flaper87 | kgriffs: Haven't read that one, I'll take a look and let you know | 15:53 |
| kgriffs | kewl, thanks | 15:53 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: fix(proxy): forward marconi headers https://review.openstack.org/47628 | 15:54 |
| openstackgerrit | Flavio Percoco proposed a change to openstack/python-marconiclient: Add a base transport class https://review.openstack.org/48287 | 15:54 |
| openstackgerrit | Flavio Percoco proposed a change to openstack/python-marconiclient: Add prepare_request function to requests.py https://review.openstack.org/48669 | 15:54 |
| flaper87 | kgriffs: ^ | 15:55 |
| flaper87 | alcabrera: ^ | 15:55 |
| flaper87 | alcabrera: you just need to look at this one: https://review.openstack.org/#/c/48287/8 | 15:55 |
| alcabrera | woot, reviews | 15:55 |
| flaper87 | no big changes, just applied kgriffs suggestion re MarconiError | 15:56 |
| alcabrera | flaper87: +2'd :) | 15:56 |
| flaper87 | alcabrera: thank you. sir. | 15:56 |
| alcabrera | zyuan: +2'd | 15:57 |
| *** malini has left #openstack-marconi | 16:00 | |
| *** malini has joined #openstack-marconi | 16:03 | |
| alcabrera | flaper87, kgriffs: I take back what I said about python-jsonschema being slow. I learned how to use it better and ran some new tests with the increased knowledge. | 16:12 |
| flaper87 | alcabrera: awesome, so glad to hear that! Any plans to use it instead of our current impl ? | 16:12 |
| alcabrera | if you use jsonschema.validate(data, schema), it's thoroughly slow (1.46ms per check). If you first instantiate a validator instance (jsonschema.Draft4Validator(schema), then do instance.validate(data) => 46.2 us per loop | 16:13 |
| alcabrera | flaper87: I'm tinkering with it for the PATCH implementation of proxy/partitions | 16:13 |
| *** gleicon___ has joined #openstack-marconi | 16:14 | |
| *** flaper87 is now known as flaper87|afk | 16:21 | |
| kgriffs | alcabrera: perf on pypy? | 16:24 |
| kgriffs | also, if it looks like something we want to use, we should have a hackday and make it scream | 16:25 |
| alcabrera | kgriffs: lemme check that out on pypy :D | 16:26 |
| *** mpanetta has quit IRC | 16:27 | |
| *** mpanetta has joined #openstack-marconi | 16:27 | |
| *** mpanetta is now known as mpanetta_lunch | 16:29 | |
| *** ykaplan has quit IRC | 16:36 | |
| alcabrera | kgriffs: the results are in: http://paste.openstack.org/show/47779/ | 16:37 |
| alcabrera | A proper microbenchmarking across pypy-2.1, python-2.7.5, and python-3.3.2 | 16:37 |
| kgriffs | u should make that a gist | 16:38 |
| kgriffs | you know, for posterity | 16:38 |
| alcabrera | +1 | 16:38 |
| kgriffs | and for reference when we get around to making an I <3 microbenchmarking website | 16:38 |
| kgriffs | hmm, 8 us is still pretty slow | 16:39 |
| kgriffs | there must be a way to optimize jsonschema | 16:39 |
| * kgriffs adds that to hackday brainstorming list | 16:39 | |
| alcabrera | kgriffs: done: https://gist.github.com/cabrera/6781406 :) | 16:40 |
| kgriffs | +1 | 16:40 |
| *** ykaplan has joined #openstack-marconi | 16:41 | |
| kgriffs | btw… https://code.google.com/p/google-mobwrite/source/browse/trunk/lib/json_validator.py | 16:41 |
| kgriffs | for simple validation | 16:41 |
| kgriffs | would you mind benching that as well? | 16:41 |
| kgriffs | that is what I used in RSE | 16:41 |
| alcabrera | hmmm | 16:42 |
| alcabrera | I'm checking it out now. | 16:43 |
| kgriffs | zyuan: https://review.openstack.org/#/c/47628/6/marconi/proxy/resources/queues.py | 16:43 |
| kgriffs | I was just thinking, it may be nice to add a # NOTE(zyuan) comment | 16:43 |
| kgriffs | explaining why capitalized is necessary | 16:43 |
| kgriffs | and also maybe a TODO to remove it once normalization is added to falcon | 16:43 |
| alcabrera | kgriffs: I'm running into an interesting result on benching that already. json.dumps is faster than json_validator.is_valid on all interpreters I have. | 16:47 |
| alcabrera | ~10x faster on pythons, and about .25 us faster on pypy. | 16:47 |
| alcabrera | pasting now | 16:48 |
| kgriffs | really? seems like it was fast back in RSE days when i tested. | 16:48 |
| kgriffs | oh | 16:48 |
| kgriffs | are you using simplejson? | 16:48 |
| alcabrera | no, I'm using native json | 16:48 |
| alcabrera | import json | 16:48 |
| kgriffs | hmm | 16:48 |
| kgriffs | that is curious. even faster on 2.6 ? | 16:48 |
| alcabrera | I haven't checked 2.6. I wonder if I have that installed...? >>> | 16:48 |
| alcabrera | >.> | 16:48 |
| kgriffs | oh, and results may differ with complex docs | 16:49 |
| kgriffs | you should try something with nested objects and stuff | 16:49 |
| kgriffs | just be sure | 16:49 |
| alcabrera | sure thing | 16:49 |
| alcabrera | kgriffs: found an error in my benchmark - retrying. This invalidates all previous data, heh. | 16:51 |
| alcabrera | I was doing something along the lines of doc = json.dumps(...); %timeit json.dumps(doc) [double dumping] | 16:51 |
| kgriffs | d00d | 16:59 |
| kgriffs | objectid.ObjectId() is slow | 16:59 |
| kgriffs | </out-of-the-blue> | 16:59 |
| alcabrera | kgriffs: http://paste.openstack.org/show/47780/ (json vs. json_validator) | 17:00 |
| alcabrera | lol | 17:00 |
| kgriffs | setting manipulate=False and letting mongod generate the id is 10 us faster | 17:00 |
| kgriffs | (slow is a relative term) | 17:00 |
| kgriffs | (one man's fast is another man's slow, esp. when that other man is kgriffs) | 17:00 |
| kgriffs | :p | 17:00 |
| alcabrera | lol, indeed | 17:01 |
| alcabrera | json_validator looks wicked slow compared to json | 17:01 |
| openstackgerrit | A change was merged to openstack/marconi: Fix queue's packages namespaces https://review.openstack.org/49174 | 17:01 |
| alcabrera | still in the us range, got about 30-40x slower than json.dumps | 17:01 |
| kgriffs | wow | 17:02 |
| kgriffs | not sure why I thought it was faster. unless it is a py26 thing | 17:02 |
| kgriffs | brb | 17:02 |
| alcabrera | kgriffs: seems to be a python 2.6 thing: http://stackoverflow.com/questions/5158773/how-to-speed-up-json-decode-and-encode-operations-in-python | 17:03 |
| alcabrera | I imagine they ported the improvements in simplejson to the stdlib from 2.6 -> 2.7 | 17:03 |
| kgriffs | gtk | 17:08 |
| *** reed has joined #openstack-marconi | 17:08 | |
| *** mpanetta_lunch is now known as mpanetta | 17:12 | |
| torgomatic | yeah, simplejson on 2.6 is way faster than stdlib, but in 2.7, they're fairly equivalent | 17:19 |
| torgomatic | I did some benchmarking at one point; results are here: https://gist.github.com/smerritt/6083828 | 17:19 |
| *** ykaplan has quit IRC | 17:21 | |
| alcabrera | torgomatic: thanks for sharing! Via reading your gist, I also learned about pypi:benchmark and pypi:benchmarker | 17:26 |
| alcabrera | cool, cool. :D | 17:26 |
| mpanetta | kgriffs: TypeError: verify_sha() takes exactly 1 argument (0 given) | 17:27 |
| mpanetta | Um, is that a class method or an instance method? heh | 17:27 |
| mpanetta | Oh! | 17:28 |
| mpanetta | I need to pass in the expected commit sha? | 17:29 |
| kgriffs | yes | 17:29 |
| kgriffs | it is a string | 17:29 |
| kgriffs | you can see the dosctring either from a REPL (help(marconi.version.verify_sha)) or by browsing teh source. | 17:30 |
| kgriffs | protip: ^^^ | 17:31 |
| kgriffs | :p | 17:31 |
| mpanetta | haha, yeah that is how I figured it out | 17:31 |
| mpanetta | Mmmm source! | 17:31 |
| mpanetta | Crap... Quote interpolation strikes again... | 17:34 |
| kgriffs | Luuuuuke. Use the source, Luke! You must learn the ways of the source, if you're to come with me to OpenStack. | 17:37 |
| mpanetta | hah | 17:37 |
| *** tvb|afk has joined #openstack-marconi | 17:57 | |
| *** JRow has joined #openstack-marconi | 18:10 | |
| *** JRow has left #openstack-marconi | 18:10 | |
| kgriffs | flaper87|afk - pls review https://review.openstack.org/#/c/45952/ | 18:25 |
| kgriffs | ! | 18:25 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: fix(proxy): forward marconi headers https://review.openstack.org/47628 | 18:27 |
| *** ykaplan has joined #openstack-marconi | 18:46 | |
| zyuan | kgriffs: what is the format of project ID? | 19:08 |
| zyuan | kgriffs: i think it make sense to validate client ID if we know it's UUID | 19:08 |
| kgriffs | it is either an integer or a uuid | 19:08 |
| zyuan | but project ID is deployment-dependent; it can be anything | 19:08 |
| zyuan | for example, if marconi is deployed outside rackspace, it really can be anything | 19:09 |
| kgriffs | i asked on the dev list, and they said it used to be a number (auto-inc primary key, I suspect), but in recent releases it is now a uuid | 19:09 |
| kgriffs | meaning, the openstack dev list | 19:09 |
| zyuan | kgriffs: ^^ which ID? | 19:09 |
| kgriffs | project ID | 19:10 |
| zyuan | then it's different from rackspace | 19:10 |
| kgriffs | let me forward the email | 19:10 |
| kgriffs | heh | 19:11 |
| kgriffs | so, looks like the best we can do is check for a sane length | 19:11 |
| zyuan | may...be? | 19:11 |
| kgriffs | i suspect that only allowing ASCII would also be fine - can't image why people would generate an ID with non-ascii chars | 19:12 |
| zyuan | UUID is fairly easy; just uuid.UUID(hex='...') you done | 19:12 |
| kgriffs | zyuan: forwarded the email | 19:12 |
| zyuan | thanks | 19:12 |
| zyuan | then... what's the desized client ID? | 19:13 |
| zyuan | UUID, which version? another thing is, if we know it's UUID, we can store the bytes form instead of the hex form in DB | 19:13 |
| zyuan | (i like binary as impl :) | 19:14 |
| zyuan | kgriffs: ^^ | 19:14 |
| *** malini is now known as malini_afk | 19:21 | |
| *** EmilienM has quit IRC | 19:27 | |
| alcabrera | zyuan, kgriffs: project-ID is deployment-dependent - it could theoretically be any string, and this may be moreso the case based on somethings I've heard about keystone. | 19:27 |
| *** EmilienM has joined #openstack-marconi | 19:30 | |
| alcabrera | mpanetta: proxy notes sent. :) | 19:31 |
| mpanetta | alcabrera: Danke :) | 19:33 |
| mpanetta | alcabrera: so I should run marconi as user marconi now, correct? Currently we are running as root. | 19:35 |
| mpanetta | er s/marconi/proxy/ | 19:35 |
| alcabrera | mpanetta: I've always run an a non-root user. :) | 19:36 |
| mpanetta | alcabrera: It is a good practice to have | 19:36 |
| alcabrera | agreed! | 19:36 |
| mpanetta | I don't even allow root logins on my systems that have internet access... Only sudo. | 19:37 |
| alcabrera | so there should be no restrictions on running as non-root, as long as you ensure that the logging directory in the confs points to a directory the user can write to. | 19:37 |
| mpanetta | probably will set it to ~marconi/log/ | 19:37 |
| mpanetta | Actually if it is looking in ~/.marconi I can still run as root for test... | 19:38 |
| mpanetta | so I can keep it as /usr/local | 19:38 |
| mpanetta | er | 19:38 |
| mpanetta | as /var/log | 19:39 |
| alcabrera | yup | 19:39 |
| mpanetta | every time I see pbr I think 'Pabst Blue Ribbon'... And I don't even drink beer... | 19:43 |
| alcabrera | mpanetta: lol | 19:55 |
| mpanetta | alcabrera: Oh, I just saw your email, feel free to answer my latest question *after* you are done fixing install ;) | 19:57 |
| *** vkmc has joined #openstack-marconi | 20:01 | |
| *** vkmc has quit IRC | 20:01 | |
| *** vkmc has joined #openstack-marconi | 20:01 | |
| zyuan | can i get this merged btw? https://review.openstack.org/#/c/47628/ | 20:04 |
| alcabrera | mpanetta: I think I figured it out. Doing 'pip install .' seems to work reliably, though it seemed like I needed to log off to get the environment to register that I installed pbr. >.> | 20:04 |
| alcabrera | zyuan: +2'd again | 20:06 |
| mpanetta | Oh ok, that was what we are doing already then... | 20:06 |
| kgriffs | zyuan: thanks for updating the commit message | 20:31 |
| *** tvb|afk has quit IRC | 20:31 | |
| *** flaper87|afk is now known as flaper87 | 20:32 | |
| kgriffs | however, i am not seeing a mention in there as to why the normalization is necessary. I'd like to see a mention of the fact that resp.headers are lowercase and falcon expects them to be captalized - maybe even a todo to remove the transform once falcon is updated to support normalization itself | 20:32 |
| kgriffs | the commit message is fine | 20:32 |
| kgriffs | i'd like to see that extra info added to the NOTE | 20:33 |
| kgriffs | sorry to nitpick on this, but I can see someone some day getting tripped up by this code without having a proper hint | 20:33 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: fix(proxy): forward marconi headers https://review.openstack.org/47628 | 20:35 |
| flaper87 | kgriffs: +2'd thanks for the hard work there | 20:38 |
| kgriffs | w00t | 20:38 |
| flaper87 | oz_akan_: thank you as well! | 20:38 |
| zyuan | yeah! | 20:38 |
| flaper87 | oz_akan_: great idea! | 20:38 |
| oz_akan_ | hi | 20:39 |
| kgriffs | http://i.imgur.com/UvgtEi6.gif | 20:39 |
| oz_akan_ | what did I do? | 20:39 |
| kgriffs | oz_akan gets kudos for the algorithm. His idea to perform the partition lookup via an array index was rather elegant. | 20:40 |
| flaper87 | kgriffs: LOOOL | 20:40 |
| zyuan | aha, mongo can store UUID directlyt | 20:41 |
| oz_akan_ | oh, yes, that is very great, pure awesomeness, as if I had 10 ideas to choose from :D | 20:41 |
| kgriffs | oz_akan_: what can you say, you're a natural! | 20:42 |
| oz_akan_ | thanks to kgriffs for getting it done. Ideas don't really matter unless they see sun light | 20:43 |
| oz_akan_ | zyuan: as well for the idea to have only one db for queues | 20:44 |
| alcabrera | kgriffs, oz_akan_: awesome work, guys! multidb is pretty sweet for the mongodb storage layer. I'm sure future storage drivers will be able to utilize a similar technique. | 20:44 |
| oz_akan_ | ..and multiple for messages | 20:44 |
| kgriffs | also thanks to zyuan | 20:46 |
| alcabrera | zyuan: +1 | 20:46 |
| kgriffs | you guys are almost as awesome as this guy. almost. :D | 20:46 |
| kgriffs | https://duckduckgo.com/?q=pop+and+lock+dance | 20:46 |
| kgriffs | btw, he is doing all of that in real time - no video tricks | 20:47 |
| kgriffs | </off-topic> | 20:47 |
| flaper87 | zyuan: +1 | 20:47 |
| openstackgerrit | A change was merged to openstack/marconi: feat(mongodb): Partition data across multiple databases https://review.openstack.org/45952 | 20:47 |
| kgriffs | just like oz_akan_ and zyuan: no tricks. they are 100% for realz | 20:48 |
| kgriffs | :p | 20:48 |
| zyuan | thanks | 20:48 |
| kgriffs | </lame-attempt-at-making-video-on-topic> | 20:48 |
| kgriffs | :D | 20:49 |
| kgriffs | aaaaaanway | 20:49 |
| kgriffs | that's the last schema-changing patch, hoooray! | 20:49 |
| zyuan | LOL | 20:49 |
| kgriffs | thanks again for everyone who helped git-r-done | 20:49 |
| zyuan | too many sudo rm /var/lib/db.... | 20:49 |
| zyuan | oops | 20:50 |
| zyuan | it's not the last | 20:50 |
| zyuan | client-id as uuid | 20:50 |
| kgriffs | alcabrera: https://review.openstack.org/#/c/47628/ | 20:52 |
| kgriffs | ready to rock on that | 20:52 |
| alcabrera | kgriffs: flaper87 beat me to it. ;) | 20:54 |
| zyuan | thanks! | 20:54 |
| *** ykaplan has quit IRC | 20:55 | |
| * flaper87 sheath his sword and bows... | 20:56 | |
| alcabrera | flaper87: +10 XP | 20:56 |
| alcabrera | flaper87: +2 GP | 20:56 |
| openstackgerrit | A change was merged to openstack/marconi: fix(proxy): forward marconi headers https://review.openstack.org/47628 | 20:56 |
| flaper87 | :D | 20:56 |
| flaper87 | brb | 20:59 |
| *** tedross has quit IRC | 21:06 | |
| alcabrera | rebasing~ | 21:11 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: proxy: mirror structure of marconi queues + bootstrap https://review.openstack.org/47642 | 21:11 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: separate config for queues and proxy https://review.openstack.org/48333 | 21:12 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: split proxy API into admin and public apps https://review.openstack.org/48495 | 21:15 |
| *** jcru has quit IRC | 21:15 | |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat (proxy/admin): allow partition modifications https://review.openstack.org/48737 | 21:22 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat (proxy/admin): allow partition modifications https://review.openstack.org/49262 | 21:23 |
| *** ayoung has quit IRC | 21:25 | |
| *** vkmc has quit IRC | 21:30 | |
| *** flaper87 is now known as flaper87|afk | 21:30 | |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: split proxy API into admin and public apps https://review.openstack.org/48495 | 21:31 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat (proxy/admin): allow partition modifications https://review.openstack.org/49262 | 21:33 |
| alcabrera | flaper87|afk, kgriffs, zyuan: The rebasing frenzy is over. Whenever you guys get the chance, I'd love a sanity check on the first three patches and a review on the last one (PATCH /v1/partitions/{partition}). :D | 21:35 |
| alcabrera | I'm out for the night. Take care, guys. | 21:35 |
| *** alcabrera has quit IRC | 21:36 | |
| *** oz_akan_ has quit IRC | 22:19 | |
| *** oz_akan_ has joined #openstack-marconi | 22:19 | |
| *** oz_akan_ has quit IRC | 22:24 | |
| *** flaper87|afk is now known as flaper87 | 22:27 | |
| *** mpanetta_ has joined #openstack-marconi | 22:28 | |
| *** amitgandhi has quit IRC | 22:29 | |
| *** mpanetta has quit IRC | 22:32 | |
| flaper87 | kgriffs: ping | 22:32 |
| flaper87 | kgriffs: could you take another look at client's patches ? https://review.openstack.org/#/q/status:open+project:openstack/python-marconiclient,n,z | 22:32 |
| flaper87 | I'd like to get those merge | 22:35 |
| flaper87 | and keep working on the client tomorrow | 22:36 |
| kgriffs | flaper87: sure | 22:40 |
| flaper87 | kgriffs: thank youuuuuuuu!!!! :D | 22:41 |
| kgriffs | my pleasure | 22:42 |
| kgriffs | :D | 22:42 |
| *** mpanetta_ is now known as mpanetta | 22:43 | |
| *** mpanetta has quit IRC | 22:55 | |
| openstackgerrit | A change was merged to openstack/python-marconiclient: Restructure client code https://review.openstack.org/48223 | 22:55 |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: fix(wsgi): Non-ASCII characters in the queue name trigger 500 status https://review.openstack.org/48776 | 22:56 |
| *** kgriffs is now known as kgriffs_afk | 23:11 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack/python-marconiclient: Add a base transport class https://review.openstack.org/48287 | 23:13 |
| openstackgerrit | Flavio Percoco proposed a change to openstack/python-marconiclient: Add prepare_request function to requests.py https://review.openstack.org/48669 | 23:13 |
| *** reed has quit IRC | 23:13 | |
| *** oz_akan_ has joined #openstack-marconi | 23:30 | |
| *** flaper87 is now known as flaper87|afk | 23:30 | |
| *** oz_akan_ has quit IRC | 23:34 | |
| openstackgerrit | A change was merged to openstack/python-marconiclient: Add common http class https://review.openstack.org/48236 | 23:40 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
