| *** vkmc has quit IRC | 00:07 | |
| *** dafter has joined #openstack-marconi | 00:16 | |
| *** dafter has quit IRC | 00:16 | |
| *** dafter has joined #openstack-marconi | 00:16 | |
| *** dafter has quit IRC | 00:20 | |
| *** malini is now known as malini_afk | 00:23 | |
| *** tedross has quit IRC | 00:26 | |
| *** amitgandhi1 has quit IRC | 00:35 | |
| *** amitgandhi has joined #openstack-marconi | 00:35 | |
| *** malini_afk is now known as malini | 00:46 | |
| *** alcabrera has joined #openstack-marconi | 00:56 | |
| alcabrera | malini, oz_akan_: Evening, guys. :) | 00:56 |
|---|---|---|
| oz_akan_ | hi alcabrera | 00:56 |
| malini | Evening :) | 00:57 |
| malini | oz_akan: I'll update the test now | 00:57 |
| oz_akan_ | malini: thanks | 00:57 |
| alcabrera | I just saw the hexadecimal email. | 00:57 |
| alcabrera | Yeah, zyuan's change was merged today that enforces UUIDs for client-id. | 00:58 |
| alcabrera | oz_akan_, malini: https://review.openstack.org/#/c/49378/ | 00:58 |
| alcabrera | That one ^^ | 00:58 |
| alcabrera | So... I imagine test code will need some updates, possibly the tsung stuff, too. | 00:59 |
| malini | yeap..I am on it | 00:59 |
| malini | oz_akan: do you just need tsung_simple1.xml ? | 00:59 |
| oz_akan_ | no | 01:00 |
| malini | alcabrera: anything generated by uuid.uuid1() shud be good enuf, rt? | 01:00 |
| *** amitgandhi has quit IRC | 01:00 | |
| alcabrera | malini: yeah, that should be fine. | 01:00 |
| oz_akan_ | I usem ike-incremental.xml | 01:00 |
| oz_akan_ | I use mike-incremental.xml | 01:00 |
| oz_akan_ | I am running it actually now | 01:00 |
| alcabrera | pass it as str(uuid.uuid1()) and you should be fine. :) | 01:01 |
| oz_akan_ | I already switched code to earlier version to be able to run the test | 01:01 |
| malini | I'll copy mike-incremental.xml & update it | 01:01 |
| oz_akan_ | thanks | 01:02 |
| malini | oz_akan_: Plz try mike-incremental.bk.xml | 01:04 |
| malini | thanks alejandro!! | 01:04 |
| oz_akan_ | thanks malini, I am waiting current running test to finish, then I will update the code again and run that test | 01:05 |
| malini | cool | 01:05 |
| oz_akan_ | what is uuid ? | 01:06 |
| alcabrera | Universally Unique Identifier, I believe. | 01:06 |
| alcabrera | Lemme double check that, oz_akan_. | 01:06 |
| oz_akan_ | can be anything? | 01:06 |
| alcabrera | oz_akan_: no, it follows a standardized format. | 01:07 |
| oz_akan_ | and has to be unique for project-id ? | 01:07 |
| malini | uuid doesnt have to be unique for the project..In our design we expect each client to have a unique id | 01:08 |
| alcabrera | oz_akan_: no. It determines the behavior of GET|POST /v1/queues/{queue}/messages. If it's the same client-id that was used to POST messages, and GET messages is called, then no messages will be shown that are associated with that client-id. | 01:08 |
| alcabrera | Here's a pretty succinct summary of the uuid format: http://en.wikipedia.org/wiki/Universally_unique_identifier#Definition | 01:09 |
| alcabrera | 32 hexadecimal digits. :D | 01:09 |
| alcabrera | We can totally fake uuids if we wanted. | 01:09 |
| alcabrera | For tests and such. | 01:09 |
| oz_akan_ | could you mail a few examples, so I can use if I need to | 01:10 |
| alcabrera | sure thing | 01:10 |
| oz_akan_ | why would the client that posted messages wouldn't see his own messages? | 01:11 |
| oz_akan_ | what is the requirement there/ | 01:11 |
| alcabrera | oz_akan_: sent UUID examples | 01:12 |
| oz_akan_ | tks | 01:12 |
| alcabrera | I keep forgetting the rationale behind it. I think the use case is in the pub-sub model, a publisher shouldn't see its own messages. | 01:13 |
| alcabrera | Since in pubsub, claims are never made. | 01:13 |
| alcabrera | Though for testing purposes, we provide a back door with ?echo=true. | 01:14 |
| oz_akan_ | ok | 01:15 |
| oz_akan_ | lb - proxy - router - queues seems very slow | 01:19 |
| oz_akan_ | so I will play the last card we have | 01:19 |
| oz_akan_ | lb - router - proxy - router - queues | 01:19 |
| alcabrera | That's going to help a lot, if the queues example tells us anything. ;) | 01:19 |
| * alcabrera likes making optimistic hypotheses | 01:20 | |
| oz_akan_ | crossing fingers | 01:25 |
| *** kgriffs_afk is now known as kgriffs | 01:36 | |
| alcabrera | kgriffs: o/ | 01:36 |
| kgriffs | hi | 01:36 |
| oz_akan_ | hi | 01:37 |
| kgriffs | re UUIDs it is to support duplex queues, as alcabrera said. But yeah, if you are only doing producer consumer it may appear to be less helpful. That being said, I think it will come in handy later since we can mine the data and get stats on how many clients are interacting with the service. | 01:38 |
| kgriffs | Anyway, it's an ongoing discussion and will evolve over time | 01:39 |
| kgriffs | </two-cents> | 01:39 |
| oz_akan_ | makes sense, have we mentioned UUID in the documentation? | 01:40 |
| kgriffs | yes | 01:40 |
| kgriffs | it was late in coming in | 01:40 |
| kgriffs | but Catherine and the client lib devs know about it now | 01:40 |
| oz_akan_ | it may seem like a burden for the developer, so would be good to understand why it is required | 01:40 |
| kgriffs | yeah, I think the dev docs explain it somewhere. It may not be in the api spec. I can check. | 01:41 |
| oz_akan_ | I was just curious | 01:41 |
| oz_akan_ | thanks | 01:41 |
| alcabrera | Hmm, good point, kgriffs. If it's not in the API spec, I can fix that up, with a paragraph explaining the rationale. | 01:41 |
| kgriffs | no worries; we've had others ask about it as well. I think it needs some refining | 01:41 |
| kgriffs | alcabrera: looks like there is a short blurb about it under "Common Headers" but it could use some elaboration or something. Well, the spec wasn't ever really meant to be an end-user document. But whatever. :D | 01:42 |
| alcabrera | lol | 01:43 |
| kgriffs | I guess until we have something on RTD, it's all people have. | 01:43 |
| kgriffs | :p | 01:43 |
| oz_akan_ | what RTD? | 01:44 |
| kgriffs | readthedocs | 01:44 |
| alcabrera | kgriffs: the client-id explanation on the specs looks good. | 01:44 |
| kgriffs | alcabrera | 01:44 |
| kgriffs | https://readthedocs.org/ | 01:44 |
| alcabrera | oz_akan_: uwsgi hosts its documentation in that style: http://uwsgi-docs.readthedocs.org/en/latest/ | 01:44 |
| kgriffs | I'd love to have a user guide and operator/admin guide on there | 01:44 |
| alcabrera | It's autogenerated by parsing either .rst files or source code comments in a certain format. | 01:45 |
| alcabrera | kgriffs: +1 | 01:45 |
| kgriffs | I need to do that for Falcon too. :p | 01:45 |
| alcabrera | kgriffs: yup, I remember that issue. Falcon needs lots of docstring love. :P | 01:45 |
| *** malini is now known as malini_afk | 01:56 | |
| *** kgriffs is now known as kgriffs_afk | 01:58 | |
| *** kgriffs_afk is now known as kgriffs | 02:03 | |
| *** amitgandhi has joined #openstack-marconi | 02:05 | |
| *** amitgandhi has quit IRC | 02:06 | |
| oz_akan_ | performance is better with a router in front of proxy but still adds significant slowness | 02:35 |
| kgriffs | you mean, the proxy adds significant slowness? | 02:38 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: proxy API improvements https://review.openstack.org/50213 | 02:40 |
| alcabrera | jenkins is gonna complain. I forgot to pep8 before submitting. :/ | 02:40 |
| alcabrera | oz_akan_: How much better is the performance with a router in front? How much slower does the proxy make things? Is it still 5-10x slower? | 02:41 |
| oz_akan_ | 736 rps, 1200 ms (30 users, 100 seconds) | 02:41 |
| oz_akan_ | with proxy | 02:41 |
| oz_akan_ | 933 rps, 30.54 ms (30 users, 100 seconds) | 02:41 |
| oz_akan_ | without | 02:41 |
| oz_akan_ | 1603.8 rps, 52.21 ms (30 users, 100 seconds) | 02:41 |
| oz_akan_ | without | 02:41 |
| oz_akan_ | last one is 50 users | 02:42 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: proxy API improvements https://review.openstack.org/50213 | 02:42 |
| oz_akan_ | 50 x 31 requet per second | 02:42 |
| oz_akan_ | alcabrera: ^^ | 02:42 |
| oz_akan_ | 1200 ms for 30 users with proxy | 02:42 |
| alcabrera | oz_akan_: thanks! | 02:42 |
| oz_akan_ | 1.2 seconds per request | 02:42 |
| alcabrera | that's pretty slow. :/ | 02:42 |
| alcabrera | Must be all the network hopping, though... hmm... | 02:43 |
| oz_akan_ | for 15 users with proxy: 507.8 rps, 44.22 ms | 02:43 |
| alcabrera | The proxy is much faster if most of the requests are cached locally in memcached | 02:43 |
| oz_akan_ | so we can do ~600 rps at 50 ms | 02:43 |
| kgriffs | is that using async workers or sync? | 02:43 |
| alcabrera | That way, the proxy avoids making the extra network hop to speak to mongodb. | 02:43 |
| alcabrera | I believe the tsung tests are biased towards mostly unique project/queues combos, which is the worst case for the proxy. | 02:44 |
| oz_akan_ | I believe proxy caches all in memcached already | 02:44 |
| oz_akan_ | test uses 20 queues | 02:44 |
| alcabrera | hmmm... | 02:44 |
| kgriffs | doesn't tsung generate random queue names? | 02:44 |
| oz_akan_ | I think it does, but there are 20 for the test I run | 02:44 |
| oz_akan_ | we don't use threads | 02:45 |
| oz_akan_ | in uwsgi | 02:45 |
| kgriffs | does it generate them all at the beginning, then use the same ones for the duration of the test? | 02:45 |
| oz_akan_ | yes, these are predefined | 02:45 |
| oz_akan_ | same queues all the time | 02:45 |
| oz_akan_ | proxy slowness comes because of being in the middle | 02:45 |
| alcabrera | Same project-id, too? | 02:45 |
| alcabrera | I mean, is the project-id pre-generated? | 02:46 |
| alcabrera | caching decisions are made on the basis of project + queue. | 02:46 |
| oz_akan_ | wouldn't matter if one of them not set | 02:47 |
| oz_akan_ | it would still cache, wouldn't it? | 02:47 |
| alcabrera | It would cache correctly if project is None. :) | 02:48 |
| alcabrera | In that case, the cache key is roughly '_/queue_name'. | 02:49 |
| alcabrera | oz_akan_: so, yes, you're right. | 02:49 |
| oz_akan_ | all in one box, sending 465 request max, w/ proxy 42.76 ms vs w/o proxy 11.95 ms | 02:49 |
| oz_akan_ | over network, difference is much more | 02:49 |
| oz_akan_ | 736 rps, 1200 ms (30 users, 100 seconds) vs 933 rps, 30.54 ms (30 users, 100 seconds) | 02:50 |
| alcabrera | oz_akan_: It seems like the breakeven point of the proxy is at about that point - ~750 rps. So if the load on a single partition ever exceeded that, the proxy would outperform the single partition. It sounds like the load would have to be pretty fierce. | 02:51 |
| alcabrera | Still, 1.2s latency under load is not cool. | 02:51 |
| oz_akan_ | alcabrera: 750 with 1200 ms latency | 02:52 |
| oz_akan_ | that is 1.2 seconds per request | 02:52 |
| alcabrera | slowness | 02:52 |
| oz_akan_ | with proxy we can't do more than 200-300 per partition | 02:52 |
| oz_akan_ | let me run a light test to talk for sure | 02:52 |
| alcabrera | kk | 02:52 |
| kgriffs | 43 vs 12 | 02:53 |
| kgriffs | that is suprising | 02:53 |
| kgriffs | seems like it shouldn't be that high | 02:53 |
| oz_akan_ | oh, I had it already | 02:53 |
| oz_akan_ | http://198.61.239.147:8000/log/20131008-0204/report.html | 02:53 |
| oz_akan_ | this with partiton 15 x 31 request max | 02:54 |
| oz_akan_ | 44 ms | 02:54 |
| oz_akan_ | 507 rps | 02:54 |
| kgriffs | sounds like there is a big bottleneck somewhere in the proxy | 02:54 |
| kgriffs | requests? | 02:54 |
| oz_akan_ | proxy code is fast | 02:55 |
| oz_akan_ | I think it is the communication | 02:55 |
| alcabrera | kgriffs: I'd need to profile it. It's too bad I can't get lazy requests async forwarding using eventlet. :/ | 02:55 |
| alcabrera | s/lazy/easy | 02:55 |
| kgriffs | oz_akan_: localhost socket communication is taking an extra 30 ms? | 02:55 |
| *** nosnos has joined #openstack-marconi | 02:55 | |
| oz_akan_ | not that much, it was somewhere between 10-15 ms | 02:56 |
| oz_akan_ | it slows down each request, and under relatively high load, user experiences each request much slower | 02:56 |
| oz_akan_ | are requests get queued | 02:57 |
| alcabrera | I'm going to go get some sleep, guys. | 02:57 |
| kgriffs | how many CPUs? | 02:57 |
| kgriffs | and how many uwsgi workers? | 02:57 |
| kgriffs | alcabrera: ttfn | 02:57 |
| kgriffs | g'night | 02:57 |
| alcabrera | Good luck with data gathering. I'll look forward to the data tomorrow morning and work from there. :) | 02:57 |
| alcabrera | night! | 02:57 |
| *** alcabrera has quit IRC | 02:57 | |
| oz_akan_ | good night | 02:58 |
| oz_akan_ | kgriffs: same amount of cpus for both tets | 02:58 |
| oz_akan_ | tests | 02:58 |
| oz_akan_ | I didn't change anything between w/ and w/o proxy tests | 02:58 |
| kgriffs | how many? | 02:59 |
| oz_akan_ | 4 cores on web servers, 8 on mongodb | 02:59 |
| oz_akan_ | so ? | 02:59 |
| kgriffs | 4 cores and how many workers? | 02:59 |
| oz_akan_ | no workers | 03:00 |
| oz_akan_ | we have standalone processes | 03:00 |
| oz_akan_ | that is faster than having workers | 03:00 |
| oz_akan_ | 8 processes | 03:00 |
| oz_akan_ | cpu load is around 50% during the test | 03:00 |
| kgriffs | 8 separate uwsgi processes, using nginx in front of them? | 03:01 |
| kgriffs | I'm actually surprised that using one uwsgi "manager" and several workers behind it isn't faster. When I tested that with RSE it was extremely fast. | 03:02 |
| oz_akan_ | we have uwsgi as router in front of these 8 processes | 03:03 |
| oz_akan_ | uwsgi router is faster than nginx | 03:03 |
| oz_akan_ | and these talk via sockets | 03:04 |
| kgriffs | hmm | 03:04 |
| kgriffs | so you tried running uwsgi -p 8 | 03:05 |
| kgriffs | and that was slower than runing 8x uwsgi -p 1 with a router in front? | 03:05 |
| oz_akan_ | yes, very slow | 03:06 |
| kgriffs | that is really suprising | 03:06 |
| kgriffs | doesn't sound right | 03:06 |
| oz_akan_ | that is when we have the proxy | 03:06 |
| kgriffs | i mean, a preforking model is slower than going over a socket? | 03:07 |
| oz_akan_ | when we do 8 workers, we saw that the master process uses most of the cpu | 03:07 |
| oz_akan_ | might be something about it | 03:07 |
| kgriffs | hmm. | 03:08 |
| oz_akan_ | but I think the problem it solves is about connections, it might be faster to establish a connection using uwsgi router | 03:08 |
| oz_akan_ | I am 100% router between proxy and queues is the fastest | 03:09 |
| oz_akan_ | I am not sure if we need router if we only have queues | 03:09 |
| kgriffs | btw, have you tested with --gevent? | 03:09 |
| oz_akan_ | 505 rps, 27.26ms queues only | 03:11 |
| oz_akan_ | 513.3 rps, 11.95 ms uwsgi router + queues | 03:11 |
| oz_akan_ | proxy code throws errors with gevent | 03:11 |
| kgriffs | how about queues? | 03:11 |
| oz_akan_ | and for queues it was not any faster then not having it | 03:11 |
| kgriffs | oic | 03:12 |
| oz_akan_ | I remember you asking this before and me pasting some benchmark results | 03:12 |
| oz_akan_ | but I can't recall numbers, how fast slow etc.. | 03:12 |
| kgriffs | yeah, rings a bell | 03:12 |
| kgriffs | ok, thanks for the info | 03:13 |
| kgriffs | I guess we can sleep on this info and figure out what to do next in the morning | 03:13 |
| kgriffs | still seems strange to me that the proxy would add so much overhead, even under load | 03:14 |
| oz_akan_ | take care | 03:14 |
| kgriffs | ttfn | 03:14 |
| oz_akan_ | think about this | 03:14 |
| oz_akan_ | I see 11 ms on marconi logs | 03:14 |
| *** kgriffs is now known as kgriffs_afk | 03:14 | |
| oz_akan_ | but tsung reports 40 ms | 03:14 |
| oz_akan_ | that is the network and queued requests overhead | 03:15 |
| *** oz_akan_ has quit IRC | 04:02 | |
| *** dafter has joined #openstack-marconi | 04:31 | |
| *** dafter has quit IRC | 04:31 | |
| *** dafter has joined #openstack-marconi | 04:31 | |
| *** oz_akan_ has joined #openstack-marconi | 05:13 | |
| *** whenry has quit IRC | 05:17 | |
| *** oz_akan_ has quit IRC | 05:18 | |
| *** Alex_Gaynor has quit IRC | 07:17 | |
| *** ykaplan has joined #openstack-marconi | 07:48 | |
| *** flaper87|afk is now known as flaper87 | 07:49 | |
| *** yassine has joined #openstack-marconi | 08:10 | |
| *** reed has joined #openstack-marconi | 08:29 | |
| *** Alex_Gaynor has joined #openstack-marconi | 09:07 | |
| *** ykaplan has quit IRC | 09:39 | |
| *** ykaplan has joined #openstack-marconi | 10:00 | |
| *** nosnos has quit IRC | 10:12 | |
| *** flaper87 is now known as flaper87|afk | 10:23 | |
| *** fifieldt has quit IRC | 10:37 | |
| *** oz_akan_ has joined #openstack-marconi | 11:12 | |
| *** oz_akan_ has quit IRC | 11:14 | |
| *** oz_akan_ has joined #openstack-marconi | 11:15 | |
| *** dafter has quit IRC | 11:22 | |
| *** dafter has joined #openstack-marconi | 11:24 | |
| openstackgerrit | ChangBo Guo proposed a change to openstack/marconi: Replace decprecated method aliases in tests https://review.openstack.org/50297 | 11:27 |
| *** alcabrera has joined #openstack-marconi | 11:44 | |
| alcabrera | Good morning! :D | 11:44 |
| *** tedross has joined #openstack-marconi | 11:58 | |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: fix: proxy mongodb storage fields overspecified https://review.openstack.org/50305 | 11:59 |
| openstackgerrit | ChangBo Guo proposed a change to openstack/marconi: Follow hacking rules about import https://review.openstack.org/50308 | 12:03 |
| openstackgerrit | ChangBo Guo proposed a change to openstack/marconi: Replace decprecated method aliases in tests https://review.openstack.org/50297 | 12:07 |
| *** ykaplan has quit IRC | 12:10 | |
| openstackgerrit | ChangBo Guo proposed a change to openstack/marconi: Follow hacking rules about import https://review.openstack.org/50308 | 12:12 |
| openstackgerrit | ChangBo Guo proposed a change to openstack/marconi: Replace decprecated method aliases in tests https://review.openstack.org/50297 | 12:15 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: fix: allow multi-update on partition storage https://review.openstack.org/50312 | 12:22 |
| *** ykaplan has joined #openstack-marconi | 12:25 | |
| *** dafter has quit IRC | 12:35 | |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: fix: stream request data rather than loading it into memory https://review.openstack.org/50328 | 12:38 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: fix: invalidate partition cache entry on delete https://review.openstack.org/50333 | 12:46 |
| *** ayoung has quit IRC | 13:02 | |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: fix: validation queue listing limits in proxy https://review.openstack.org/50342 | 13:11 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: fix: validate queue listing limits in proxy https://review.openstack.org/50342 | 13:11 |
| *** jraim_ has joined #openstack-marconi | 13:16 | |
| *** jraim has quit IRC | 13:17 | |
| *** jraim_ is now known as jraim | 13:17 | |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: Log names of drivers being loaded https://review.openstack.org/49843 | 13:31 |
| *** malini_afk is now known as malini | 13:32 | |
| alcabrera | malini: good morning! | 13:32 |
| malini | good morning!! | 13:32 |
| alcabrera | I've got soooo many tiny proxy fixes this morning, including the queue limits being ignored on GET /v1/queues. :D | 13:33 |
| *** ykaplan has quit IRC | 13:38 | |
| *** jcru has joined #openstack-marconi | 13:39 | |
| *** amitgandhi has joined #openstack-marconi | 13:43 | |
| *** dafter has joined #openstack-marconi | 13:43 | |
| *** amitgandhi has quit IRC | 13:45 | |
| *** amitgandhi has joined #openstack-marconi | 13:45 | |
| *** amitgandhi has quit IRC | 13:56 | |
| *** amitgandhi has joined #openstack-marconi | 13:57 | |
| *** kgriffs_afk is now known as kgriffs | 14:02 | |
| *** ykaplan has joined #openstack-marconi | 14:03 | |
| *** vkmc has joined #openstack-marconi | 14:05 | |
| *** ayoung has joined #openstack-marconi | 14:08 | |
| *** ayoung_ has joined #openstack-marconi | 14:16 | |
| *** ayoung has quit IRC | 14:17 | |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: feat(test): queue context for proxy https://review.openstack.org/49830 | 14:29 |
| *** jergerber has joined #openstack-marconi | 14:40 | |
| *** acabrera has joined #openstack-marconi | 14:57 | |
| zyuan | alcabrera: where is the problem within the current proxy? | 14:57 |
| zyuan | and what kind of design can address this? | 14:58 |
| acabrera | zyuan: the problem is that the current proxy requires data lookup as well as an additional network hop. | 14:58 |
| kgriffs | we would like to remove the sharding from the data plane | 14:59 |
| kgriffs | the fundamental design is flawed | 14:59 |
| zyuan | acabrera: but any proxy need data lookup and network forwarding... | 14:59 |
| zyuan | kgriffs: then how to do sharding?> | 14:59 |
| kgriffs | the sharding should just pick which DB connection to use, and should not be piping any data | 14:59 |
| *** alcabrera has quit IRC | 15:00 | |
| kgriffs | zyuan: the more traditional way | 15:00 |
| *** acabrera is now known as alcabrera | 15:00 | |
| zyuan | hmm | 15:00 |
| kgriffs | i.e., we have a layer that sits above the DB client library | 15:00 |
| kgriffs | and picks which DB client/connection to use | 15:00 |
| *** ayoung_ is now known as ayoung | 15:00 | |
| kgriffs | this has been done with good success by a number of startups that have suddenly ran into scaling problems | 15:00 |
| zyuan | then how marconi server talks to end-user? | 15:01 |
| kgriffs | I think the meat of the current proxy code will be able to be used | 15:01 |
| zyuan | directly? | 15:01 |
| kgriffs | we are pushing the routing logic down a layer | 15:01 |
| kgriffs | I will be out there later this week, and we can discuss | 15:01 |
| kgriffs | brb | 15:03 |
| malini | kgriffs: did you hear from flaper87 yet ? | 15:05 |
| *** dafter has quit IRC | 15:07 | |
| *** dafter has joined #openstack-marconi | 15:09 | |
| *** dafter has quit IRC | 15:09 | |
| *** dafter has joined #openstack-marconi | 15:09 | |
| kgriffs | nope | 15:12 |
| kgriffs | stand by | 15:12 |
| kgriffs | making a minor change to the patch | 15:17 |
| kgriffs | (self.assertEquals ==> self.assertEqual) | 15:18 |
| alcabrera | back | 15:21 |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: fix(mongo): Queue listing may include queues from other projects https://review.openstack.org/50176 | 15:21 |
| alcabrera | kgriffs: +2 | 15:22 |
| kgriffs | flaper87 is reviewing from mobile | 15:26 |
| malini | wooot | 15:27 |
| alcabrera | awesome | 15:27 |
| alcabrera | flaper87|afk: Thanks! | 15:27 |
| kgriffs | give me a few more mins | 15:28 |
| kgriffs | to finish travel arrangments | 15:28 |
| alcabrera | I'm taking a moment to try to fix the eventlet/requests problem. I have a unit test almost ready in the eventlet code base. | 15:28 |
| alcabrera | I'll disengage from that when you're ready, kgriffs. | 15:29 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: Replace deprecated method aliases in tests https://review.openstack.org/50297 | 15:34 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: feat(test): queue context for proxy https://review.openstack.org/49830 | 15:37 |
| openstackgerrit | A change was merged to openstack/marconi: fix(mongo): Queue listing may include queues from other projects https://review.openstack.org/50176 | 15:41 |
| kgriffs | malini: ^^^ | 15:41 |
| malini | kgriffs: | 15:41 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: Replace deprecated method aliases in tests https://review.openstack.org/50297 | 15:43 |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: feat(test): queue context for proxy https://review.openstack.org/49830 | 15:44 |
| *** ykaplan has quit IRC | 15:53 | |
| *** yassine has quit IRC | 16:06 | |
| *** ykaplan has joined #openstack-marconi | 16:10 | |
| zyuan | kgriffs: ping | 16:14 |
| *** ykaplan has quit IRC | 16:30 | |
| *** malini is now known as malini_afk | 16:40 | |
| *** malini_afk is now known as malini | 16:47 | |
| *** alcabrera is now known as alcabrera|afk | 16:55 | |
| *** oz_akan__ has joined #openstack-marconi | 17:04 | |
| *** alcabrera|afk is now known as alcabrera | 17:07 | |
| *** oz_akan_ has quit IRC | 17:07 | |
| *** mpanetta has joined #openstack-marconi | 17:29 | |
| *** mpanetta has quit IRC | 17:30 | |
| *** mpanetta has joined #openstack-marconi | 17:31 | |
| malini | zyuan: looks like we dont validate client UUID for all requests ..is tht intentional ? | 17:36 |
| zyuan | malini: it is | 17:36 |
| malini | why ? | 17:36 |
| zyuan | malini: we only calidate it when the UUID is needed | 17:36 |
| zyuan | we only read client id when it's needed | 17:36 |
| malini | which api's need the client-id ? | 17:36 |
| zyuan | only post message and listing message | 17:36 |
| kgriffs | Later if we want to, e.g., write it to logs we will need to always read it | 17:37 |
| malini | thx!! | 17:37 |
| zyuan | our policy was: user always submit client-id | 17:37 |
| kgriffs | but, you know, YAGNI | 17:37 |
| zyuan | and my patch extend the policy to: user always submit correct client-id | 17:37 |
| malini | ok..thx!! | 17:37 |
| zyuan | but we only check it to defend ourself. | 17:37 |
| *** jcru has quit IRC | 17:58 | |
| *** dafter has quit IRC | 17:58 | |
| *** dafter has joined #openstack-marconi | 18:00 | |
| *** dafter has quit IRC | 18:05 | |
| *** dafter has joined #openstack-marconi | 18:20 | |
| *** dafter has quit IRC | 18:20 | |
| *** dafter has joined #openstack-marconi | 18:20 | |
| *** kgriffs is now known as kgriffs_afk | 18:33 | |
| *** dafter has quit IRC | 18:39 | |
| *** dafter has joined #openstack-marconi | 18:39 | |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: WIP https://review.openstack.org/50437 | 18:42 |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: Use oslo.config directly instead of common.config https://review.openstack.org/49550 | 18:42 |
| *** kgriffs_afk is now known as kgriffs | 18:42 | |
| *** tvb|afk has joined #openstack-marconi | 18:42 | |
| *** tvb|afk has quit IRC | 18:42 | |
| *** tvb|afk has joined #openstack-marconi | 18:42 | |
| kgriffs | alcabrera: ^^^ | 18:43 |
| kgriffs | Jenkins will fail on it because I still have some refactoring to do | 18:43 |
| kgriffs | but I wanted to put it up there so you can see where I'm headed | 18:43 |
| alcabrera | I'll check it out. | 18:44 |
| alcabrera | kgriffs: Thanks! | 18:44 |
| kgriffs | yw! | 18:44 |
| *** dafter has quit IRC | 18:44 | |
| alcabrera | kgriffs: comments inline | 18:50 |
| alcabrera | It's a good start. I'll have a patch ready before I go home today with the structural groundwork for the schema changes/admin-API changes. | 18:51 |
| alcabrera | So... in about an hour. :P | 18:51 |
| kgriffs | kk | 18:53 |
| alcabrera | kgriffs: PUT /v1/shard/{name} {'weight': integer, 'location': string::url, 'options': {...}} | 18:55 |
| alcabrera | Also | 18:55 |
| alcabrera | PATCH as above | 18:55 |
| alcabrera | location looks like: mongodb://localhost:27010, redis://<address>/<port>. We can make location handling driver friendly by encapsulating it in a urlparse object or something like that (some common interface). StorageConnection.[type => 'mongodb', host, port, raw => as given] | 18:57 |
| kgriffs | oic | 19:00 |
| kgriffs | so we could parse redis uri out | 19:00 |
| alcabrera | yup | 19:00 |
| kgriffs | in the redis driver | 19:00 |
| kgriffs | and mongo could take it directly | 19:00 |
| kgriffs | +1 | 19:01 |
| kgriffs | although | 19:01 |
| kgriffs | hmm | 19:01 |
| kgriffs | what does that get us vs. just putting everything under options? | 19:01 |
| alcabrera | it gives us a common, guaranteed field. | 19:02 |
| alcabrera | options can be anything | 19:02 |
| alcabrera | by common/guaranteed, I can do stricter validation using jsonschema. | 19:02 |
| kgriffs | ok | 19:02 |
| alcabrera | kgriffs: I've got some proxy fixes we can leverage if we get them reviewed by tomorrow morning. | 19:14 |
| kgriffs | ok | 19:14 |
| alcabrera | The links to those I think will help us were emailed earlier today. Would you prefer if I posted them on an etherpad? :) | 19:15 |
| *** tvb|afk has quit IRC | 19:18 | |
| *** ayoung has quit IRC | 19:18 | |
| *** malini is now known as malini_afk | 19:25 | |
| kgriffs | alcabrera: sure, etherpad | 19:28 |
| *** dafter has joined #openstack-marconi | 19:31 | |
| *** dafter has quit IRC | 19:31 | |
| *** dafter has joined #openstack-marconi | 19:31 | |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack/marconi: feat: storage sharding schema for marconi-queues https://review.openstack.org/50456 | 19:34 |
| *** kgriffs is now known as kgriffs_afk | 19:34 | |
| alcabrera | kgriffs_afk: https://etherpad.openstack.org/proxy-patches | 19:36 |
| *** kgriffs_afk is now known as kgriffs | 19:52 | |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: WIP: Sharding https://review.openstack.org/50437 | 19:53 |
| kgriffs | alcabrera: thx | 19:54 |
| *** dafter has quit IRC | 19:54 | |
| kgriffs | that WIP is now testing green | 19:54 |
| kgriffs | so, refactoring are OK | 19:54 |
| kgriffs | however, the sharding isn't actually doing anything. | 19:54 |
| kgriffs | :p | 19:54 |
| kgriffs | let me make it just always return the same thing, and then add a test that turns it on | 19:55 |
| alcabrera | kgriffs: sweet. I'm starting to see the picture on your side of things. I've got some awesome scribbles here for the next steps on my end. | 19:57 |
| alcabrera | The next step for me is to implement the sharding storage controller, which allows one to handle shard storage over on the admin side. | 19:57 |
| alcabrera | *shard registration | 19:58 |
| *** ayoung has joined #openstack-marconi | 19:58 | |
| alcabrera | then, I'll tackle the shard admin resource, integrating it with the schema and the storage controller. | 19:58 |
| *** tvb|afk has joined #openstack-marconi | 19:59 | |
| *** tvb|afk has quit IRC | 19:59 | |
| *** tvb|afk has joined #openstack-marconi | 19:59 | |
| kgriffs | interesting thought | 19:59 |
| kgriffs | you could spin up a gazillion sqlites | 19:59 |
| kgriffs | and shard queues across them | 19:59 |
| alcabrera | After that, I've got: integrate storage_lookup with resources (/v1/queues/{queue}/*_), add caching, and test the whole thing. | 19:59 |
| kgriffs | :p | 19:59 |
| alcabrera | lol | 19:59 |
| alcabrera | marconi gets crazier every day. :P | 20:00 |
| alcabrera | Innovation on demand. | 20:00 |
| kgriffs | "integrate storage_lookup with resources" | 20:00 |
| kgriffs | can you elaborate? | 20:00 |
| alcabrera | Sure thing. | 20:00 |
| kgriffs | Seems like you just need to flesh out sharding.py | 20:00 |
| alcabrera | All endpoints of the form given above (those involving a queue) need to figure out what storage connection to use. At the transport layer, it'll probably be as simple as self.sharder.lookup_storage(project, queue) => client; client.op(). | 20:01 |
| alcabrera | at the storage/sharding layer, it'll take that fleshing out that you mention. :P | 20:02 |
| kgriffs | did you see my latest patchset? | 20:02 |
| alcabrera | kgriffs: briefly - I've glanced. | 20:02 |
| kgriffs | I added the plumbing to the transport resources | 20:02 |
| kgriffs | they will end up calling into sharding.Manager | 20:03 |
| kgriffs | so, you don't need to do anything in transport.wsgi | 20:03 |
| alcabrera | kgriffs: ah, I see now. | 20:04 |
| alcabrera | Gotcha, cool. | 20:04 |
| kgriffs | in fact, all you have to do is address the two TODO's in storage.sharding | 20:04 |
| alcabrera | That integration step is already taken care of. | 20:04 |
| kgriffs | yep | 20:04 |
| kgriffs | kapow! | 20:04 |
| alcabrera | Haha. :P | 20:05 |
| alcabrera | I had another idea on storage-shard options. | 20:05 |
| *** jcru has joined #openstack-marconi | 20:05 | |
| alcabrera | options are 'optional'. I'm thinking storage drivers are required to provide sane defaults for those options, and passing in a new set of options is something that can bee done at PUT /v1/shards/{name} time, but isn't required. | 20:06 |
| kgriffs | yes | 20:06 |
| kgriffs | my thought was the __init__ has kwargs with default options | 20:06 |
| alcabrera | This'll reduce the amount of storage needed for options details for the common case: homogenous shards | 20:06 |
| kgriffs | those would be overridden by "options" | 20:06 |
| alcabrera | *homogeneous | 20:06 |
| alcabrera | kgriffs: exatly | 20:06 |
| alcabrera | *exactly | 20:06 |
| kgriffs | what do you think about replacing driver_name with uri | 20:06 |
| alcabrera | we're on the same page on that, cool | 20:06 |
| alcabrera | kgriffs: +1 on that substitution. The name is no longer identifying. It's more like driver.type at this point. | 20:07 |
| alcabrera | And even that's redundant, given a proper URI | 20:07 |
| kgriffs | i mean, you could pull out the first part of the uri | 20:07 |
| alcabrera | redis://*, mongodb://*, leveldb://* | 20:07 |
| kgriffs | and give that to stevedore | 20:07 |
| kgriffs | then pass the uri into the driver | 20:07 |
| alcabrera | yep, those were my thoughts. :D | 20:07 |
| kgriffs | cool | 20:07 |
| kgriffs | let me do that | 20:07 |
| alcabrera | awesome | 20:08 |
| alcabrera | I'm wrapping up for the day. | 20:08 |
| kgriffs | kk | 20:08 |
| alcabrera | Any final questions/thoughts/concerns before I head out? | 20:08 |
| kgriffs | that's it | 20:08 |
| kgriffs | thanks! | 20:08 |
| alcabrera | cool, cool. I'll handle sharding storage/resource tomorrow. See you! Safe travels. :) | 20:08 |
| kgriffs | ttfn | 20:09 |
| *** alcabrera has quit IRC | 20:09 | |
| *** tvb|afk has quit IRC | 20:45 | |
| *** malini_afk is now known as malini | 20:53 | |
| *** dafter has joined #openstack-marconi | 20:53 | |
| *** dafter has quit IRC | 20:53 | |
| *** dafter has joined #openstack-marconi | 20:53 | |
| amitgandhi | kgriffs: to get messages by ids | 21:02 |
| amitgandhi | is it by ids as a query string? | 21:02 |
| kgriffs | yes | 21:06 |
| kgriffs | /messages?ids=x,y,z | 21:06 |
| *** malini is now known as malini_afk | 21:46 | |
| *** dafter has quit IRC | 21:54 | |
| *** tvb|afk has joined #openstack-marconi | 21:58 | |
| openstackgerrit | Zhihao Yuan proposed a change to openstack/marconi: feat(validation): project id length https://review.openstack.org/50496 | 22:05 |
| *** tvb|afk has quit IRC | 22:09 | |
| *** mpanetta has quit IRC | 22:26 | |
| *** oz_akan__ has quit IRC | 22:27 | |
| *** mpanetta has joined #openstack-marconi | 22:27 | |
| *** mpanetta has quit IRC | 22:31 | |
| *** amitgandhi has quit IRC | 22:35 | |
| *** jergerber has quit IRC | 22:36 | |
| *** jcru has quit IRC | 22:40 | |
| *** mpanetta has joined #openstack-marconi | 22:57 | |
| *** mpanetta has quit IRC | 23:08 | |
| openstackgerrit | Kurt Griffiths proposed a change to openstack/marconi: feat: Storage sharding foundation https://review.openstack.org/50437 | 23:08 |
| *** mpanetta has joined #openstack-marconi | 23:11 | |
| *** kgriffs is now known as kgriffs_afk | 23:20 | |
| *** vkmc has quit IRC | 23:43 | |
| *** mpanetta has quit IRC | 23:52 | |
| *** mpanetta has joined #openstack-marconi | 23:52 | |
| *** mpanetta has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!