| *** amitgandhi has joined #openstack-marconi | 00:49 | |
| *** malini_afk is now known as malini | 00:51 | |
| *** oz_akan_ has joined #openstack-marconi | 00:54 | |
| *** nosnos has joined #openstack-marconi | 01:06 | |
| *** fandikurnia01 has joined #openstack-marconi | 01:29 | |
| *** fandikurnia01 has quit IRC | 01:59 | |
| *** fandikurnia01 has joined #openstack-marconi | 02:04 | |
| *** oz_akan_ has quit IRC | 02:06 | |
| *** oz_akan_ has joined #openstack-marconi | 02:06 | |
| *** oz_akan_ has quit IRC | 02:09 | |
| *** fandikurnia01 has quit IRC | 02:25 | |
| *** oz_akan_ has joined #openstack-marconi | 02:37 | |
| *** amitgandhi has quit IRC | 03:23 | |
| *** malini is now known as malini_afk | 03:28 | |
| *** fandikurnia01 has joined #openstack-marconi | 03:58 | |
| *** fandikurnia01 has quit IRC | 04:15 | |
| *** fandikurnia01 has joined #openstack-marconi | 04:16 | |
| *** oz_akan_ has quit IRC | 04:18 | |
| *** oz_akan_ has joined #openstack-marconi | 04:18 | |
| *** torgomatic has quit IRC | 04:19 | |
| *** russellb has quit IRC | 04:19 | |
| *** fandikurnia01 has quit IRC | 04:21 | |
| *** russellb has joined #openstack-marconi | 04:22 | |
| *** torgomatic has joined #openstack-marconi | 04:23 | |
| *** oz_akan_ has quit IRC | 04:23 | |
| *** oz_akan_ has joined #openstack-marconi | 05:29 | |
| *** oz_akan_ has quit IRC | 05:34 | |
| *** flaper87|afk is now known as flaper87 | 06:39 | |
| *** whenry has quit IRC | 06:41 | |
| *** key4 has quit IRC | 08:48 | |
| *** key4 has joined #openstack-marconi | 08:48 | |
| *** key4 has quit IRC | 10:11 | |
| *** key4 has joined #openstack-marconi | 10:12 | |
| *** nosnos has quit IRC | 10:26 | |
| *** nosnos has joined #openstack-marconi | 10:27 | |
| *** nosnos has quit IRC | 10:27 | |
| *** nosnos has joined #openstack-marconi | 10:28 | |
| *** nosnos has quit IRC | 10:35 | |
| *** nosnos has joined #openstack-marconi | 10:36 | |
| *** nosnos has quit IRC | 10:40 | |
| *** cthulhup has quit IRC | 10:47 | |
| *** tedross has joined #openstack-marconi | 11:39 | |
| *** vkmc has joined #openstack-marconi | 11:42 | |
| *** malini_afk is now known as malini | 11:48 | |
| *** malini is now known as malini_afk | 13:00 | |
| *** oz_akan_ has joined #openstack-marconi | 13:20 | |
| *** oz_akan_ has quit IRC | 13:23 | |
| *** oz_akan_ has joined #openstack-marconi | 13:24 | |
| *** amitgandhi has joined #openstack-marconi | 13:29 | |
| *** malini_afk is now known as malini | 13:58 | |
| *** whenry has joined #openstack-marconi | 14:05 | |
| *** acabrera has joined #openstack-marconi | 14:36 | |
| acabrera | Morning! | 14:40 |
|---|---|---|
| flaper87 | acabrera: GOOOD MORNING! | 14:40 |
| * flaper87 is reviewing acabrera's proxy patch | 14:40 | |
| * acabrera cannot settle on a nick :P | 14:40 | |
| acabrera | woot, flaper87! | 14:40 |
| acabrera | It's a hefty bunch of patches. | 14:40 |
| flaper87 | acabrera: I'm reviewing this: https://review.openstack.org/#/c/44356/ | 14:41 |
| flaper87 | got there following dependency's chain | 14:41 |
| *** JRow has joined #openstack-marconi | 14:42 | |
| flaper87 | acabrera: I meant to say, this: https://review.openstack.org/#/c/43909/ | 14:42 |
| flaper87 | :D | 14:42 |
| acabrera | ahh, round 1 of the whole thing. :) | 14:42 |
| flaper87 | acabrera: did you want me to review another one first? | 14:43 |
| acabrera | Quick note on that one - for the whole round-robin/select when creating a queue, I'm thinking of simplifying that portion and deferring the load balancing logic to the partition URL. | 14:43 |
| acabrera | Hmm... | 14:43 |
| flaper87 | I'm a bit confused because that one still uses redis | 14:44 |
| flaper87 | and I thought you were working on multi-storage support | 14:44 |
| acabrera | Yeah, the Redis logic is getting replaced by oslo.cache, and the multi-storage support - I worked on that yesterday. | 14:44 |
| acabrera | So where to start...? | 14:45 |
| flaper87 | yup, that's my question :P | 14:45 |
| flaper87 | because I've the feeling I'm reviewing the wrong patch | 14:45 |
| flaper87 | but the dependencies' chain led me there | 14:45 |
| acabrera | I'd say right where you are - it's the best place to get the big picture design of the thing - routes, resources. | 14:45 |
| flaper87 | kk | 14:45 |
| * flaper87 does that and STFU | 14:45 | |
| flaper87 | actually | 14:45 |
| acabrera | Just keep in mijnd that the redis details will get replaced soon. | 14:46 |
| * flaper87 STFU and then does that | 14:46 | |
| acabrera | :) | 14:46 |
| flaper87 | kk, +1 for replacing that | 14:46 |
| * flaper87 loves reviews that get simpler | 14:46 | |
| flaper87 | malini: helllooooooooooooo | 14:46 |
| acabrera | Ask me anything while you're reading - it's a hefty bunch of patches and if you're confused on any points, I'm happy to help guide you through how I got to the current design. :D | 14:46 |
| flaper87 | malini: btw, I got an environment with marconi + keystone setup | 14:47 |
| malini | flaper87: yayyy!!! | 14:47 |
| flaper87 | I promisse to test against that from now on and don't screw your auth tests anymore | 14:47 |
| flaper87 | :D | 14:47 |
| malini | now you can enjoy the weekend in peace! | 14:47 |
| flaper87 | weekend ? what's that ? | 14:47 |
| * flaper87 is now confused | 14:47 | |
| flaper87 | is there something called weekend ? | 14:48 |
| malini | you know, the thing that comes between friday & monday. | 14:48 |
| flaper87 | why do people never tell me these things. | 14:48 |
| flaper87 | ? | 14:48 |
| malini | you shud have paid attention in kindergarten ;) | 14:48 |
| flaper87 | aaahhh, I think I did, I just have really bad memory so, no idea what they taught me there | 14:49 |
| malini | :D | 14:49 |
| malini | me neither..I had a troubled KG life :D | 14:49 |
| acabrera | lol | 14:51 |
| acabrera | weekends... | 14:51 |
| flaper87 | btw, tomorrow marconi will be migrated under openstack/ | 14:52 |
| acabrera | Python Cookbook 3e is on sale today: http://shop.oreilly.com/category/deals/programmers-day.do | 14:52 |
| acabrera | Oohh, migration. :D | 14:53 |
| flaper87 | you'll have to update your repo remote | 14:53 |
| malini | new home!!! <:o) | 14:53 |
| openstackgerrit | Zhihao Yuan proposed a change to stackforge/marconi: WPI: test(proxy): catalog tested https://review.openstack.org/46362 | 14:59 |
| acabrera | zyuan: I'll check that out after I catch up on morning news. :) | 15:00 |
| *** kgriffs_afk is now known as kgriffs | 15:06 | |
| acabrera | kgriffs: o/ | 15:07 |
| kgriffs | yo yo yo | 15:08 |
| kgriffs | oooh, this is tempting: http://shop.oreilly.com/product/9781593272203.do?code=DEAL | 15:09 |
| acabrera | yeeess | 15:10 |
| acabrera | I have that one. It's totally the spiritual succeesor to Advanced Unix Programming. | 15:10 |
| acabrera | It's so thorough it's scary. | 15:10 |
| acabrera | 50+ glorious chapters on every nook and cranny of Linux system's programming. | 15:11 |
| acabrera | *60+ | 15:11 |
| kgriffs | hmm | 15:13 |
| kgriffs | no mention of ALSA | 15:13 |
| kgriffs | ? | 15:13 |
| flaper87 | kgriffs: gooood morning | 15:15 |
| zyuan | hehe | 15:16 |
| acabrera | kgriffs: nope - no mention of sound libraries at all. I suppose that's higher level than systems. :P | 15:16 |
| acabrera | git pissed: https://github.com/chrishunt/git-pissed | 15:16 |
| kgriffs | LOOOL | 15:17 |
| acabrera | tests, queues, and storage come up a lot in marconi logs. :P | 15:18 |
| kgriffs | acabrera: this one is also tempting me - http://shop.oreilly.com/product/9781118612101.do?code=DEAL | 15:18 |
| * kgriffs is such a geek | 15:18 | |
| acabrera | That one tempted me, too, kgriffs. :D | 15:19 |
| acabrera | I checked it out on amazon for "Peek Inside" and ultimately, I ended up comparing it to CLRS Algorithms: http://www.amazon.com/Introduction-Algorithms-Thomas-H-Cormen/dp/0262033844 | 15:20 |
| acabrera | I'd pick the latter over Essential Algorithms | 15:20 |
| zyuan | oh, that's the text book used by many schools | 15:21 |
| zyuan | introdution algorithms | 15:21 |
| acabrera | zyuan: Yeah, I still haven't found anything better than that. :) | 15:21 |
| acabrera | I need to get a copy sometime. I don;'t have my 2e of it any more. | 15:22 |
| zyuan | (i still haven't read that) | 15:22 |
| zyuan | ( i read some others...) | 15:22 |
| zyuan | about CLRS, berkley has a series of videos | 15:23 |
| zyuan | well... altually those lectures are widely available on internet during the recent years... | 15:25 |
| acabrera | I've made note of the algorithm courses on Coursera, but I hadn't noticed Berkeley's past offerings. | 15:27 |
| acabrera | Link? | 15:27 |
| zyuan | can't find... i saved them on my computer serveral years ago, and forgot where i got them... | 15:29 |
| *** JRow has quit IRC | 15:29 | |
| zyuan | ChanServ: found. it's actually from MIT: http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-046j-introduction-to-algorithms-sma-5503-fall-2005/video-lectures/ | 15:33 |
| zyuan | acabrera: ^^ | 15:35 |
| acabrera | ahh, ocw content. classic. | 15:37 |
| acabrera | zyuan: Thanks! | 15:37 |
| acabrera | Too bad they start calling everything after lecture 19 "advanced topics" :P | 15:38 |
| zyuan | my hope is to learn how to implement STL; i have no further pursue on algorithms... | 15:40 |
| zyuan | acabrera: check out this video http://channel9.msdn.com/Events/GoingNative/2013/Cpp-Seasoning the first 30mins are about algorithms | 15:47 |
| zyuan | my favoriate part on goingnative | 15:47 |
| flaper87 | btw guys, can I get your thoughts about the code structure I proposed the other day? | 15:50 |
| zyuan | where? | 15:50 |
| flaper87 | If we agree it makes sense, I think that should be the first thing to do after marconi is migrated under openstack | 15:50 |
| flaper87 | zyuan: some therpad, let me try to find it again | 15:50 |
| flaper87 | https://etherpad.openstack.org/marconi-tree | 15:51 |
| zyuan | looks deep | 15:52 |
| flaper87 | 1 level deeper, yes | 15:52 |
| flaper87 | I think it makes sense to separate the 3 main serivces, it's not that we HAVE to | 15:52 |
| kgriffs | I still think "marconi-queueing" is a little awkward | 15:52 |
| kgriffs | can anyone think of a better name? | 15:53 |
| flaper87 | could be marconi-service | 15:53 |
| kgriffs | marconi-api | 15:54 |
| kgriffs | other ideas? | 15:54 |
| kgriffs | (brainstorming) | 15:54 |
| flaper87 | marconi-api sounds good to me | 15:54 |
| * acabrera joins brainstorm | 15:54 | |
| flaper87 | but I would call the package queues (just to give it some context) | 15:55 |
| kgriffs | I'm adding them to the etherpad so we can see how the different options look in context | 15:55 |
| flaper87 | (in case we change the package structure) | 15:55 |
| kgriffs | flaper87: do you think notifications will have a separate/standalone API? | 16:02 |
| *** malini is now known as malini_afk | 16:02 | |
| flaper87 | kgriffs: mmh, I don't think so. | 16:06 |
| flaper87 | mmh, actually, it might make sense | 16:07 |
| flaper87 | but I haven't put many thoughts on that | 16:08 |
| *** malini_afk is now known as malini | 16:09 | |
| acabrera | I think amitgandhi has put some thought into notifications. | 16:10 |
| kgriffs | ok, for now we just need to decide on directory structure | 16:10 |
| acabrera | I have a suspicion that there will be a separate API, since the purpose of a notification is different than that of a message bus. | 16:10 |
| kgriffs | "queues" is easier to spell | 16:11 |
| flaper87 | acabrera: there'll be a separte API | 16:11 |
| flaper87 | what I'm not sure of is whether it'll run as a standalone service | 16:11 |
| flaper87 | I can see that working either way | 16:11 |
| flaper87 | as long as the user doesn't have to deal with that | 16:11 |
| flaper87 | I think it's fine | 16:11 |
| kgriffs | "queues" services /v1/queues | 16:12 |
| kgriffs | "notifications" services "/v1/notifications" | 16:12 |
| kgriffs | or something | 16:12 |
| flaper87 | kgriffs: yup | 16:12 |
| kgriffs | notifications will need to be more flexible, optional | 16:12 |
| *** JRow has joined #openstack-marconi | 16:13 | |
| acabrera | I'm happy with the proposed directory structure. | 16:13 |
| acabrera | +1 for the common folder, too. | 16:13 |
| kgriffs | ok, so I vote for reorging with (cmd, common, proxy, queues) | 16:14 |
| acabrera | It seems like that's the kind of utilities we might be able to give back to oslo | 16:14 |
| acabrera | +1 kgriffs | 16:14 |
| flaper87 | acabrera: +1 | 16:14 |
| * flaper87 votes for that as well | 16:14 | |
| amitgandhi | i see notifications being a seperate product that uses marconi | 16:15 |
| amitgandhi | but yeh something like v1/notiications | 16:15 |
| flaper87 | acabrera: sorry, very slow review. :( I added some comments | 16:27 |
| acabrera | no worries, flaper87! Thanks so much for looking at it. :D | 16:27 |
| acabrera | I'm heading to the comments now. | 16:28 |
| amitgandhi | kgriffs: flaper87: any idea what Get Actions is about: https://wiki.openstack.org/wiki/Marconi/specs/api/v1/responsecodes#Get_Actions | 16:28 |
| kgriffs | oh, that's old cruft | 16:29 |
| amitgandhi | ok im deleting it | 16:29 |
| flaper87 | amitgandhi: what kgriffs said | 16:34 |
| flaper87 | :D | 16:34 |
| flaper87 | sorry, my brain is working in slow montion | 16:34 |
| acabrera | flaper87: responded to all your review comments. Thanks for doing a thorough run through on that. :) | 16:35 |
| *** JRow has quit IRC | 16:35 | |
| acabrera | brb office lunch. :) | 16:46 |
| *** acabrera is now known as acabrera_away | 16:46 | |
| *** acabrera_away is now known as acabrera | 17:05 | |
| acabrera | zyuan: unit test patch reviewed. Thanks! | 17:18 |
| acabrera | I'm going to spend the next hour or so putting together/submitting PyCon talk proposals. | 17:19 |
| acabrera | I have three in mind. | 17:20 |
| kgriffs | is Falcon one of them? | 17:22 |
| kgriffs | :D | 17:22 |
| zyuan | LOL | 17:22 |
| acabrera | lol | 17:22 |
| acabrera | Actually, that was proposal #4. :P | 17:22 |
| acabrera | I'm on the fence about presenting that one, but I might. | 17:22 |
| zyuan | how many sessions in total | 17:23 |
| zyuan | on the conference? | 17:23 |
| zyuan | you submit multiple proposals, not all will be presented, i guess | 17:24 |
| acabrera | I intend to give one 30 minute talk. :) | 17:24 |
| acabrera | I'm submitting multiple proposals to increase my chances of being accepted. :D | 17:24 |
| acabrera | My three target topics are: 1) Migrating to Python 3 - Single Code Base w/ Py2.6 + 2.7 support, 2) Python 3 Pains: Concurrency in Python, 3) [Use Case] Developing a Scalable, Distributed Message Bus w/ Python (Marconi), 4) Falcon: High-performance API Framework | 17:24 |
| acabrera | s/three/four | 17:24 |
| openstackgerrit | Zhihao Yuan proposed a change to stackforge/marconi: WPI: test(proxy): catalog tested https://review.openstack.org/46362 | 17:25 |
| malini | need to WFH rest of the day..will be back online in ~45 min | 17:25 |
| acabrera | If I end up getting selected to speak on #4, expect me to bug you a lot, kgriffs. :P | 17:25 |
| acabrera | malini: see you again soon. :) | 17:25 |
| *** malini is now known as malini_afk | 17:26 | |
| openstackgerrit | Zhihao Yuan proposed a change to stackforge/marconi: test(proxy): partitions https://review.openstack.org/46362 | 17:27 |
| acabrera | zyuan: Your unit tests tend to contain cleverly hidden references to nifty anime. I like it. :D | 17:29 |
| kgriffs | acabrera: I may propose the Falcon one | 17:29 |
| kgriffs | let me consider the idea over the weekend | 17:30 |
| zyuan | ChanServ: ;) | 17:30 |
| zyuan | acabrera: ;) | 17:30 |
| zyuan | ... i always complete your id with c<Tab> | 17:31 |
| acabrera | hehe | 17:31 |
| acabrera | cppcabrera isn't here today. ;P | 17:31 |
| * acabrera likes cycling through closely related nicknakes | 17:32 | |
| acabrera | s/nicknanes/nicknames | 17:32 |
| acabrera | kgriffs: PyCon talk proposals are due Monday. :P | 17:32 |
| zyuan | wow | 17:32 |
| acabrera | I've been stewing ideas for about a month now, and I decided to last-minute blitz the proposals. >.> | 17:32 |
| kgriffs | acabrera: gtk, i will email you to let you know by tomorrow re Falcon talk | 17:33 |
| acabrera | awesome, kgriffs | 17:34 |
| acabrera | I'll prepare a falcon proposal, just in case, but I'll only submit my initial three today. | 17:34 |
| openstackgerrit | Zhihao Yuan proposed a change to stackforge/marconi: test(proxy): partition related endpoints https://review.openstack.org/46362 | 17:43 |
| acabrera | kgriffs: correction - talsk proposals are due Sunday Sept. 15th. Lighting talks/tutorials are also accepted and these are due by October 15th. | 17:51 |
| *** malini_afk is now known as malini | 18:17 | |
| kgriffs | acabrera: proxy? | 18:29 |
| acabrera | ready, kgriffs | 18:29 |
| kgriffs | ok | 18:30 |
| kgriffs | so, first thing I wanted to clarify was how queues are directed to partitions and webheads within those partitions | 18:30 |
| acabrera | Yes, that's a good place to start. I think I overengineered that portion of the proxy. | 18:31 |
| acabrera | Since given a URL to a load balancer as the entry point to a partition, the logic can be handled else where. | 18:32 |
| acabrera | (If a load balancer is even used - leaving options open!) | 18:32 |
| acabrera | I'm thinking of modifying partition handling so that an operator need only specify a single URL rather than a list of URLs, and select() will return the URL to the weight-selected partition. | 18:33 |
| kgriffs | let's try this: http://demo.tutorialzine.com/2012/08/nodejs-drawing-game/ | 18:33 |
| kgriffs | oops | 18:33 |
| kgriffs | http://flockdraw.com/danjvx | 18:34 |
| acabrera | sweet | 18:34 |
| acabrera | cool, that's the image that came to my mind after reading your comment, kgriffs. | 18:37 |
| kgriffs | ok, so.. | 18:37 |
| acabrera | LB level 2 handled at partition level | 18:37 |
| kgriffs | you can either have basically a VIP per partition | 18:38 |
| kgriffs | or you have to have all the web heads in the catalog | 18:38 |
| kgriffs | seems like we thought about this before and decided to skip the VIP idea - can't remember why | 18:38 |
| kgriffs | another hop/latency? | 18:38 |
| acabrera | Latency per hop is about ~.7 ms, IIRC. | 18:39 |
| acabrera | Intra-DC | 18:39 |
| acabrera | 0.7 ms | 18:39 |
| acabrera | In any case, that ventures into the realm of things done outside of the proxy. | 18:40 |
| acabrera | Partition deployment is another topic entirely, if we agree that LB level 2 should be handled outside of the proxy. | 18:40 |
| acabrera | Thoughts, kgriffs? | 18:40 |
| kgriffs | hmmm | 18:41 |
| kgriffs | If it is just a plain TCP LB it should be almost no overhead other than network latency | 18:41 |
| acabrera | agreed | 18:42 |
| kgriffs | so, cons for proxy going directly to partition nodes? | 18:42 |
| kgriffs | (fwiw 10g links will make latency a non-issue) | 18:43 |
| acabrera | it's over-specified, IMO, and probably hurts the weighting system unless the deployer very carefully sets weights. | 18:43 |
| kgriffs | i mean, you do the following | 18:43 |
| kgriffs | so, weights would only be used when creating a queue (new catalog entry) | 18:44 |
| kgriffs | for other requests, you just lookup the partition for the queue, pick one of the webheads at random (or round-robin) in that partition, and forward on the request | 18:45 |
| acabrera | Ah, I see. | 18:45 |
| kgriffs | #breakout(breakout()) weighting | 18:46 |
| acabrera | lol | 18:46 |
| kgriffs | (have a different thought on that to share in a min | 18:46 |
| kgriffs | so, the cons I see to what I just outlined is you are flattening your operations namespace | 18:46 |
| acabrera | create queue -> assign to partition, forward request -> select node from partition - the conn here is that we enforce a load-balancing strategy at the proxy level. | 18:47 |
| acabrera | I'd love to leave that decision open to the deployer. | 18:47 |
| kgriffs | you end up having to deal with MIA nodes and stuff in the proxy, updating the catalog when adjusting a given partition's makeup is a pain, etc. | 18:47 |
| kgriffs | oz_akan_: ping | 18:47 |
| acabrera | yup, I hadn't even considered node cleansing | 18:47 |
| kgriffs | personally, I vote for the VIP tree idea | 18:48 |
| kgriffs | Service VIP ——> Partition VIPs | 18:48 |
| acabrera | one partition, one URL - that's my winning idea, which goes in line with VIP / partition | 18:48 |
| kgriffs | that localizes all the partition ops and we don't have to write half of haproxy in marconi-proxy | 18:48 |
| acabrera | s/winning/favored | 18:48 |
| acabrera | It also simplifies the proxy-storage driver implementation requirements. | 18:49 |
| acabrera | Previously, I was requiring storage drivers handle LB decisions - baaaad. | 18:49 |
| kgriffs | yep | 18:49 |
| kgriffs | you said it | 18:49 |
| acabrera | yup. :D | 18:49 |
| kgriffs | so, an operator can deploy basically a single partition with a single VIP (LB) | 18:50 |
| kgriffs | OR, | 18:50 |
| kgriffs | they can clone a bunch of those (no changes needed) | 18:50 |
| acabrera | or~ | 18:50 |
| kgriffs | stick some marconi-proxy boxes in front | 18:50 |
| acabrera | single node / partition - wimpy partitions style | 18:50 |
| kgriffs | which themselves are fronted by a LB VIP (public) | 18:50 |
| kgriffs | sound good? | 18:51 |
| acabrera | yup | 18:51 |
| kgriffs | ok, now regarding weighting | 18:51 |
| acabrera | one partition, one URL - noted and added to my proxy TODOs. | 18:51 |
| kgriffs | my question is this: | 18:52 |
| kgriffs | is there a simple way to add a dynamic element to weighting such that hot queues are less likely to end up together on the same partition? | 18:53 |
| acabrera | hmm... | 18:53 |
| acabrera | how would a hot queue be determined before it is acted upon/created? It seems like some sort of hint would have to be passed at creation time. | 18:54 |
| acabrera | X-Queue-Hotness: Extreme | 18:54 |
| kgriffs | LOL | 18:54 |
| kgriffs | how about some kind of heuristic? | 18:54 |
| kgriffs | what *do* we know | 18:55 |
| kgriffs | 1. how may queues are already assigned to a partition | 18:55 |
| kgriffs | 2. How many queues a project has created already | 18:55 |
| acabrera | If the proxy tracked how many queues were kept at each partition and how many *somethings* were done using that queue, we could approximate a partition's current "queue" load. | 18:56 |
| acabrera | That could be used as a heuristic together with weight. | 18:56 |
| kgriffs | 3. Historical stats of a partition load (maybe based on statsd or something from the partition's LB) | 18:56 |
| kgriffs | acabrera: sounds promising | 18:56 |
| acabrera | I'd like to see that in a later patch, but the base algorithm is very amenable to expansion by heuristic. | 18:57 |
| kgriffs | can you register a blueprint? we can look at it after we get a solid 1.0 proxy done | 18:57 |
| acabrera | The base idea is that we select from partitions by creating a partition-selection-spectrum. Modifying the length of each partotion's spectrum using a heuristic would be a one line change. | 18:57 |
| acabrera | +1 kgriffs - I'll add that bp now. :) | 18:57 |
| kgriffs | to summarize, we want to factor in a partition's load to the weighting | 18:57 |
| kgriffs | you would take static weight combined with load factor | 18:58 |
| kgriffs | ok, moving on | 18:58 |
| kgriffs | re caching | 18:58 |
| kgriffs | I was thinking to not put that in the storage driver after all | 18:59 |
| oz_akan_ | kgriffs: yesp | 18:59 |
| kgriffs | it's actually common pattern in web apps, for example, to just layer cache on top of storage | 18:59 |
| acabrera | re caching: I agree. | 19:00 |
| kgriffs | oz_akan_: do you have any major concerns with having each proxy be fronted with a LB, say haproxy? | 19:00 |
| kgriffs | sorry, not proxy | 19:00 |
| kgriffs | partition | 19:00 |
| kgriffs | each partition has a VIP | 19:00 |
| kgriffs | and you would do simple TCP load balancing | 19:00 |
| acabrera | in_cache? | yes = read from cache, no = check primary store, update cache if found | 19:00 |
| kgriffs | ok, good on that | 19:01 |
| kgriffs | the only other thing is when migration lands, don't forget about cache invalidation | 19:01 |
| kgriffs | re what i will be working on, it will be some kind of in-process backend for oslo.cache | 19:01 |
| kgriffs | after that, redis unless someone beats me to it. :p | 19:02 |
| acabrera | doesn't that already exist with the memory backend for oslo.cache? | 19:02 |
| kgriffs | acabrera: sort of. it doesn't do LRU | 19:02 |
| acabrera | ahh, I see. | 19:03 |
| kgriffs | also, i was toying with the idea of doing a backend for uwsgi's cache | 19:03 |
| acabrera | That sounds like the kind of policy that would be great to apply to all oslo.cache backends. | 19:03 |
| kgriffs | that one is interesting because it supports p2p replication | 19:03 |
| acabrera | optional LRU spec | 19:03 |
| kgriffs | and it's shared between workers | 19:03 |
| oz_akan_ | kgriffs: in front of proxies I think we will have cloud load balancers | 19:03 |
| oz_akan_ | kgriffs: so do you mean another later between cloud load balancer and proxies? | 19:04 |
| kgriffs | yes | 19:04 |
| oz_akan_ | later = layer | 19:04 |
| kgriffs | each partition would be fronted by it's own proxy | 19:04 |
| kgriffs | otherwise we end up writing half of haproxy into marconi-proxy | 19:04 |
| kgriffs | also means you don't have to update the catalog whenever you add/remove/replace webheads | 19:04 |
| oz_akan_ | plan was to have marconi-proxy instances to serve for all partitions | 19:05 |
| kgriffs | one moment | 19:05 |
| oz_akan_ | otherwise, we would just put marconi-proxy functionality on marconi app | 19:06 |
| oz_akan_ | I think that is why we started to call it proxy | 19:06 |
| oz_akan_ | instead of placement service | 19:06 |
| oz_akan_ | we may install haproxy on marcon-proxy instances, I am not sure how that would work | 19:07 |
| kgriffs | https://etherpad.openstack.org/queuing-scratch | 19:07 |
| acabrera | #link https://blueprints.launchpad.net/marconi/+spec/smarter-partition-selection | 19:08 |
| *** gordonsim has quit IRC | 19:20 | |
| kgriffs | so | 19:24 |
| acabrera | wb | 19:24 |
| kgriffs | catalog should be out-of-band | 19:24 |
| kgriffs | meaning | 19:24 |
| kgriffs | request comes in | 19:24 |
| kgriffs | proxy asks catalog driver for list of webheads OR single proxy href (whatever we decide) | 19:24 |
| kgriffs | proxy then forwards request to one of webheads OR the proxy | 19:25 |
| kgriffs | (latter proxy is haproxy on localhost) | 19:25 |
| kgriffs | make sense? | 19:25 |
| acabrera | yews | 19:25 |
| acabrera | *yes | 19:25 |
| kgriffs | so, the catalog driver doesn't care which webhead gets the request, it is just informational only | 19:25 |
| oz_akan_ | I think it gets overly complicated | 19:26 |
| kgriffs | now, the proxy can just hit webheads round-robin using in-memory counter | 19:26 |
| oz_akan_ | each marconi-proxy has haproxy | 19:26 |
| kgriffs | if one fails, it just tries the next | 19:26 |
| kgriffs | the assumption would be that the operator deploys enough webheads that it is extremely unlikely for all of them to be down at the same time | 19:27 |
| kgriffs | proxy can blacklist the webhead in cache with expiration | 19:27 |
| kgriffs | (TTL for blacklist is configurable) | 19:27 |
| kgriffs | seems like this may not be very hard to implement after all? | 19:28 |
| kgriffs | given this means another cache lookup, having in-process, hierarchical would be a real help | 19:28 |
| kgriffs | or just keep in-process cache for blacklist | 19:28 |
| kgriffs | will be small, and won't take long for all workers on all proxies to blacklist a node if it is down hard | 19:29 |
| kgriffs | if it is only down for a second or two, then you don't want everyone blacklisting anyway | 19:29 |
| * kgriffs is done | 19:29 | |
| * acabrera reads and thinks | 19:29 | |
| oz_akan_ | I like the idea marconi-proxy sending requests to web servers instead of sending to another proxy | 19:30 |
| oz_akan_ | if it has to be another proxy, it could be local haproxy | 19:30 |
| oz_akan_ | I don't like another component as it is one more moving part | 19:30 |
| oz_akan_ | and doing round-robin and taking dead node out, as kgriffs described, shouldn't be so difficult | 19:31 |
| acabrera | I like the idea on "one partition, one URL" - having marconi-proxy hitting web servers directly w/o additional load balancing could be accomplished right not by regstering a partition containing a single URL per webhead, | 19:32 |
| acabrera | I don't like the idea of doing round-robin in the proxy. | 19:32 |
| acabrera | in the marconi-proxy | 19:32 |
| acabrera | That forces a load balancing algorithm on future deployers. | 19:32 |
| oz_akan_ | lets don't like idea compared to complexity to implement but with the complexity it will bring to the environment | 19:33 |
| oz_akan_ | future deployers? | 19:33 |
| oz_akan_ | marconi-proxy can support other algorithms later on | 19:33 |
| acabrera | That will lead to a bloated marconi-proxy over time, while these load balancing algorithms are already a solved problem by other software. | 19:34 |
| acabrera | I'm in favor of marconi-proxy caring about the details of partitions, but not about the various web servers registered under a partition (if more than 1, which I believe is bad idea to have more than one node/contact point per partition). | 19:35 |
| oz_akan_ | acabrera: assuming that marconi web servers will be identical, there is tiny chance someone will ever need anything other than round-robin | 19:37 |
| oz_akan_ | and if needed, you could just configure one endpoint under marconi-proxy, which will actually be a proxy and then do what ever algorithm is supported | 19:38 |
| acabrera | so the argument to have marconi-proxy implement round robin is that it avoids adding another dependency... | 19:39 |
| oz_akan_ | yes, that is my argument | 19:39 |
| kgriffs | guys, consider that you could just have a list of "webheads" with two entries that go to two haproxies or a single entry going to a VIP for the partition | 19:40 |
| kgriffs | then an operator can do advanced LB stuff | 19:40 |
| kgriffs | otherwise they can just let marconi-proxy do it's simple LB algorithm | 19:41 |
| kgriffs | s/it's/its | 19:41 |
| oz_akan_ | yes, round robin is just enough, I think for ever. | 19:42 |
| acabrera | alright | 19:42 |
| acabrera | so given this discussion - keep marconi-proxy as-is? (round-robin, lists of nodes per partition)? | 19:42 |
| oz_akan_ | acabrera: haven't you already implemented that anywyas? | 19:42 |
| acabrera | yup. :P | 19:42 |
| oz_akan_ | so why are we loosing time? | 19:42 |
| acabrera | making sure it's the right way to go. | 19:43 |
| oz_akan_ | makes sense | 19:43 |
| kgriffs | acabrera: not really as-is, i mean the storage driver doesn't do the lb | 19:45 |
| kgriffs | or has that already been moved? | 19:45 |
| acabrera | alright, marconi proxy stays as is. The algorithm for round robin became much easier to use with the mongo driver patch. All a storage driver needs to do is provide a list of [(name, weight), ...] and then pass the chosen partition to round_robin(), which yields a node from that partition after 2 more DB queries... | 19:45 |
| oz_akan_ | so even if we had decided to have a proxy, we could just set up current implementation to point to one endpoint which would be a load balancer / proxy | 19:45 |
| acabrera | kgriffs: as-is since the mongo patch | 19:45 |
| acabrera | I simplified it. :) | 19:45 |
| acabrera | storage doesn't LB. | 19:46 |
| kgriffs | wait a sec | 19:46 |
| oz_akan_ | storage driver doesn't have to do lb, does it? | 19:46 |
| kgriffs | storage driver ===> catalog driver | 19:46 |
| oz_akan_ | LB is done on transportation level | 19:46 |
| kgriffs | +! | 19:46 |
| kgriffs | catalog driver just returns info | 19:46 |
| acabrera | storage driver --> (catalogue + partitions) | 19:46 |
| oz_akan_ | marcon-proxy will be reading from mongodb. so HA is provided by mogno | 19:47 |
| kgriffs | that's fine, but it shouldn't have any logic to pick a webhead within the partition | 19:47 |
| kgriffs | i mean, the catalog driver shouldn't | 19:47 |
| oz_akan_ | yes | 19:47 |
| acabrera | I need to further move the round-robin index out of storage and place that in-process. | 19:47 |
| kgriffs | I imagine handler like: | 19:47 |
| kgriffs | on_get()... | 19:47 |
| oz_akan_ | queues will be mapped to partition and partition mapped to web servers | 19:47 |
| kgriffs | def on_get(…): | 19:48 |
| kgriffs | 1. get catalog entry for queue | 19:49 |
| kgriffs | 2. pick href from list of webheads in partition record from catalog | 19:49 |
| kgriffs | 3. forward request to that href | 19:49 |
| kgriffs | (2) will use the simple round-robin, in-proc blacklisting discussed earllier | 19:49 |
| acabrera | hmm... | 19:50 |
| oz_akan_ | 1- get partition - web server mapping | 19:50 |
| kgriffs | (1) is the only thing the storage driver thingy is responsible for | 19:50 |
| kgriffs | (2) is a helper function | 19:50 |
| acabrera | I need to make some more updates, since I don't currently support per request LB. | 19:50 |
| acabrera | I need to make it so that queue stores a partition name, rather than an href. | 19:50 |
| kgriffs | you just have a RoundRobin() object instance for (2) | 19:51 |
| oz_akan_ | to keep mapping data small, I think we should map queue to partition id | 19:51 |
| oz_akan_ | and parition id to web servers | 19:51 |
| kgriffs | that is shared by on_* methods | 19:51 |
| oz_akan_ | or urls | 19:51 |
| kgriffs | that instance keeps a blacklist and counter | 19:51 |
| acabrera | +1 oz_akan_ - I was just going to use partition name. | 19:51 |
| kgriffs | rr.choose(list_of_webheads) | 19:51 |
| kgriffs | like random.choice | 19:51 |
| oz_akan_ | guy, I will catch acabrera on Monday to go over the plan, I need to finish some stuff now | 19:51 |
| oz_akan_ | guy =guya | 19:52 |
| kgriffs | kk | 19:52 |
| oz_akan_ | guys... | 19:52 |
| acabrera | k | 19:52 |
| *** vkmc has quit IRC | 20:01 | |
| *** tedross has quit IRC | 20:19 | |
| zyuan | acabrera: http://openmp.org/wp/2013/09/clang/ | 20:30 |
| zyuan | looks good http://clang-omp.github.io/ | 20:31 |
| *** amitgandhi has quit IRC | 20:32 | |
| *** amitgandhi has joined #openstack-marconi | 20:37 | |
| acabrera | sweet | 20:45 |
| acabrera | IMO, there's no reason AFAICT to continue using gcc, except to compile the linux kernel. | 20:47 |
| acabrera | seems like clang will surpass gcc in every respect before too much longer. | 20:47 |
| zyuan | hahahaha linux kernel | 20:47 |
| zyuan | acabrera: the thing left is transactional memory. gcc's libitm already support haswell's HTM instructions | 20:58 |
| *** acabrera has quit IRC | 21:12 | |
| *** openstackgerrit has quit IRC | 21:14 | |
| *** flaper87 is now known as flaper87|afk | 21:27 | |
| *** amitgandhi has quit IRC | 21:29 | |
| *** oz_akan_ has quit IRC | 21:37 | |
| *** vkmc has joined #openstack-marconi | 22:53 | |
| *** reed has joined #openstack-marconi | 23:01 | |
| reed | i just read that this channel is a source of good laughs :) | 23:01 |
| reed | http://www.openstack.org/blog/2013/09/open-mic-spotlight-flavio-percoco/ | 23:02 |
| kgriffs | hehe | 23:26 |
| kgriffs | hang out with us for a week and you'll see what he means. :p | 23:27 |
| reed | will do :) set the channel to autojoin | 23:30 |
| *** kgriffs is now known as kgriffs_afk | 23:40 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!