| *** haomaiwang has quit IRC | 00:51 | |
| *** haomaiwang has joined #openstack-marconi | 00:54 | |
| *** nosnos has joined #openstack-marconi | 00:55 | |
| *** fifieldt__ is now known as fifieldt | 02:30 | |
| *** haomai___ has joined #openstack-marconi | 02:33 | |
| *** haomaiwang has quit IRC | 02:34 | |
| *** jeffreycoho has quit IRC | 03:20 | |
| *** nosnos has quit IRC | 03:36 | |
| *** nosnos has joined #openstack-marconi | 04:15 | |
| *** haomai___ has quit IRC | 04:44 | |
| *** haomaiwa_ has joined #openstack-marconi | 04:45 | |
| *** haomai___ has joined #openstack-marconi | 05:00 | |
| *** haomaiwa_ has quit IRC | 05:03 | |
| *** amalagon has joined #openstack-marconi | 06:16 | |
| *** haomai___ has quit IRC | 06:53 | |
| *** haomaiwang has joined #openstack-marconi | 06:53 | |
| *** jamiehannaford has joined #openstack-marconi | 06:56 | |
| *** AAzza_afk is now known as AAzza | 07:01 | |
| *** chandan_kumar has quit IRC | 07:05 | |
| *** haomaiw__ has joined #openstack-marconi | 07:08 | |
| *** haomaiwang has quit IRC | 07:10 | |
| *** haomaiw__ has quit IRC | 07:14 | |
| *** flaper87|afk is now known as flaper87 | 07:24 | |
| *** ykaplan has joined #openstack-marconi | 08:18 | |
| *** seiflotfy has left #openstack-marconi | 09:12 | |
| *** AAzza is now known as AAzza_afk | 09:13 | |
| *** AAzza_afk is now known as AAzza | 09:13 | |
| *** davidhadas_ has quit IRC | 09:32 | |
| *** AAzza is now known as AAzza_afk | 09:34 | |
| *** ykaplan has quit IRC | 09:37 | |
| *** norman has joined #openstack-marconi | 09:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/marconi: Updated from global requirements https://review.openstack.org/99033 | 09:48 |
|---|---|---|
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/python-marconiclient: Updated from global requirements https://review.openstack.org/99093 | 09:52 |
| *** openstackgerrit has quit IRC | 10:06 | |
| *** openstackgerrit has joined #openstack-marconi | 10:07 | |
| *** davidhadas has joined #openstack-marconi | 10:09 | |
| *** haomaiwa_ has joined #openstack-marconi | 10:13 | |
| *** haomaiwa_ has quit IRC | 10:23 | |
| *** haomaiwang has joined #openstack-marconi | 10:23 | |
| *** haomaiw__ has joined #openstack-marconi | 10:25 | |
| *** haomaiwang has quit IRC | 10:28 | |
| *** ykaplan has joined #openstack-marconi | 10:29 | |
| *** openstackgerrit has quit IRC | 10:35 | |
| *** openstackgerrit has joined #openstack-marconi | 10:36 | |
| *** vkmc has joined #openstack-marconi | 10:47 | |
| vkmc | morning all! o/ | 10:50 |
| *** davidhadas___ has joined #openstack-marconi | 11:03 | |
| *** davidhadas has quit IRC | 11:06 | |
| flaper87 | vkmc: morning :) | 11:06 |
| vkmc | flaper87, hey flaaaa :) | 11:08 |
| flaper87 | vkmc: soooo, I gotta summarize the call we had on Friday | 11:09 |
| flaper87 | I'll do that today | 11:09 |
| vkmc | flaper87, sure, whenever you have a moment I'm looking forward to it | 11:10 |
| vkmc | flaper87, probably you want to send it to the ML, so everyone is up to date | 11:10 |
| flaper87 | the sooner I do it, the better. I'm starting to forget things from that call | 11:10 |
| flaper87 | :P | 11:10 |
| vkmc | lol | 11:11 |
| vkmc | that's the bad part of video conferences... there is no log -.- | 11:11 |
| flaper87 | add to that my really bad memory | 11:13 |
| vkmc | haha | 11:13 |
| vkmc | I also have a bad memory... like Nemo's Dory | 11:14 |
| *** nosnos has quit IRC | 11:15 | |
| vkmc | bbs | 11:44 |
| *** norman has quit IRC | 12:19 | |
| *** norman has joined #openstack-marconi | 12:20 | |
| norman | hi all, how can I own the new bugs? The 'assigned to' is not editable | 12:23 |
| *** davidhadas___ has quit IRC | 12:24 | |
| flaper87 | norman: you can assign it to yourself | 12:34 |
| flaper87 | norman: what bug is it? | 12:34 |
| *** sriram has joined #openstack-marconi | 12:37 | |
| norman | flaper87: 1328720 about the pymongo's cert options | 12:39 |
| flaper87 | bug 1328720 | 12:39 |
| flaper87 | bug #1328720 | 12:39 |
| flaper87 | stupid bot | 12:39 |
| norman | I refreshed my browser, it seems that it's editable again | 12:39 |
| *** sriram has quit IRC | 12:40 | |
| *** sriram has joined #openstack-marconi | 12:40 | |
| norman | thanks, it owned by me now -:) | 12:40 |
| *** jmckind has joined #openstack-marconi | 12:43 | |
| flaper87 | norman: np | 12:50 |
| *** haomaiw__ has quit IRC | 12:58 | |
| *** Obulpathi has joined #openstack-marconi | 13:26 | |
| *** Obulpathi has quit IRC | 13:26 | |
| *** Obulpathi has joined #openstack-marconi | 13:27 | |
| *** tonytan4ever has joined #openstack-marconi | 13:47 | |
| *** malini1 has joined #openstack-marconi | 13:52 | |
| *** balajiiyer has joined #openstack-marconi | 13:54 | |
| *** tonytan4ever has quit IRC | 14:21 | |
| malini1 | hola!!! | 14:23 |
| malini1 | flaper87: can you please review this when you get a chance https://review.openstack.org/#/c/93295/ ? | 14:24 |
| vkmc | hola malini1! | 14:27 |
| malini1 | hello vkmc!! | 14:27 |
| malini1 | how are you? | 14:27 |
| vkmc | all good and you? :) | 14:28 |
| malini1 | good.for a change, it feels like I had a long enough weekend :D | 14:29 |
| sriram | nice! | 14:29 |
| *** rwsu has joined #openstack-marconi | 14:29 | |
| sriram | and hola! :P | 14:29 |
| *** cpallares has joined #openstack-marconi | 14:29 | |
| *** kgriffs|afk is now known as kgriffs | 14:33 | |
| *** kgriffs is now known as kgriffs|afk | 14:34 | |
| *** kgriffs|afk is now known as kgriffs | 14:35 | |
| *** chandan_kumar has joined #openstack-marconi | 14:36 | |
| vkmc | hola sriram! | 14:36 |
| cpallares | hi vkmc :) | 14:39 |
| vkmc | hi cpallares! | 14:43 |
| cpallares | vkmc: how are you doing? | 14:44 |
| vkmc | cpallares, all good around here and you? | 14:45 |
| vkmc | freeeeeeeeezing | 14:45 |
| vkmc | >.> | 14:45 |
| cpallares | vkmc: Oh? Too cold outside? It's too hot here >_< | 14:45 |
| cpallares | vkmc: Maybe we can trade half the weather :P | 14:46 |
| vkmc | cpallares, too cold for my taste yeah... I love the idea :P | 14:47 |
| *** tonytan4ever has joined #openstack-marconi | 14:47 | |
| sriram | kgriffs: ping | 14:54 |
| kgriffs | yo | 14:54 |
| sriram | how are you today? :) | 14:55 |
| sriram | wrt https://review.openstack.org/#/c/91804/ , I liked flaper87's comments. That is how I had tried to implement it earlier. | 14:56 |
| sriram | but storage is not version aware, how do we go about doing it at the storage layer rather than at transport layer. | 14:56 |
| sriram | kgriffs: ^ | 14:56 |
| sriram | it = queue create logic. | 14:56 |
| kgriffs | looking | 14:57 |
| *** tonytan4ever has quit IRC | 15:03 | |
| *** tonytan4ever has joined #openstack-marconi | 15:04 | |
| *** tonytan4ever has quit IRC | 15:11 | |
| *** tonytan4ever has joined #openstack-marconi | 15:15 | |
| *** amitgandhi has joined #openstack-marconi | 15:16 | |
| kgriffs | sriram: I replied to flaper87's comments | 15:25 |
| kgriffs | TL;DR I'm having trouble understanding how the upsert approach would be any better | 15:25 |
| sriram | upsert would involve 2 queries instead of one insert, so it might be slower wrt to performance. | 15:26 |
| flaper87 | I think the lazy-create is something we should leave to the storage layer. Storage drivers will do it in the best way possible for the driver | 15:27 |
| *** tonytan4_ has joined #openstack-marconi | 15:27 | |
| flaper87 | I don't think checking existence will be faster because it still needs to send data from/to marconi to/from the storage | 15:27 |
| flaper87 | instead, with the upsert, we'd send everything to the storage and let it handle everything using the index | 15:27 |
| kgriffs | flaper87: but you can cache the existence check | 15:28 |
| flaper87 | IIRC, the previous version that used upsert had already taken care of the possible-counter-bug | 15:28 |
| kgriffs | flaper87: also, in the case of mongodb, does an upsert lock the db? | 15:28 |
| flaper87 | yeah but if we're using redis, we still need to send the request to redis and get it back | 15:28 |
| flaper87 | kgriffs: not during the check | 15:29 |
| flaper87 | (AFAIK) | 15:29 |
| flaper87 | kgriffs: and yes, I meant exactly that. Use upsert for v1.1 and check for existence in v1.0 | 15:29 |
| kgriffs | how do you avoid resetting the coutner? | 15:29 |
| flaper87 | kgriffs: you would write the counter just when inserting the document | 15:30 |
| flaper87 | (unless I'm missing something) | 15:30 |
| sriram | self._collection.update({'p_q': scoped_name, 'c': counter},{'$set': {'m': {}}}, upsert=True) | 15:30 |
| sriram | This is what is in there before. | 15:30 |
| kgriffs | doesn't an upsert insert a new document or update the existing? in the latter case, it would reset the counter | 15:30 |
| kgriffs | wait | 15:31 |
| kgriffs | been too long since I've written upsert code | 15:31 |
| *** tonytan4ever has quit IRC | 15:31 | |
| * kgriffs looks that up | 15:31 | |
| flaper87 | kgriffs: yes but the upsert query is: "{'p_q': scoped_name, 'c': counter}" where counter is a 0 counter | 15:31 |
| flaper87 | (IIRC) | 15:31 |
| * flaper87 needs to read the old patch-set again | 15:32 | |
| flaper87 | I know it worked | 15:32 |
| flaper87 | :P | 15:32 |
| kgriffs | hmm | 15:32 |
| sriram | yeah it worked, I tested it :) | 15:32 |
| flaper87 | kgriffs: see that the query doesn't check just on the `p_q` field but `p_q` and counter | 15:33 |
| kgriffs | ok. let me walk through this | 15:33 |
| flaper87 | that's to force the insert to happen *just* when the queue doesn't exist | 15:33 |
| kgriffs | if the queue does not exist | 15:33 |
| kgriffs | you will insert a queue with the given query | 15:33 |
| kgriffs | "{'p_q': scoped_name, 'c': counter}" | 15:33 |
| kgriffs | plus, setting metadata to an empty document, {} | 15:34 |
| kgriffs | correct? | 15:34 |
| flaper87 | there's still a bug in that query, because it basically unsets the metadata | 15:34 |
| flaper87 | yeah | 15:34 |
| kgriffs | flaper87: yeah, I was going to ask about that | 15:34 |
| sriram | unsets the metadata? | 15:35 |
| kgriffs | if the queue does exist, it does not recreate it, but does reset the metadata | 15:35 |
| sriram | ohh. | 15:35 |
| flaper87 | sriram: the query sets metadata to {} regardless the .... you figured it out | 15:35 |
| flaper87 | :P | 15:35 |
| sriram | I got it. :P | 15:35 |
| kgriffs | so... how do you fix it? | 15:37 |
| kgriffs | self._collection.update({'p_q': scoped_name, 'c': counter, 'm': {}},{}, upsert=True) | 15:37 |
| flaper87 | kgriffs: that's one way. Another way is not setting the metadata | 15:37 |
| flaper87 | it'll be set whenever is needed | 15:37 |
| flaper87 | and it's a waste of space to keep it there, empty. | 15:38 |
| kgriffs | yeah | 15:38 |
| flaper87 | Although that helps with updates paddings | 15:38 |
| flaper87 | still, I think not setting it is better | 15:38 |
| kgriffs | so | 15:39 |
| kgriffs | self._collection.update({'p_q': scoped_name, 'c': counter},{}, upsert=True) | 15:39 |
| kgriffs | assuming most of the time the queue *does* exist | 15:39 |
| flaper87 | yup | 15:39 |
| kgriffs | which is faster, doing a cached check for whether the queue exists, or doing this upsert that turns into a noop if the thing exists? | 15:40 |
| kgriffs | two things to check: 1. does update always lock the db | 15:40 |
| kgriffs | 2. benchmark the two approaches | 15:40 |
| kgriffs | (if only we had a way to benchmark this...) | 15:40 |
| kgriffs | benchmarking will depend on whether we are hitting a localhost cache or a central cache (i.e., memcached/redis) | 15:41 |
| kgriffs | errr | 15:41 |
| kgriffs | sorry, conflated that | 15:41 |
| kgriffs | a localhost could be in-process (memory driver), memcached or redis (when those drivers land) | 15:41 |
| flaper87 | yeah, the local memory will definitely be faster although I expect most deployments to use a central one | 15:42 |
| flaper87 | s/central/remote/ | 15:42 |
| kgriffs | ...or a hybrid (similar to the way L1, L2 caches work on a CPU) | 15:42 |
| kgriffs | ...but I digress | 15:42 |
| kgriffs | flaper87: what's the status on the memcached driver? | 15:42 |
| flaper87 | kgriffs: bryan was working on it, I'm not sure if he kept doing so or just abandoned it. | 15:43 |
| flaper87 | but there's a bigger thing going on | 15:43 |
| flaper87 | kgriffs: https://review.openstack.org/#/c/97155/ | 15:43 |
| flaper87 | check that review out | 15:43 |
| kgriffs | "currently unused" | 15:44 |
| kgriffs | well, by integrated projects anyway. :p | 15:44 |
| kgriffs | flaper87: "Keystone has used..." | 15:45 |
| kgriffs | is keystone middleware using dogpile as well, or just the service? | 15:45 |
| flaper87 | yeah, so, I've an on-going review there | 15:47 |
| flaper87 | part of it is that it's being used | 15:47 |
| flaper87 | that said, I'm all for writing a wrapper on top of dogpile | 15:47 |
| flaper87 | that would bring more supported backends for caching | 15:47 |
| flaper87 | kgriffs: I think the middleware uses dogpile as well | 15:48 |
| flaper87 | brb | 15:48 |
| *** ykaplan has quit IRC | 15:49 | |
| sriram | ok, one more thing. coming back to my intial question | 15:49 |
| sriram | if we move logic to storage layer, wont it affect v1 api as well? | 15:49 |
| sriram | maybe I'm missing something here. | 15:49 |
| flaper87 | sriram: you have to keep the check for v1.0 *in the transport* | 15:50 |
| flaper87 | *just for v1.0* | 15:50 |
| flaper87 | v1.1 will rely on the storage layer | 15:50 |
| kgriffs | sriram: it's sort of the inverse from what you are doing now | 15:51 |
| kgriffs | instead of checking in v1.1, you check in v1.0 and raise an error if it does *not* exist | 15:52 |
| kgriffs | but still, given that we still have to do a check, what does moving to an upsert buy us? | 15:52 |
| sriram | I see. I'm getting some clarity now. Sorry for the confusion | 15:53 |
| kgriffs | Just feels like we are moving things around and ending up with something equivalent, with the added possibility that the upsert may be slower (not proven, but possible) | 15:53 |
| sriram | I'm not sure how we would benchmark this to prove which is faster | 15:54 |
| sriram | maybe mongo's .explain() might help here. | 15:54 |
| kgriffs | mmm | 15:54 |
| kgriffs | you could try both ways | 15:54 |
| kgriffs | oh, actually, we can't really benchmark this because our only option for caching right now is memory, since there are no other drivers for oslo.cache | 15:55 |
| kgriffs | blah | 15:55 |
| kgriffs | I think we should decide whether there is some design reason to change back to upsert | 15:56 |
| sriram | hmm | 15:56 |
| kgriffs | if it is actually equivalent, design-wise, to what you have now, then there is no point changing it again | 15:56 |
| sriram | good point. | 15:56 |
| kgriffs | esp. given the performance risk, but even without that | 15:56 |
| kgriffs | consider: | 15:56 |
| kgriffs | in both approaches you have to do a check for queue existence. in one approach, you do it in v1.0, while in the other you do it in v1.1 | 15:59 |
| * kgriffs hopes he got that right | 15:59 | |
| sriram | Yes | 15:59 |
| *** alcabrera|afk is now known as alcabrera | 15:59 | |
| kgriffs | in one approach, you also modify create() to be an upsert rather than an insert | 16:00 |
| sriram | kgriffs: yes, that was the earlier approach | 16:00 |
| kgriffs | what is the benefit? | 16:00 |
| sriram | we let the storage layer handle this. | 16:00 |
| sriram | maybe some other backend has a really cool way of doing it | 16:01 |
| sriram | and that costs us way less in terms of time. | 16:01 |
| sriram | so its better to off load that work to the storage controller. | 16:01 |
| sriram | those are the pros there. | 16:01 |
| kgriffs | I can't see how the storage controller would be able to be any *faster* than cache | 16:01 |
| sriram | Umm, yes. | 16:02 |
| kgriffs | it may be close, or equivalent, but not faster | 16:02 |
| kgriffs | I think the one argument that may make sense is that future API versions will not have to do the "exists" check | 16:02 |
| kgriffs | we just put it in v1.0 and subsequent APIs don't need to do it | 16:03 |
| kgriffs | but then, we lose the opportunity for any possible performance gains by caching...unless we implement that in each driver | 16:03 |
| kgriffs | hmmm | 16:03 |
| sriram | caching every driver is additional hassle, and makes the code harder to read in some cases. | 16:04 |
| kgriffs | yeah | 16:04 |
| *** tonytan4_ has quit IRC | 16:05 | |
| kgriffs | on the other hand, we will have to do the exists check in v1.1, 2.0, 2.1 | 16:05 |
| sriram | do we decide this at the meeting tomorrow? | 16:05 |
| kgriffs | it still feels like a wash to me. No clear winner. So I would just go with what you have right now. Optimize for least amount of code change. | 16:06 |
| sriram | we could have more people weigh in with their opinions. | 16:06 |
| sriram | All right | 16:06 |
| kgriffs | sriram: nah, I don't think this is that critical of a decision. | 16:06 |
| flaper87 | the whole point was exactly that, I don't think future API versions will have to do the exists | 16:07 |
| kgriffs | If Flavio has a super good reason why we should change more code than what you have currently, then do that. otherwise, leave as is | 16:07 |
| kgriffs | flaper87: like I said, I don't think the decision is super critical either way, just as long as we make sure we can still make things fast. | 16:08 |
| sriram | bbl lunch | 16:08 |
| kgriffs | we can, in both ways, because we can cache at transport or driver layer if needed | 16:08 |
| flaper87 | sure | 16:08 |
| * sriram will think on this as he munches food | 16:08 | |
| flaper87 | anyway, I guess we can leave it as-is | 16:09 |
| kgriffs | I'm still curious about whether upsert takes a db lock | 16:09 |
| flaper87 | we'll figure this out later | 16:09 |
| flaper87 | I think the "query" part of it doesn't | 16:09 |
| flaper87 | kgriffs: btw, We'll have the mongodb summit next week. If there's anything I should bring up there, let me know | 16:09 |
| flaper87 | I'll also ask this | 16:09 |
| kgriffs | kk | 16:10 |
| *** jergerber has joined #openstack-marconi | 16:12 | |
| *** vkmc has quit IRC | 16:14 | |
| *** tonytan4ever has joined #openstack-marconi | 16:16 | |
| *** AAzza_afk is now known as AAzza | 16:40 | |
| *** balajiiyer has quit IRC | 16:41 | |
| *** tonytan4ever has quit IRC | 16:42 | |
| *** tonytan4ever has joined #openstack-marconi | 16:43 | |
| *** tonytan4ever has quit IRC | 16:48 | |
| *** balajiiyer has joined #openstack-marconi | 16:50 | |
| *** balajiiyer has quit IRC | 16:50 | |
| *** balajiiyer has joined #openstack-marconi | 16:50 | |
| openstackgerrit | Sriram Madapusi Vasudevan proposed a change to openstack/marconi: Implement Lazy Create Queue in v1.1 API https://review.openstack.org/91804 | 16:57 |
| *** vkmc has joined #openstack-marconi | 17:14 | |
| *** kgriffs is now known as kgriffs|afk | 17:17 | |
| *** balajiiyer has quit IRC | 17:21 | |
| *** jmckind has quit IRC | 17:25 | |
| *** jamiehannaford has quit IRC | 17:26 | |
| *** balajiiyer has joined #openstack-marconi | 17:27 | |
| vkmc | flaper87, about AMQP, do we have a possible way to make it work or are we completely blocked? | 17:27 |
| vkmc | flaper87, (a preview would be great to kill the anxiety) | 17:29 |
| *** reed has joined #openstack-marconi | 17:36 | |
| *** kgriffs|afk is now known as kgriffs | 17:38 | |
| *** kgriffs is now known as kgriffs|afk | 17:39 | |
| *** ChanServ changes topic to "OpenStack Queuing and Notification Service || Smile :D || Meetings every Tuesday @ 15:00 UTC || Wiki: https://wiki.openstack.org/wiki/Marconi || Paste: http://paste.openstack.org/ || Send messages and make some noise :D" | 17:39 | |
| *** tonytan4ever has joined #openstack-marconi | 17:40 | |
| *** malini2 has joined #openstack-marconi | 18:01 | |
| *** haomaiwang has joined #openstack-marconi | 18:02 | |
| *** malini1 has quit IRC | 18:04 | |
| *** alcabrera is now known as alcabrera|afk | 18:05 | |
| *** AAzza is now known as AAzza_afk | 18:17 | |
| *** AAzza_afk is now known as AAzza | 18:23 | |
| *** tonytan4ever has quit IRC | 18:32 | |
| *** tonytan4ever has joined #openstack-marconi | 18:33 | |
| *** tonytan4ever has quit IRC | 18:43 | |
| *** tonytan4ever has joined #openstack-marconi | 18:44 | |
| *** tonytan4_ has joined #openstack-marconi | 18:45 | |
| *** tonytan4_ has quit IRC | 18:47 | |
| *** tonytan4_ has joined #openstack-marconi | 18:48 | |
| *** tonytan4ever has quit IRC | 18:48 | |
| *** vkmc_ has joined #openstack-marconi | 18:53 | |
| *** vkmc has quit IRC | 18:56 | |
| *** vkmc_ is now known as vkmc | 18:56 | |
| *** ametts has joined #openstack-marconi | 19:02 | |
| *** AAzza is now known as AAzza_afk | 19:07 | |
| *** reed has quit IRC | 19:09 | |
| sriram | malini: Where does the wiki for all the status codes for marconi live? I'm trying to find it. | 19:11 |
| vkmc | sriram, https://wiki.openstack.org/wiki/Marconi/specs/api/v1#HTTP_Response_Codes | 19:13 |
| sriram | thanks vkmc, I had been scrolling past that :P | 19:13 |
| vkmc | haha is too small | 19:14 |
| vkmc | np | 19:14 |
| *** reed has joined #openstack-marconi | 19:14 | |
| *** malini2 has quit IRC | 19:25 | |
| *** alcabrera|afk is now known as alcabrera | 19:28 | |
| *** cpallares has quit IRC | 19:38 | |
| *** ykaplan has joined #openstack-marconi | 19:38 | |
| *** jdprax has joined #openstack-marconi | 19:41 | |
| *** haomaiwang has quit IRC | 19:52 | |
| *** jamiehannaford has joined #openstack-marconi | 20:13 | |
| *** jdprax has quit IRC | 20:27 | |
| *** jdprax has joined #openstack-marconi | 20:27 | |
| *** jdprax has quit IRC | 20:27 | |
| *** sriram has quit IRC | 20:35 | |
| *** kgriffs|afk is now known as kgriffs | 20:48 | |
| kgriffs | there is one for v1.1 as well | 20:54 |
| kgriffs | https://wiki.openstack.org/wiki/Marconi/specs/api/v1.1/errors | 20:55 |
| kgriffs | vkmc, sriram: ^^^ | 20:55 |
| vkmc | thanks kgriffs! | 20:55 |
| kgriffs | malini: when should I schedule this bug? https://bugs.launchpad.net/marconi/+bug/1273335 | 20:56 |
| *** tonytan4_ has quit IRC | 21:05 | |
| vkmc | kgriffs, which is the status of this bp? https://blueprints.launchpad.net/python-marconiclient/+spec/cli | 21:07 |
| vkmc | kgriffs, is it scheduled for K? | 21:07 |
| *** Obulpathi has quit IRC | 21:09 | |
| kgriffs | looking | 21:09 |
| kgriffs | ah, OK | 21:13 |
| kgriffs | Flavio did some basic plumbing on that | 21:13 |
| kgriffs | https://github.com/openstack/python-marconiclient/blob/master/marconiclient/queues/cli.py | 21:13 |
| kgriffs | I don't think it does much right now, though | 21:13 |
| * vkmc clicks | 21:14 | |
| vkmc | cool | 21:14 |
| kgriffs | https://review.openstack.org/#/c/87937/ | 21:15 |
| vkmc | it really caught my attention... and since AMQP is a bit stalled right now I was looking for something to contribute with in parallel | 21:15 |
| flwang | hi guys when is this week meeting? | 21:16 |
| *** tonytan4ever has joined #openstack-marconi | 21:16 | |
| kgriffs | ok, looks like it just does queue CRUD | 21:16 |
| *** jamiehannaford has quit IRC | 21:16 | |
| vkmc | yup :) | 21:17 |
| kgriffs | flwang: we haven't moved it yet, so it is at the same time. 1500 UTC. That's like, 3am for you or something? | 21:17 |
| flwang | got it | 21:17 |
| flwang | 3am for me, gosh :D | 21:17 |
| kgriffs | yeah, we'll have to move it. I'll bring it up in tomorrow's meeting. :p | 21:18 |
| flwang | I will read the meeting minutes | 21:18 |
| vkmc | :o too early and too late at the same time | 21:18 |
| flwang | vkmc: yep, I'm based in New Zealand | 21:19 |
| vkmc | flwang, yaay so nice :) | 21:20 |
| flwang | vkmc: sort of :D | 21:21 |
| kgriffs | vkmc: re the CLI, it would be good to sync with flaper87, but I don't see any reason why you couldn't lend a hand | 21:28 |
| vkmc | kgriffs, k :) thanks | 21:30 |
| *** tonytan4ever has quit IRC | 21:34 | |
| flwang | kgriffs: when is the m-2 due date? | 21:38 |
| kgriffs | here's the full schedule | 21:39 |
| kgriffs | https://wiki.openstack.org/wiki/Juno_Release_Schedule | 21:39 |
| flwang | kgriffs: cool, thanks | 21:39 |
| vkmc | kgriffs, regarding that schedule... I was thinking that maybe I could check some of the others backend storages | 21:41 |
| vkmc | if that is a priority over the cli for Naav | 21:41 |
| flwang | wow, so Naav has been the official name, right? | 21:42 |
| kgriffs | yep, that was voted on by the team | 21:42 |
| flwang | by us or by TC? :) | 21:42 |
| vkmc | oh... spoiler alert :( | 21:42 |
| kgriffs | vkmc: re backends... did we determine that AMQP 1.0 can't work with even part of the 1.x API? | 21:43 |
| flwang | TBH, I love Marconi | 21:43 |
| vkmc | kgriffs, I still have to chat with flaper87 about it | 21:43 |
| kgriffs | yeah, someone snagged the Marconi trademark... I was sworn to silence, but I'll just say it was rather disingenuous of the trademark holder | 21:44 |
| kgriffs | out of the short list the team brainstormed: "Our trademark counsel did a search and recommended Zaqar or Naav as the best potential names with low-to-moderate risk. Tamtam came up as a moderate risk based on TamTamy, but we could still likely pursue it because the software is quite distinguishable. Both Raven and Copper pose a higher risk, because they are registered to companies in dev / cloud spaces (Rave | 21:44 |
| kgriffs | nflow and Copper.io respectively)" | 21:44 |
| *** balajiiyer has quit IRC | 21:45 | |
| flwang | kgriffs: how to pronounce Naav? is it like [na:vi] ? | 21:46 |
| vkmc | is like... dragging the a | 21:47 |
| kgriffs | yeah. "Na" is like the "kno" in "knot" | 21:48 |
| kgriffs | you drag out the middle a bit (like slurring two notes together of the same value, if you are a musician) | 21:49 |
| kgriffs | something like that, anyway. :p | 21:50 |
| kgriffs | vkmc: back on the topic of backend drivers | 21:51 |
| kgriffs | I've been discussing with marconi core reviewers what they want to do about that. It's a rather complicated question. | 21:52 |
| vkmc | kgriffs, it is yeah | 21:52 |
| vkmc | maybe we want to focus on redis and amqp right now? | 21:52 |
| vkmc | we have to define the API unified model so we can go through a secure path | 21:53 |
| kgriffs | yeah. The API is the sticking point. Also the question of durability. | 21:53 |
| kgriffs | redis should be super fast (TBD), but is not durable | 21:53 |
| kgriffs | MongoDB is reasonably fast, durable, but AGPL so some people don't want to deploy it | 21:54 |
| kgriffs | AMQP 1.0 brokers are pretty fast, durable, but don't support the entire API | 21:55 |
| kgriffs | Kafka is insanely fast, durable, but doesn't support the entire API or perhaps any of it | 21:55 |
| vkmc | well.. redis and mongo are alike | 21:56 |
| kgriffs | AMQP is also nice because it gives operators another way to scale their brokers | 21:56 |
| vkmc | and amqp and kafka too.. | 21:56 |
| vkmc | in a high level of abstraction of course | 21:57 |
| kgriffs | redis and mongo can support the entire API | 21:57 |
| vkmc | they follow the same arch | 21:57 |
| kgriffs | AMQP and Kafka are more opinionated about the semantics | 21:57 |
| kgriffs | Flavio also mentioned Elasticsearch should be able to support the entire API | 21:57 |
| kgriffs | and we could just go all out and roll our own thing using LevelDB as a building block | 21:58 |
| vkmc | sounds good | 21:58 |
| kgriffs | I kind of feel like we need to either decide to just go with 1.x design for J and only work on things that can support all of it | 21:59 |
| kgriffs | or | 21:59 |
| kgriffs | reboot the project and design the API so it will be lovely for AMQP and Kafka | 21:59 |
| kgriffs | or | 21:59 |
| vkmc | thing is... why do we want to give support for brokers like amqp and kafka? just for efficiency reasons? | 21:59 |
| kgriffs | two reasons | 22:00 |
| kgriffs | one is that Kafka has some impressive benchmarks, albeit those aren't going over HTTP | 22:00 |
| kgriffs | and using a compiled, highly-optimized language | 22:01 |
| kgriffs | :p | 22:01 |
| vkmc | and can't we achieve something similar by using other storage backends as elasticsearch? | 22:01 |
| kgriffs | second reason is that Flavio was thinking we could help AMQP brokers scale out | 22:01 |
| kgriffs | vkmc: that's the million-dollar question | 22:02 |
| kgriffs | we don't really know how fast a more generic nosql backend can be | 22:02 |
| vkmc | hehe we will need to do some benchmarks | 22:02 |
| kgriffs | or how fast Qpid or Kafka would be if put behind a python-based, REST interface | 22:02 |
| kgriffs | actually, there were three reasons | 22:03 |
| kgriffs | last one was that it has been posited that there is something inherently less efficient about the current design - that if we simplify it drastically, we could make it much faster. | 22:04 |
| kgriffs | s/design/API | 22:04 |
| kgriffs | That is probably true, at least for task distribution workloads | 22:04 |
| vkmc | yes that's true | 22:04 |
| *** alcabrera is now known as alcabrera|afk | 22:05 | |
| kgriffs | but then again, it is hard to know for sure without benchmarks | 22:05 |
| flaper87 | o/ | 22:05 |
| flaper87 | 'sup folks | 22:05 |
| vkmc | we have to figure out if the flexibility provided is important for naav users... if that pay off the reduced efficiency | 22:05 |
| flaper87 | vkmc: so, spoler, it's quite possible that the result of this whole AMQP research ends up in being a no-go for the backend | 22:06 |
| vkmc | adjusting ourselves to strict queue semantics would make us to change the api... a lot | 22:06 |
| flaper87 | however, having a amqp 1.0 transport still makes sense | 22:06 |
| flaper87 | that said, I'll write everything on an etherpad as promissed and bring it up in tomorrow's meeting | 22:06 |
| vkmc | thanks flaper87 :) | 22:07 |
| flaper87 | I believe there are some semplifications we can do in the API but even with those semplifications I don't think we'll be able to support AMQP backends | 22:07 |
| *** amitgandhi has quit IRC | 22:07 | |
| vkmc | :/ | 22:07 |
| flaper87 | vkmc: since you've done lot of research on the AMQP side, would you be interested in writing an AMQP 1.0 transport | 22:08 |
| flaper87 | ? | 22:08 |
| flaper87 | Azure Queues has support for AMQP1.0 which is really interesting | 22:09 |
| flaper87 | they reported a 20x times performance improvement after enabling it. | 22:09 |
| flaper87 | Not saying the same will happen with Marconi | 22:09 |
| vkmc | sure I'd be glad :) | 22:09 |
| vkmc | I was looking into ElasticSearch as well | 22:10 |
| flaper87 | but the AMQP 1.0 protocol puts marconi in the position to sit along side other amqp technologies | 22:10 |
| vkmc | but maybe we want to stick with Redis storage and AMQP transport for now? | 22:10 |
| flaper87 | vkmc: funny you said that. I started playing with it | 22:10 |
| flaper87 | I mean, working on a storage for ES | 22:10 |
| vkmc | it looks cool | 22:10 |
| flaper87 | I've used ES in the past already | 22:10 |
| vkmc | haha ok I won't take your toy away :) | 22:10 |
| flaper87 | if you prefer to stick to storage technologies, I'm happy to leave that work to you | 22:10 |
| flaper87 | vkmc: really, I wan't you to work on something you find interesting | 22:11 |
| kgriffs | BTW, not sure how much it matters, but one more data point: All the feedback we have gotten from Rackspace's Marconi deployment has never been "hey, we don't like the API" but "hey, I want to do like 10,000 req/sec, can you guys handle that?" I think we can do that today on optimized MongoDB hardware, but it may not be as cost-efficient as some other backend. | 22:11 |
| flaper87 | since you started looking at it, it may probably make sense | 22:11 |
| flaper87 | kgriffs: agreed | 22:11 |
| kgriffs | also, there is the question of how much performance gain we get from using a non-HTTP protocol | 22:11 |
| flaper87 | kgriffs: that's something I want share w/ you tomorrow too | 22:12 |
| flaper87 | re the API | 22:12 |
| kgriffs | ok | 22:12 |
| vkmc | flaper87, whatever is best for Marconi :) | 22:12 |
| flaper87 | TL;DR: I don't think it's bad | 22:12 |
| flaper87 | vkmc: and for you ;) | 22:12 |
| flaper87 | kgriffs: what do you think about the ES backend? | 22:13 |
| vkmc | flaper87, thx... I could continue with AMQP... this time in the transport side | 22:13 |
| vkmc | flaper87, so we can get something from the research done | 22:13 |
| kgriffs | i will say, however, that I've done a different service using python, over HTTP, that was pretty darn fast. Not Kafka-fast, but fast enough to make people do a double-take when I told them the benchmarks. | 22:13 |
| flaper87 | vkmc: it's your call | 22:13 |
| kgriffs | this is why we __neeeeeed__ benchmarks | 22:13 |
| vkmc | k | 22:13 |
| vkmc | flaper87, should I drop the pseudo-amqp-storage-driver? | 22:14 |
| flaper87 | kgriffs: I don't believe HTTP is "slow for the task" | 22:14 |
| flaper87 | vkmc: yeah, lets stop that work there and change direction | 22:14 |
| kgriffs | flaper87: no, what you said is we should migrate everything to Go | 22:14 |
| vkmc | (y) | 22:14 |
| flaper87 | don't trash it, though | 22:14 |
| vkmc | oh no it's in my repo | 22:14 |
| * kgriffs trolling for fun and profit | 22:14 | |
| flaper87 | kgriffs: LIAR, I said rust | 22:14 |
| flaper87 | vkmc: great | 22:15 |
| flaper87 | vkmc: so, what do you pick? ES store or AMQP transport ? | 22:15 |
| vkmc | flaper87, AMQP transport :) | 22:15 |
| flaper87 | vkmc: cool | 22:16 |
| flaper87 | vkmc: so, some hints there | 22:16 |
| flaper87 | vkmc: take a look at how Azure Queues implemented it | 22:16 |
| flaper87 | weeeellllll | 22:16 |
| vkmc | noted, thanks :) | 22:16 |
| flaper87 | that was a fun thing to say given the fact that it's Microsoft | 22:16 |
| flaper87 | I meant to say, read the API docs | 22:16 |
| flaper87 | :D | 22:16 |
| vkmc | shh, we keep that under Marconi's rug | 22:16 |
| vkmc | of course | 22:17 |
| flaper87 | vkmc: thanks | 22:18 |
| flaper87 | a looooooooooooooooooooooooot | 22:18 |
| vkmc | :) | 22:18 |
| kgriffs | remember, great artists steal | 22:18 |
| kgriffs | ...with a grain of salt | 22:19 |
| vkmc | totally haha | 22:19 |
| vkmc | oh and flaper87, what do you think about the cli right now? | 22:19 |
| vkmc | for marconi client | 22:20 |
| kgriffs | specifically: https://blueprints.launchpad.net/python-marconiclient/+spec/cli | 22:21 |
| vkmc | yes, sorry | 22:21 |
| vkmc | thanks kgriffs | 22:21 |
| kgriffs | p.s. - need some help triaging bugz | 22:21 |
| kgriffs | https://bugs.launchpad.net/python-marconiclient | 22:21 |
| flaper87 | vkmc: it's a WIP | 22:21 |
| * kgriffs goes away again | 22:21 | |
| flaper87 | :D | 22:21 |
| flaper87 | kgriffs: I'll triage them | 22:21 |
| kgriffs | excellent | 22:21 |
| flaper87 | vkmc: want to help with that? | 22:21 |
| vkmc | flaper87, yeah it would be cool! | 22:21 |
| flaper87 | vkmc: cool I just added support for queues | 22:22 |
| flaper87 | I was focusing on plumbing everything | 22:22 |
| vkmc | flaper87, kgriffs shared it with me... it looks great | 22:22 |
| flaper87 | it should be pretty straightforward to add support for other things | 22:22 |
| vkmc | yes.. it's indeed really helpful to have it as an example | 22:22 |
| flaper87 | great. Thanks, vkmc | 22:26 |
| flaper87 | :) | 22:26 |
| vkmc | np :) | 22:26 |
| vkmc | kgriffs, flaper87 I just updated the AMQP bp... if there is something wrong let me know | 22:33 |
| flaper87 | vkmc: ok | 22:34 |
| flaper87 | thanks | 22:34 |
| flaper87 | It can always be revisited in the future | 22:34 |
| vkmc | of course | 22:34 |
| vkmc | maybe I should add it? | 22:34 |
| flaper87 | sure | 22:34 |
| *** jergerber has quit IRC | 22:35 | |
| vkmc | done | 22:35 |
| flaper87 | great, thanks again vkmc | 22:36 |
| flaper87 | ok, gtg now | 22:36 |
| flaper87 | :) | 22:36 |
| flaper87 | take care | 22:36 |
| vkmc | ttfn flaper87 o/ you too! | 22:36 |
| vkmc | enjoy the rest of your evening | 22:36 |
| *** flaper87 is now known as flaper87|afk | 22:37 | |
| *** kgriffs is now known as kgriffs|afk | 22:47 | |
| openstackgerrit | A change was merged to openstack/marconi: Sync from oslo-incubator https://review.openstack.org/100093 | 23:41 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!