Thursday, 2014-01-16

« Wednesday, 2014-01-15 Index Friday, 2014-01-17 »
*** flwang has quit IRC00:03
*** jcru has quit IRC00:11
*** flwang has joined #openstack-marconi00:19
*** amitgandhi has joined #openstack-marconi00:23
*** amitgandhi has quit IRC00:28
*** kgriffs_afk is now known as kgriffs00:52
*** kgriffs is now known as kgriffs_afk01:07
*** amitgandhi has joined #openstack-marconi01:14
*** oz_akan_ has joined #openstack-marconi01:17
*** amitgandhi has quit IRC01:18
*** nosnos has joined #openstack-marconi01:33
*** reed has quit IRC01:35
openstackgerritFei Long Wang proposed a change to openstack/python-marconiclient: Add /health support to the client  https://review.openstack.org/6678702:06
*** amitgandhi has joined #openstack-marconi02:14
*** amitgandhi has quit IRC02:19
*** balajiiyer has joined #openstack-marconi02:23
*** fifieldt has joined #openstack-marconi02:47
*** oz_akan_ has quit IRC02:58
*** flwang has quit IRC03:04
*** ayoung_ has joined #openstack-marconi03:14
*** amitgandhi has joined #openstack-marconi03:15
*** amitgandhi has quit IRC03:20
*** lbragstad has joined #openstack-marconi04:08
*** oz_akan_ has joined #openstack-marconi04:14
*** amitgandhi has joined #openstack-marconi04:16
*** amitgandhi has quit IRC04:20
*** reed has joined #openstack-marconi04:26
*** reed has quit IRC04:36
*** amitgandhi has joined #openstack-marconi05:16
*** amitgandhi has quit IRC05:21
*** nosnos_ has joined #openstack-marconi05:27
*** lbragstad has quit IRC05:27
*** nosnos has quit IRC05:28
*** balajiiyer has quit IRC05:31
*** oz_akan_ has quit IRC06:06
*** nosnos_ has quit IRC06:10
*** nosnos has joined #openstack-marconi06:10
*** oldfish has joined #openstack-marconi06:12
*** oldfish has quit IRC06:14
*** flwang has joined #openstack-marconi06:14
*** amitgandhi has joined #openstack-marconi06:17
*** amitgandhi has quit IRC06:21
*** chandankumar has joined #openstack-marconi06:23
*** oz_akan_ has joined #openstack-marconi06:38
*** oz_akan_ has quit IRC06:43
*** amitgandhi has joined #openstack-marconi07:18
*** amitgandhi has quit IRC07:22
*** flaper87|afk is now known as flaper8707:35
*** oz_akan_ has joined #openstack-marconi07:39
*** oz_akan_ has quit IRC07:44
*** jamieh has joined #openstack-marconi08:08
*** amitgandhi has joined #openstack-marconi08:18
openstackgerritSascha Peilicke proposed a change to openstack/python-marconiclient: Sync with global requirements  https://review.openstack.org/6704608:20
openstackgerritFlavio Percoco proposed a change to openstack/python-marconiclient: Add shard support to the client  https://review.openstack.org/6482908:22
*** amitgandhi has quit IRC08:23
*** oz_akan_ has joined #openstack-marconi08:39
*** oz_akan_ has quit IRC08:44
*** yassine has joined #openstack-marconi09:18
*** amitgandhi has joined #openstack-marconi09:19
*** amitgandhi has quit IRC09:23
*** oz_akan_ has joined #openstack-marconi09:40
openstackgerritA change was merged to openstack/python-marconiclient: Let Hacking choose the right dependencies on its own  https://review.openstack.org/6652609:41
*** oz_akan_ has quit IRC09:45
*** nosnos has quit IRC09:51
*** ykaplan has joined #openstack-marconi09:51
*** flwang has quit IRC09:58
*** amitgandhi has joined #openstack-marconi10:20
*** amitgandhi has quit IRC10:24
*** oz_akan_ has joined #openstack-marconi10:41
*** oz_akan_ has quit IRC10:42
*** oz_akan_ has joined #openstack-marconi10:43
*** oz_akan_ has quit IRC10:47
*** ykaplan has quit IRC10:49
*** flaper87 is now known as flaper87|afk10:59
*** amitgandhi has joined #openstack-marconi11:20
*** amitgandhi has quit IRC11:25
*** ykaplan has joined #openstack-marconi11:59
*** oz_akan_ has joined #openstack-marconi12:21
*** amitgandhi has joined #openstack-marconi12:21
*** amitgandhi has quit IRC12:26
*** tedross has joined #openstack-marconi12:41
*** flaper87|afk is now known as flaper8713:06
*** flwang has joined #openstack-marconi13:15
*** oz_akan_ has quit IRC13:18
*** amitgandhi has joined #openstack-marconi13:22
*** amitgandhi has quit IRC13:26
*** oz_akan_ has joined #openstack-marconi13:29
*** fifieldt has quit IRC13:31
openstackgerritSascha Peilicke proposed a change to openstack/python-marconiclient: Sync with global requirements  https://review.openstack.org/6704613:33
*** oz_akan_ has quit IRC13:33
*** mpanetta has joined #openstack-marconi13:37
*** ayoung_ has quit IRC13:42
flwangflaper87: ping13:56
flaper87flwang: pong13:56
flaper87flwang: glance or marconi?13:57
flwangflaper87: a question about oslo.messaging :)13:57
flaper87glance13:57
*** haomaiwang has quit IRC14:03
*** haomaiwang has joined #openstack-marconi14:04
*** cpallares has joined #openstack-marconi14:08
*** oz_akan_ has joined #openstack-marconi14:12
*** haomaiwa_ has joined #openstack-marconi14:14
cpallaresgood mornin'14:15
*** jcru has joined #openstack-marconi14:15
flaper87cpallares: gooood morning14:17
cpallaresflaper87: o/14:17
cpallareshow are you today?14:17
*** haomaiwang has quit IRC14:17
flaper87cpallares: I'm doing fine, how are you?14:18
*** ykaplan has quit IRC14:18
*** oz_akan_ has quit IRC14:18
*** oz_akan_ has joined #openstack-marconi14:19
openstackgerritCindy Pallares proposed a change to openstack/marconi: Create communication between API request & storage  https://review.openstack.org/6716814:24
flaper87woooooooooooooooooooooooooow14:25
flaper87lemme look at THAT!14:25
*** whenry has joined #openstack-marconi14:26
cpallares:D14:28
*** amitgandhi has joined #openstack-marconi14:28
cpallaresYou should also give some love to this one http://review.openstack.org/#/c/61476/14:28
cpallaresflaper87: ^14:28
*** ykaplan has joined #openstack-marconi14:29
* flaper87 will hug all those patches today14:30
*** balajiiyer has joined #openstack-marconi14:34
*** amitgandhi has quit IRC14:37
*** amitgandhi has joined #openstack-marconi14:38
*** lbragstad has joined #openstack-marconi14:46
*** kgriffs_afk is now known as kgriffs14:50
*** malini has joined #openstack-marconi14:53
*** alcabrera has joined #openstack-marconi15:03
alcabreraGood morning! :D15:03
*** rwsu has joined #openstack-marconi15:07
*** jmckind has joined #openstack-marconi15:15
cpallaresHello alcabrera, good morning to you.15:17
alcabreracpallares: Good morning!15:21
alcabreraHow are you?15:21
*** ayoung_ has joined #openstack-marconi15:26
kgriffsflaper87: quick review here plz - https://review.openstack.org/#/c/65521/15:27
flaper87kgriffs: looking15:27
flaper87alcabrera: good morning. How are you doing?15:28
flaper87kgriffs: I thought I had reviewed that patch :/15:28
flaper87doone15:28
alcabreraflaper87: good morning! I'm feeling hungry. It's one of those days where I've forgotten to eat breakfast since waking up. >.>15:29
kgriffsflaper87: thanks!15:32
kgriffsflaper87: different topic15:32
kgriffsI am thinking about introducing two-phase commit when posting messages to mongodb15:33
alcabreraI'm out for a little while - headed to an appointment. See you guys a little after noon!15:33
*** alcabrera has quit IRC15:33
kgriffsthat would allow us to say that--even in the case of a nasty network failure--either the entire message batch was posted, or none were.15:34
kgriffsmy idea was to post messages with a boolean flag, e.g., "committed"15:34
kgriffsand then message queries would filter out anything where committed == False15:34
kgriffsdownside is that this requires an extra write on posting messages, and an extra filter on reading messages. The latter probably won't impact performance significantly.15:35
kgriffsflaper87: thoughts?15:35
flaper87mmhhh, a question first :)15:36
kgriffsshoot15:36
flaper87will this be part of the API or just related to the mongodb backend? As you explained, I'd assume the later15:36
kgriffsgreat question15:37
kgriffsso15:37
flaper87As you explained it*15:37
kgriffsv1.1 removes the "partial" field15:37
kgriffsbut there is still a chance for partial enqueuing if there is a sudden, prolonged network failure in the middle of sending the messages over.15:38
kgriffsor, consider that the master mongod crashes hard15:39
kgriffswe get autoreconnect15:39
kgriffsand then have to figure out what, if anything was replicated to a secondary15:39
kgriffsI guess this all assumes a few things about how mongod works15:39
kgriffsiirc, mongod writes each doc as it comes in, rather than waiting for the entire set in the request to arrive before getting to work.15:40
flaper87indeed, my concern is that I don't want to bring this concepts to the API15:40
flaper87kgriffs: that's correct15:40
flaper87mongod writes happen in a FIFO fashion15:41
flaper87if it stays within the mongodb backend boundaries then I think it's fine15:41
flaper87bringing that to the API will make implementations like amqp even harder15:41
flaper87we'll have queue flavors, though15:42
flaper87that could solve that issue15:42
flaper87but lets not get there yet15:42
flaper87that being said. I don't think reads will be a huge impact since it's an eq match on an indexed field15:42
flaper87the updates may be an issue, though.15:43
kgriffskk. we can try it and benchmark to see.15:43
flaper87I guess you're thinking to do all this before returning to the client15:43
kgriffscorrect15:43
flaper87ok, cool15:43
kgriffsotherwise, you have to do some messy hacks to figure out which ids made it (if any) so you can tell the client15:44
kgriffswrt putting this in the API15:44
kgriffsdo you mean, we should not require atomic message postings for all backends?15:44
flaper87thinking aloud. Would it be that bad to have a partial insert in case of network failure ?15:45
flaper87from a user stand point, what would that mean?15:45
maliniflaper87: depends on what response you send back15:45
kgriffssuppose i am using marconi for metering so I can bill a customer15:45
kgriffsif I get a partial insert, I *have* to know what succeeded and what failed. Otherwise the customer could get overbilled if I resend some messages.15:46
kgriffsso, i can catch socket and autoreconnect errors and retry, plus query to see which of the messages made it in15:47
kgriffsbut at some point i will have to give up for a prolonged network partition15:47
kgriffsand in that case there is no way for me to discover which messages succeeded in order to inform the client, since by definition, I can't talk to the DB.15:48
flaper87mmh, but that's different than saying partial inserts are bad.15:49
kgriffson the other hand, if I do two-phase, then i can just tell the client to assume nothing succeeded and it is safe for them to resend the entire batch.15:49
flaper87I mean, if we can tell what messages were not inserted, then we're offering some level of safety15:49
kgriffsflaper87: well, they are only bad if you can't tell the client which messages succeeded15:49
flaper87:)15:49
kgriffsexactly15:49
kgriffsbut we can't always tell15:50
kgriffsthat's what I'm concerned about15:50
flaper87because we removed the partial stuff15:50
flaper87:P15:50
flaper87jokes apart15:50
kgriffsheh15:50
flaper87FWIW, mongodb 2.6 will introduce a better error reporting (AFAIU)15:51
kgriffsi mean, we could still do partial stuff, but at some point you have to just give up if the outage lasts too long15:51
flaper87and yes, YOU ARE WELCOME, mongodb community15:51
kgriffsw00t15:51
flaper87why should we wait? What if we return back to the client as soon as the failure happens instead of retrying ?15:52
flaper87it'd be safer for the client to get back to it and let it retry than retrying ourselves from the same node15:52
kgriffswe could do that.15:52
kgriffsbut that doesn't solve the problem of not knowing what succeeded15:52
flaper87because mongodb <2.6 makes that very hard15:53
kgriffssay I get a socket error. I immediately return to the client and they have no idea which messages succeeded15:53
flaper87kgriffs: true that15:53
flaper87socket errors15:53
kgriffsnetworking sucks.15:54
* kgriffs hides15:54
flaper87kgriffs: networking sucks BIG TIME15:54
flaper87kgriffs: where are you hiding? quick quick15:54
kgriffs:D15:54
kgriffswrt AMQP15:54
kgriffsi don't believe message delivery is guaranteed, is it?15:54
flaper87at least we could hack from the same place, that would remove any network partition in our conversation15:55
flaper87:D15:55
malinifyi..we have marconi on the agenda for today's openstack-qa meeting15:55
flaper87kgriffs: there are a few ways to guarantee message delivery15:56
maliniif anybody is interested in joining its @ 1700 UTC15:56
flaper87malini: AWESOME! Glad to know that :D15:56
kgriffsmalini: kk15:57
maliniHopefully tht'll get some folks to look at our reviews seriously15:57
kgriffsflaper87: so, if we said that all backends had to have atomic commits of batch messages, would that work for AMQP?15:57
kgriffsi guess that is a little different from guaranteed delivery15:57
kgriffsperhaps we could say this in the v1.1 spec: "When using the official SQL and MongoDB drivers, Marconi will either succeed in enqueuing all messages in the submitted batch, or fail to enqueue any of them."16:00
flaper87kgriffs: that depends on whether the backend supports batch inserts or not but yes, that sounds reasonable16:01
kgriffsso, for v2.0 we could define some "optional" operations. batch inserts could be one of those.16:02
flaper87TBH, given the performance expectations we've for marconi and the underlying backends, I'd rather send 50 messages than just sending 1 batch insert16:02
flaper87Based on the reliability needs of some use cases like billing16:03
flaper87kgriffs: re for v2.0: +1. I don't think the AMQP backend will happen anytime soon. It's likely to happen for Jth16:04
flaper87I'm happy that we already discussed that, though.16:04
kgriffshmm16:05
kgriffsso, how about this16:05
kgriffserr16:05
kgriffsso, i think we have at least two options16:05
kgriffsA. do not guarantee atomic batch inserts and tell people to do one at a time if they need to be safe16:06
kgriffsB. guarantee atomic batch inserts for some drivers/queue flavors, but not for others16:06
kgriffsC. best-effort, where we tell them, if possible, which messages succeeded and which did not16:07
kgriffsordered by complexity of implementation ^^^16:08
kgriffsTBH, I like A over C since C is a PITA for client devs16:08
kgriffsSo, A or B?16:15
flaper87sorry, got distracted16:15
kgriffsI am leaning toward trying B and if it is too slow, doing A16:16
flaper87I think B is fine16:16
kgriffsB-2: guarantee atomic for everyone16:16
kgriffs(forgot that option)16:16
kgriffsand by "everyone" I mean "every backend"16:16
flaper87read my mind, I was going to add the atomic part16:16
kgriffs"every backend that supports batch posts"16:17
flaper87if a backend can't provide atomic batch, then it better don't provide batch inserts at all16:17
kgriffskewl16:17
kgriffssounds like a plan16:17
flaper87rock on16:17
kgriffswe need to some docs for driver devs!16:17
kgriffsmalini: we will need to benchmark my two-phased commit patch before allowing it to be merged16:18
*** ykaplan has quit IRC16:18
malinikgriffs: sorry I wasnt following the conversation16:19
malinido we already have the patches in place ?16:19
malinias in ready for review16:20
kgriffsmalini: no code yet; I was waiting to discuss with the team.16:21
malinikgriffs: its really easy to run the benchmark tests..But the hard part is getting it deployed in an env16:22
maliniWe shud come up with a better strategy for tht16:22
kgriffsso, I'll get to work on this. It will make fixing the autoreconnect and socket error bugs easier.16:22
kgriffsmalini: for sure16:22
kgriffsAUTOMATE ALL THE THINGS16:22
malinikgriffs: does the benchmarking need to happen in a full blown env, or is it enough to run it against devstack?16:23
kgriffssrsl. WHAT IS WRONG with gate lately? https://review.openstack.org/#/c/65521/16:25
kgriffsmalini: as long as it hits mongodb we should be fine16:25
kgriffsthis will add a little bit of time to message posting16:25
kgriffsso we want to measure the impact16:25
malinikgriffs: tht makes it a lot easier !! I'll work on adding a non-voting job to run the tsung tests against devstack16:26
maliniAdding a bp16:26
maliniwell no..I'll update an existing16:26
maliniwe have too many load-test bps already16:27
kgriffsheh16:28
kgriffscool beans16:28
*** kgriffs is now known as kgriffs_afk16:35
*** amitgandhi has quit IRC16:45
*** ayoung_ is now known as admiyo16:51
openstackgerritA change was merged to openstack/python-marconiclient: Add shard support to the client  https://review.openstack.org/6482917:06
flaper87w000000000000000000t17:10
flaper87it took way too long17:10
*** amitgandhi has joined #openstack-marconi17:16
*** amitgandhi has quit IRC17:20
*** amitgandhi has joined #openstack-marconi17:28
*** amit has joined #openstack-marconi17:30
*** amit is now known as Guest5505917:31
*** Guest55059 has quit IRC17:32
*** amitgandhi has quit IRC17:33
*** amitgandhi has joined #openstack-marconi17:33
maliniflaper87: is there anywhere I can find a list of core devs for each project?17:34
malinia wiki maybe ?17:34
*** amitgandhi has quit IRC17:41
*** chandankumar has quit IRC17:41
*** whenry has quit IRC17:52
flaper87malini: gerrit17:57
*** amitgandhi has joined #openstack-marconi17:58
flaper87https://review.openstack.org/#/admin/groups/104,members17:58
flaper87malini: https://review.openstack.org/#/admin/groups/17:58
flaper87back in a bit17:58
*** amitgandhi has quit IRC17:58
malinithanks flaper87!17:58
*** amitgandhi has joined #openstack-marconi17:59
*** alcabrera has joined #openstack-marconi17:59
maliniweird why they dont have the infra core there ..18:02
*** reed has joined #openstack-marconi18:02
*** lbragstad has left #openstack-marconi18:09
*** whenry has joined #openstack-marconi18:16
*** jamieh has quit IRC18:25
*** yassine has quit IRC18:33
openstackgerritA change was merged to openstack/marconi: test: Use srmock header_dict in preparation for Falcon 0.1.8  https://review.openstack.org/6552118:36
*** kgriffs_afk is now known as kgriffs18:50
*** openstackgerrit has quit IRC18:52
*** openstackgerrit has joined #openstack-marconi18:52
*** alcabrera is now known as alcabrera|afk19:53
*** alcabrera|afk is now known as alcabrera20:12
*** malini has left #openstack-marconi20:24
*** kgriffs is now known as kgriffs_afk20:45
*** cpallares has quit IRC21:28
*** jbernard has joined #openstack-marconi21:38
*** amitgandhi has quit IRC21:39
*** amitgandhi has joined #openstack-marconi21:51
jbernardflaper87: there is only the python-marconi package, correct? no -doc or client packages that i can see21:52
flaper87jbernard: correct21:52
flaper87jbernard: and thanks for reminding us that we've failed at documenting things21:52
flaper87:P21:53
jbernardyou're welcome ;)21:53
jbernardDB dependency - up to the user, or do we depend on a default?21:53
jbernardim guessing the latter21:53
*** amitgandhi has quit IRC22:00
*** balajiiyer has quit IRC22:01
*** amitgandhi has joined #openstack-marconi22:01
*** mpanetta has quit IRC22:07
flaper87jbernard: the default would be sqlite22:08
flaper87which is not good for production envs22:08
flaper87so, I'd expect users to change that22:08
flaper87but yeah, the default is sqlite22:08
jbernardand the daemon is to be run as the marconi user/group?22:09
*** alcabrera has quit IRC22:10
*** oz_akan_ has quit IRC22:21
*** whenry has quit IRC22:43
*** tedross has quit IRC22:47
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(mongodb): Ensure batch message posts are atomic (all or nothing)  https://review.openstack.org/6732023:11
flaper87kgriffs_afk: weren't you afk ?23:12
flaper87that was fast23:12
flaper87:D23:12
*** jcru has quit IRC23:14
*** flaper87 is now known as flaper87|afk23:19
*** admiyo has quit IRC23:26
*** amitgandhi has quit IRC23:37
« Wednesday, 2014-01-15 Index Friday, 2014-01-17 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!