Friday, 2014-09-26

« Thursday, 2014-09-25 Index Saturday, 2014-09-27 »
*** echevemaster has joined #openstack-zaqar02:02
*** jeffrey4l has joined #openstack-zaqar02:06
*** jeffrey4l_ has joined #openstack-zaqar02:17
*** jeffrey4l has quit IRC02:21
*** jeffrey4l_ is now known as jeffrey4l03:11
*** vkmc has quit IRC03:34
*** zhiyan has quit IRC03:54
*** zhiyan has joined #openstack-zaqar03:57
*** echevemaster has quit IRC04:05
*** jeffrey4l has quit IRC04:22
*** jeffrey4l has joined #openstack-zaqar04:40
openstackgerritJeffrey Zhang proposed a change to openstack/zaqar: Capture the stdout and stderr based on environment  https://review.openstack.org/12427305:18
*** fifieldt has joined #openstack-zaqar05:21
*** jeffrey4l has quit IRC05:22
*** jeffrey4l has joined #openstack-zaqar05:34
*** AAzza_afk is now known as AAzza06:00
*** AAzza is now known as AAzza_afk06:03
openstackgerritOpenStack Proposal Bot proposed a change to openstack/zaqar: Imported Translations from Transifex  https://review.openstack.org/12428506:07
*** AAzza_afk is now known as AAzza06:18
*** flaper87|afk is now known as flaper8707:06
*** boris-42 has quit IRC07:15
*** reed has quit IRC07:43
*** tmu has quit IRC07:46
*** tmu has joined #openstack-zaqar07:46
*** jeffrey4l has quit IRC07:59
*** tmu has quit IRC08:18
*** tmu has joined #openstack-zaqar08:18
*** sphoorti has joined #openstack-zaqar08:58
zhiyanhi folks, seems I found a few defects with my zaqar try, could give me some inputs?09:24
zhiyanflaper87: ^^09:24
flaper87zhiyan: shoot09:26
flaper87what's up?09:26
zhiyanflaper87: there are two minor things.09:26
zhiyanfirst is when i run server i got "No handlers could be found for logger "zaqar.common.cli""09:26
flaper87zhiyan: ah yeah, probably failing to load the conf defaults or something09:27
zhiyanthen i saw https://bugs.launchpad.net/zaqar/+bug/1201562 , a fixed defect. but i'm interested seems current codebase doesn't include it09:27
flaper87zhiyan: did you run it with zaqar-server -d -v ?09:27
zhiyan-v only09:27
zhiyanflaper87: so not sure if we miss that fix in current codebase for a reason?09:28
flaper87zhiyan: mmh, it's weird the fix is not in the codebase :/09:28
zhiyanyep09:28
flaper87probably a regression09:28
zhiyanthen i re-add that print line to _fail(), and it doesn't work too...but with a minor change to print(...), then I got thing to be correct.09:29
zhiyanflaper87: i think so frankly. so probably we need to put it back. i can help.09:29
zhiyanflaper87: so do a double-check with you first.09:30
flaper87zhiyan: +1 awesome, thanks09:30
zhiyanflaper87: nice :)09:30
zhiyanflaper87: the second one is about requirement09:30
zhiyani run the server with zaqar-server -v, then failed by pymongo lib requirement...but i can't found it in requirement.txt. so see we miss it?09:31
zhiyans/see/seems09:31
zhiyanflaper87: or some reason i missed team dislike put pymongo req in there.09:32
flaper87zhiyan: yeah, it's part of the test-requirements09:33
flaper87we don't want to force anyone to install things they don't need09:33
flaper87for example, if you just want to run zaqar+redis you don't need pymongo09:33
zhiyanhum..ok makesense, currently i just used mongodb as storage.09:33
zhiyanyep09:33
zhiyansensible09:33
zhiyanflaper87: that's all for me, thanks!09:33
flaper87any time09:34
zhiyan:)09:34
openstackgerritZhi Yan Liu proposed a change to openstack/zaqar: Fix regression: No handlers could be found for logger when start  https://review.openstack.org/12435009:54
flaper87zhiyan: btw, all tests are failing because of this: https://review.openstack.org/#/c/123252/09:55
flaper87once that patch lands, we'll recheck all patches09:55
zhiyanflaper87: ok, good to know09:56
zhiyanflaper87: seems the change just for 'queue' -> 'messag' ?09:57
flaper87zhiyan: yup09:57
zhiyanah, ok09:57
zhiyanfine, more common for the zaqar service. (ok, back to barrier）09:59
*** malini has joined #openstack-zaqar10:33
*** AAzza is now known as AAzza_afk10:36
*** sphoorti has quit IRC10:43
*** AAzza_afk is now known as AAzza10:47
*** AAzza is now known as AAzza_afk10:53
*** malini has quit IRC11:01
*** malini has joined #openstack-zaqar11:01
*** AAzza_afk is now known as AAzza11:05
*** AAzza is now known as AAzza_afk11:26
*** prashanthr_ has joined #openstack-zaqar11:44
*** prashanthr_ has quit IRC12:06
*** malini has quit IRC12:09
*** AAzza_afk is now known as AAzza12:11
*** sgotliv has joined #openstack-zaqar12:17
*** AAzza is now known as AAzza_afk12:18
*** vkmc has joined #openstack-zaqar12:20
*** vkmc has quit IRC12:20
*** vkmc has joined #openstack-zaqar12:20
vkmcgood morningg12:22
*** AAzza_afk is now known as AAzza12:30
flaper87vkmc: goooooooooooooooood morning12:34
vkmcflaper87, hiiiiiiii12:37
*** prashanthr_ has joined #openstack-zaqar12:45
vkmcstill waiting for tempest :/12:55
*** jeffrey4l has joined #openstack-zaqar12:56
flaper87:(13:00
* flaper87 finally got this fucking mongodb sharded environment setup13:04
flaper87now, to test zaqar13:05
*** sriram1 has joined #openstack-zaqar13:06
vkmccoool13:06
*** sriram1 is now known as sriram13:06
* flaper87 has no idea what key he should shard on13:08
flaper87vkmc: want to do something cool?13:08
flaper87vkmc: what are you working on now? (If I may know)13:08
*** jchai has joined #openstack-zaqar13:09
sriramdid you guys have a look at the bash bug?13:12
flaper87sriram: bash bug?13:12
* flaper87 lost13:12
flaper87or wrong channel ?13:12
flaper87:D13:12
sriramflaper87: heh13:12
flaper87sriram: dude, zaqar is written in python, not bash13:12
sriramflaper87: of course :)13:13
sriramwas interesting and pretty scary actually.13:14
vkmcflaper87, more Rally and the func tests13:16
vkmcflaper87, what is the cool thing you mentioned? :)13:16
flaper87vkmc: help me figure out a good shard key to use13:19
flaper87:D13:19
*** sgotliv has quit IRC13:20
vkmcflaper87, hmm let's think13:21
flaper87vkmc: how familiar are you with mongodb shard keys?13:21
flaper87vkmc: http://docs.mongodb.org/manual/core/sharding-introduction/13:22
flaper87you may want to go through that13:22
vkmcflaper87, fairly familiar13:22
vkmccool13:22
vkmcand this http://docs.mongodb.org/manual/tutorial/choose-a-shard-key/13:22
vkmcso its advised to use a high cardinality key13:25
vkmcwith high degree of randomness13:25
vkmcwhat are the options?13:26
*** mpanetta has joined #openstack-zaqar13:27
flaper87right but not to random, otherwise we're fucked13:27
flaper87vkmc: https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/messages.py#L66-L10413:27
flaper87I think the marker index is probably our best choice13:28
flaper87although we don't always have a marker13:28
vkmcI just cited this http://docs.mongodb.org/manual/tutorial/choose-a-shard-key/#create-a-shard-key-that-has-high-degree-of-randomness13:28
flaper87LOL13:29
flaper87mmh, actually, if we make the marker index something like: PROJECT_SCOPE -> hashed and marker -> 113:29
flaper87I think we can get something good13:29
flaper87mmh wait13:29
*** malini has joined #openstack-zaqar13:30
vkmcwouldn't it make sense to use the queue is as the key?13:32
flaper87mmhh, that woul keep all messages for 1 queue in a single shard13:33
vkmcso.. by using the marker you would allow a queue to be distributed in one or several shards13:34
flaper87if we use p_q+marker it would start from shardX (balancing on queue) and then it'll add messages sequentialy13:35
flaper87right, the problem with the marker, though, is that it'd end up writing *always* on the last shard13:35
flaper87since it's progressive13:35
vkmcyea13:35
flaper87which will make writes painful13:35
vkmchmm13:36
flaper87these are the fields we have:13:39
flaper87https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/messages.py#L118-L12813:39
vkmcyes, I'm checking those13:39
vkmcwhat if..13:42
flaper87vkmc: .. are not good for indexes13:42
flaper87:P13:42
vkmcI'm measuring the level of stupidity of my idea before saying it :D13:43
flaper87hahah13:43
vkmcI was thinking on using ttl, and then taking advantage of that to make some kind of priority selection13:44
vkmcbut sharding is time/resources consuming and that won't be useful13:45
flaper87the thing about ttl is that I don't think it'll be changed much13:45
flaper87as in, most use-cases will stick to the default ttl13:45
vkmc-1 randomness13:45
vkmcok13:45
vkmcalso, in the ttl idea, starving may ocurr and that won't be a good feature haha13:46
vkmcfeature as in bug13:46
flaper87hahaha13:46
flaper87I'm afraid we won't be able to optimize for both cases13:47
flaper87read-heavy and write-heavy13:47
flaper87and I'm leaning towards optimizing for writes13:47
vkmccan we apply some operations before doing the sharding?13:47
flaper87vkmc: like what?13:48
vkmcI don't think it's possible but maybe13:48
vkmcyou said that the problem with p_q+marker is that it would end up writting always on the last shard13:48
flaper87right, that's my assumption13:49
vkmcif you could randomize it a little13:49
flaper87I'm going to test that13:49
vkmcyou may mitigate that issue13:49
vkmcok, let me know how that works13:50
vkmcnow I'm curious13:50
flaper87vkmc: I should've installed this sharded environment somewhere public13:51
flaper87:/13:51
flaper87so, that idea may work13:51
flaper87not sure, lets see13:51
vkmcflaper87, no problem13:51
flaper87vkmc: another idea would be to swap marker and p_q13:53
flaper87I really have no idea how compound shard keys are used13:53
flaper87:P13:53
* flaper87 goes and reads the fucking manual13:53
vkmchaha let's figure it out13:53
*** sgotliv has joined #openstack-zaqar13:53
*** AAzza has quit IRC14:20
*** AAzza_afk has joined #openstack-zaqar14:21
*** AAzza_afk is now known as AAzza14:21
*** AAzza_afk has joined #openstack-zaqar14:22
*** AAzza has quit IRC14:25
*** AAzza_afk is now known as AAzza14:25
*** sgotliv has quit IRC14:45
*** sriram has quit IRC14:48
*** sriram has joined #openstack-zaqar14:48
flaper87unproductive friday14:50
flaper87vkmc: news on tempest?14:50
vkmcflaper87, still in the queue14:50
flaper87damnit14:51
vkmcflaper87, how are the shard keys treating you?14:51
flaper87I just set up MMS14:51
flaper87to make monitoring easier14:51
flaper87but, using p_q+k sends all writes to a single shard14:51
flaper87as expected14:51
vkmchmm14:52
vkmc>.> c'mon tempest15:07
*** prashanthr_ has quit IRC15:08
flaper87vkmc: https://www.dropbox.com/s/w2ufh784co1m5pf/insert.jpg?dl=015:14
vkmcflaper87, what is representing the y axis?15:16
flaper87sorry got distracted15:16
flaper87all graphs sow the ops in each shard15:17
flaper87there are 4 shards15:17
vkmcit's ok, I live distracted15:17
vkmccool15:17
flaper87the graph at the left-top corner15:17
flaper87is the last shard15:17
flaper87shard number 415:17
flaper87and you can see that all writes are going there15:17
flaper87(the kinda yellow line)15:18
vkmcffh as expected15:18
*** kgriffs|afk is now known as kgriffs15:18
flaper87vkmc: now I'll try swaping the keys15:20
vkmcflaper87, what are the other operations? reads and?15:21
vkmclines blue, white, green15:21
vkmctum tum tum15:23
*** sgotliv has joined #openstack-zaqar15:25
*** jeffrey4l has quit IRC15:27
* flaper87 back15:30
flaper87the other operations are queries, updates15:30
flaper87basically the balancer moving chunks15:31
vkmc+115:35
flaper87sooooooooooooooooooooooooooooooo15:41
flaper87it looks like I was right15:41
vkmcswaping the keys.. same happens?15:42
flaper87not the same, writes are distributed15:42
flaper87it writes a bunch on the first shard, then the second, then the third, then the fourth15:43
flaper87writes are not random but it, at least, does not hit just 1 shard15:43
flaper87vkmc: refresh15:43
flaper87vkmc: the link I sent before15:43
flaper87and it completed the task way faster than before15:46
vkmcflaper87, wow that yellow line15:46
vkmcthats great15:46
flaper87I'll test it with 2 producers15:47
vkmcok :)15:47
flaper87no idea why I'm not using zaqar-bench15:47
flaper87probably because I've no idea how to use it15:47
flaper87:P15:47
vkmchaha15:47
flaper87but I think it's still not good15:48
flaper87I mean, we are not writing to the same shard which is good for different queues15:48
flaper87but for this same queue we would be writing to the same shard15:48
flaper87As in, we will switch shards sequentially15:49
vkmcthen... reads will be harder15:49
flaper87well, writes will be harder for queue A15:49
flaper87since all producers would be writing to the same shard15:49
* flaper87 is not happy15:52
flaper87:(15:52
vkmcno :(15:53
vkmcsomething to make you happy15:54
vkmctempest patch already passed the gates, 15 min left to get merged15:55
* vkmc doesn't lie when she say she stares at zuul sometimes15:55
flaper87LOOOL15:56
flaper87that's great news15:56
flaper87merged15:59
flaper87finally15:59
flaper87lets recheck all the things15:59
vkmc:')15:59
vkmcthat's the best moment of my week16:00
vkmcs/that/this16:00
vkmclol16:00
flaper87LOL16:01
flaper87ok, I helpped you with some16:01
vkmcflaper87, thx16:02
vkmcfloooood16:03
*** jchai is now known as jchai_afk16:03
*** reed has joined #openstack-zaqar16:04
vkmcI'll keep track of them16:08
flaper87while this new shard key seems to work better, I don't think it's optimal16:09
flaper87:(16:09
vkmcwhat is your expectation?16:10
flaper87if only I knew that16:11
flaper87jokes apart16:11
vkmc:)16:11
vkmcI wanna help16:11
flaper87I really think we won't be able to optimize for both casses16:11
flaper87cases16:11
flaper87whatever16:11
flaper87so, this kind of writes, will hurt heavy-write queues but benefit reads16:12
*** sgotliv has quit IRC16:12
vkmccan we have a different key depending on the use case?16:12
flaper87nope16:12
flaper87:/16:12
vkmclike ... a configuration value in zaqar.conf16:13
vkmcdang16:13
*** jchai_afk is now known as jchai16:21
flaper87mmh, actually, it may be a good shard key after all16:23
flaper87I was testing it with 1 queue16:23
flaper87which made the marker sequence standard16:23
flaper87if we test it with more queues, the range keeps chaning and it adds some randomness16:23
flaper87I wouldn't call it randomness but my brain is tired and I dunno how to call it16:24
*** alcabrera|afk is now known as alcabrera16:25
vkmc:)16:36
vkmcnon determinism16:40
flaper87are you describing my brain?16:41
vkmcI couldn't possible ever do that with finite time and space flaper8716:41
flaper87it actually sounded pretty accurate... :P16:42
flaper87moaaaaaaaaaaaaaar tests, moaaaaaaaaaaaaaaaaar16:42
vkmchahaha16:42
vkmcmy brain is tired today too16:44
vkmcI woke up with a headache and have been trying to ignore it since then16:45
vkmcits winning >.>16:45
flaper87oh mmh, then I'll give you a brain teaser, I'm pretty sure that will help16:51
flaper87huhauhauhauahuahu16:51
vkmclol16:56
flaper87vkmc: refresh :)16:58
flaper87it's not perfect but it could be better16:58
vkmcflaper87, looks better17:02
vkmcits marker and p_q?17:02
vkmcwith more producers?17:02
flaper87yup17:02
vkmc+117:02
flaper87on different queues17:02
vkmcsounds a more realistic scenario17:03
vkmcbrb17:03
*** kgriffs is now known as kgriffs|afk17:08
*** alcabrera is now known as alcabrera|afk17:19
*** kgriffs|afk is now known as kgriffs17:32
*** kgriffs is now known as kgriffs|afk17:35
*** kgriffs|afk is now known as kgriffs17:40
*** kgriffs is now known as kgriffs|afk18:03
*** kgriffs|afk is now known as kgriffs18:15
*** kgriffs is now known as kgriffs|afk18:17
*** malini has quit IRC18:57
*** malini has joined #openstack-zaqar18:59
*** sriram2 has joined #openstack-zaqar19:00
*** sriram has quit IRC19:01
*** sriram2 is now known as sriram19:02
*** jchai is now known as jchai_afk19:10
*** sgotliv has joined #openstack-zaqar19:20
*** sgotliv has quit IRC19:23
*** sgotliv has joined #openstack-zaqar19:24
*** boris-42 has joined #openstack-zaqar19:27
*** alcabrera|afk is now known as alcabrera19:36
*** malini has quit IRC20:00
*** jchai_afk is now known as jchai20:03
*** alcabrera is now known as alcabrera|afk20:33
vkmcflaper87, still around?20:40
*** sriram has quit IRC20:51
openstackgerritA change was merged to openstack/zaqar: Replace misleading max_message_size option name  https://review.openstack.org/12322321:10
*** jchai has quit IRC21:10
*** sgotliv has quit IRC21:39
*** alcabrera|afk is now known as alcabrera21:44
*** AAzza is now known as AAzza_afk21:59
*** ametts has quit IRC22:08
*** mpanetta has quit IRC22:33
boris-42vkmc zoo could you make some benchmark for zaqar?22:52
boris-42vkmc on top of cleanup22:52
boris-42vkmc at least something, just to setup gates=)22:52
boris-42vkmc if you have some questions just ping me22:52
vkmcboris-42, sure thing22:53
boris-42vkmc thanks22:53
vkmcboris-42, sorry I didn't upload it sooner, I had some other stuff in my plate22:53
boris-42vkmc no worries22:54
*** AAzzal has joined #openstack-zaqar22:58
*** AAzzal is now known as AAzza22:59
*** AAzza_afk has quit IRC22:59
*** echevemaster has joined #openstack-zaqar23:53
« Thursday, 2014-09-25 Index Saturday, 2014-09-27 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!