| *** shakamunyi has joined #openstack-marconi | 00:23 | |
| *** shakamunyi has quit IRC | 00:48 | |
| *** shakamunyi has joined #openstack-marconi | 01:13 | |
| *** saikrishna has joined #openstack-marconi | 01:13 | |
| *** saikrishna has quit IRC | 01:53 | |
| *** flwang has quit IRC | 01:56 | |
| *** renlt has joined #openstack-marconi | 02:15 | |
| *** flwang has joined #openstack-marconi | 02:36 | |
| *** shakamunyi has quit IRC | 02:57 | |
| *** cpallares has quit IRC | 03:18 | |
| *** alcabrera|afk has quit IRC | 04:08 | |
| *** alcabrera|afk has joined #openstack-marconi | 04:20 | |
| *** haomaiwang has joined #openstack-marconi | 04:31 | |
| *** haomaiwang has quit IRC | 04:32 | |
| *** haomaiwang has joined #openstack-marconi | 04:33 | |
| *** flwang has quit IRC | 04:38 | |
| *** flwang has joined #openstack-marconi | 05:12 | |
| *** prashanthr_ has joined #openstack-marconi | 05:22 | |
| *** haomai___ has joined #openstack-marconi | 05:22 | |
| *** haomaiwang has quit IRC | 05:22 | |
| *** haomaiwang has joined #openstack-marconi | 05:26 | |
| *** chandan_kumar has joined #openstack-marconi | 05:26 | |
| *** cpallares has joined #openstack-marconi | 05:27 | |
| *** haomai___ has quit IRC | 05:29 | |
| *** cpallares has quit IRC | 05:32 | |
| *** flwang has quit IRC | 05:49 | |
| *** flwang has joined #openstack-marconi | 05:51 | |
| prashanthr_ | flwang: Hi :) | 06:18 |
|---|---|---|
| *** flwang has quit IRC | 06:22 | |
| *** flwang has joined #openstack-marconi | 06:26 | |
| flwang | prashanthr_: hi | 06:40 |
| prashanthr_ | Thanks a lot for all the help for Gsoc :). Was able to make through :) | 06:41 |
| flwang | my pleasure | 06:42 |
| *** flwang has quit IRC | 06:46 | |
| *** flwang has joined #openstack-marconi | 06:58 | |
| *** flwang has quit IRC | 07:09 | |
| *** mkoderer has joined #openstack-marconi | 07:09 | |
| *** flwang has joined #openstack-marconi | 07:10 | |
| *** Manishanker has joined #openstack-marconi | 07:14 | |
| *** jamie_h has joined #openstack-marconi | 07:24 | |
| *** jamie_h has quit IRC | 07:25 | |
| *** flaper87|afk is now known as flaper87 | 07:29 | |
| flwang | flaper87: hey | 07:33 |
| flaper87 | yoy yo | 07:33 |
| flaper87 | how are you doing? | 07:33 |
| flwang | flaper87: good | 07:34 |
| flwang | flaper87: not sure if you know that, my travel is cancelled :( | 07:35 |
| flaper87 | flwang: damnit :( | 07:36 |
| flaper87 | so sad to hear that | 07:36 |
| flaper87 | :( | 07:36 |
| *** flwang has quit IRC | 07:42 | |
| *** prashanthr_ has quit IRC | 07:46 | |
| *** openstackgerrit has quit IRC | 07:53 | |
| *** ykaplan has joined #openstack-marconi | 07:54 | |
| *** prashanthr_ has joined #openstack-marconi | 08:00 | |
| *** flwang has joined #openstack-marconi | 08:19 | |
| flwang | flaper87: sorry, I was disconnected | 08:19 |
| flaper87 | flwang: no worries | 08:32 |
| flaper87 | I said, It's sad that you won't be there :( | 08:32 |
| flwang | flaper87: yep | 08:32 |
| flwang | hopefully my next summit travel can be sponsored :) | 08:33 |
| *** prashanthr_ has quit IRC | 08:57 | |
| *** haomaiwang has quit IRC | 09:25 | |
| *** flwang has quit IRC | 09:58 | |
| *** renlt has quit IRC | 10:43 | |
| *** vkmc has joined #openstack-marconi | 10:59 | |
| *** vkmc has quit IRC | 10:59 | |
| *** vkmc has joined #openstack-marconi | 10:59 | |
| *** haomaiwa_ has joined #openstack-marconi | 11:21 | |
| *** ykaplan has quit IRC | 11:31 | |
| *** ykaplan has joined #openstack-marconi | 11:59 | |
| *** sriram has joined #openstack-marconi | 12:41 | |
| *** flaper87 has quit IRC | 12:52 | |
| *** flaper87|afk has joined #openstack-marconi | 12:56 | |
| *** flaper87|afk is now known as flaper87 | 12:56 | |
| *** flaper87 has quit IRC | 12:56 | |
| *** flaper87 has joined #openstack-marconi | 12:56 | |
| *** ChanServ sets mode: +o flaper87 | 12:56 | |
| *** amitgandhi has joined #openstack-marconi | 13:00 | |
| *** amitgandhi has quit IRC | 13:00 | |
| *** mwagner_lap has quit IRC | 13:00 | |
| *** amitgandhi has joined #openstack-marconi | 13:03 | |
| *** jchai has joined #openstack-marconi | 13:04 | |
| *** amitgandhi has quit IRC | 13:09 | |
| *** amitgandhi has joined #openstack-marconi | 13:09 | |
| *** flwang has joined #openstack-marconi | 13:27 | |
| *** alcabrera|afk is now known as alcabrera | 13:28 | |
| alcabrera | Good morning! :) | 13:29 |
| flaper87 | alcabrera: goooooood morning | 13:31 |
| alcabrera | flaper87: heya! How are you? :) | 13:31 |
| flaper87 | alcabrera: I'm doing great, you? How was your weekend? | 13:32 |
| alcabrera | It was wonderful, flaper87. I hung out with Jess a bunch, did a little reading, writing, and cooking! | 13:32 |
| alcabrera | I made a 32-egg egg salad and a whole mega-bowl of cole slaw. Out fridge is happy. <3 | 13:32 |
| flaper87 | wow, sounds like you had a very productive and fun weekend! :D | 13:33 |
| flaper87 | glad to hear that | 13:33 |
| alcabrera | :) | 13:34 |
| alcabrera | how about you, flaper87? | 13:34 |
| flaper87 | I, and I know this is hard to believe, spent the weekend away from my laptop and out in the open. | 13:35 |
| flaper87 | hung out w/ GF and friends at the lake, had great food, relaxed etc | 13:35 |
| alcabrera | yeeahhh! | 13:36 |
| alcabrera | That sounds great! :D | 13:36 |
| alcabrera | a little time away from computer is good. <3 | 13:36 |
| *** openstackgerrit has joined #openstack-marconi | 13:36 | |
| flaper87 | yeah, definitely! | 13:37 |
| flaper87 | Happy Star Wars day, btw! (A bit late, it was yday) | 13:37 |
| alcabrera | thanks! May the 4th be with you, flaper87. ;) | 13:37 |
| alcabrera | I had a funny moment on Sunday -- related to 'away from computer' | 13:38 |
| alcabrera | I set out to study around 5pm | 13:38 |
| alcabrera | an hour maybe | 13:38 |
| alcabrera | aaanddd | 13:38 |
| alcabrera | it was 9:30pm when I looked at the book again, and thought, "yeah, I'd rather just keeping talking to my friend and writing" | 13:38 |
| alcabrera | spent the whole time away from computing. :P | 13:39 |
| alcabrera | honestly surprised myself a bit | 13:39 |
| *** mpanetta has joined #openstack-marconi | 13:41 | |
| flaper87 | hehehe, yeah! Sometimes it's good, actually, I know, deep down in my brain, that it's best to just stay away from it. I've become an expert on the art of ignoring my own brain | 13:41 |
| *** mpanetta has quit IRC | 13:41 | |
| *** malini_afk is now known as malini | 13:42 | |
| *** mpanetta has joined #openstack-marconi | 13:42 | |
| flaper87 | with all that said, the next 3 months will be a no-stop-geek-work period so | 13:42 |
| flaper87 | (at least for me) | 13:42 |
| alcabrera | busy, busy flaper87! | 13:43 |
| *** rektide_ is now known as rektide | 13:44 | |
| *** Obulpathi has joined #openstack-marconi | 13:50 | |
| *** Obulpathi has quit IRC | 13:50 | |
| *** Obulpathi has joined #openstack-marconi | 13:50 | |
| *** openstackgerrit has quit IRC | 13:51 | |
| *** cpallares has joined #openstack-marconi | 13:53 | |
| *** cpallares has left #openstack-marconi | 13:53 | |
| *** mpanetta has quit IRC | 13:54 | |
| *** mpanetta_ has joined #openstack-marconi | 13:54 | |
| alcabrera | Out for a bit. Catch y'all later! | 14:00 |
| *** alcabrera is now known as alcabrera|afk | 14:00 | |
| malini | jchai: ping | 14:02 |
| jchai | malini: hello | 14:02 |
| malini | jchai: good morning!! | 14:02 |
| *** whenry has joined #openstack-marconi | 14:03 | |
| malini | jchai: I see we still dont have much luck with xenserver https://review.openstack.org/#/c/87762/ :( | 14:03 |
| jchai | No. The job doesn't seem to start now. | 14:03 |
| malini | jchai: can you try posting in #openstack-qa | 14:04 |
| malini | somebody there might know | 14:04 |
| jchai | OK. Let me request it once more and if it doesn't run, I'll drop there | 14:04 |
| malini | thanks jchai! | 14:05 |
| *** ykaplan has quit IRC | 14:08 | |
| flaper87 | malini: good morning :) | 14:11 |
| *** Manishanker has quit IRC | 14:14 | |
| sriram | good morning everyone! | 14:16 |
| sriram | \o o/ | 14:16 |
| *** rwsu has joined #openstack-marconi | 14:18 | |
| *** mwagner_lap has joined #openstack-marconi | 14:18 | |
| malini | hello flaper87!! | 14:18 |
| flaper87 | sriram: gooooood morning | 14:20 |
| *** cpallares has joined #openstack-marconi | 14:20 | |
| sriram | flaper87: good morning :) | 14:22 |
| malini | flaper87: all set for the trip? | 14:23 |
| flaper87 | I have a ticket and a valid passport, that should be enough, right? | 14:24 |
| flaper87 | :P | 14:24 |
| *** rossk has joined #openstack-marconi | 14:26 | |
| *** prashanthr_ has joined #openstack-marconi | 14:27 | |
| malini | flaper87: yes :D | 14:29 |
| malini | & laundry the night before the trip, might be good as well :D | 14:29 |
| malini | I am excited to meet everybody | 14:30 |
| *** balajiiyer has joined #openstack-marconi | 14:41 | |
| *** rossk has quit IRC | 14:41 | |
| cpallares | o/ malini! | 14:45 |
| malini | hello cpallares!!! | 14:46 |
| cpallares | malini: Did we decide on a restaurant? | 14:46 |
| cpallares | malini: how are you doing? :D | 14:46 |
| malini | cpallares: good..how are you? | 14:46 |
| malini | cpallares: We are going to meet at the CNN center,& grab something from their food court | 14:46 |
| cpallares | malini: I'm doing good too :) | 14:46 |
| malini | we might end up going to the centennial park right across the street | 14:47 |
| malini | cpallares: kgriffs|afk will be sending an email on tht | 14:47 |
| malini | flaper87, kgriffs|afk: We need to add our design session etherpads here https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#Marconi | 14:48 |
| *** kgriffs|afk is now known as kgriffs | 14:49 | |
| kgriffs | malini: thanks for the reminder; I'll set up some pads. Everyone feel free to seed them before your session | 14:51 |
| flaper87 | malini: lol at laundry... /me usually prepares his bag 20 mins before leaving | 14:51 |
| kgriffs | I'll send out a notice to the mailing list | 14:51 |
| malini | thanks kgriffs! | 14:51 |
| kgriffs | wrt the dinner, watch for an evite | 14:52 |
| flaper87 | malini: When is the CNN thing? I know we w.... kgriffs thanks :D | 14:52 |
| malini | :D | 14:52 |
| kgriffs | malini, mpanetta_: I had some questions/suggestions about the benchmark you did | 14:53 |
| kgriffs | got a minute to discuss? | 14:53 |
| malini | kgriffs: sure | 14:53 |
| kgriffs | ok, so first off, is journaling enabled (would be in the connection string for MongoDB iirc) | 14:54 |
| kgriffs | flaper87: related question; what is the value of enabling journaling if you are already replicating to multiple secondaries? | 14:55 |
| flaper87 | kgriffs: if the master dies before replication happens and you don't have journaling enabled then something may go wrong | 14:55 |
| flaper87 | however, we can also consider journal as enough quorum to get a response back from mongo | 14:56 |
| flaper87 | that is, as soon as the record is journaled, we can tell mongo to consider the write operation complete | 14:56 |
| flaper87 | that's all done with the write concern and putting it in the mongodb URI should be enough. I'm not 100% sure about the URI, I'll check | 14:57 |
| flaper87 | kgriffs: I mentioned that we should have a performance hackthon during the summit | 14:57 |
| flaper87 | thoughts? | 14:57 |
| flaper87 | Also, I consider all these journal, replication and scaling bits very deployment-dependent | 14:58 |
| flaper87 | (the ones related to the storage driver) | 14:58 |
| flaper87 | not saying we shouldn't care. What I'm saying is that we should find a configuration that we can tag as the recommended one | 14:58 |
| flaper87 | This setup should take all those bits into consideration but it should also be aware that there are some trade-offs that can't be avoided | 14:59 |
| kgriffs | flaper87: +1 for both performance hackathon and for coming up with deployment recipes for our docs | 14:59 |
| flaper87 | and then, around that recommended solution we can have things like: "If you disable journaling you get X and loose Y" | 14:59 |
| kgriffs | flaper87: if we have write concern as "majority" then write's arent' ack'd until they have been replicated, correct? If so, I feel like journaling is less important. | 15:00 |
| flaper87 | kgriffs: it depends on the number of nodes you use in the write concern | 15:00 |
| kgriffs | unless you are concerned about catastrophic failure in the DC | 15:00 |
| * flaper87 gets the docs link | 15:00 | |
| flaper87 | I would prefer having journal + 1 replica as opposed to all replicas and no journal | 15:01 |
| kgriffs | flaper87: well, consider your have 1 primary and two secondaries. | 15:01 |
| kgriffs | then majority means primary + 1 secondary | 15:01 |
| flaper87 | not necessarily primary, note that journal != primary | 15:01 |
| *** Obulpathi has quit IRC | 15:02 | |
| flaper87 | journal means the record is journaled but it may not be present in the db | 15:02 |
| flaper87 | ... yet. | 15:02 |
| * flaper87 gets the link for real this tiem | 15:02 | |
| flaper87 | http://docs.mongodb.org/manual/core/write-concern/ | 15:02 |
| kgriffs | flaper87: ok | 15:02 |
| flaper87 | (that's for 2.6) | 15:02 |
| kgriffs | ok, just to be clear | 15:03 |
| kgriffs | there is an option in the uri for enabling journaling | 15:03 |
| kgriffs | and once that is enabled, you can also add that in your write concern | 15:03 |
| kgriffs | correct? | 15:03 |
| flaper87 | (the link is for 2.6, the rest is valid for other versions >= journal_was_added) | 15:04 |
| flaper87 | journaling has to be enabled server side, what you can do in the URI is use journal as write concern | 15:04 |
| kgriffs | oh | 15:05 |
| kgriffs | oic | 15:05 |
| kgriffs | http://docs.mongodb.org/manual/reference/connection-string/ | 15:05 |
| kgriffs | so, uri.journal is part of the write concern, and enabling journaling is what you do in the mongo config | 15:05 |
| kgriffs | flaper87: correct? | 15:05 |
| kgriffs | according to core/write-concern: "This write concern ensures that MongoDB can recover the data following a shutdown or power interruption." | 15:06 |
| kgriffs | if you have redundant power supplies on the rack as well as backup generator power for your DC, and your secondaries are in different availability zones within the DC, I see very little value to journaling | 15:06 |
| kgriffs | well, at least for messages | 15:07 |
| kgriffs | if this were financial transactions for ATM machines, I would be more concerned. :p | 15:07 |
| * kgriffs is reading this now: http://docs.mongodb.org/manual/core/journaling/ | 15:08 | |
| kgriffs | mpanetta_: ping | 15:08 |
| flaper87 | kgriffs: http://docs.mongodb.org/manual/core/write-concern/ | 15:08 |
| flaper87 | damnit | 15:08 |
| flaper87 | kgriffs: http://docs.mongodb.org/manual/reference/connection-string/#write-concern-options | 15:08 |
| flaper87 | kgriffs: you found it first | 15:08 |
| flaper87 | T_T | 15:09 |
| flaper87 | kgriffs: that's correct | 15:09 |
| kgriffs | flaper87: it was my future self. Being a transdimensional being has it's advantages. | 15:09 |
| flaper87 | :P | 15:09 |
| flaper87 | I still think there's some value in having journal+1 replica | 15:10 |
| flaper87 | well, wait | 15:10 |
| flaper87 | actually, now I'm thinking | 15:10 |
| kgriffs | flaper87: you should watch Noein. It will blow your mind. | 15:11 |
| *** jchai is now known as jchai_afk | 15:11 | |
| kgriffs | flaper87: i'm not saying journaling isn't useful, but that it depends on your deployment environment, plus your tolerance for loosing messages | 15:12 |
| flaper87 | from a queuing perspective, there's probably little value in journaling since if something bad happens to the master, it'll likely not be up soon-enough for the journal to be useful | 15:12 |
| flaper87 | that is, probably messages will be consumed by the time the master node comes back up | 15:12 |
| flaper87 | but again, this is very dependent | 15:12 |
| kgriffs | yeah | 15:12 |
| mpanetta_ | kgriffs: pong | 15:13 |
| kgriffs | so, we should give some smart guidance in the docs on when you might want to enable journaling, and when you may get away not doing so. | 15:13 |
| kgriffs | mpanetta_: hey, we were wondering whether journaling is enabled on the benchmark cluster for MongoDB. Can you share the mongo config file? | 15:13 |
| kgriffs | also, can you share the connection string? | 15:13 |
| mpanetta_ | The cluster is firewalled, you won't be able to connect | 15:14 |
| flaper87 | However, replicating to secondaries may add some latency (depending on the network) | 15:14 |
| kgriffs | flaper87: So, someone like ObjectRocket can enable journaling almost for free. But if you have slower disk I/O, journaling can be pretty costly. | 15:14 |
| flaper87 | mpanetta_: how dare you firewall the server from us? | 15:14 |
| mpanetta_ | I don't trust robots :P | 15:14 |
| *** mpanetta_ is now known as mpanetta | 15:14 | |
| flaper87 | mpanetta: damnit, fair enough | 15:15 |
| kgriffs | mpanetta_: I don't need to connect, just want to see the marconi and mongo conf files | 15:15 |
| mpanetta | Ok | 15:15 |
| kgriffs | mpanetta: hey, I'm not a robot! | 15:15 |
| kgriffs | well.. not all of me | 15:15 |
| *** jchai_afk is now known as jchai | 15:15 | |
| flaper87 | kgriffs: re ObjRocket, yes! That's a fair statement | 15:15 |
| mpanetta | Are you the 6 million dollar man? | 15:15 |
| flaper87 | kgriffs: if you are, throw 1 this way | 15:16 |
| flaper87 | and thanks | 15:16 |
| flaper87 | this = my | 15:16 |
| flaper87 | (just in case) | 15:16 |
| kgriffs | mpanetta: I'm not at liberty to say what I am, for your safety as well as mine. | 15:16 |
| mpanetta | https://gist.github.com/bentwire/2c7dd823910bcc8a8509 <--- mongodb.conf template | 15:16 |
| flaper87 | mmhhh, how many replicas? | 15:17 |
| flaper87 | mpanetta: what's the mongo URI ? | 15:17 |
| flaper87 | and why is your username on GH bentwire ? | 15:18 |
| * flaper87 is now confused | 15:18 | |
| mpanetta | flaper87: Because That is the account I created so long ago ;) | 15:18 |
| mpanetta | 3 | 15:18 |
| mpanetta | Th RS is 3 | 15:18 |
| kgriffs | mpanetta, flaper87, malini: OK, so it looks like journaling is disabled on the benchmark cluster | 15:18 |
| mpanetta | primary and 2 secondaries | 15:18 |
| flaper87 | mpanetta: are read from slaves enabled ? | 15:18 |
| sriram | so for shard creation, this was the URI : mongodb://mng3a-cqt-ord,mng1a-cqt-ord,mng2a-cqt-ord:27017/?replicaSet=test-rs1&w=2&readPreference=secondaryPreferred | 15:19 |
| kgriffs | hmm | 15:19 |
| flaper87 | holy crap, mmmh... what have we done? | 15:19 |
| kgriffs | w=2 looks bad | 15:19 |
| mpanetta | Is it? | 15:19 |
| kgriffs | if you loose one secondary, you will deadlock | 15:19 |
| kgriffs | it should be "majority" IMO | 15:19 |
| kgriffs | w=majority | 15:19 |
| flaper87 | yup | 15:20 |
| mpanetta | uhoh | 15:20 |
| kgriffs | lose | 15:20 |
| kgriffs | s/oo/o | 15:20 |
| mpanetta | The catalog db URI is similar | 15:20 |
| mpanetta | Gotta log off my IRC is messed up | 15:21 |
| *** mpanetta has quit IRC | 15:21 | |
| *** mpanetta has joined #openstack-marconi | 15:22 | |
| kgriffs | sriram: good morning! | 15:22 |
| *** mpanetta has joined #openstack-marconi | 15:23 | |
| *** mpanetta has quit IRC | 15:23 | |
| sriram | good morning kgriffs! | 15:23 |
| *** whenry has quit IRC | 15:23 | |
| kgriffs | sriram: thanks for helping with the benchmarking project | 15:23 |
| *** mpanetta has joined #openstack-marconi | 15:23 | |
| kgriffs | sriram: and... you will be at the summit, right? | 15:24 |
| sriram | sure, and of course :) | 15:24 |
| kgriffs | sweet | 15:24 |
| kgriffs | so, I have been making myself a lot of thoughts about benchmarking. I looked at rally, and also have been thinking about Mike's suggestions. | 15:24 |
| kgriffs | sriram: what do you think about this idea: | 15:25 |
| kgriffs | Create a POC marconi-bench project. The project would take one of our primary messaging patterns, and simulate it using python-marconiclient, but patched to use gevent version of requests | 15:26 |
| sriram | that sounds really cool! | 15:26 |
| mpanetta | I had a suggestion? :P | 15:27 |
| kgriffs | then, we sample metrics and pipe them to either beekeeper or graphite. Samples would go in the load generators, the consumers, as well as in some WSGI middleware and maybe even at the DB driver layer | 15:27 |
| kgriffs | mpanetta: your suggestion to benchmark end-to-end scenarios so the results where more meaningful | 15:27 |
| mpanetta | Ah yes | 15:28 |
| mpanetta | I have slept since then :P | 15:28 |
| kgriffs | I don't believe rally is up to the task today, but we could contribute to the project and get it there eventually | 15:28 |
| *** whenry has joined #openstack-marconi | 15:28 | |
| kgriffs | first step is to do a POC so we can have a baseline of what we want to achieve. plus, I'd like this sooner than we would be able to do it via Rally. | 15:29 |
| kgriffs | so, my plan is | 15:29 |
| kgriffs | 1. POC | 15:29 |
| kgriffs | 2. Work with rally to build out necessary support | 15:29 |
| kgriffs | 3. merge POC with rally | 15:29 |
| kgriffs | flaper87, malini: thoughts? | 15:29 |
| mpanetta | WHat is rally again? | 15:29 |
| *** haomaiwa_ has quit IRC | 15:29 | |
| mpanetta | Is that the OS BM tool? | 15:29 |
| kgriffs | https://wiki.openstack.org/wiki/Rally | 15:29 |
| *** haomaiwang has joined #openstack-marconi | 15:30 | |
| kgriffs | also, I mentioned beekeeper: https://github.com/worldline/beekeeper | 15:30 |
| * flaper87 reads backlog | 15:31 | |
| malini | kgriffs: tht is a good idea | 15:32 |
| malini | it will help us get more stats than just per api calls | 15:33 |
| flaper87 | kgriffs: it's a good idea. | 15:33 |
| flaper87 | I'd also like to run other queuing systems benches on MArconi | 15:33 |
| flaper87 | that is, the known existing benchmark rules for messaging systems | 15:34 |
| flaper87 | take them, translate them to marconi and run them | 15:34 |
| malini | boris-42: do you have any thoughts on this? | 15:34 |
| kgriffs | flaper87: ok | 15:34 |
| kgriffs | malini: is boris-42 on the Rally team? | 15:34 |
| malini | yes | 15:35 |
| kgriffs | ok | 15:35 |
| kgriffs | malini: http://summit.openstack.org/cfp/details/179 | 15:37 |
| kgriffs | flaper87: ^^^ | 15:37 |
| *** haomaiwang has quit IRC | 15:37 | |
| kgriffs | I think it would be good for some of us to attend that. We should connect with Boris and chat with him afterwards to share our use cases for Rally and find out what gaps Rally has that we can help with. | 15:38 |
| kgriffs | sriram: ^^^ | 15:38 |
| mpanetta | I can go. | 15:38 |
| kgriffs | mpanetta: seet. | 15:38 |
| *** haomaiwa_ has joined #openstack-marconi | 15:38 | |
| kgriffs | sweet | 15:38 |
| flaper87 | mpanetta++ | 15:38 |
| kgriffs | I just added it to my own agenda | 15:38 |
| sriram | Yes, I'll go as well | 15:39 |
| kgriffs | ok, so about that POC | 15:39 |
| kgriffs | how far can we get working on this today and tomorrow? I want to help too. Let me bring up a pad | 15:39 |
| kgriffs | https://etherpad.openstack.org/p/juno-marconi-benchmarking | 15:40 |
| *** haomaiwa_ has quit IRC | 15:43 | |
| *** openstackgerrit has joined #openstack-marconi | 15:43 | |
| malini | kgriffs: I talked to boris-42 a while back & rally was not yet ready to support our 2000 rps then | 15:44 |
| *** rossk has joined #openstack-marconi | 15:44 | |
| malini | We shuld sync up with the rally folks again during the summit | 15:45 |
| kgriffs | ok | 15:47 |
| kgriffs | we will also need rally to be able to run different types of clients simultaneously - It wasn't clear from their docs whether Rally can do that yet. | 15:48 |
| malini | kgriffs: what do you mean by 'diff types of client' ? | 15:49 |
| kgriffs | like, we need some message producers running at the same time as some consumers, and we want to measure the rate that messages are being produced as well as consumed. | 15:50 |
| kgriffs | basically, modeling a more real-world use case | 15:50 |
| malini | but tht will vary depending on the app tht uses Marconi, rt? | 15:51 |
| malini | are we looking at metrics for real crazy fast prod & consumer ? | 15:52 |
| kgriffs | malini: well, I'd like to be able to spin up enough load generator processes and consumer processes to max out the benchmark deployment of Marconi | 15:56 |
| kgriffs | basically, we need enough generators to feed X consumers | 15:56 |
| kgriffs | we will have to experiment to figure out the numbers. :p | 15:56 |
| malini | we need a virtual white board *-) | 15:57 |
| kgriffs | X greenlets across Y processes across Z VMs | 15:57 |
| kgriffs | mmm. I should diagram this | 15:57 |
| kgriffs | flaper87: https://review.openstack.org/#/c/91909/ | 15:57 |
| kgriffs | malini: also need your approval if you agree ^^^ | 15:58 |
| kgriffs | I want to do this before I cut a new pypi release | 15:58 |
| kgriffs | </random> | 15:58 |
| openstackgerrit | A change was merged to openstack/python-marconiclient: Updated from global requirements https://review.openstack.org/91565 | 16:03 |
| *** prashanthr_ has quit IRC | 16:05 | |
| boris-42 | malini I'm here=) | 16:08 |
| boris-42 | kgriffs ping | 16:10 |
| boris-42 | kgriffs what do you mean by different type of clients? | 16:11 |
| kgriffs | boris-42: so, we would like to simulate an end-to-end scenario. We would have X client that are producing messages as fast as they can, and Y clients that are consuming those messages. | 16:13 |
| kgriffs | so you have two different behaviors happening concurrently | 16:14 |
| boris-42 | kgriffs hm it will be a bit tricky | 16:15 |
| flaper87 | kgriffs: but we already bumpped the version | 16:15 |
| flaper87 | I guess we'll have to do the milestone-proposed thing | 16:15 |
| kgriffs | flaper87: but we did not tag it yet | 16:15 |
| boris-42 | kgriffs e.g. benchmark in rally meaning is some function that will be repeated multiply times (from randomly chosen user) | 16:16 |
| boris-42 | kgriffs and there is another thing called benchmark context | 16:16 |
| boris-42 | kgriffs it allows you to build any environment before running benchmark and cleanup it after running | 16:17 |
| flaper87 | kgriffs: +@ | 16:17 |
| kgriffs | boris-42: so, I guess what we would need is for the ability to run N of those benchmarks/environments simultaneously | 16:17 |
| flaper87 | erm, +2 | 16:17 |
| * flaper87 has to go through 10 proxies to use gerrit | 16:18 | |
| flaper87 | this is insane | 16:18 |
| kgriffs | boris-42: not sure if I am understanding correctly. I'd love to meet with you during the summit to discuss in detail | 16:18 |
| boris-42 | kgriffs i have desing seesion | 16:18 |
| kgriffs | flaper87: WAH? | 16:18 |
| boris-42 | kgriffs about rally | 16:18 |
| flaper87 | kgriffs: yeah, just like that. my ISP hasn't fixed the blocked IP "issue" | 16:19 |
| boris-42 | kgriffs http://junodesignsummit.sched.org/event/82bbddb45555dc39fa6029ee56e7dc4e | 16:19 |
| *** chandan_kumar has quit IRC | 16:19 | |
| boris-42 | kgriffs btw rally is run on every patch in keystone in glance | 16:19 |
| boris-42 | kgriffs take a look here https://review.openstack.org/#/c/90405/ | 16:20 |
| *** balajiiyer has quit IRC | 16:20 | |
| kgriffs | boris-42: cool. I'd love to run benchmarks on Marconi for every patch | 16:20 |
| malini | boris-42: do these run against devstack? | 16:20 |
| boris-42 | malini yep | 16:21 |
| boris-42 | malini kgriffs so take a look at this file | 16:21 |
| kgriffs | boris-42: for the moment we are going to do a POC benchmark but our plan is to migrate to rally | 16:21 |
| boris-42 | kgriffs malini https://review.openstack.org/#/c/90405/3/rally-scenarios/keystone.yaml | 16:21 |
| boris-42 | kgriffs malini you can specify here any amount of benchmarks with any load | 16:21 |
| boris-42 | and result will be in this job check-rally-dsvm-keystone | 16:21 |
| boris-42 | in case of keystone | 16:22 |
| malini | it'll be a good if boris-42 can look at our POC idea, to make sure it can integrate well into Rally | 16:22 |
| openstackgerrit | A change was merged to openstack/python-marconiclient: chore: Classify as Beta (was pre-alpha) https://review.openstack.org/91909 | 16:22 |
| boris-42 | malini kgriffs and you are getting output like here http://logs.openstack.org/05/90405/3/check/check-rally-dsvm-keystone/81a809d/rally-plot/results.html.gz | 16:22 |
| boris-42 | ^ open in chrome=) | 16:22 |
| boris-42 | malini yep it will be nice=) | 16:23 |
| boris-42 | malini actually we don't support at this moment versioning cause we are trying to adopt rally | 16:23 |
| boris-42 | malini to meet everybody requierments | 16:23 |
| boris-42 | malini so generating big load | 16:24 |
| boris-42 | malini aka 2k request per sec will be hard in gates | 16:24 |
| boris-42 | malini cause at this moment we can request only one node from nodepool | 16:24 |
| boris-42 | malini and marconi will work on the same node with rally | 16:25 |
| boris-42 | malini so it may make troubles | 16:25 |
| malini | boris-42: does rally have distributed load gen? | 16:25 |
| boris-42 | malini we are working in that direction | 16:25 |
| boris-42 | malini actually there are 3 separated task | 16:25 |
| boris-42 | 1. visualization of data | 16:25 |
| boris-42 | 2. storing big amount of data | 16:26 |
| boris-42 | 3. doing this stuff with distributed load gen (yahoo is going to integrate task flow in rally so we will be able to do it) | 16:26 |
| boris-42 | I am working on point 2 and thinking about point 1 | 16:26 |
| boris-42 | when we finish all these steps we will be able to produce distributed load | 16:27 |
| boris-42 | but we will have still issues with gates | 16:27 |
| boris-42 | malini cause we have only one node => no need in distributed load generator | 16:27 |
| malini | thanks boris-42! Hope to meet you at the summit | 16:28 |
| boris-42 | malini me too=) | 16:28 |
| boris-42 | malini kgriffs so guys my thoughts are next | 16:28 |
| *** Obulpathi has joined #openstack-marconi | 16:28 | |
| boris-42 | malini kgriffs we should start with adding marconi client to rally and making first simple benchmark | 16:28 |
| boris-42 | malini kgriffs that just put something to queue and consume from one user | 16:28 |
| boris-42 | malini kgriffs so you'll get familiar with rally and we will be able to better understand what we actually need | 16:29 |
| malini | sure boris-42! We'll try to get something simple done, before the summit | 16:30 |
| boris-42 | malini kgriffs I think that distributed load generator is not only one change that is required in rally | 16:30 |
| boris-42 | malini so let's make it today?) | 16:30 |
| boris-42 | malini simple benchmarks are quite easy (copy-paste stuff) | 16:30 |
| malini | boris-42: I am necks down in something else :( | 16:31 |
| boris-42 | malini heh | 16:31 |
| boris-42 | malini so when you will have about 1hrs | 16:31 |
| boris-42 | malini or even less just ping me=) | 16:31 |
| malini | sure I will | 16:31 |
| boris-42 | malini I'll help to start=) | 16:32 |
| malini | thanks boris-42! | 16:32 |
| boris-42 | malini np | 16:32 |
| boris-42 | kgriffs malini btw does devstack install marconi ? | 16:32 |
| malini | yes | 16:32 |
| boris-42 | malini kgriffs I mean in gates? | 16:32 |
| malini | boris-42: only in our experimental jobs & at the marconi jenkins pipeline | 16:32 |
| boris-42 | malini ahh | 16:33 |
| boris-42 | malini we will need to add dsvm-with-marconi job | 16:33 |
| boris-42 | malini and one more rally job | 16:33 |
| boris-42 | malini to be able to run rally or use experimental one | 16:33 |
| malini | boris-42: we have a dsvm job | 16:34 |
| malini | http://logs.openstack.org/55/76555/6/check/check-tempest-dsvm-marconi/fb49f06/ | 16:34 |
| malini | eg | 16:34 |
| boris-42 | malini nice=) | 16:34 |
| boris-42 | malini ok then no worries about running rally against it=) | 16:34 |
| malini | cool! | 16:34 |
| *** Obulpathi has quit IRC | 16:40 | |
| kgriffs | mpanetta, malini: are you doing LB on the benchmark deployment? | 16:44 |
| kgriffs | or is it a single web head? | 16:44 |
| mpanetta | Yes | 16:44 |
| kgriffs | mpanetta: haproxy or something else? | 16:44 |
| mpanetta | Nope | 16:44 |
| mpanetta | cloud LB | 16:44 |
| kgriffs | oh. What do you think about using haproxy? That would give the community one less "black-box" piece to factor into when digesting the results | 16:46 |
| kgriffs | also gives us a little bit more consistent results, I would think, since we would own more of the box than we get as one of many tenants in cloud LB | 16:46 |
| kgriffs | (more noisy neighbor in cloud LB) | 16:46 |
| *** reed has joined #openstack-marconi | 16:49 | |
| kgriffs | mpanetta: p.s. - are you benchmarking with SSL termination? | 16:51 |
| kgriffs | and keystone auth? | 16:51 |
| mpanetta | I think they disabled keystone auth | 16:51 |
| mpanetta | You would have to ask sriram that | 16:51 |
| kgriffs | sriram: ^^^ | 16:51 |
| mpanetta | no ssl term AFAIK | 16:52 |
| kgriffs | mpanetta: how many web heads? 4? | 16:52 |
| mpanetta | yep | 16:52 |
| *** Obulpathi has joined #openstack-marconi | 16:57 | |
| *** Obulpathi has quit IRC | 16:58 | |
| *** Obulpathi has joined #openstack-marconi | 16:59 | |
| *** alcabrera|afk is now known as alcabrera | 17:04 | |
| *** balajiiyer has joined #openstack-marconi | 17:04 | |
| sriram | keystone auth was in place. 4 web heads, behind a loab balancer, with a mongo replica set | 17:09 |
| *** openstackgerrit has quit IRC | 17:20 | |
| *** openstackgerrit has joined #openstack-marconi | 17:25 | |
| kgriffs | sriram: memcached for caching keystone auth? | 17:25 |
| sriram | yep | 17:25 |
| kgriffs | sriram: ssl termination? | 17:26 |
| sriram | no ssl, just ip | 17:29 |
| *** abettadapur has joined #openstack-marconi | 17:31 | |
| *** Obulpathi has quit IRC | 17:53 | |
| kgriffs | sriram: memcached is colocated on each web head? | 18:02 |
| sriram | yes, they are colocated. | 18:04 |
| kgriffs | ok. Do you have time today and tomorrow to work on this POC with me? https://etherpad.openstack.org/p/juno-marconi-benchmarking | 18:14 |
| sriram | sure, can I let you know about this soon? | 18:18 |
| kgriffs | sriram: yeah, just ping me. I was going to work on it myself, but wanted to see who would be able to help divide up the work | 18:20 |
| kgriffs | flaper87: "git push gerrit 0.0.1" | 18:20 |
| kgriffs | ! [remote rejected] 0.0.1 -> 0.0.1 (prohibited by Gerrit) | 18:20 |
| * kgriffs is a sad panda | 18:21 | |
| kgriffs | mpanetta: can you help me? I'd like to set up a graphite box that our benchmarking cluster can push to | 18:23 |
| kgriffs | I'd also like to set up haproxy instead of using cloud LB | 18:23 |
| mpanetta | Oh | 18:47 |
| mpanetta | I don't think I have enough time to do that :( | 18:47 |
| kgriffs | I can do the legwork. let me ping you. | 18:50 |
| flaper87 | kgriffs: you'll have to ping ttx | 19:05 |
| flaper87 | I think they never gave you / me access | 19:05 |
| flaper87 | :/ | 19:05 |
| kgriffs | strange. why did it work last time for 0.0.1a1 ? | 19:05 |
| flaper87 | kgriffs: because he did it :) | 19:05 |
| kgriffs | mm, my name is on the last tag | 19:06 |
| * kgriffs is confused | 19:06 | |
| flaper87 | oh, mmh | 19:06 |
| flaper87 | then, no idea | 19:06 |
| *** Obulpathi has joined #openstack-marconi | 19:23 | |
| *** Obulpathi has quit IRC | 19:24 | |
| *** Obulpathi has joined #openstack-marconi | 19:24 | |
| *** ametts has joined #openstack-marconi | 19:28 | |
| *** AAzza has joined #openstack-marconi | 19:28 | |
| *** malini is now known as malini_afk | 19:38 | |
| *** megan_w|afk is now known as megan_w | 19:41 | |
| *** AAzza has quit IRC | 19:45 | |
| *** cpallares has quit IRC | 19:50 | |
| kgriffs | flaper87: I pinged ttx earlier, no response. Can you help me try to reach him? If we both keep pinging him, we'll have a better chance getting this resolved sooner rather than later. | 19:52 |
| kgriffs | flaper87: p.s. - we should add a doc to the wiki on how to release | 19:52 |
| kgriffs | flaper87: like this https://wiki.openstack.org/wiki/Savanna/How_To_Release | 19:52 |
| flaper87 | kgriffs: IIRC he's off on vacation | 20:08 |
| flaper87 | but yeah, I'll try tor each him tomorrow | 20:08 |
| flaper87 | re the wiki page, the process is the same for everyone, I wonder why that page is not in some common place | 20:09 |
| flaper87 | perhaps we should update the existing one | 20:09 |
| *** reed has quit IRC | 20:09 | |
| *** reed has joined #openstack-marconi | 20:12 | |
| *** kgriffs is now known as kgriffs|afk | 20:13 | |
| *** abettadapur has quit IRC | 20:15 | |
| *** alcabrera is now known as alcabrera|afk | 20:17 | |
| *** megan_w is now known as megan_w|afk | 20:27 | |
| *** sriram has quit IRC | 20:30 | |
| *** megan_w|afk is now known as megan_w | 20:32 | |
| *** balajiiyer has left #openstack-marconi | 20:32 | |
| *** kgriffs|afk is now known as kgriffs | 20:50 | |
| *** mpanetta has quit IRC | 20:53 | |
| amitgandhi | kgriffs: flaper87: forgot the etherpad link for places we should incorporate marconi, but lets add mistral to that list - they use task queues | 21:07 |
| kgriffs | kk | 21:08 |
| *** boris-42 has quit IRC | 21:15 | |
| *** boris-42 has joined #openstack-marconi | 21:16 | |
| *** jchai has quit IRC | 21:18 | |
| *** amitgandhi has quit IRC | 21:48 | |
| *** mwagner_lap has quit IRC | 21:48 | |
| *** Obulpathi has quit IRC | 22:03 | |
| *** megan_w is now known as megan_w|afk | 22:08 | |
| *** vkmc has quit IRC | 22:54 | |
| *** kgriffs is now known as kgriffs|afk | 23:31 | |
| *** flaper87 is now known as flaper87|afk | 23:43 | |
| *** mwagner_lap has joined #openstack-marconi | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!