| *** jeromatron has joined #magnetodb | 00:42 | |
| *** keith_newstadt has joined #magnetodb | 01:02 | |
| *** keith_newstadt has quit IRC | 01:07 | |
| *** keith_newstadt has joined #magnetodb | 01:08 | |
| *** jeromatron has quit IRC | 01:21 | |
| *** vivekd has joined #magnetodb | 03:23 | |
| *** jeromatron has joined #magnetodb | 03:35 | |
| *** rushiagr_away is now known as rushiagr | 03:37 | |
| *** rushiagr is now known as rushiagr_away | 03:51 | |
| *** rushiagr_away is now known as rushiagr | 04:36 | |
| *** jeromatron has quit IRC | 05:10 | |
| *** ajayaa has joined #magnetodb | 05:15 | |
| *** ajayaa has quit IRC | 06:16 | |
| *** jeromatron has joined #magnetodb | 06:22 | |
| *** ajayaa has joined #magnetodb | 06:29 | |
| *** jeromatron has quit IRC | 06:53 | |
| *** k4n0 has joined #magnetodb | 07:08 | |
| ajayaa | Hi. Why are the notifications of priority 'debug' not sent to amqp? | 07:17 |
|---|---|---|
| ajayaa | When an item is added to a table, a notification of priority debug is generated, which is not sent to the amqp at all. | 07:18 |
| ajayaa | *without any log or warning | 07:19 |
| ajayaa | ikhudoshyn ^^ | 07:19 |
| *** romainh has joined #magnetodb | 07:49 | |
| *** rushiagr is now known as rushiagr_away | 09:12 | |
| ikhudoshyn | ajayaa, hm, looks like smth with notification->logging driver. it is configured in *.conf | 09:13 |
| ikhudoshyn | cwang is the one who implemented it | 09:14 |
| ikhudoshyn | now i'm working on moving to oslo.notify but patch is in progress | 09:14 |
| ajayaa | ikhudoshyn, https://github.com/stackforge/magnetodb/blob/master/magnetodb/common/notifier/rpc_notifier.py#L43-45 | 09:19 |
| *** rushiagr_away is now known as rushiagr | 09:19 | |
| ikhudoshyn | oh, I see | 09:21 |
| ikhudoshyn | )) | 09:21 |
| ikhudoshyn | ho particular reason for that from our side | 09:22 |
| ikhudoshyn | magnetodb.openstack.common.* is a code common for all OS projects | 09:22 |
| ikhudoshyn | initially it was just copy-pasted from one project to another | 09:23 |
| ikhudoshyn | now it is extracted as set of oslo.* projects | 09:23 |
| ikhudoshyn | we're working on moving our stuff to use it | 09:23 |
| ikhudoshyn | the only idea that comes to my mind is they try to save MQ from being flooded by numerous debug messages | 09:24 |
| openstackgerrit | Illia Khudoshyn proposed a change to stackforge/magnetodb: Add 'last_updated' to 'table_info' https://review.openstack.org/124317 | 09:25 |
| ajayaa | ikhudoshyn, Even I thought the same. But some of the operations in mdb are sent with debug priority and some are sent with info priority. | 09:27 |
| ajayaa | for e.g., operations such as put_item, query are debug level. | 09:27 |
| ajayaa | If we are going to od billing via ceilometer these things need to reach message queue. | 09:28 |
| ikhudoshyn | in my notification patch i replaced all notifications with debug level to use info | 09:28 |
| ajayaa | ikhudoshyn, cool. :) | 09:29 |
| ikhudoshyn | i really hope it will get in eventually | 09:29 |
| ikhudoshyn | btw that fact about ingoring debug ones -- great to know, thanks | 09:29 |
| ikhudoshyn | i guess not mavy of us were aware of it | 09:30 |
| ikhudoshyn | * many | 09:30 |
| ajayaa | ikhudoshyn, Are there any other consumers of notifications other than ceilometer? | 09:33 |
| ajayaa | I feel like, notifications in current format does not help too much with ceilometer. | 09:34 |
| ikhudoshyn | as for notification format, oslo.notify provides sort of a spec for notification format | 09:35 |
| ikhudoshyn | we're just to use it | 09:35 |
| ikhudoshyn | kinda OS-way)) | 09:35 |
| ajayaa | ikhudoshyn, the content of notification. | 09:35 |
| ikhudoshyn | what do you mean? | 09:35 |
| ajayaa | * I should have put in the above way. | 09:35 |
| ikhudoshyn | http://docs.openstack.org/developer/oslo.messaging/notifier.html | 09:36 |
| ajayaa | for e.g. If a user does a query, then message contains query parameters and user and project. | 09:36 |
| ikhudoshyn | here is the spec | 09:36 |
| ikhudoshyn | and event_type | 09:36 |
| ajayaa | It should also contain how many rows were returned or how many bytes were returned. | 09:37 |
| ikhudoshyn | well I don't see any reason why not to do so)) | 09:37 |
| ikhudoshyn | if u think we need (or you need it) - just raise the discussion, register BP.. | 09:38 |
| ikhudoshyn | it's open source any way | 09:38 |
| ajayaa | okay. I will come up with the specs and blueprint. | 09:39 |
| ajayaa | ikhudoshyn, Would it not depend on your patch? | 09:39 |
| ikhudoshyn | sure | 09:39 |
| ikhudoshyn | well it shouldn't | 09:40 |
| ikhudoshyn | they won't be able to get merged, but it's ok | 09:40 |
| ajayaa | okay. Thanks. | 09:40 |
| ikhudoshyn | when team approves it it sometimes requires to get rebased first, but it's a normal process | 09:41 |
| ajayaa | I am familiar with rebase process. I have gone through it several times. :) | 09:41 |
| ikhudoshyn | sure) but sometimes it annoys so much)) | 09:42 |
| *** isviridov_away is now known as isviridov | 09:42 | |
| isviridov | Hello ikhudoshyn, ajayaa | 09:47 |
| ajayaa | Hi isviridov, | 09:48 |
| isviridov | Just looked at you coversation. | 09:48 |
| ajayaa | What so you think? | 09:49 |
| ikhudoshyn | isviridov, o/ | 09:49 |
| ajayaa | do* | 09:49 |
| isviridov | We are not sending the message on every event, because it will flood the queue. | 09:49 |
| isviridov | It was discussed the following workagounds | 09:49 |
| isviridov | 1. implement notification driver what will agregate the messages | 09:50 |
| ajayaa | isviridov, who is the intended consumer of messages? | 09:50 |
| isviridov | 2. use monitorign api for usage data | 09:51 |
| ajayaa | and what do we aim to achieve via sending messages? | 09:51 |
| isviridov | ajayaa, it is expeccted to be celiometer or any other billing of monitoring system. | 09:52 |
| ajayaa | isviridov, Let's say that I want to know the amount of bytes user has read already. | 09:52 |
| ajayaa | If we don't send notifications related to query and scan to message bus, then these kind of answers are impossible know. | 09:53 |
| isviridov | Notifications sound good. But how many notification are you going to send? | 09:54 |
| ajayaa | Right now we can send one notifications per api call, but going forward if the load on message bus increases considerably we can think of sending these notifications at some time interval. | 09:55 |
| openstackgerrit | Andrei V. Ostapenko proposed a change to stackforge/magnetodb: Application log rotation https://review.openstack.org/126519 | 09:56 |
| ajayaa | For e.g. if you do a query, then we can send a notification containing the number of bytes the user has read. | 09:57 |
| isviridov | We have to think about the load from the beginning. Even now we have installation what process up to 40k requests per second. It means 40k messages in the queue for celiometer. Do you think it is possible to handle such a stream by celiometer now? | 09:59 |
| ajayaa | isviridov, I am not sure. We need to ask ceilometer guys about it. But scalability of rabbitmq could be mitigated by using zeromq. | 10:01 |
| ajayaa | But using monitoring api does not give a complete idea of usage information. | 10:02 |
| isviridov | Not sure that it is true for celiometer ;) | 10:02 |
| ajayaa | It just gives you number of items and size of table. | 10:02 |
| ajayaa | isviridov, How do you measure read request by an user? | 10:03 |
| isviridov | ajayaa, what stops you from extending it? And add counters you need? | 10:03 |
| isviridov | ajayaa, we don't. What the goal? | 10:03 |
| ajayaa | isviridov, The goal is to measure everything user is consuming. | 10:04 |
| ajayaa | Not just size of tables or number of items. | 10:04 |
| isviridov | ajayaa, let us start from the list and set the priority for each metric. | 10:06 |
| openstackgerrit | Andrei V. Ostapenko proposed a change to stackforge/magnetodb: Monitoring health check request https://review.openstack.org/125108 | 10:06 |
| isviridov | I believe the ML is good place for such discussion. | 10:06 |
| ajayaa | isviridov, okay. | 10:06 |
| isviridov | I think we can implement everything with internal counters and exposing it via monitoring api or/and sending agregated information to queue. But with taking into account the expected load, it woun't be so easy as just send message on each request | 10:08 |
| ajayaa | isviridov, I agree with you. | 10:09 |
| ajayaa | isviridov, We need to decide what to send via message bus and what to expose via monitoring api. | 10:10 |
| isviridov | Yes | 10:11 |
| *** romainh has quit IRC | 10:13 | |
| isviridov | But anyhow we have to clarify celiometer's capacity as well. | 10:14 |
| ajayaa | isviridov, Yes. Also, We need to quantify the work-load of magnetodb as well. | 10:16 |
| ajayaa | As you said you have a setup of mdb which is serving 40K rps. | 10:16 |
| ajayaa | What is the maximum work-load we are expecting mdb to take on? | 10:17 |
| isviridov | It should be designed to handle 100K per second | 10:19 |
| rushiagr | isviridov: 'it' == 'magnetodb'?? | 10:22 |
| isviridov | rushiagr, yeap | 10:29 |
| rushiagr | isviridov: I have a problem with that :) When cassandra scales infinitely, it would be very weird to provide a scheme on top of it which doesn't scale that much | 10:30 |
| rushiagr | isviridov: why would people use magnetodb then, if we're not assuring them that they don't need to care when their load increases when they start getting more customers/traffic spikes? :) | 10:31 |
| isviridov | Hello rushiagr | 10:31 |
| rushiagr | isviridov: hey | 10:32 |
| isviridov | rushiagr, ideally we will have linear scalability and theoretically infinity capacity of requests per second and stored items as well. | 10:34 |
| isviridov | But we have to have some figure in mind to make better decisions. | 10:34 |
| openstackgerrit | Andrei V. Ostapenko proposed a change to stackforge/magnetodb: Monitoring health check request https://review.openstack.org/125108 | 10:35 |
| isviridov | rushiagr, what are your expectations from mdb? | 10:36 |
| rushiagr | isviridov: I think to make better decisions, we should target scalability as much as cassandra can provide | 10:37 |
| rushiagr | isviridov: that should be our ultimate goal | 10:37 |
| rushiagr | isviridov: sorry, in a meeting. Will come back soon to carry on the discussion | 10:38 |
| isviridov | rushiagr, looking forward | 10:39 |
| *** romainh has joined #magnetodb | 10:51 | |
| rushiagr | isviridov: I'm back | 11:04 |
| rushiagr | isviridov: maybe I interpreted your statement wrongly. What I understood was that you said that mdb should target 100k ops per second only | 11:05 |
| rushiagr | isviridov: in the process of scaling linearly, we will need to scale mdb nodes as well | 11:06 |
| rushiagr | isviridov: again, I have not read through the discussion which brought this topic up | 11:06 |
| isviridov | rushiagr, it was a figure to show the ammount of expected load | 11:07 |
| rushiagr | isviridov: but probably it is just a simple thing, like putting multiple mdb nodes behind a load balancer | 11:07 |
| rushiagr | isviridov: this is the short-term goal right? longer term we might want mdb to scale beyond this too..? | 11:08 |
| isviridov | rushiagr, mdb is linear scalable now. But there is always practical limit, what really depends on work load | 11:09 |
| rushiagr | isviridov: you mean to say practical limit on the operations the virtual machine on which MDB is running can perform? | 11:10 |
| *** openstack has joined #magnetodb | 14:14 | |
| openstackgerrit | Andrei V. Ostapenko proposed a change to stackforge/magnetodb: Monitoring health check request https://review.openstack.org/125108 | 14:19 |
| *** rushiagr_away is now known as rushiagr | 14:21 | |
| openstackgerrit | Andrei V. Ostapenko proposed a change to stackforge/magnetodb: Monitoring health check request https://review.openstack.org/125108 | 14:28 |
| *** jeromatron has joined #magnetodb | 14:47 | |
| *** jeromatron has quit IRC | 15:07 | |
| *** jeromatron has joined #magnetodb | 15:07 | |
| openstackgerrit | Illia Khudoshyn proposed a change to stackforge/magnetodb: Use oslo.notify for notifications https://review.openstack.org/126489 | 15:18 |
| *** ajayaa has quit IRC | 15:51 | |
| *** jeromatron has quit IRC | 15:52 | |
| *** k4n0 has quit IRC | 16:00 | |
| *** rushiagr is now known as rushiagr_away | 16:01 | |
| *** ajayaa has joined #magnetodb | 16:10 | |
| *** jeromatron has joined #magnetodb | 16:56 | |
| *** romainh has left #magnetodb | 16:59 | |
| *** ajayaa has quit IRC | 17:13 | |
| *** jeromatron has quit IRC | 17:25 | |
| *** jeromatron has joined #magnetodb | 17:45 | |
| *** jeromatron has quit IRC | 18:37 | |
| *** jeromatron has joined #magnetodb | 18:38 | |
| *** charlesw has quit IRC | 21:30 | |
| *** vnaboychenko has joined #magnetodb | 22:22 | |
| *** jeromatron has quit IRC | 22:45 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!