Friday, 2013-08-30

*** fnaval__ has joined #openstack-metering00:03
*** fnaval___ has joined #openstack-metering00:04
*** dina_belova has joined #openstack-metering00:04
*** fnaval_ has quit IRC00:05
*** fnaval__ has quit IRC00:07
*** fnaval___ has quit IRC00:08
*** dina_belova has quit IRC00:09
*** evanjfraser has joined #openstack-metering00:13
*** gordc has joined #openstack-metering00:15
*** zul has joined #openstack-metering00:36
*** gordc has quit IRC00:50
*** flwang has quit IRC00:52
*** dina_belova has joined #openstack-metering01:05
*** zul has quit IRC01:06
*** dina_belova has quit IRC01:10
*** sandywalsh has quit IRC01:27
*** sandywalsh has joined #openstack-metering01:45
openstackgerritA change was merged to openstack/ceilometer: Imported Translations from Transifex  https://review.openstack.org/4421601:53
*** d34dh0r53 has joined #openstack-metering01:59
*** d34dh0r53 has quit IRC02:02
*** shaneduan is now known as shaneduan[afk]02:04
*** changbl has joined #openstack-metering02:05
*** dina_belova has joined #openstack-metering02:06
openstackgerritA change was merged to openstack/ceilometer: Adds Hyper-V compute inspector  https://review.openstack.org/4370902:08
*** dina_belova has quit IRC02:10
*** flwang has joined #openstack-metering02:19
*** anteaya has quit IRC02:27
openstackgerritlitong01 proposed a change to openstack/ceilometer: db2 implementation does not sort results of get_resources  https://review.openstack.org/4439002:31
openstackgerritlitong01 proposed a change to openstack/ceilometer: add tests for _query_to_kwargs func  https://review.openstack.org/4379602:31
*** evanjfraser has quit IRC02:38
openstackgerritLianhao Lu proposed a change to openstack/ceilometer: Added pollsters for the hardware agent  https://review.openstack.org/4307402:52
openstackgerritLianhao Lu proposed a change to openstack/ceilometer: Adding hardware-agent  https://review.openstack.org/4307202:52
openstackgerritLianhao Lu proposed a change to openstack/ceilometer: Added hardware agent's inspector and snmp implementation  https://review.openstack.org/4307302:52
*** dina_belova has joined #openstack-metering03:06
*** dina_belova has quit IRC03:10
openstackgerritlitong01 proposed a change to openstack/ceilometer: db2 implementation does not sort results of get_resources  https://review.openstack.org/4439003:23
openstackgerritFengqian.gao proposed a change to openstack/ceilometer: Add pagination support for sqlalchemy database  https://review.openstack.org/3545403:26
openstackgerritFengqian.gao proposed a change to openstack/ceilometer: Add pagination parameter to the database backends of storage  https://review.openstack.org/4258203:26
openstackgerritFengqian.gao proposed a change to openstack/ceilometer: Add pagination support to selected APIs  https://review.openstack.org/3745403:26
openstackgerritFengqian.gao proposed a change to openstack/ceilometer: Change pagination query method in mongodb  https://review.openstack.org/4186903:26
openstackgerritFengqian.gao proposed a change to openstack/ceilometer: Add a fake UUID to Meter database model named meter_id  https://review.openstack.org/4427803:26
*** litong has quit IRC03:48
*** shang has joined #openstack-metering03:56
*** dina_belova has joined #openstack-metering04:07
*** fnaval_ has joined #openstack-metering04:08
*** fnaval_ has quit IRC04:09
*** fnaval_ has joined #openstack-metering04:10
*** shaneduan[afk] is now known as shaneduan04:11
*** dina_belova has quit IRC04:11
*** boris-42 has joined #openstack-metering04:46
*** SergeyLukjanov has joined #openstack-metering05:06
*** dina_belova has joined #openstack-metering05:07
*** dina_belova has quit IRC05:12
*** gordc has joined #openstack-metering05:19
*** SergeyLukjanov has quit IRC05:35
*** gordc has quit IRC05:43
*** dina_belova has joined #openstack-metering05:57
*** eglynn has joined #openstack-metering06:00
*** dina_belova has quit IRC06:01
*** dina_belova has joined #openstack-metering06:02
openstackgerritJenkins proposed a change to openstack/ceilometer: Imported Translations from Transifex  https://review.openstack.org/4440506:04
*** dina_belova has quit IRC06:06
*** shang has quit IRC06:06
*** shaneduan is now known as shaneduan[afk]06:26
*** SergeyLukjanov has joined #openstack-metering06:31
*** SergeyLukjanov has quit IRC06:33
*** tasdomas_afk is now known as tasdomas06:42
*** Bada has joined #openstack-metering06:45
*** Bada has quit IRC06:45
*** boris-42 has quit IRC06:49
openstackgerritA change was merged to openstack/ceilometer: Base Alarm history persistence model  https://review.openstack.org/4384806:54
*** dina_belova has joined #openstack-metering06:58
*** shang has joined #openstack-metering07:01
*** yolanda has joined #openstack-metering07:08
*** dina_belova has quit IRC07:08
*** dina_belova has joined #openstack-metering07:09
openstackgerritA change was merged to openstack/ceilometer: Add pagination parameter to the database backends of storage  https://review.openstack.org/4258207:10
*** shang has quit IRC07:11
*** dina_belova has quit IRC07:13
*** eglynn has quit IRC07:22
openstackgerritLianhao Lu proposed a change to openstack/ceilometer: Added pollsters for the hardware agent  https://review.openstack.org/4307407:29
*** shang has joined #openstack-metering07:29
*** janisg has joined #openstack-metering07:32
*** shang has quit IRC07:39
*** shang has joined #openstack-metering07:39
*** shaneduan[afk] is now known as shaneduan07:56
*** eglynn has joined #openstack-metering08:06
*** shaneduan is now known as shaneduan[afk]08:10
*** Alexei_987 has joined #openstack-metering08:13
*** dina_belova has joined #openstack-metering08:20
*** dina_belova has quit IRC08:24
*** Fengqian has joined #openstack-metering08:28
*** Fengqian has quit IRC08:33
*** Fengqian has joined #openstack-metering08:35
*** Fengqian has quit IRC08:47
openstackgerritA change was merged to openstack/ceilometer: Extra indexes cleanup  https://review.openstack.org/4271608:48
*** Fengqian has joined #openstack-metering08:57
Fengqianhello08:57
Fengqiani have two questions about ceiolmeter pagination-query bp. One is the sort key of get_meters and another is get_resources of mongoDB08:58
Fengqianhttps://review.openstack.org/#/c/44278/08:59
FengqianA fake UUID for Meter, actually the combination of resource_id and counter_name, is it make sense here?09:00
*** dina_belova has joined #openstack-metering09:20
*** dina_belova has quit IRC09:25
jd__hey Fengqian09:29
jd__a lot of ping timeout :)09:30
jd__<jd__> if you want a unique id, message_id is a unique id  [10:44]09:30
jd__<jd__> so what your code is doing is useless09:30
jd__<jd__> that's the first point09:30
jd__a unique field is needed for pagination of samples09:30
jd__for the pagination of meters, there's no table with meters, the meter table stores _samples_ not meters (meters are  the counter_name field basically)09:30
Fengqianif we use message_id, how can users get this value from API level, when they using get_meters?09:32
FengqianThe model of Meter does not have message_id, right?09:32
jd__Fengqian: it wouldn't make sense09:39
jd__you have to build a unique value based on the fields present in the Meter model in the API09:40
jd__probably in the API, not in the storage driver09:40
jd__and split it to values you can paginate on in he API, and passes it on the storage drivers09:40
*** boris-42 has joined #openstack-metering09:45
jd__Fengqian: do I make sense?09:47
Fengqianyes, it is not the best solution that use resource_id + counter_name:).  But how can we do if we do not want to change database schema?09:49
jd__Fengqian: you can compoute that dynamically no?09:50
jd__I don't why you need to _store_ it09:50
Fengqianyes, we surely can compute it. The reason i store it because i think the user may need this value from API level if they do not want to compute it09:52
Fengqianotherwise, user have to compute it and pass in the value09:52
llujd__, I think we can dynamically compute 'resource_id + counter_name' for meter, but not message_id, message_id is only for sample, agreed?09:53
Fengqianyes, message_id is only for sample09:54
*** flwang has quit IRC09:54
Fengqiandynamically compute 'resource_id + counter_name',  i think that means we will add the compute part in API level, right?09:56
Fengqiani mean, if user do not compute that, we must do that part09:56
lluFengqian, user should never do the computing, the meter_id is opaque to them09:57
Fengqiansure:) i mean when GET/meters/limit=1?marker_value=******09:58
*** shaneduan[afk] is now known as shaneduan10:01
jd__llu: agreed10:03
jd__Fengqian: yes compute in the API part, that was my point10:03
*** fnaval_ has quit IRC10:05
*** fnaval_ has joined #openstack-metering10:06
Fengqianok, got it10:06
llujd__: one more thought about the pagination API. Do we want to expose the whole feature(marker/limit/sort_keys/sort_dirs) to the API user?10:09
lluor we just follow other openstack servier only expose (marker/limit) to the API user.10:09
*** fnaval_ has quit IRC10:10
*** shaneduan is now known as shaneduan[afk]10:10
lluexposing the full featured pagination  seems make the restful API complicated.10:11
jd__llu: I'd think expose everything10:11
jd__llu: it's _options_10:11
jd__:-)10:11
llujd__: ok10:12
*** Fengqian has quit IRC10:20
openstackgerritA change was merged to openstack/ceilometer: Fix wrong UniqueConstraint name  https://review.openstack.org/4271510:21
*** dina_belova has joined #openstack-metering10:21
jd__sileht: do you have an ongoing patch on the mongodb connection problem?10:22
silehtjd__, I have made some test with only netloc for the connection pool of mongo, it seems mongoclient create a new connection each time you request a db different than the one used at the creation of the MongoClient object10:23
jd__facepalm10:23
*** boris-42 has quit IRC10:24
*** dina_belova has quit IRC10:26
silehtjd__, I have read that in pymongo doc: 'applications in which more than 100 threads are in requests at once should disable the max_pool_size option'10:26
jd__yeah but we don't use the pool10:26
jd__(I think)10:26
silehtjd__, we use it10:26
jd__that's why we have our own pool10:27
jd__if we use it, it doesn't work then :-)10:28
silehtjd__, \o/10:31
silehtjd__, I have added max_pool_size=None and put maxConns=32 and it seems to work now10:32
silehtjd__, does this fix seem resonnable ?10:34
*** llu is now known as llu_away10:44
*** shang has quit IRC10:44
silehtjd__, max_pool_size=None workaround the problem but ~23000 connections are opened (and never closed) by pymongo10:55
* sileht out for lunch10:56
*** flwang has joined #openstack-metering10:59
*** shaneduan[afk] is now known as shaneduan11:02
*** shaneduan is now known as shaneduan[afk]11:11
*** dina_belova has joined #openstack-metering11:21
*** dina_belova has quit IRC11:26
*** boris-42 has joined #openstack-metering11:37
*** boris-42 has quit IRC11:40
jd__sileht: how can you have 23k open connections with maxConns=32?11:52
silehtjd__, 4211:53
jd__okay11:53
silehtjd__, it seems to be only TIME_WAIT connection ...11:55
jd__fair enough11:55
*** thomasm has joined #openstack-metering12:01
thomasmhey all!12:03
silehtjd__, do you think we must change max_pool_size=None for test only or not ?12:05
jd__sileht: I don't now yet, I'm reading pymongo12:06
silehtjd__, I suspected something with the thread handling of pymongo and eventlet12:08
jd__sileht: we had to remove use_greenlet=True recently12:08
*** yolanda has quit IRC12:09
silehtjd__, yes to let pymongo use the monkey patched threading module12:10
*** dina_belova has joined #openstack-metering12:22
*** dina_belova has quit IRC12:27
swannhi sileht, jd__ : We can ask Linux to recycle more quickly connections (in TIME_WAIT) by setting the sysctl param : net.ipv4.tcp_tw_recycle=112:38
swannif can help ..12:39
swannobviously, it's not resolve problem if it's connection leak.12:39
jd__swann: yeah, this isn't going to help much as we're not goint to be root to run the test suite :-)12:40
swannjd__: understandable12:41
*** boris-42 has joined #openstack-metering12:43
*** anteaya has joined #openstack-metering12:50
*** gordc has joined #openstack-metering12:52
*** bpokorny has joined #openstack-metering12:55
annegentle_go go go jd__12:57
annegentle_:)12:57
jd__hey annegentle_ !12:59
jd__did you see my question on -infra and do they make sense to you?12:59
annegentle_jd__: reading there too12:59
*** shaneduan[afk] is now known as shaneduan13:02
*** shaneduan is now known as shaneduan[afk]13:11
*** cflma has joined #openstack-metering13:14
*** cflma has quit IRC13:15
*** sandywalsh_ has joined #openstack-metering13:20
swannsileht, eglynn : Hi to you, I've may be an issue concerning the alarm with autoscaling stuff, I would like your feedback.13:22
*** dina_belova has joined #openstack-metering13:22
eglynnswann: what's the problem?13:23
swannI opened a bug for Heat https://bugs.launchpad.net/heat/+bug/1215840 and I pushed a patch https://review.openstack.org/#/c/43698/4 (not yet? aprouved)13:23
* eglynn looking13:23
swannwith this patch the ceilometer-alarm-singleton can't evaluate the alarm, more precisly this is the API which return a 400 HTTPBadRequest on the statistic query.13:23
*** sandywalsh has quit IRC13:24
swannhttp://paste.openstack.org/show/45465/13:24
*** dina_belova has quit IRC13:27
eglynnswann: one sec, I'll take a look ...13:28
swanneglynn: I've time :)13:28
*** flwang has quit IRC13:28
swanneglynn: basiclly, I add the stackId in alarm rules and it seems the API don't like caractere like ':' or  it's the alarm-evaluator which build a bad query13:32
eglynnswann: one sec, just finishing up something else13:32
*** litong has joined #openstack-metering13:39
*** flwang has joined #openstack-metering13:42
openstackgerritJulien Danjou proposed a change to openstack/ceilometer: doc: fix storage backend features status  https://review.openstack.org/4445313:47
*** shaneduan[afk] is now known as shaneduan13:47
*** boris-42 has quit IRC13:48
*** anteaya is now known as ColinMochrie13:50
eglynncheck first in mongo14:02
eglynn$ mongo ceilometer14:02
eglynn> db.meter.find({'counter_name': 'image.download'})14:03
eglynn(for example)14:03
eglynnsorry wrong channel for the above ^^^14:04
eglynnplease ignore14:05
litong@thomasm, ping14:14
thomasmlitong, hello!14:14
litongplease see this patchset. https://review.openstack.org/#/c/44390/14:14
litongI realize that with current code, I can not add the sort because the sort14:15
thomasmlitong, I just commented there. So, with the sort key set up, I can aggregate like in the mongo driver, correct?14:15
thomasmlitong, oh?14:15
litongbecause the latest code does not have sort_key and sort_dir in yet.14:15
litongwith what I added in that patchset, you can add these code I put out in your patchset.14:16
litongso they can go together.14:16
litong@thomasm, making sense?14:16
litong@thomasm, u there?14:18
thomasmlitong, yeah, sorry reading the patch14:19
thomasmlitong, So, I'm not sure I understand why we're checking for mongo in a db2 driver?14:19
litong@thomasm, actually the change will support sort.14:19
litong@thomasm, because of jenkins.14:19
litongwe do not have db2 on jenkins, so when test db2 driver, I use mongdo.14:20
litong@thomasm, that is the reason. If we never wanted to test db2 driver on jenkins, these code go away.14:20
thomasmlitong, So, this is just a temporary work around?14:20
litong@thomasm, as long as we do not have db2 on jenkins and we want to test the db2 driver, these code need to be in.14:21
thomasmlitong, Oh, okay14:21
litong@thomasm, this also is a work around for developers, and I bet not many of them has db2 on their machine.14:22
litongso these work around are important.14:22
thomasmlitong, What's confusing me is I see it implementing the sort keys for DB2 but not for mongo?14:23
*** dina_belova has joined #openstack-metering14:23
litong@thomasm, ?14:24
*** ColinMochrie is now known as anteaya14:25
thomasmlitong, I posted a comment on the line: https://review.openstack.org/#/c/44390/14:25
litong@thomasm, see the comments. we do not need sort when we get the resource ids from meter collection.14:28
*** dina_belova has quit IRC14:28
*** shaneduan is now known as shaneduan[afk]14:28
litong@thomasm, the sort is done when query resource collection.14:28
*** SergeyLukjanov has joined #openstack-metering14:29
thomasmlitong, Oh, I'm sorry - I misunderstood. :)14:29
*** alexb_ has quit IRC14:29
thomasmlitong, I see it now, heh. Thought that sort was a part of the query above. Eyes tricked me.14:29
litong@thomasm, yeah. the other reason I can not use the code for mongodb is that db2 has some limited aggreagte functions.14:30
litong@thomasm, for example, it does not have $first, $push, $addToSet implemented.14:30
thomasmlitong, That's concerning. Reason being, we need those to grab min/max and latest representation from the samples.14:31
litong@thomasm, so that the aggregate method can not be used, so I have to keep the original logic.14:31
thomasmBecause the resource collection has no concept of time, we need to use samples in order to filter latest representation within a time limit.14:31
thomasmlitong, Or, at the very least, add timestamp to the resource collection for last updated since messages can be out of order and we don't want to overwrite a newer state with an older one.14:32
eglynnswann: sorry, just getting back to this now14:32
litong@thomasm, changing schema right now is probably not a good idea.14:32
eglynnswann: k, so the problem is that the ceilo API service uses ast.literal_eval to interpret the qury parameters14:33
litong@thomasm, that will also causing changes to migration, I do not know if you want to do that now.14:33
eglynnswann: when you use a colon in the user metadata value, this confuses the AST14:33
thomasmlitong, Yeah, so I guess we'll have to do the aggregation manually, then?14:34
thomasmin the Python14:34
eglynnswann: (... as it's expecting that to have a certain meaning, e.g. as key:value separator in a dict definition)14:34
litong@thomasm, that is where the code and logic will differ for mongodb and db2.14:35
eglynnswann: so it's not an issue per se with the alarming logic14:35
eglynnswann: more just an undocumented restriction on queryable user metadata14:35
eglynnswann: (within the API service)14:35
thomasmlitong, Gotcha. Alright, so I can at least get the timestamp sorted data back. Thanks. Let me pull this down and see what I can do with it. =]14:36
eglynnswann: was there a particular reason that you needed the colon in the user metadata value?14:36
litong@thomasm, since the sort_key, sort_dir are not in yet, I can not really make these changes.14:36
litongI will here and let me know what I can do to help.14:37
litongas far as the sort goes, the code and logic are in that patch and I've tested last night, they all work against both mongdb and db2.14:37
*** tasdomas is now known as tasdomas_afk14:39
*** boris-42 has joined #openstack-metering14:40
thomasmlitong, Where do sort_key and sort_dir need to be added?14:40
eglynnswann: did that make sense?14:41
litong@thomasm, I thought you are adding these parameters to get_resources and get_meters method.14:41
litong@thomasm, the latest code in upstream does not have these in the method signature.14:42
swanneglynn: I understand the limitation of the api even I don't know what is AST ..14:43
thomasmlitong, No, I'm just correcting the drivers to return the latest resource representation - that's the bug I'm fixing. -- https://bugs.launchpad.net/ceilometer/+bug/120854714:43
eglynnswann: AST = "Abstract Syntax Tree"14:43
eglynnswann: http://docs.python.org/2/library/ast.html14:43
swanneglynn: the colon come from the stackId which is generated by Heat14:43
eglynnswann: can you sanitize this?14:44
eglynnswann: e.g. map ':' to '_'14:44
thomasmlitong, So, whenever somebody calls the function it needs to return the latest resource_metadata according to the timestamp. It was originally just undefined behavior based on how the driver handled grabbing non-aggregated samples from an aggregated query.14:44
*** sandywalsh_ has quit IRC14:44
thomasmlitong, This also means being able to do that within a start/end time period so people can get resource representation for a previous period of time.14:45
swanneglynn: don't know, I have to talk with heat guys if it's acceptable for them14:45
eglynnswann: k14:45
litong@thomasm, so get_resources and get_meters will have to sort regardless.14:45
litongthe sort keys will have to be what?14:46
thomasmlitong, timestamp, and their sample ID to tie-break.14:46
thomasmlitong, Not sure if we want to sort on anything else.14:47
litong@thomasm, the sample ID is just the _id, right?14:47
litong@thomasm, why do you want to sort on that?14:48
thomasmlitong, I believe so, yes.14:48
thomasmlitong, To tie break in cases where the timestamps are the same14:48
openstackgerritgordon chung proposed a change to openstack/ceilometer: add tests for _query_to_kwargs func  https://review.openstack.org/4379614:49
litong@thomasm, if I add the sort on timestamp and resource_id, will that satisfy your patch?14:49
thomasmlitong, I think so. Because the timestamps are far more precise in mongo, we should be fine.14:51
litong@sileht, ping14:51
litong@thomasm, you want to add that to db2 so that you have one patchset everything is together, or you want me to add that in a different patchset.14:52
litongI assume that you are adding the sort to mongodb.14:52
swanneglynn: I pretty sure it will be rejected to map ':' to '_' in Heat side, because I use in Heat template the special {'Ref' : 'AWS::StackId'} to build 'matching_rules' for the alarm and I have to introduce a new {'Ref' : StackIdWithoutColon} ..14:52
thomasmlitong, I already added it to mongo, so it should work when we use the same base here for sort keys.14:53
thomasmlitong, I just pulled the file down and I'm going to start trying to make it pass my test. :)14:53
thomasmlitong, So, I can't use .aggregate, really?14:54
litong@thomasm, ok, that is great. let me know if you want me to do or provide anything.14:54
litong@thomasm, you can with some limitations.14:54
thomasmlitong, Absolutely. Thanks for the help!14:54
swanneglynn: I'll see with angus. A solution will be to use only UUID of tenant and stack.14:55
swannand not ARN14:55
eglynnswann: yep, UUIDs would certainly be simpler14:55
swanneglynn: k14:55
thomasmlitong, Alright. Let me see what I can do. Is there any documentation on what I can and can't use for aggregate?14:56
litongfor db2 or for mongodb?14:56
thomasmdb215:00
*** dina_belova has joined #openstack-metering15:00
*** sandywalsh_ has joined #openstack-metering15:00
thomasmlitong, since I'm going to first try aggregation, I want to be sure I'm using supported features.15:01
litong@thomasm, in get_resources db2 impl, please do not use aggregate. since db2 aggreate does not have $first, $last, $push, $addToSet15:02
litong@thomasm, to use aggreagte to accomplish what mongodb did, you will need these, there is no way around that.15:03
litong@thomasm, so just keep what I have there by adding sort into find method.15:03
litong@thomasm, the sort is basically adding a sort parameter to find with a direction.15:04
thomasmlitong, Alright - I'll aggregate in the Python. That's going to be a lot of memory use for larger clouds, I think.15:08
thomasmI can yield it resource by resource, though.15:08
litong@thomasm, if you do that, cores may not like it.15:09
litong@thomasm, the impact probably will be quite big. I would prefer to use the method available on server side.15:09
litongin this case for db2, still try to use find with combination of sort, and use aggregate for mongodb.15:10
thomasmlitong, Then I don't see how this driver can support what we need. We shouldn't be using the resource collection because it has no regard to time.15:10
openstackgerritMehdi Abaakouk proposed a change to openstack/ceilometer: Disable the pymongo pooling feature  https://review.openstack.org/4446515:10
thomasmlitong, Samples can be stored out of order, and we're requesting the latest representation of each resource, further complicated by a time period filter if desired.15:11
litong@thomasm, in that case, you would use the timestamp in meter, you can sort there, I think.15:11
openstackgerritMehdi Abaakouk proposed a change to openstack/ceilometer: Disable the pymongo pooling feature  https://review.openstack.org/4446515:12
thomasmlitong, Hmm, I've never really worked with this before, so I'm mostly feeling my way around. That's probably why I'm confused.15:12
thomasmlitong, So, I can get the max timestamp for each resource's sample and pull that, then?15:12
thomasmsamples*15:13
litong@thomasm, I think you can do a search on meter with timestamp (whatever the way you like), get these resources ids, then15:13
litong@thomasm, using these resources ids to get all the resources from resource collection.15:13
litongin that case, the resource sorted by user_id, project_id.15:14
litongwhen you sort in meter collection, you sort by timestamp and other condition you may have.15:14
thomasmlitong, But that still ends us up with the same problem, the resource collection has  the latest stored sample for that resource, not necessarily the latest proper representation. So, filtering on a time period won't always return correct data. There are two problems:15:14
thomasmlitong, When storing the data, there is no timestamp in the resource collection to check against, so we could overwrite with an old state, and when retrieving we could be retrieving an old state, not the latest.15:15
thomasmlitong, So, that's why we have to pull that state from the samples, not the resource collection.15:16
thomasmlitong, unless we want to start storing historical metadata in the resource collection, or externalize the metadata altogether.15:16
thomasmlitong, regardless, the resource collection has no respect for time, so we can't trust it completely.15:17
thomasmlitong, Does that make sense?15:17
litong@thomasm, the only timestamp we have is from meter collection, so I am not sure if you are proposing a new field for resource?15:17
thomasmlitong, That's what I'm saying, that is how we have to filter, sort, and get the representation, that's the source of truth.15:18
thomasmlitong, the resource collection is not, with the current implementation.15:18
thomasmlitong, Say I'm filtering for the last week, saying I want to see what these resources were from 10 days ago to 3 days ago, the resource collection will give me what that resource is today, not what it was back then.15:19
litong@thomasm, in that case, I do not think we want to do that in H3, adding a field to resource table I thnk is a big deal.15:19
*** boris-42 has quit IRC15:20
thomasmlitong, And since the aggregation on the samples can't be done on DB2 yet, we either have to do it manually (possibly not performant) or we have to raise the NotImplementedError and just understand that we can't fully trust that function in DB2, unless I'm missing something.15:20
litong@thomasm, use find for db2, and use aggregate for mongodb.15:21
litongin any case, the only timestamp you have is from meter collection.15:22
litongthat is what you will have to play with regardless which method you use.15:22
thomasmlitong, Yeah, that's what I'm concerned about without DB side aggregation. :\15:22
litong@thomasm, find is done on the db side for db2.15:23
litong@thomasm, so is the sort.15:23
thomasmlitong, Sure, maybe I don't fully understand all of the things find can do.15:23
litongwhen you say aggregate, I assume that you mean db.meter.aggregate, not writing python code to do aggreation manually.15:24
thomasmlitong, Correct15:25
thomasmlitong, Which we can't do in DB2, it sounds like.15:25
thomasmlitong, It already works fine with the mongo driver15:25
litong@thomasm, you can with limitations.15:25
litong@thomasm, right, you are adding sort as far as I can tell.15:25
thomasmlitong, Does aggregate for DB2 support $max?15:26
litong@thomasm, yes, it does.15:27
litongit does support $min and $max, that is about it.15:27
litongit does not support $first, $last, or $push.15:27
thomasmlitong, So, what if I were to get each resource's max timestamp from the meter collection and then use that to find each resource's sample associated with that?15:27
thomasmlitong, So, it'd be a map of resource_id to it's max timestamp, and therefore it's latest sample.15:28
litongI would think you want to get a list of resource ids whose timestamp fall into the range that you want to, then get these resources from resource collection15:28
thomasmlitong, But that doesn't necessarily represent the latest state of that resource15:29
litong$max is fine, since you only need to get one unique resource id from meter.15:29
litong@thomasm, I thought that you want to use a range, between 10 days ago and 3 days ago.15:30
thomasmlitong, The resource may have changed since 3 days ago. So, what I'm looking for is what it was like then, and what I'll get from the resource collection is what it is now.15:31
litong@thomasm, oh, wow, you are looking for a history. I did not get that,15:33
litong@thomasm, if that is the case, I do not think you can do that.15:33
litongI donot think anywhere we keep a history of resource.15:34
litongif it is changed, then it is changed15:34
thomasmlitong, yeah. And, even if I'm not specifying a period of time, I could get an old state (because samples can be stored out of order) since there's no timestamp check when updating a resource in the collection.15:34
thomasmlitong, we do in the samples15:34
litongbut that is meter, not necessarily the resource.15:34
litong10 days ago, the resource may not be the resource 3 days ago, (I mean resource, not meter)15:35
thomasmlitong, each sample has a snapshot of the resource metadata and the same attributes we keep updating for that resource. That's why the mongo driver and sqlalchemy driver uses the meter collection/table instead of the resource one.15:35
dragondm^yup.15:36
thomasm:]15:37
dragondmthe way they do it is a little odd (fetching the resource data from the sample) but I can see why it's needed. If you want to fetch the state of a resource for billing, you want the state at the end of the billing period, not whatever it is now.15:37
thomasmyep15:39
thomasmlitong, So, if that's the case, should we just expect this not to be implemented for that driver until it's supported?15:40
thomasmlitong, If you check out my patch here: https://review.openstack.org/#/c/44277/15:41
thomasmthat's where I just said to raise the NotImplementedError since it can't really support it yet.15:41
thomasmlitong, And it just skips those tests until we can implement that piece. Until then we just can't fully trust that function.15:42
openstackgerritJulien Danjou proposed a change to openstack/ceilometer: doc: start the Ceilometer Guide  https://review.openstack.org/4447015:42
litong@thomasm, let me take a look.15:43
*** boris-42 has joined #openstack-metering15:43
openstackgerritJulien Danjou proposed a change to openstack/ceilometer: doc: start the Ceilometer Guide  https://review.openstack.org/4447015:43
*** fnaval_ has joined #openstack-metering15:43
openstackgerritJulien Danjou proposed a change to openstack/ceilometer: doc: start the Ceilometer Guide  https://review.openstack.org/4447015:43
litong@thomasm, you can not mark the entire method notimplemented.15:43
litongthat will cause problems for db2.15:44
*** fnaval_ has quit IRC15:44
litong@thomasm, whatever test case fails, you just need to add that test case to test_impl_db2 to overwrite.15:44
litongso that the jenkins will work ok.15:45
litong@thomasm, you mark the entire method not implemented, then when db2 is hooked up, nothing will work.15:45
*** fnaval_ has joined #openstack-metering15:45
thomasmlitong, Sure. That was the direction I was given since it can't really handle the base functionality of returning the trusted latest resource representation. I'm honestly fine with that, but ultimately it needs to be addressed.15:46
thomasmjd__, ping15:47
litong@thomas, let's say you add something that db2 driver fails, then you copy the test case in test_impl_db2.py, overwrite that method,15:47
litong@thomasm, what you are doing is killing the entire method.15:47
thomasmlitong, I understand. Like I said, the direction I was given was to raise that since the function doesn't satisfy the basic need of returning the latest resource representation within the time period. :\ I'm not familiar at all with the DB2 work, but I suppose the reason was to be sure we put constraints on what's considered a complete driver vs. not complete.15:49
thomasmlitong, Sorry. If the reviewers are fine with me putting passes in for those tests cases in the test_impl_db2.py, I'm alright with that. I already wrote up the bug, so we can address it.15:49
litongthe problem when you do what you did, you invalidate the entire method which has worked for serve other purposes.15:49
dragondmIt is a bug, which will need to be fixed in the db2 driver.15:49
*** boris-42 has quit IRC15:50
litong@dragondm, a method can provide many functions, it is ok until you add something break that, then you can not simply mark the entire method not implemented.15:51
dragondmactually the method does not work. It returns incorrect data.15:51
dragondmthomas' patch is to fix the bug for other drivers.15:51
jd__thomasm: pong15:51
litong@dragondm, that is correct, but that fix made the entire db2 method not implemented.15:52
dragondmAs far as raising in the method itself, vs putting some kind of skip in the tests, that's a ptl call.15:52
*** boris-42 has joined #openstack-metering15:52
thomasm^^ that's why15:52
jd__litong: either the method works and passes tests, either it doesn't pass tests; if it doesn't it needs to be fixed, if we can't fix it, it must just raises NotImplemented15:52
jd__thomasm: I think litong proposed a fix you can merge right?15:52
litong@jd__, it works ok, until something new added to break it. when that happens, if you mark entire method not implemented, then we got15:53
litongproblem,15:53
litong@jd__, ibm is pulling ceilometer code into its product, we can not mark some methods not implemented.15:53
jd__litong: then IBM need to fix the driver to match what the API should return15:54
litong@jd__, someone early marked get_meter_statistics method entire notImplemented. I am dealing with this exactly problem now.15:54
jd__litong: we don't want to have drivers returning data to the API that are not correct15:55
litong@jd__, IBM is trying to provide full mongodb functions in DB2, but it takes time,15:55
jd__thomasm: I get that15:55
litongif the backend does not provide some methods, the client provided function is limited,15:55
jd__litong: agreed, but in this case basic assumptions were broken15:55
thomasmThe sort patch doesn't really address the actual problem. I have a sort key, but sorting in a resource collection doesn't really solve it - it's not holding historical data, nor is it the source of truth for the latest representation within a time period.15:56
litong@jd__, but what put in are working, it breaks when Thomas changed something and without using db2 specific methods/work around15:56
dragondmYah, it's understandable that that is a bind, but thomas' patch is not adding new functionality. It's fixing a bug that produces incorrect output. We need to mark it somehow that "this doesn't actually work"15:56
jd__litong: I don't want to be rude or anything, you just need to provide a fix on top of thomasm's patch so he can use it, there's no middle ground here15:57
thomasmlitong, dragondm, jd__: As I understand it, the initial implementation wasn't based on the intended behavior.15:57
litong@thomasm, not true,15:58
litongit does return the resources existed in the db.15:58
dragondmNot the right one, tho.15:58
litonglet's say you simply want to get resources.15:58
jd__I don't negotiate driver comformance to the API15:58
jd__:-)15:58
thomasmlitong, The intended behavior is to return the latest representation of a resource for a time period, meaning it needs to support historical data, also it needs to respect time.15:58
thomasmlitong, why would you want to without respect for what that resource is or what it was?16:02
litong@thomas, you simply want to get a list of resources and then you do whatever with them as you please.16:03
litong@thomasm , remember we only provide API and most likely it is the application calling these apis and do things with the results.16:04
thomasmlitong, What's the point of getting a resource_id and nothing else?16:04
thomasmBecause that's the only thing that isn't subject to change, it seems like.16:04
dragondmyup, which is why it's an issue. As is, if you use the api for fetching billing info (for example), you will bill on the wrong thing. The metadata will be incorrect.16:05
dragondm(if the backend is db2)16:05
thomasmyep ^^16:05
thomasmlitong, that's where the problem lies16:06
litong@thomasm, that is where problem occurs when you mark the entire method not implemented.16:06
litong@Thomassm, it is returning at least id, now it says not implemented.16:07
litongtell me they are the same thing.16:07
*** bpokorny has quit IRC16:08
thomasmlitong, I don't understand why you would want a resource_id and nothing else. The purpose of that function is to return a representation of resources based on some criteria. Since that representation is subject to change, the lack of respect for time invalidates its use.16:08
thomasmlitong, a resource_id means nothing without the attributes regarding it.16:09
*** bpokorny has joined #openstack-metering16:09
thomasmIt's just a key. If the value(s) change over time, there's nothing you can do without being able to know what time they were that way.16:09
litong@thomasm, v2/resources works, now fails.16:09
thomasmlitong, but it doesn't work as intended.16:10
thomasmlitong, Which means people may see a difference in what's returned going from, say, mongodb to db2.16:10
litong@thomasm, if intended is to specify conditions, then it should be v2/resource?some_conditions.16:10
*** dina_belova has quit IRC16:11
thomasmlitong, The default conditions aren't satisifed with this current implementation either.16:11
thomasmlitong, That's why16:11
*** dina_belova has joined #openstack-metering16:11
litong@thomasm, either you fix the db2 driver or I fix it, but simply mark it entire not implemented is not good16:12
litong@thomasm, that is what I am saying.16:12
jd__litong: we all know that16:13
jd__time would be better spent fixing it than talking about it16:13
jd__:)16:14
litong@jd__, agreed, just do not like to simply mark a method not implemented, that cause a big head for IBM. WE ARE PULLING CEILOMETER to our product.16:14
jd__litong: I hope you realize this isn't an argument?16:15
thomasmlitong, the tests I added were to verify an expected behavior from the default conditions. The expected behavior wasn't originally in the tests, but since I was addressing the observed problem I wanted to verify that it worked in all drivers going forward.16:15
jd__it will get you nowhere arguing that16:15
openstackgerritFengqian.gao proposed a change to openstack/ceilometer: Add pagination query support for get_samples  https://review.openstack.org/4449216:15
openstackgerritFengqian.gao proposed a change to openstack/ceilometer: Add pagination support for sqlalchemy database  https://review.openstack.org/3545416:15
openstackgerritFengqian.gao proposed a change to openstack/ceilometer: Add pagination support to selected APIs  https://review.openstack.org/3745416:15
openstackgerritFengqian.gao proposed a change to openstack/ceilometer: Change pagination query method in mongodb  https://review.openstack.org/4186916:15
*** dina_belova has quit IRC16:16
thomasmlitong, What I'm trying to do is be sure we document what these drivers are supposed to do in the base tests. That will always cause some issues as we try to align things, but it's important to do so we can confidently say what our API will and will not do.16:19
thomasmThe tests will pass because they just skip when a NotImplementedError is thrown, but yes, if this is pulled down on a DB2 backend, v2/resources would fail.16:21
thomasmUntil it's corrected16:21
flwanggordc: can you please revisit https://review.openstack.org/#/c/41818/ again? thanks16:23
*** dina_belova has joined #openstack-metering16:23
gordcflwang: sure. i'll take a look at it after i look at another patch.16:24
flwanggordc: thanks, appreciate it16:24
*** shaneduan[afk] is now known as shaneduan16:25
*** dina_belova has quit IRC16:47
*** dina_belova has joined #openstack-metering16:48
*** fnaval_ has quit IRC16:48
*** dina_belova has quit IRC16:52
*** dina_belova has joined #openstack-metering16:53
*** fnaval_ has joined #openstack-metering17:03
*** shardy is now known as shardy_afk17:21
openstackgerritA change was merged to openstack/ceilometer: Fix empty metadata issue of instance  https://review.openstack.org/4181817:31
*** dina_belova has quit IRC17:37
*** dina_belova has joined #openstack-metering17:38
*** Alexei_987 has quit IRC17:41
*** dina_belova has quit IRC17:42
*** sandywalsh_ has quit IRC17:43
*** fnaval_ has quit IRC17:48
*** sandywalsh_ has joined #openstack-metering17:56
*** dina_belova has joined #openstack-metering17:58
*** eglynn has quit IRC18:01
*** gordc has quit IRC18:04
*** gordc has joined #openstack-metering18:11
*** eglynn has joined #openstack-metering18:36
*** eglynn has quit IRC18:41
openstackgerritlitong01 proposed a change to openstack/ceilometer: db2 distinct call results are different from mongodb call.  https://review.openstack.org/4451418:46
*** litong has quit IRC18:54
openstackgerritSandy Walsh proposed a change to openstack/ceilometer: Reject duplicate events  https://review.openstack.org/4419119:00
*** dina_belova has quit IRC19:09
*** dina_belova has joined #openstack-metering19:10
*** dina_belova has quit IRC19:12
*** dina_belova has joined #openstack-metering19:13
*** SergeyLukjanov has quit IRC19:13
*** SergeyLukjanov has joined #openstack-metering19:15
*** boris-42 has quit IRC19:17
*** eglynn has joined #openstack-metering19:25
*** dina_belova has quit IRC19:27
*** dina_belova has joined #openstack-metering19:27
*** dina_belova has quit IRC19:28
*** dina_bel_ has joined #openstack-metering19:28
*** SergeyLukjanov has quit IRC19:30
*** eglynn has quit IRC19:30
*** thomasm has quit IRC19:35
*** SergeyLukjanov has joined #openstack-metering19:38
*** bpokorny has quit IRC19:43
*** openstack has joined #openstack-metering19:44
*** bpokorny has joined #openstack-metering19:44
*** eglynn has joined #openstack-metering19:44
openstackgerritSvetlana Shturm proposed a change to openstack/ceilometer: Fix wrong migrations  https://review.openstack.org/4453920:08
*** dina_bel_ has quit IRC20:30
*** dina_belova has joined #openstack-metering20:30
*** gordc has quit IRC20:31
*** SergeyLukjanov has quit IRC20:34
*** dina_belova has quit IRC20:35
*** shaneduan is now known as shaneduan[afk]20:42
*** shaneduan[afk] is now known as shaneduan20:45
*** thomasm has joined #openstack-metering20:56
*** eglynn has quit IRC20:56
*** shaneduan is now known as shaneduan[afk]21:02
openstackgerritThomas Maddox proposed a change to openstack/ceilometer: Fix to return latest resource metadata  https://review.openstack.org/4427721:05
*** eglynn has joined #openstack-metering21:08
openstackgerritThomas Maddox proposed a change to openstack/ceilometer: Fix to return latest resource metadata  https://review.openstack.org/4427721:09
*** openstackgerrit has quit IRC21:25
*** openstackgerrit has joined #openstack-metering21:25
*** shaneduan[afk] is now known as shaneduan21:26
*** thomasm has quit IRC21:40
*** dina_belova has joined #openstack-metering21:41
*** dina_belova has quit IRC21:45
*** eglynn has quit IRC21:47
*** bpokorny has quit IRC21:50
openstackgerritRussell Bryant proposed a change to openstack/ceilometer: Sync rpc from oslo-incubator  https://review.openstack.org/4456122:18
*** shakayumi has joined #openstack-metering22:26
*** alexb_ has joined #openstack-metering22:49
*** shanewang1 has joined #openstack-metering23:05
*** shanewang1 has left #openstack-metering23:05
*** alexb_ has quit IRC23:20
*** zul has joined #openstack-metering23:36
*** yjiang5 is now known as bmqq12323:42

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!