| openstackgerrit | Sandy Walsh proposed a change to openstack/ceilometer: Reject duplicate events https://review.openstack.org/44191 | 00:17 |
|---|---|---|
| *** herndon has quit IRC | 00:20 | |
| *** krtaylor has quit IRC | 00:25 | |
| *** krtaylor has joined #openstack-metering | 00:33 | |
| *** shaneduan is now known as shaneduan[afk] | 00:33 | |
| *** shardy has quit IRC | 01:08 | |
| *** fnaval_ has joined #openstack-metering | 01:19 | |
| *** fnaval_ has quit IRC | 01:19 | |
| *** fnaval_ has joined #openstack-metering | 01:20 | |
| *** d34dh0r53 has joined #openstack-metering | 01:59 | |
| *** d34dh0r53 has quit IRC | 02:02 | |
| *** anteaya has quit IRC | 02:05 | |
| *** zul has quit IRC | 02:20 | |
| *** thomasm has joined #openstack-metering | 02:22 | |
| *** d34dh0r53 has joined #openstack-metering | 02:28 | |
| *** d34dh0r53 has quit IRC | 02:33 | |
| openstackgerrit | A change was merged to openstack/ceilometer: Adds group by statistics for MongoDB driver https://review.openstack.org/43043 | 02:42 |
| *** SergeyLukjanov has joined #openstack-metering | 02:50 | |
| *** thomasm has quit IRC | 02:54 | |
| *** SergeyLukjanov has quit IRC | 03:34 | |
| *** dina_belova has joined #openstack-metering | 03:54 | |
| *** dina_belova has quit IRC | 03:56 | |
| *** dina_belova has joined #openstack-metering | 03:57 | |
| *** dina_belova has quit IRC | 04:01 | |
| *** boris-42 has joined #openstack-metering | 04:30 | |
| *** sandywalsh has quit IRC | 04:37 | |
| openstackgerrit | Lianhao Lu proposed a change to openstack/ceilometer: Adding hardware-agent https://review.openstack.org/43072 | 04:49 |
| openstackgerrit | Lianhao Lu proposed a change to openstack/ceilometer: Added hardware agent's inspector and snmp implementation https://review.openstack.org/43073 | 04:49 |
| *** dina_belova has joined #openstack-metering | 05:07 | |
| *** sandywalsh has joined #openstack-metering | 05:07 | |
| *** SergeyLukjanov has joined #openstack-metering | 05:11 | |
| *** dina_belova has quit IRC | 05:12 | |
| *** SergeyLukjanov has quit IRC | 05:13 | |
| *** SergeyLukjanov has joined #openstack-metering | 05:16 | |
| *** eglynn has joined #openstack-metering | 05:16 | |
| *** SergeyLukjanov has quit IRC | 05:21 | |
| *** SergeyLukjanov has joined #openstack-metering | 05:23 | |
| *** SergeyLukjanov has quit IRC | 05:32 | |
| *** SergeyLukjanov has joined #openstack-metering | 05:48 | |
| *** SergeyLukjanov has quit IRC | 05:53 | |
| *** SergeyLukjanov has joined #openstack-metering | 05:56 | |
| *** SergeyLukjanov has quit IRC | 06:02 | |
| openstackgerrit | Jenkins proposed a change to openstack/ceilometer: Imported Translations from Transifex https://review.openstack.org/44216 | 06:04 |
| *** flwang has quit IRC | 06:38 | |
| *** infiniti has quit IRC | 06:45 | |
| *** flwang has joined #openstack-metering | 06:59 | |
| llu | sileht: ping | 07:18 |
| sileht | llu, pong | 07:20 |
| llu | sileht: about your comment of patch https://review.openstack.org/#/c/43074/, what's special about the NotImplementedError? | 07:21 |
| llu | jd__: about the devstack patch of changing mongodb to mysql, is that ok even mysql still doesn't support metadata query? | 07:22 |
| *** evanjfraser_ has quit IRC | 07:23 | |
| sileht | llu, I'm just afraid if a inspector doesn't Implement a pollster, I will have my log file growing quickly with Exception related to something I can fix (as operator), I'm sure supervision guys take care of Exception message, and don't take care about level like INFO ? | 07:25 |
| *** yolanda has joined #openstack-metering | 07:25 | |
| sileht | llu, for now; it's a detail, we have only one inspector and it implements all pollsters, so | 07:27 |
| llu | sileht: ok. What if I change the inspector base's methods into abstract method, so illegal inspectors can not be loaded at start time | 07:28 |
| llu | sileht: actually, we can have multiple inspectors, it's defined in the pipeline as in the 'resources', e.g. snmp://10.1.1.1 means using inspector snmp to poll from host 10.1.1.1 | 07:29 |
| sileht | llu, this just force the new inspect to raise NotImplementedError itself, we have already done that for db backend | 07:30 |
| sileht | llu, when I say one inspector, this is for one inpector class not instance of it | 07:31 |
| sileht | llu, I like the new resources: in the pipeline | 07:31 |
| *** SergeyLukjanov has joined #openstack-metering | 07:32 | |
| llu | sileht, ok, i'll change the severity to INFO for NotImplementedError | 07:32 |
| llu | sileht: that idea is from jd__ | 07:33 |
| *** shardy has joined #openstack-metering | 07:33 | |
| *** SergeyLukjanov has quit IRC | 07:33 | |
| *** SergeyLukjanov has joined #openstack-metering | 07:36 | |
| llu | jd__: about the devstack patch to have mysql as default DB. Is it ok even if sqlalchemy doesn't support metadata query? | 07:39 |
| llu | oops, I've already asked. Jammed head. | 07:39 |
| *** SergeyLukjanov has quit IRC | 07:51 | |
| *** eglynn has quit IRC | 07:55 | |
| *** SergeyLukjanov has joined #openstack-metering | 07:58 | |
| *** SergeyLukjanov has quit IRC | 07:59 | |
| openstackgerrit | Lianhao Lu proposed a change to openstack/ceilometer: Added pollsters for the hardware agent https://review.openstack.org/43074 | 08:00 |
| jd__ | hi | 08:09 |
| *** eglynn has joined #openstack-metering | 08:25 | |
| *** SergeyLukjanov has joined #openstack-metering | 08:27 | |
| *** SergeyLukjanov has quit IRC | 08:54 | |
| *** evanjfraser has joined #openstack-metering | 08:59 | |
| *** evanjfraser has quit IRC | 09:07 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/ceilometer: Import middleware from Oslo https://review.openstack.org/44247 | 09:13 |
| openstackgerrit | Julien Danjou proposed a change to openstack/ceilometer: alarm: generate alarm_id in API https://review.openstack.org/44065 | 09:13 |
| *** flwang has quit IRC | 09:17 | |
| openstackgerrit | A change was merged to openstack/ceilometer: Adds else and TODO in statistics storage tests https://review.openstack.org/43993 | 09:18 |
| *** evanjfraser has joined #openstack-metering | 09:33 | |
| *** flwang has joined #openstack-metering | 09:43 | |
| *** evanjfraser has quit IRC | 09:49 | |
| openstackgerrit | Fei Long Wang proposed a change to openstack/ceilometer: Fix empty metadata issue of instance https://review.openstack.org/41818 | 09:54 |
| openstackgerrit | A change was merged to openstack/ceilometer: Import middleware from Oslo https://review.openstack.org/44247 | 10:50 |
| *** yolanda has quit IRC | 11:36 | |
| *** yolanda has joined #openstack-metering | 11:38 | |
| openstackgerrit | Fei Long Wang proposed a change to openstack/ceilometer: Fix empty metadata issue of instance https://review.openstack.org/41818 | 11:38 |
| *** thomasm has joined #openstack-metering | 11:59 | |
| thomasm | Hey all! | 12:02 |
| *** SergeyLukjanov has joined #openstack-metering | 12:15 | |
| openstackgerrit | Thomas Maddox proposed a change to openstack/ceilometer: Addresses the latest resource metadata not being returned in the MongoDB, SQLAlchemy, and HBase drivers. The DB2 driver appears to be lacking support for grouping on the meter collection, so that was written up for someone more familiar with the DB2 NoSQL https://review.openstack.org/44277 | 12:19 |
| *** SergeyLukjanov has quit IRC | 12:20 | |
| thomasm | Ahhh? didn't realize it wouldn't wrap my text on the site. I apologize in advance. :( | 12:21 |
| thomasm | jd__: Review up for the metadata bug fix: https://review.openstack.org/#/c/44277/ | 12:24 |
| thomasm | Anyone I ought to add specifically? | 12:24 |
| *** Ruetobas has joined #openstack-metering | 12:29 | |
| *** jergerber has joined #openstack-metering | 12:30 | |
| openstackgerrit | Thomas Maddox proposed a change to openstack/ceilometer: Fix to return latest resource metadata https://review.openstack.org/44277 | 12:36 |
| openstackgerrit | Thomas Maddox proposed a change to openstack/ceilometer: Fix to return latest resource metadata https://review.openstack.org/44277 | 12:40 |
| *** Ruetobas has quit IRC | 12:42 | |
| *** sandywalsh has quit IRC | 12:46 | |
| openstackgerrit | Fengqian.gao proposed a change to openstack/ceilometer: Add a fake UUID to Meter database model named meter_id https://review.openstack.org/44278 | 12:46 |
| openstackgerrit | Fengqian.gao proposed a change to openstack/ceilometer: Add pagination parameter to the database backends of storage https://review.openstack.org/42582 | 12:46 |
| openstackgerrit | Fengqian.gao proposed a change to openstack/ceilometer: Add pagination support to selected APIs https://review.openstack.org/37454 | 12:46 |
| openstackgerrit | Fengqian.gao proposed a change to openstack/ceilometer: Change pagination query method in mongodb https://review.openstack.org/41869 | 12:46 |
| *** Ruetobas has joined #openstack-metering | 12:49 | |
| *** anteaya has joined #openstack-metering | 12:51 | |
| *** fnaval_ has quit IRC | 12:54 | |
| *** fnaval_ has joined #openstack-metering | 12:55 | |
| *** shanewang has joined #openstack-metering | 12:56 | |
| *** shanewang has left #openstack-metering | 12:56 | |
| *** fnaval_ has quit IRC | 12:59 | |
| *** bpokorny has joined #openstack-metering | 13:01 | |
| *** sandywalsh has joined #openstack-metering | 13:01 | |
| *** flwang has left #openstack-metering | 13:03 | |
| *** gordc has joined #openstack-metering | 13:21 | |
| *** fnaval_ has joined #openstack-metering | 13:21 | |
| *** changbl has quit IRC | 13:43 | |
| *** herndon has joined #openstack-metering | 13:49 | |
| *** herndon has quit IRC | 13:52 | |
| *** janisg has quit IRC | 14:19 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/ceilometer: alarm api: rename counter_name to meter_name https://review.openstack.org/44292 | 14:20 |
| openstackgerrit | Julien Danjou proposed a change to openstack/ceilometer: alarm: generate alarm_id in API https://review.openstack.org/44065 | 14:20 |
| *** flwang has joined #openstack-metering | 14:21 | |
| flwang | jd__: ping | 14:23 |
| jd__ | ah caught! | 14:23 |
| jd__ | flwang: pong | 14:23 |
| flwang | jd__: lol, could you please revisit my patch https://review.openstack.org/#/c/41818/ ? it had been approved by you | 14:24 |
| jd__ | sure | 14:24 |
| jd__ | don't worry I review twice a day | 14:24 |
| flwang | jd__: cool, you're the super man | 14:24 |
| *** herndon has joined #openstack-metering | 14:26 | |
| flwang | dhellmann_: ping | 14:26 |
| *** herndon has quit IRC | 14:29 | |
| *** dhellmann_ is now known as dhellmann | 14:29 | |
| dhellmann | flwang: pong | 14:29 |
| dhellmann | jd__: I'm starting to worry about the storage drivers drifting apart feature wise. sandywalsh's change to reject repeated events made me think maybe we should be tracking those differences as bugs. What do you think? | 14:30 |
| *** janisg has joined #openstack-metering | 14:30 | |
| jd__ | that sounds fair to me | 14:31 |
| flwang | I'm running a weird issue, for resource show, if I specified a non-existed id and set the CM debug level as False, CM will return 500 instead of 400 | 14:31 |
| flwang | any thought? | 14:31 |
| sandywalsh | dhellmann, we were talking about that internally yesterday too. thomasm has a bug fix that's pulling him into the hbase code and that's not our wheelhouse. | 14:31 |
| dhellmann | jd__: we ask devs who make inconsistent changes to open the bugs? or expect maintainers to keep up? | 14:31 |
| jd__ | flwang: bug? | 14:31 |
| jd__ | dhellmann: I expect maintainer to keep up | 14:31 |
| dhellmann | sandywalsh: yeah, I don't think it's fair to expect everyone to be able to update all the drivers | 14:31 |
| jd__ | dhellmann: I don't think adding burden on developers about DB2 and HBase is likely to help | 14:31 |
| jd__ | nor fair | 14:32 |
| jd__ | and I don't want to drive down the development because of that neither | 14:32 |
| dhellmann | flwang: it sounds like something is causing an exception with that configuration | 14:32 |
| sandywalsh | the problem he was running into was the tests were failing under hbase, so he had to do something. | 14:32 |
| *** alexb_ has joined #openstack-metering | 14:32 | |
| dhellmann | jd__: but as a deployer, how do *I* know which features are in which driver? | 14:32 |
| jd__ | dhellmann: we should document it | 14:32 |
| sandywalsh | if there was some way to say "don't run this test under <hbase>" that would be awesome | 14:32 |
| dhellmann | jd__: right, and who is "we"? that's why I was saying the dev should open a bug | 14:33 |
| jd__ | dhellmann: documenting drivers features could be a blocker for Havana if you ask me | 14:33 |
| dhellmann | for example, maybe sandywalsh could open bugs against the other drivers for this new feature | 14:33 |
| sandywalsh | dhellmann, sure thing | 14:33 |
| jd__ | dhellmann: well I can write documentation if I'm forced too | 14:33 |
| dhellmann | haha | 14:33 |
| dhellmann | thanks, sandywalsh | 14:33 |
| thomasm | sandywalsh, dhellmann, jd__: We can just break out the Mixin that tells the DB scenarios to run on all backends and just mix in the ones we need for each set of tests? | 14:33 |
| dhellmann | thomasm: yeah, exactly | 14:33 |
| jd__ | what problem does that solve? | 14:34 |
| dhellmann | it avoids failing tests for a feature that is not in all drivers | 14:34 |
| thomasm | dhellmann, sandywalsh, jd__: The ultimate problem is that all of the drivers don't support the features we are advertising. | 14:34 |
| jd__ | dhellmann: we already skip on NotImplementedError | 14:34 |
| dhellmann | oh, ok | 14:34 |
| thomasm | dhellmann, sandywalsh, jd__: Because we didn't have full testing around the required features, things like ordering weren't being supported. | 14:34 |
| jd__ | that was I was asking which problem you thought it would fix | 14:34 |
| dhellmann | I didn't realize we did that | 14:35 |
| jd__ | I wish that could extract us automatic documentation | 14:35 |
| jd__ | but for the time being I can write it manually | 14:35 |
| sandywalsh | but in this case, it was implemented and started failing once the semantics of the feature changed. | 14:35 |
| thomasm | dhellmann, sandywalsh, jd__: So, if we are going to allow drivers to exist that aren't full-featured with the most used ones, then we have to work around it. | 14:35 |
| *** alexb_ has quit IRC | 14:35 | |
| jd__ | thomasm: we already do that a lot I think | 14:35 |
| thomasm | jd__, Yep, and the reason I started making changes to the other drivers was because I wanted that to stop. | 14:36 |
| thomasm | heh | 14:36 |
| jd__ | what changes are you talking about? | 14:36 |
| thomasm | jd__, https://review.openstack.org/#/c/44277/ | 14:36 |
| flwang | dhellmann: http://paste.openstack.org/show/45378/ | 14:37 |
| thomasm | The driver is supposed to return filtered resources with the latest attributes from that filtering | 14:37 |
| thomasm | HBase couldn't support that because it wasn't storing historical metadata | 14:37 |
| dragondm | I think it's more a case of "I need to fix an issue with db backends X, Y, Z, and I know nothing about backends Q and Z, I need a way to merge my tests for the first ones and not run the test on the last two til' someone who knows about it can fix those" | 14:37 |
| thomasm | DB2 doesn't support that because it uses the resource collection which doesn't have historical metadata | 14:37 |
| flwang | you can just set the Debug as False, and then query a non-existed resource | 14:38 |
| jd__ | thomasm: in this case you've 2 options, the better being try to do thinsg in Python, the other one being raise NotImplementedError and that's it | 14:38 |
| flwang | dhellmann: recreate steps | 14:38 |
| jd__ | thomasm: and ping the DB2/HBase authors so that they may be able to help eventually | 14:39 |
| thomasm | dhellmann, jd__, sandywalsh: If we're okay with some drivers falling behind like that, then we would have to go a work around, like dragondm mentioned. | 14:39 |
| thomasm | jd__, How can I do that when the fundamental purpose of that function is flawed? | 14:39 |
| jd__ | thomasm: you raise at the start of the method? | 14:39 |
| dhellmann | dragondm: I think the NotImplementedError solution is nice and explicit, and handles the tests, too | 14:39 |
| thomasm | jd__, Oh, you mean just for the get_resources function just raise that? | 14:40 |
| jd__ | the question boils down to "is it better to ship not working drivers or bugged drivers?" | 14:40 |
| thomasm | jd__, Exactly, yeah. | 14:40 |
| dhellmann | thomasm: well, I don't *like* it but I really don't see a way to avoid them drifting apart | 14:40 |
| thomasm | dhellmann, haha, yeah :( | 14:40 |
| jd__ | if we ship non working driver we have a chance that people come up and fixes it, if we release bugged drivers with no test, nobody's going to notice | 14:40 |
| dhellmann | jd__: bugs should be fixed, but "not feature complete" seems a reasonable compromise | 14:40 |
| jd__ | dhellmann: well, question is how do you fix a bug if you don't know/have HBase/DB2? | 14:41 |
| *** alexb_ has joined #openstack-metering | 14:42 | |
| thomasm | jd__, I can see the bug in the code anyway. The problem is the one reference those two drivers have to the metadata isn't an honest source anyway. It's completely dependent on your architecture and the nature of network comms. | 14:42 |
| jd__ | I've opened a bug for doc https://bugs.launchpad.net/ceilometer/+bug/1218419 | 14:42 |
| thomasm | dhellmann, sandywalsh ^^ | 14:42 |
| thomasm | dragondm, ^^ | 14:42 |
| jd__ | thomasm: well the test case is the same for all drivers anyway, right? | 14:43 |
| thomasm | jd__, Yeah, it stores the messages out of order, and all drivers would have failed that. I fixed SQLAlchemy and MongoDB without a schema change, HBase wasn't storing historical so that would fail anyway. | 14:43 |
| thomasm | That's why I changed HBase to start storing historical metadata | 14:44 |
| thomasm | along with the other historical attributes in the samples | 14:44 |
| *** alexb_ has quit IRC | 14:44 | |
| dhellmann | jd__: well, bugs should be fixed by the team, not every individual developer | 14:45 |
| dhellmann | jd__: I meant we should try to avoid releasing stuff we know is buggy, but if it's just not complete I think we have to live with and document that case | 14:45 |
| jd__ | thomasm: cool then | 14:45 |
| dhellmann | thomasm: yeah, that sounds like the way to fix it | 14:46 |
| jd__ | dhellmann: well I think we agree, it's just that sometimes you can't fix the bugs | 14:46 |
| dhellmann | jd__: hence "try" :-) | 14:46 |
| *** changbl has joined #openstack-metering | 14:47 | |
| jd__ | :-) | 14:47 |
| thomasm | jd__, dhellmann, I'd never really worked with any of these before (SQLAlchemy, PyMongo, HappyBase, etc?). So, I just did what made sense to me, and the DB2 stuff, according to your comment, jd__, was missing some features that were really necessary to fix the bug. | 14:47 |
| thomasm | jd__, dhellmann: So, I chatted with gordc and we got it written up and sent to some folks who knew far more about it than me. | 14:47 |
| thomasm | So? should I err on the side of raising the NotImplementedError, then? | 14:49 |
| thomasm | I can make that change | 14:49 |
| thomasm | And push up a new patch set without a mixin change | 14:49 |
| jd__ | I don't know, dhellmann says the solution is to "try" :) | 14:49 |
| *** alexb_ has joined #openstack-metering | 14:49 | |
| thomasm | Oh, try to fix the DB2 stuff? | 14:49 |
| jd__ | but my call is raising NotImplementedCorrectly | 14:49 |
| jd__ | :-) | 14:49 |
| jd__ | thomasm: yes | 14:50 |
| thomasm | I had a few go's at it. Hmmm, do you know of any documentation on what is supported by PyMongo for DB2 NoSQL? | 14:50 |
| thomasm | jd__, dhellmann ^^ | 14:50 |
| thomasm | I've had a hell of a time trying to find any | 14:51 |
| gordc | thomasm: i've passed on the bug to litong, regarding documentation you'll be hardpressed to find it... it's a very very new feature db2 is working on. | 14:52 |
| thomasm | gordc, Cool. I'll follow up there. =] | 14:54 |
| *** shakayumi has joined #openstack-metering | 14:54 | |
| gordc | thomasm: is it blocking you or can you bypass it by just raising a notimplemented? | 14:55 |
| *** litong has joined #openstack-metering | 14:55 | |
| dhellmann | thomasm: yeah, raise NotImplemented in cases where you can detect that something needs to be added to the driver | 14:57 |
| thomasm | gordc, dhellmann: Okay, awesome. Thanks for the help! =] | 14:57 |
| dhellmann | thomasm: unfortunately I don't know anything about the mongo stuff in db2 -- that was a surprise to me | 14:57 |
| thomasm | gordc, Nope, not blocking, then. | 14:57 |
| thomasm | dhellmann, haha me too | 14:57 |
| litong | @dhellmann, @thomasm, | 14:59 |
| litong | I think you are adding more test cases to the base, right? | 14:59 |
| thomasm | litong, yep | 14:59 |
| litong | db2 test cases basically inherit from all the base test cases. | 14:59 |
| thomasm | litong, Yeah, so it fails these tests | 15:00 |
| litong | the easy way is to add a new empty test cases in test_impl_db2.py to simply pass. | 15:00 |
| openstackgerrit | Terri Yu proposed a change to openstack/ceilometer: Add group by statistics tests in API v2 tests https://review.openstack.org/44130 | 15:00 |
| litong | the other way is that I can look at the sorting, and make sure that the sorting is implemented in db2 get_sample. | 15:00 |
| litong | I think that is where the problem is. | 15:00 |
| thomasm | litong, I think the concern there was the lack of documentation, whereas raising NotImplemented is a little more noisy to motivate people. | 15:01 |
| thomasm | It was get_resources that wasn't sorted - I didn't look at the others. | 15:01 |
| thomasm | messages out of order cause undefined behavior when pulling metadata. | 15:01 |
| thomasm | for a resource | 15:01 |
| litong | so it is sorted by timestamp? | 15:02 |
| thomasm | It needs to be, yeah. | 15:02 |
| thomasm | But it's not, because it just goes to the latest representation in the resource collection - which doesn't take the timestamp filtering into account for resource metadata at that time. | 15:03 |
| litong | I am looking at it now. get_resources method for db2 and mongo are the same, I do not think there is difference | 15:03 |
| litong | there is no difference, sorry | 15:03 |
| thomasm | Mongo uses the meter collection, DB2 uses the resource collection | 15:04 |
| litong | in get_resources method? | 15:04 |
| thomasm | mongo aggregates on the meter collection, db2 filters down and gets distinct resource_ids then pulls those from the resource collection | 15:04 |
| thomasm | yeah | 15:04 |
| litong | that must be recent changes I suspect | 15:05 |
| thomasm | litong, https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/impl_db2.py#L341-L414 | 15:05 |
| litong | db2 driver was mostly copied from mongo driver with changes that specific to db2. | 15:05 |
| litong | my env is rebuilding now, this is my top priority. | 15:05 |
| thomasm | litong, Okay =] | 15:06 |
| litong | once I have it, I will check, I do not think it is hard to figure out what went wrong. | 15:06 |
| thomasm | litong, Thanks so much! | 15:06 |
| litong | @thomasm, thanks, | 15:06 |
| litong | @thomasm, I will make changes now, I believe that method in mongo impl recently changed to aggregate. | 15:09 |
| litong | the problem with aggregate is that it does not support multiple partitions or sharding, I do not know if anybody was aware of that. | 15:10 |
| dragondm | yup, it was a few weeks agon | 15:10 |
| dragondm | (see: https://review.openstack.org/#/c/35297/) | 15:15 |
| dragondm | litong ^ | 15:15 |
| litong | @dragondm, yeah, saw that now. was not aware. but do you know aggregate actually has problem with partitioned mongodb? | 15:17 |
| litong | @dragondm, I think that the document says it won't support that. | 15:17 |
| *** alexb_ has quit IRC | 15:18 | |
| dragondm | Didn't know that. I'm guessing Julien didn't either. | 15:18 |
| thomasm | Hmmm interesting | 15:22 |
| thomasm | litong, so then we should probably just do the sort in the query and then do the min/max on our own? | 15:24 |
| thomasm | i.e. in the Python | 15:25 |
| litong | @thomasm, I am not sure, probably need some discussion, did you see the doc regarding the restriction? | 15:25 |
| *** shaneduan[afk] is now known as shaneduan | 15:25 | |
| litong | @thomasm, just wanted to make sure we are on the same page. | 15:26 |
| thomasm | litong, The one that jd__ put in? | 15:26 |
| litong | @thomasm, I was talking about the aggregate restriction on mongodb. | 15:26 |
| thomasm | litong, No, I haven't seen that one | 15:26 |
| litong | @thomasm, hold on, let me get a link | 15:27 |
| thomasm | litong, thanks! | 15:27 |
| litong | http://docs.mongodb.org/manual/core/aggregation-pipeline-sharded-collections/ | 15:31 |
| litong | @thomasm, I may have seen older docs. | 15:31 |
| litong | that links says it does support sharded collections. | 15:31 |
| litong | @thomasm, http://docs.mongodb.org/manual/core/aggregation-pipeline-limits/ | 15:32 |
| thomasm | litong, So the limitations are more hardware based? | 15:33 |
| litong | looks like it. | 15:34 |
| litong | from that doc, seems we are ok. | 15:34 |
| thomasm | litong, yeah. I wonder if we ought to profile that a bit, though. Just to be sure we don't penalize people beyond reason, haha. | 15:34 |
| litong | if I found other docs on this, I will contact you . Somewhere I remember I read something about the restrictions of using aggregate method. | 15:34 |
| thomasm | Not sure what beyond reason would be | 15:34 |
| thomasm | litong, Okay. Thanks! =] | 15:35 |
| litong | @thomasm, compare map_reduce and aggregate is not always easy. | 15:35 |
| thomasm | Yeah | 15:35 |
| sileht | thomasm, litong to get what works or not in each backend you can look for skipped test in the testr result file | 15:40 |
| dragondm | Of course, there are other issues there as well. AFAIK, ceilometer doesn't specify a shard key on any of it's collections. | 15:40 |
| sileht | in more than 90% the reason is NotImplementedError | 15:40 |
| thomasm | dragondm, none that I saw | 15:40 |
| sileht | dragondm, I'm not sure that ceilometer should be aware of the shard key | 15:48 |
| openstackgerrit | A change was merged to openstack/ceilometer: alarm: generate alarm_id in API https://review.openstack.org/44065 | 15:50 |
| terriyu | sileht: where's the testr result file? | 15:53 |
| sileht | terriyu, for example: http://logs.openstack.org/47/44247/1/gate/gate-ceilometer-python26/3109bf1/testr_results.html.gz | 15:53 |
| sileht | terriyu, you need to export some env variables to generate it locally | 15:54 |
| sileht | ie: NOSE_WITH_XUNIT=1 NOSE_WITH_HTML_OUTPUT=1 NOSE_HTML_OUT_FILE=nose_results.html | 15:54 |
| terriyu | sileht: I thought we don't use nose anymore? | 15:54 |
| *** changbl has quit IRC | 15:55 | |
| sileht | terriyu, correct, perhaps they are some compatibility | 15:55 |
| jd__ | probably indeed :) | 15:56 |
| terriyu | sileht: I was interested because I wanted to see the skipped tests | 15:56 |
| terriyu | I asked jd__ this question recently :) | 15:56 |
| sileht | terriyu, on gate look on the py26 not py27 because all mongo tests are skipped in py27 | 15:57 |
| terriyu | sileht: uhh, all the mongo tests are skipped?? Isn't that bad? | 15:57 |
| sileht | jd__, we will have a issue soon (very soon) with the one mongo db connection per test | 15:57 |
| sileht | terriyu, nop because it run on py26 | 15:58 |
| jd__ | sileht: ? | 15:59 |
| sileht | jd__, when I have enabled testscenario (and enabled tests for db2), I have grow the number of max connections to 128 | 15:59 |
| sileht | but according the last pagination changeset, this is not enough http://logs.openstack.org/54/37454/18/check/gate-ceilometer-python26/ec57492/console.html | 15:59 |
| sileht | because the patch add many new tests | 15:59 |
| jd__ | that's caused by DB2 right? | 16:00 |
| terriyu | jd__: hmm, maybe this is related to my MongoDB connection failure problems?? | 16:01 |
| sileht | db2 have x2 the number of connections needed | 16:01 |
| jd__ | terriyu: it looks like it | 16:01 |
| sileht | jd__, terriyu I was having this issue in the past and I have just increase the maxConns of mongod | 16:01 |
| jd__ | but I saw you raised it but that I think that's because DB2 don't have a pooling mechanism | 16:01 |
| terriyu | sileht: how do you increase the max number of connections? | 16:02 |
| sileht | DB2 use a mongod database in reality | 16:02 |
| sileht | jd__, ^ | 16:02 |
| jd__ | sileht: I know that but it doesn't use the pooling we built in impl_mongodb | 16:02 |
| sileht | terriyu, is run-test.sh | 16:02 |
| jd__ | sileht: unless I missed it :) | 16:02 |
| sileht | jd__, I have not checking this point | 16:02 |
| sileht | jd__, it seems use pool too | 16:03 |
| terriyu | sileht: thanks! | 16:03 |
| jd__ | sileht: *(#@ | 16:04 |
| jd__ | je donne ma langue au chat | 16:04 |
| sileht | if you take a look on http://logs.openstack.org/47/44247/1/gate/gate-ceilometer-python27/94a7aaa/testr_results.html.gz | 16:05 |
| sileht | you will see that 795 tests have been skipped | 16:05 |
| sileht | only 180 in py26 | 16:05 |
| sileht | so we can have ~600 tests running in the same times | 16:06 |
| sileht | each tests use his own database | 16:06 |
| sileht | a new database create a new item in the pool and a new connection | 16:06 |
| terriyu | sileht, jd__ : when I raise my max connections from 128 --> 256, then the test suite passes | 16:06 |
| jd__ | terriyu: that fits what's sileht's saying | 16:06 |
| terriyu | I still get the message "No module named deprecated" | 16:06 |
| jd__ | -'s | 16:07 |
| * jd__ grumbles | 16:07 | |
| sileht | I don't know this module :p | 16:07 |
| terriyu | No module named give_me_more_connections_me_hungry | 16:08 |
| sileht | jd__, do you kown if infra will be happy with our way to run tests ? | 16:09 |
| *** janisg has quit IRC | 16:11 | |
| jd__ | sileht: the maxConns was here because infra had a limit on the number of processes or something, and therefore mongodb exploded | 16:11 |
| jd__ | so I set it very low (32) to be sure we don't break it on CI | 16:12 |
| jd__ | I was already suprised 128 was OK on CI | 16:12 |
| sileht | jd__, perhaps we needs to limit the concurrency in testr | 16:12 |
| *** changbl has joined #openstack-metering | 16:12 | |
| jd__ | sileht: what about we pool on host:port rather than the full URL? | 16:13 |
| jd__ | the problem is that pymongo doesn't close the connection | 16:13 |
| sileht | jd__, it works too | 16:15 |
| jd__ | what's your bug number terriyu? | 16:17 |
| sileht | jd__, ok I will propose something for that I prefer your method | 16:17 |
| jd__ | sileht: well I think it'll be better and will work :) | 16:17 |
| *** shaneduan is now known as shaneduan[afk] | 16:18 | |
| *** dina_belova has joined #openstack-metering | 16:22 | |
| terriyu | jd__: oh, I didn't post it yet | 16:23 |
| *** shaneduan[afk] is now known as shaneduan | 16:23 | |
| terriyu | jd__: hold on | 16:24 |
| jd__ | terriyu: ok, sileht will need it to include in its fix | 16:24 |
| * sileht needs to upgrade its .tox dir sniff | 16:26 | |
| *** SergeyLukjanov has joined #openstack-metering | 16:26 | |
| terriyu | jd__ sileht : https://bugs.launchpad.net/ceilometer/+bug/1218488 | 16:30 |
| sileht | jd__, it does works because the we cache the MongoClient object that connect to the db not only to mongod instance | 16:34 |
| sileht | jd__, forget | 16:35 |
| *** herndon has joined #openstack-metering | 16:36 | |
| herndon | sandywalsh: ping | 16:38 |
| *** shakayumi has quit IRC | 16:45 | |
| *** dina_belova has quit IRC | 16:54 | |
| *** dina_belova has joined #openstack-metering | 16:55 | |
| *** dina_belova has quit IRC | 16:59 | |
| sandywalsh | herndon, howdy | 17:31 |
| herndon | hey there | 17:34 |
| herndon | just wanted to chat about your patch since there have been a few back and forths | 17:35 |
| *** eglynn has quit IRC | 17:35 | |
| herndon | Is message_id a key name that is more universal than I am aware of? | 17:36 |
| herndon | Have to admit, I have not been tracking the notification work that closely :( | 17:36 |
| sandywalsh | nothing other than it's the name in the notification | 17:38 |
| sandywalsh | the event.id vs. event_id thing is my biggest twitch (but I'm not hung up on it) | 17:39 |
| *** Bada has joined #openstack-metering | 17:39 | |
| sandywalsh | my spidey senses are saying that'll cause confusion down the road | 17:40 |
| Bada | Hi guys, does anybody knows what this mean : [-] Could not load 'accumulator': pbr>=0.5.21,<1.0 ? | 17:40 |
| sandywalsh | Bada, check that you've got the proper version of pbr installed | 17:41 |
| sandywalsh | Bada, is this under devstack? | 17:42 |
| Bada | sandywalsh not it's not | 17:42 |
| Bada | sandywalsh what is pbr ? | 17:42 |
| sandywalsh | Bada, is a central library to other openstack components (Python Build Reasonableness or something) | 17:43 |
| sandywalsh | pip install -U pbr | 17:43 |
| Bada | yeah found it https://github.com/openstack-dev/pbr | 17:43 |
| herndon | sandywalsh, message_id it is then. | 17:43 |
| Bada | sandywalsh, it works ! | 17:45 |
| Bada | sandywalsh, brilliant many thanks | 17:45 |
| herndon | hm, although it may cause confusion in the api when the query for event id looks up message_id. Maybe the event api should change? | 17:45 |
| herndon | or the api should look at event.is, but I think that is less useful than message_id | 17:46 |
| herndon | s/event.is/event.id | 17:46 |
| sandywalsh | Bada, np | 17:48 |
| sandywalsh | herndon, can we change it in the api? | 17:49 |
| sandywalsh | hmm, but I see your point | 17:49 |
| herndon | seems like offloading the confusion to the consumer of the api | 17:49 |
| sandywalsh | it's kind of klunky to ask for event_id and map that to message_id | 17:49 |
| herndon | yeah. | 17:50 |
| sandywalsh | well, the caller would only see event_id | 17:50 |
| herndon | ignore my previous comment | 17:50 |
| sandywalsh | shall we live with message_id and see how it feels? If it's a real sticky point, I'll change it. | 17:52 |
| *** SergeyLukjanov has quit IRC | 17:52 | |
| *** shakayumi has joined #openstack-metering | 17:52 | |
| *** SergeyLukjanov has joined #openstack-metering | 17:53 | |
| herndon | Yeah, let's do that. I already did the event_id -> message_id conversion in one patch. I really like that you've moved it into event, that feels a lot better than querying traits to lookup the id of an event. | 17:53 |
| sandywalsh | cool ... k, stay tuned. Thanks for the feedback! | 17:53 |
| *** dina_belova has joined #openstack-metering | 17:53 | |
| gordc | sileht: are we waiting for something before we approve this: https://review.openstack.org/#/c/43848? just want to know so i don't superseded something. | 17:56 |
| sileht | gordc, I have missing the Approved button ! | 17:57 |
| gordc | sileht: cool. i'll approve... unless you'd like to. :) | 17:57 |
| sileht | gordc, ok, I'm not with my computer actually | 17:58 |
| gordc | sileht: done, carry on with your evening :) | 18:00 |
| *** dina_belova has quit IRC | 18:03 | |
| *** dina_belova has joined #openstack-metering | 18:04 | |
| *** thomasm has quit IRC | 18:05 | |
| sileht | gordc, thanks | 18:06 |
| *** shakayumi has quit IRC | 18:08 | |
| *** dina_belova has quit IRC | 18:08 | |
| *** Bada has quit IRC | 18:15 | |
| *** dina_belova has joined #openstack-metering | 18:24 | |
| *** thomasm has joined #openstack-metering | 18:38 | |
| *** herndon has quit IRC | 18:43 | |
| *** dina_belova has quit IRC | 18:51 | |
| *** dina_belova has joined #openstack-metering | 18:52 | |
| *** dina_belova has quit IRC | 18:57 | |
| *** yolanda has quit IRC | 18:58 | |
| *** boris-42 has quit IRC | 19:03 | |
| *** alexb_ has joined #openstack-metering | 19:07 | |
| *** eglynn has joined #openstack-metering | 19:17 | |
| *** boris-42 has joined #openstack-metering | 19:32 | |
| *** SergeyLukjanov has quit IRC | 19:32 | |
| *** alexb_ has quit IRC | 19:38 | |
| *** alexb_ has joined #openstack-metering | 19:40 | |
| *** SergeyLukjanov has joined #openstack-metering | 19:41 | |
| *** SergeyLukjanov has quit IRC | 19:59 | |
| *** dina_belova has joined #openstack-metering | 20:02 | |
| *** dina_belova has quit IRC | 20:07 | |
| *** boris-42 has quit IRC | 20:09 | |
| openstackgerrit | Eoghan Glynn proposed a change to openstack/ceilometer: Base Alarm history persistence model https://review.openstack.org/43848 | 20:15 |
| openstackgerrit | Eoghan Glynn proposed a change to openstack/ceilometer: Plug alarm history logic into the API https://review.openstack.org/43849 | 20:15 |
| openstackgerrit | Eoghan Glynn proposed a change to openstack/ceilometer: Alarm history storage implementation for mongodb https://review.openstack.org/43850 | 20:15 |
| *** sandywalsh has quit IRC | 20:23 | |
| *** sandywalsh has joined #openstack-metering | 20:35 | |
| *** eglynn has quit IRC | 20:48 | |
| openstackgerrit | Thomas Maddox proposed a change to openstack/ceilometer: Fix to return latest resource metadata https://review.openstack.org/44277 | 20:50 |
| *** shaneduan is now known as shaneduan[afk] | 20:58 | |
| *** evanjfraser has joined #openstack-metering | 21:01 | |
| *** eglynn has joined #openstack-metering | 21:02 | |
| *** dina_belova has joined #openstack-metering | 21:03 | |
| *** dina_belova has quit IRC | 21:08 | |
| *** shaneduan[afk] is now known as shaneduan | 21:17 | |
| *** shanewang has joined #openstack-metering | 21:26 | |
| *** shanewang has left #openstack-metering | 21:26 | |
| *** shaneduan is now known as shaneduan[afk] | 21:29 | |
| *** alexb_ has quit IRC | 21:31 | |
| *** litong has quit IRC | 21:41 | |
| openstackgerrit | Sandy Walsh proposed a change to openstack/ceilometer: Reject duplicate events https://review.openstack.org/44191 | 21:49 |
| *** dina_belova has joined #openstack-metering | 22:03 | |
| *** dina_belova has quit IRC | 22:08 | |
| *** alexb_ has joined #openstack-metering | 22:10 | |
| *** bpokorny has quit IRC | 22:24 | |
| *** changbl has quit IRC | 22:26 | |
| *** eglynn has quit IRC | 22:31 | |
| *** litong has joined #openstack-metering | 22:33 | |
| *** evanjfraser has quit IRC | 22:36 | |
| *** thomasm has quit IRC | 22:48 | |
| *** gordc has quit IRC | 22:54 | |
| *** dina_belova has joined #openstack-metering | 23:04 | |
| *** dina_belova has quit IRC | 23:09 | |
| *** shaneduan[afk] is now known as shaneduan | 23:29 | |
| *** evanjfraser has joined #openstack-metering | 23:31 | |
| *** evanjfraser has quit IRC | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!