| *** slogan621 has joined #openstack-telemetry | 00:00 | |
| *** llu has quit IRC | 00:01 | |
| *** jaypipes has quit IRC | 00:04 | |
| *** rbak has quit IRC | 00:04 | |
| *** ddieterly has joined #openstack-telemetry | 00:22 | |
| *** terriyu has quit IRC | 00:38 | |
| *** thorst has joined #openstack-telemetry | 00:44 | |
| *** thorst has quit IRC | 00:51 | |
| *** thorst has joined #openstack-telemetry | 00:51 | |
| *** thorst has quit IRC | 00:56 | |
| *** liusheng has joined #openstack-telemetry | 01:03 | |
| *** thumpba has joined #openstack-telemetry | 01:06 | |
| *** ljxiash has joined #openstack-telemetry | 01:06 | |
| *** thumpba has quit IRC | 01:12 | |
| *** ildikov has quit IRC | 01:15 | |
| *** ildikov has joined #openstack-telemetry | 01:16 | |
| *** thorst has joined #openstack-telemetry | 01:23 | |
| *** thorst has quit IRC | 01:23 | |
| *** thorst has joined #openstack-telemetry | 01:23 | |
| *** thorst has quit IRC | 01:24 | |
| *** thorst has joined #openstack-telemetry | 01:25 | |
| *** thorst has quit IRC | 01:29 | |
| *** jaypipes has joined #openstack-telemetry | 01:33 | |
| *** slogan621 has quit IRC | 01:37 | |
| *** pradk has quit IRC | 01:38 | |
| *** boris-42 has joined #openstack-telemetry | 01:50 | |
| *** Liuqing has joined #openstack-telemetry | 01:58 | |
| openstackgerrit | Rohit Jaiswal proposed openstack/ceilometer: Consistent publisher_id and event_type for polling and api https://review.openstack.org/240342 | 02:00 |
|---|---|---|
| *** prashantD has quit IRC | 02:01 | |
| *** jwcroppe has quit IRC | 02:21 | |
| *** chaozhechen_ has joined #openstack-telemetry | 02:29 | |
| *** CheneyChen has joined #openstack-telemetry | 02:29 | |
| *** fawadkhaliq has joined #openstack-telemetry | 02:42 | |
| openstackgerrit | liusheng proposed openstack/aodh: Don't send notificaton when recording alarm change https://review.openstack.org/246727 | 02:45 |
| *** ddieterly has quit IRC | 02:46 | |
| *** Liuqing has quit IRC | 02:55 | |
| *** Liuqing has joined #openstack-telemetry | 02:55 | |
| *** thorst has joined #openstack-telemetry | 02:57 | |
| *** thorst has quit IRC | 02:57 | |
| *** ddieterly has joined #openstack-telemetry | 03:05 | |
| *** ddieterl_ has joined #openstack-telemetry | 03:06 | |
| *** ddieterly has quit IRC | 03:10 | |
| *** prashantD has joined #openstack-telemetry | 03:26 | |
| *** lvdongbing has joined #openstack-telemetry | 03:27 | |
| lvdongbing | Lianhao Lu, are you there? | 03:31 |
| *** liusheng has quit IRC | 03:35 | |
| *** ViswaV has quit IRC | 03:45 | |
| openstackgerrit | liusheng proposed openstack/aodh: Don't send notificaton when recording alarm change https://review.openstack.org/246727 | 03:47 |
| *** liusheng has joined #openstack-telemetry | 03:48 | |
| *** ViswaV has joined #openstack-telemetry | 03:49 | |
| *** prashantD_ has joined #openstack-telemetry | 03:55 | |
| *** prashantD has quit IRC | 03:56 | |
| *** lvdongbing has quit IRC | 04:04 | |
| *** lvdongbing has joined #openstack-telemetry | 04:04 | |
| *** fawadkhaliq has quit IRC | 04:06 | |
| *** khushbu has joined #openstack-telemetry | 04:07 | |
| *** khushbu has quit IRC | 04:08 | |
| *** ddieterl_ has quit IRC | 04:11 | |
| *** hparekh2 has quit IRC | 04:20 | |
| *** fawadkhaliq has joined #openstack-telemetry | 04:27 | |
| *** hparekh has joined #openstack-telemetry | 04:27 | |
| openstackgerrit | liusheng proposed openstack/ceilometer: Move the content of ReleaseNotes to README.rst https://review.openstack.org/246743 | 04:52 |
| *** jwcroppe has joined #openstack-telemetry | 05:03 | |
| openstackgerrit | liusheng proposed openstack/ceilometer: Fix a indent nit of enforce_limit method https://review.openstack.org/246744 | 05:04 |
| openstackgerrit | liusheng proposed openstack/ceilometer: Fix an indent nit of enforce_limit method https://review.openstack.org/246744 | 05:05 |
| openstackgerrit | yuntongjin proposed openstack/ceilometer-specs: event to sample publisher https://review.openstack.org/223926 | 05:05 |
| *** ddieterly has joined #openstack-telemetry | 05:11 | |
| *** ddieterly has quit IRC | 05:17 | |
| *** prashantD_ has quit IRC | 05:19 | |
| *** jaypipes has quit IRC | 05:20 | |
| *** khushbu_ has joined #openstack-telemetry | 05:23 | |
| *** khushbu_ has quit IRC | 05:24 | |
| *** khushbu_ has joined #openstack-telemetry | 05:27 | |
| *** khushbu_ has quit IRC | 05:34 | |
| *** khushbu_ has joined #openstack-telemetry | 05:39 | |
| *** khushbu_ has quit IRC | 05:46 | |
| *** jwcroppe has quit IRC | 05:50 | |
| *** jwcroppe has joined #openstack-telemetry | 05:50 | |
| *** jwcroppe has quit IRC | 05:54 | |
| *** nadya_ has joined #openstack-telemetry | 06:01 | |
| *** Liuqing has quit IRC | 06:10 | |
| *** Liuqing has joined #openstack-telemetry | 06:11 | |
| *** ddieterly has joined #openstack-telemetry | 06:13 | |
| *** nadya_ has quit IRC | 06:18 | |
| *** ddieterly has quit IRC | 06:19 | |
| *** changbl has quit IRC | 06:22 | |
| *** changbl has joined #openstack-telemetry | 06:23 | |
| *** liusheng has quit IRC | 06:36 | |
| *** rcernin has joined #openstack-telemetry | 06:37 | |
| *** liusheng has joined #openstack-telemetry | 06:37 | |
| *** Liuqing has quit IRC | 07:05 | |
| *** Liuqing has joined #openstack-telemetry | 07:06 | |
| *** ddieterly has joined #openstack-telemetry | 07:16 | |
| *** ddieterly has quit IRC | 07:20 | |
| *** eglynn has quit IRC | 07:32 | |
| *** hparekh has quit IRC | 07:33 | |
| openstackgerrit | liusheng proposed openstack/ceilometer: Move the content of ReleaseNotes to README.rst https://review.openstack.org/246743 | 07:34 |
| *** hparekh has joined #openstack-telemetry | 07:40 | |
| *** belmoreira has joined #openstack-telemetry | 07:57 | |
| *** Ala has joined #openstack-telemetry | 07:58 | |
| *** liusheng has quit IRC | 07:59 | |
| *** liusheng has joined #openstack-telemetry | 08:00 | |
| *** eglynn has joined #openstack-telemetry | 08:12 | |
| *** ddieterly has joined #openstack-telemetry | 08:17 | |
| *** safchain has joined #openstack-telemetry | 08:20 | |
| *** shardy has joined #openstack-telemetry | 08:21 | |
| *** ddieterly has quit IRC | 08:21 | |
| *** Liuqing has quit IRC | 08:23 | |
| *** Liuqing has joined #openstack-telemetry | 08:24 | |
| *** fawadkhaliq has quit IRC | 08:41 | |
| *** fawadkhaliq has joined #openstack-telemetry | 08:42 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/python-gnocchiclient: resouce id must be url quoted https://review.openstack.org/246802 | 08:52 |
| *** Liuqing has quit IRC | 08:53 | |
| *** Liuqing has joined #openstack-telemetry | 08:54 | |
| *** fawadkhaliq has quit IRC | 08:57 | |
| openstackgerrit | Béla Vancsics proposed openstack/ceilometer: Reduced source code by extracting duplicated code https://review.openstack.org/232020 | 09:00 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: add a __repr__ for AggregatedTimeSerie https://review.openstack.org/245076 | 09:04 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: implement an integer sampling attribute https://review.openstack.org/245075 | 09:04 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: make offset conversion consistent https://review.openstack.org/245074 | 09:04 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: archive_policy: enforce types https://review.openstack.org/245073 | 09:04 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: _carbonara: dedicated methods to store raw timeserie https://review.openstack.org/245072 | 09:04 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: storage: support storage upgrade https://review.openstack.org/245070 | 09:04 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: cli: allow to upgrade in 2 passes https://review.openstack.org/245071 | 09:04 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: deprecate TimeSerieArchive https://review.openstack.org/240905 | 09:04 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: Rename dbsync to upgrade https://review.openstack.org/245069 | 09:04 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: Add missing PrettyTable dependency https://review.openstack.org/246807 | 09:04 |
| *** yassine__ has joined #openstack-telemetry | 09:12 | |
| *** nadya_ has joined #openstack-telemetry | 09:15 | |
| *** ddieterly has joined #openstack-telemetry | 09:17 | |
| *** ddieterly has quit IRC | 09:22 | |
| *** Liuqing has quit IRC | 09:32 | |
| *** lvdongbing has quit IRC | 09:36 | |
| *** ljxiash has quit IRC | 09:50 | |
| *** r-mibu has left #openstack-telemetry | 09:59 | |
| *** Liuqing has joined #openstack-telemetry | 10:04 | |
| *** eglynn has quit IRC | 10:15 | |
| *** ddieterly has joined #openstack-telemetry | 10:18 | |
| *** ddieterly has quit IRC | 10:22 | |
| *** jwcroppe has joined #openstack-telemetry | 10:30 | |
| *** jwcroppe has quit IRC | 10:34 | |
| *** fawadkhaliq has joined #openstack-telemetry | 10:36 | |
| *** ildikov has quit IRC | 11:05 | |
| *** exploreshaifali has joined #openstack-telemetry | 11:05 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: add a __repr__ for AggregatedTimeSerie https://review.openstack.org/245076 | 11:13 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: implement an integer sampling attribute https://review.openstack.org/245075 | 11:13 |
| *** prashantD has joined #openstack-telemetry | 11:17 | |
| *** prashantD has quit IRC | 11:21 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: use gnocchiclient instead of requests https://review.openstack.org/237538 | 11:38 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: Use keystoneauth1 instead of manual setup https://review.openstack.org/237537 | 11:38 |
| *** sergio__nubeliu has joined #openstack-telemetry | 11:43 | |
| *** exploreshaifali has quit IRC | 11:49 | |
| *** exploreshaifali has joined #openstack-telemetry | 11:53 | |
| *** ildikov has joined #openstack-telemetry | 11:54 | |
| *** exploreshaifali has quit IRC | 12:03 | |
| *** khushbu_ has joined #openstack-telemetry | 12:05 | |
| *** khushbu_ has quit IRC | 12:05 | |
| *** khushbu_ has joined #openstack-telemetry | 12:09 | |
| *** khushbu_ has quit IRC | 12:10 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: add a __repr__ for AggregatedTimeSerie https://review.openstack.org/245076 | 12:15 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: deprecate TimeSerieArchive https://review.openstack.org/240905 | 12:15 |
| *** khushbu has joined #openstack-telemetry | 12:16 | |
| *** khushbu has quit IRC | 12:16 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: deprecate TimeSerieArchive https://review.openstack.org/240905 | 12:16 |
| *** khushbu_ has joined #openstack-telemetry | 12:19 | |
| *** ddieterly has joined #openstack-telemetry | 12:19 | |
| *** khushbu_ has quit IRC | 12:20 | |
| *** wolsen has quit IRC | 12:20 | |
| *** alejandrito has joined #openstack-telemetry | 12:23 | |
| *** ddieterly has quit IRC | 12:24 | |
| *** ddieterly has joined #openstack-telemetry | 12:24 | |
| *** jkraj has joined #openstack-telemetry | 12:28 | |
| *** thorst has joined #openstack-telemetry | 12:28 | |
| *** khushbu has joined #openstack-telemetry | 12:33 | |
| *** khushbu has quit IRC | 12:33 | |
| alejandrito | jd__, why is it that when querying the measures for a metric that has this archive policy (http://pastebin.com/7ff9e0Bm) every new day, i see the points from 00:00 hs of today and not the day/s after ? | 12:36 |
| alejandrito | jd__, are we interpreting something wrong ? maybe about definition of the policy? | 12:37 |
| jd__ | alejandrito: can you paste me what you get and what you would expect? | 12:37 |
| alejandrito | jd__, sure | 12:39 |
| *** fawadkhaliq has quit IRC | 12:42 | |
| alejandrito | jd__, http://pastebin.com/f6bWma8A i see the measures from today since 00:00hs here is a metric show output ( http://pastebin.com/FNVP0tUm ) | 12:49 |
| jd__ | alejandrito: can you show me the details of the metric ? like its archive policy | 12:54 |
| jd__ | your archive policy is pretty weird too :) | 12:58 |
| alejandrito | jd__, its thought for show back purposes :) this one is no good ? http://pastebin.com/FNVP0tUm | 13:01 |
| jd__ | you keep some datapoints for 60 years | 13:01 |
| alejandrito | jd__, yup :D its like saying ... FOR EVER | 13:04 |
| jd__ | alejandrito: I'm still waiting for the metric details, can you show me? :) | 13:05 |
| alejandrito | jd__, sorry this one ? http://pastebin.com/FNVP0tUm | 13:06 |
| *** gordc has joined #openstack-telemetry | 13:07 | |
| *** gordc_ has joined #openstack-telemetry | 13:09 | |
| *** jwcroppe has joined #openstack-telemetry | 13:10 | |
| *** gordc has quit IRC | 13:13 | |
| *** CheneyChen has quit IRC | 13:13 | |
| *** chaozhechen_ has quit IRC | 13:13 | |
| jd__ | ah yeah | 13:14 |
| jd__ | thanks | 13:14 |
| jd__ | so yeah looks like you miss points | 13:14 |
| *** jwcroppe has quit IRC | 13:15 | |
| *** jwcroppe has joined #openstack-telemetry | 13:15 | |
| *** jwcroppe_ has joined #openstack-telemetry | 13:17 | |
| *** jwcroppe has quit IRC | 13:20 | |
| sergio__nubeliu | jd__: yes but there is a pattern, in all cases we miss points from the day before and older | 13:24 |
| *** prashantD has joined #openstack-telemetry | 13:25 | |
| *** ljxiash has joined #openstack-telemetry | 13:28 | |
| *** prashantD has quit IRC | 13:29 | |
| *** prashantD has joined #openstack-telemetry | 13:30 | |
| *** Liuqing_ has joined #openstack-telemetry | 13:31 | |
| *** prashantD has quit IRC | 13:31 | |
| *** tomoiaga has joined #openstack-telemetry | 13:31 | |
| jd__ | sergio__nubeliu: what do you mean? | 13:31 |
| jd__ | ah right | 13:32 |
| jd__ | yeah something is weird | 13:32 |
| *** Liuqing has quit IRC | 13:33 | |
| *** gordc_ has quit IRC | 13:34 | |
| jd__ | sergio__nubeliu: alejandrito if you can send me the Carbonara file stored in Ceph that'd probably help me | 13:39 |
| jd__ | maybe it's a normal, maybe not but I'm confused | 13:40 |
| openstackgerrit | Rohit Jaiswal proposed openstack/ceilometer: Consistent publisher_id and event_type for polling and api https://review.openstack.org/240342 | 13:42 |
| *** kbyrne has joined #openstack-telemetry | 13:43 | |
| sergio__nubeliu | jd__: alejandrito will send you the file in a few minutes | 13:44 |
| *** chaozhechen_ has joined #openstack-telemetry | 13:49 | |
| *** ddieterly has quit IRC | 13:54 | |
| *** julim has joined #openstack-telemetry | 13:55 | |
| *** dan-t has joined #openstack-telemetry | 13:56 | |
| *** changbl has quit IRC | 14:01 | |
| *** exploreshaifali has joined #openstack-telemetry | 14:04 | |
| *** bapalm has joined #openstack-telemetry | 14:08 | |
| *** boris-42 has quit IRC | 14:08 | |
| *** jaypipes has joined #openstack-telemetry | 14:23 | |
| alejandrito | jd__, im back | 14:23 |
| *** lsmola has quit IRC | 14:26 | |
| *** ddieterly has joined #openstack-telemetry | 14:26 | |
| *** ddieterly has quit IRC | 14:26 | |
| *** ddieterly has joined #openstack-telemetry | 14:27 | |
| alejandrito | jd__, before going into the ceph carbonara content, im having metricd giving this : http://pastebin.com/cCR0xwBN | 14:27 |
| alejandrito | jd__, can it have something to do with "missing" data ? | 14:27 |
| jd__ | alejandrito: that explains probably everything indeed | 14:28 |
| jd__ | since it creates an empty timeserie each time | 14:28 |
| jd__ | that explains why you're losing metrics | 14:29 |
| *** lsmola has joined #openstack-telemetry | 14:29 | |
| jd__ | alejandrito: I imagine you see no reason for that data corruption to happen on your side? Ceph is OK? | 14:29 |
| jd__ | sileht: you have any idea why so many read errors would happen on Ceph? | 14:30 |
| sileht | jd__, not really | 14:30 |
| *** lsmola has quit IRC | 14:31 | |
| alejandrito | jd__, the first message happened today at 00:48 im looking at ceph right now. | 14:31 |
| jd__ | ok | 14:31 |
| jd__ | keep me in touch | 14:31 |
| *** openstackgerrit has quit IRC | 14:31 | |
| alejandrito | jd__, sileht to see if i see something ... because the measure_ are still being created | 14:32 |
| alejandrito | jd__, we have no error on CEPH and the health out put is OK | 14:32 |
| *** openstackgerrit has joined #openstack-telemetry | 14:32 | |
| jd__ | alejandrito: you got no write issues before 00:48? | 14:32 |
| alejandrito | jd__, nope ... and our developers told us that they saw same behaviour past week, but ceph has a 180 days uptime with no errors whatsoever | 14:33 |
| jd__ | alejandrito: ok,I'm gonna try to write a patch to grab more info on this | 14:35 |
| *** jkraj has quit IRC | 14:37 | |
| *** ljxiash has quit IRC | 14:39 | |
| *** ljxiash has joined #openstack-telemetry | 14:41 | |
| *** bapalm has quit IRC | 14:43 | |
| jd__ | alejandrito: what coordination_url do you use? | 14:43 |
| *** bapalm has joined #openstack-telemetry | 14:43 | |
| jd__ | alejandrito: how many computers are running gnocchi-metricd? | 14:43 |
| *** ddieterly has quit IRC | 14:44 | |
| *** lsmola has joined #openstack-telemetry | 14:44 | |
| alejandrito | jd__, wow ... the critical thing about this ... is that the corrupted messages keep appearing and all my metrics / measures are dissapearing :O , i hace just only one metricd vm running ... let me double check de coordinator url | 14:44 |
| *** fawadkhaliq has joined #openstack-telemetry | 14:44 | |
| *** lsmola has quit IRC | 14:45 | |
| *** lsmola has joined #openstack-telemetry | 14:45 | |
| *** lsmola has quit IRC | 14:45 | |
| alejandrito | jd__, coordination_url = file:///var/lib/gnocchi/locks | 14:45 |
| *** lsmola has joined #openstack-telemetry | 14:46 | |
| jd__ | alejandrito: ok and how many servers run metricd? | 14:46 |
| alejandrito | jd__, just one with one worker | 14:46 |
| jd__ | alejandrito: ok thanks! | 14:47 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: use gnocchiclient instead of requests https://review.openstack.org/237538 | 14:53 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: Use keystoneauth1 instead of manual setup https://review.openstack.org/237537 | 14:53 |
| *** ddieterly has joined #openstack-telemetry | 14:57 | |
| alejandrito | jd__, sileht well ... i can confirm that the data corruption messages keep appearing and all the measures are gone for all the metrics | 14:58 |
| alejandrito | jd__, sileht they appear in this manner http://pastebin.com/LXqqPQS0 | 14:59 |
| jd__ | yeah makes sense, if I can say so | 14:59 |
| *** rbak has joined #openstack-telemetry | 15:01 | |
| *** ddieterly has quit IRC | 15:01 | |
| *** ddieterly has joined #openstack-telemetry | 15:01 | |
| alejandrito | jd__, im just keeping everything in this state ( holding the developers ) just for you to debug something here if you want to take advantage of this "corrupted" env | 15:05 |
| jd__ | alejandrito: yeah thanks, I'm trying to write a patch with more debug stuff | 15:06 |
| alejandrito | jd__, one side question in what gnocchi resource type can i put all this samples ? ( we kept all this ones in ceilometer ) http://pastebin.com/j0MqXjUW | 15:08 |
| *** llu-laptop has joined #openstack-telemetry | 15:09 | |
| *** llu-laptop is now known as llu | 15:10 | |
| *** Liuqing_ has quit IRC | 15:11 | |
| jd__ | alejandrito: generic? | 15:12 |
| jd__ | not sure we have anything for that yet | 15:12 |
| *** exploreshaifali has quit IRC | 15:12 | |
| alejandrito | jd__, oka | 15:13 |
| alejandrito | jd__, one resource type could be "hypervisor" for example | 15:13 |
| alejandrito | jd__, waiting for you patch to debug ^_^ | 15:13 |
| *** larainema has quit IRC | 15:16 | |
| *** larainema has joined #openstack-telemetry | 15:18 | |
| openstackgerrit | Lianhao Lu proposed openstack/ceilometer: Fix an indent nit of enforce_limit method https://review.openstack.org/246744 | 15:21 |
| jd__ | alejandrito: can you try this http://paste.openstack.org/show/479265/ ? it's a simple approach that will first tell us what's read | 15:21 |
| jd__ | and why it's corrupted | 15:21 |
| jd__ | I'd like to see if it's a totally empty file, or a partial file for example | 15:21 |
| alejandrito | jd__, trying RIGHT NOW ! | 15:22 |
| jd__ | ok thanks | 15:22 |
| jd__ | I'm gonna check librados doc at the same time as I'm not really familiar with it, to see if I can grab extra info | 15:22 |
| jd__ | alejandrito: just restart metricd after applying the patch, that's the only one affected here | 15:23 |
| alejandrito | jd__, oka perfect, can i ask how to apply the patch ? | 15:23 |
| * alejandrito :$ | 15:23 | |
| jd__ | alejandrito: go to the source dir of Gnocchi and type: patch -p1 -i yourpatch | 15:24 |
| jd__ | it should apply it seamlessly :) | 15:24 |
| jd__ | you can download the raw patch here http://paste.openstack.org/raw/479265/ | 15:24 |
| alejandrito | jd__, perfect, yourpatch is the filename ? and then python setup.py install ? | 15:25 |
| jd__ | yes | 15:26 |
| *** chaozhechen_ has quit IRC | 15:27 | |
| *** pradk has joined #openstack-telemetry | 15:28 | |
| *** yprokule has joined #openstack-telemetry | 15:28 | |
| openstackgerrit | Zi Lian Ji proposed openstack/ceilometer-specs: Enable LBaaS V2 for Ceilometer https://review.openstack.org/244139 | 15:28 |
| *** ildikov has quit IRC | 15:28 | |
| *** yprokule has quit IRC | 15:30 | |
| *** yprokule has joined #openstack-telemetry | 15:30 | |
| *** yprokule has quit IRC | 15:32 | |
| *** yprokule has joined #openstack-telemetry | 15:32 | |
| alejandrito | jd__, ok, running | 15:33 |
| alejandrito | jd__, want me to pastebin you what i see ? or want the whole file by mail ? | 15:33 |
| jd__ | alejandrito: just paste me what you see | 15:33 |
| alejandrito | jd__, i'll just paste when the data corription message appears | 15:34 |
| jd__ | alejandrito: 👍 | 15:34 |
| alejandrito | jd__, http://pastebin.com/LV76HUmR | 15:37 |
| jd__ | lol damn it | 15:37 |
| jd__ | alejandrito: I'm gonna fix my debug ptch now :) | 15:39 |
| jd__ | Python and bytes… | 15:39 |
| alejandrito | jd__, hahahaha ! great | 15:39 |
| jd__ | alejandrito: http://paste.openstack.org/show/479271/ | 15:42 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: deprecate TimeSerieArchive https://review.openstack.org/240905 | 15:50 |
| alejandrito | jd__, testing | 15:53 |
| *** annasort_ has quit IRC | 15:54 | |
| alejandrito | jd__, should i apply this one over the already patched ? or over the original one ? because its giving me this : http://pastebin.com/0Mb5LvTJ | 15:55 |
| jd__ | alejandrito: no revert the original one | 15:56 |
| jd__ | git checkout -f | 15:56 |
| jd__ | that should remove the previous patch | 15:56 |
| alejandrito | jd__, great | 15:56 |
| alejandrito | jd__, patched, restarting everything | 15:56 |
| *** pradk_ has joined #openstack-telemetry | 15:58 | |
| alejandrito | jd__, running ... didnt hit a corrupted message still | 16:00 |
| *** liamji has joined #openstack-telemetry | 16:06 | |
| alejandrito | jd__, http://pastebin.com/8RkKwhQQ | 16:07 |
| *** tomoiaga has quit IRC | 16:08 | |
| *** rcernin has quit IRC | 16:09 | |
| jd__ | alejandrito: ok http://paste.openstack.org/show/479279/ this is a simpler version with only the len of the content | 16:11 |
| jd__ | alejandrito: just send me the corrupted file then if you can | 16:11 |
| alejandrito | jd__, that log will give me a ceph corrupted filename , so i can send you that file via email ? is that it ? | 16:12 |
| *** rickyrem has joined #openstack-telemetry | 16:13 | |
| *** jaypipes has quit IRC | 16:13 | |
| jd__ | alejandrito: no, do you need the filename? | 16:13 |
| jd__ | this will just give me the size of the corrupted file | 16:13 |
| alejandrito | jd__, hmmmm ... then i didnt got what i need to do based on that patch, sorry J :( | 16:14 |
| jd__ | alejandrito: the 3rd one? | 16:14 |
| alejandrito | jd__, yeahp | 16:14 |
| jd__ | yeah it does not print out any file name | 16:14 |
| jd__ | just the length of the corrupted data | 16:14 |
| jd__ | the filename should be pretty easy it has the metric id in it | 16:14 |
| alejandrito | jd__, oka doing it | 16:15 |
| sileht | jd__, alejandrito the object name in rados is 'gnocchi_<metric_id>_<aggregation_method>' | 16:15 |
| jd__ | hm except that I think metricd will replace it by a non corrupted version | 16:16 |
| sileht | true | 16:16 |
| jd__ | alejandrito: wait a sec, I'll update the patch to remove the creation of a new file when corrupted data | 16:16 |
| alejandrito | jd__, great so i can send you the original file | 16:16 |
| *** rickyrem1 has joined #openstack-telemetry | 16:16 | |
| alejandrito | jd__, WAITING | 16:17 |
| jd__ | exactly | 16:17 |
| *** rickyrem has quit IRC | 16:17 | |
| jd__ | alejandrito: http://paste.openstack.org/show/479281/ should do it | 16:17 |
| alejandrito | jd__, applying | 16:17 |
| *** wolsen has joined #openstack-telemetry | 16:19 | |
| *** edmondsw has quit IRC | 16:19 | |
| *** Ephur has joined #openstack-telemetry | 16:20 | |
| *** fawadkhaliq has quit IRC | 16:20 | |
| *** jfluhmann has joined #openstack-telemetry | 16:22 | |
| alejandrito | jd__, http://pastebin.com/YsdvFsT6 | 16:23 |
| *** jwcroppe_ has quit IRC | 16:24 | |
| jd__ | alejandrito: hum you're sure you've applied it and reinstalled? | 16:24 |
| jd__ | looks like the old error | 16:24 |
| *** belmoreira has quit IRC | 16:24 | |
| alejandrito | jd__, oh good ... i hate myself ... sorry J | 16:24 |
| jd__ | I know I'm bad but I hope not that bad | 16:24 |
| jd__ | :D | 16:24 |
| alejandrito | jd__, AJJAJAAJAJAJAJA my bad | 16:25 |
| *** exploreshaifali has joined #openstack-telemetry | 16:25 | |
| alejandrito | jd__, running | 16:25 |
| openstackgerrit | Béla Vancsics proposed openstack/ceilometer: Reduced the complexity of the send method https://review.openstack.org/247015 | 16:26 |
| alejandrito | jd__, http://paste.openstack.org/show/479288/ getting rados object ... via email ? | 16:28 |
| *** edmondsw has joined #openstack-telemetry | 16:28 | |
| jd__ | alejandrito: is it big? | 16:28 |
| jd__ | email should be ok I don't think it's that big | 16:28 |
| alejandrito | jd__, let me check | 16:29 |
| jd__ | 16384 is not big | 16:29 |
| jd__ | but it is really suspiscious indeed | 16:29 |
| jd__ | cc sileht | 16:29 |
| jd__ | is it a block size or something sileht ? | 16:29 |
| sileht | jd__, from the rados API point of view we didn't deal with block size | 16:31 |
| jd__ | hmhm | 16:31 |
| alejandrito | jd__, rados -p gnocchi get gnocchi_a2beb7f1-65c1-473c-8f6d-fc912452a6bb_mean | 16:32 |
| alejandrito | jd__, 18K email ? | 16:32 |
| jd__ | alejandrito: awesome | 16:32 |
| jd__ | yup yup | 16:32 |
| alejandrito | jd__, sending | 16:32 |
| jd__ | i'll decode msgpack manually so fun | 16:32 |
| alejandrito | jd__, hahahahaah ! if it is to know the root cause ... hope you get a good one ! | 16:33 |
| alejandrito | jd__, emailing | 16:33 |
| openstackgerrit | Béla Vancsics proposed openstack/ceilometer: Reduced complexity of get_meter_statistics method https://review.openstack.org/247021 | 16:34 |
| sileht | jd__, bs is 8192 | 16:34 |
| sileht | jd__, I was wrong, we deal with the block size in the driver | 16:34 |
| jd__ | sileht: with the offset in reading? | 16:35 |
| sileht | jd__, yes | 16:35 |
| sileht | jd__, I guess the culprit is around this piece of code | 16:35 |
| alejandrito | jd__, sileht email sent | 16:36 |
| jd__ | sileht: though alejandrito just said the result of getting with rados -p is 18K | 16:36 |
| jd__ | so not sure Gnocchi is wrong | 16:36 |
| alejandrito | yeahp ... maybe because the file just got data after being recreated because the previous corruption | 16:37 |
| jd__ | sileht: -rw------- 1 jd staff 18168 Nov 18 17:37 gnocchi_corrupted_ceph | 16:38 |
| jd__ | the file is not corrupted it's brand new | 16:38 |
| jd__ | but it is 18168 not 16384 | 16:38 |
| jd__ | so yeah you're right we're miss-reading it for whatever reason | 16:38 |
| openstackgerrit | Pradeep Kilambi proposed openstack/gnocchi: Ensure file basepath exists https://review.openstack.org/245348 | 16:39 |
| openstackgerrit | Béla Vancsics proposed openstack/ceilometer: Reduced the complexity of the get_events method https://review.openstack.org/247029 | 16:46 |
| *** jwcroppe has joined #openstack-telemetry | 16:46 | |
| openstackgerrit | Zi Lian Ji proposed openstack/ceilometer-specs: Enable LBaaS V2 for Ceilometer https://review.openstack.org/244139 | 16:47 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: ceph: fix computation of read offset https://review.openstack.org/247031 | 16:48 |
| sileht | jd__, WDT ? https://review.openstack.org/247031 | 16:48 |
| jd__ | sileht: makes no sense to me | 16:50 |
| sileht | jd__, I will write a test | 16:50 |
| jd__ | sileht: even read(object_name, offset=0) should read the whole file | 16:50 |
| *** liamji has quit IRC | 16:50 | |
| jd__ | not only 16k | 16:50 |
| jd__ | IIUC http://docs.ceph.com/docs/v0.94/rados/api/python/ | 16:50 |
| sileht | jd__, read use a 8k buffer | 16:51 |
| jd__ | ah yeah | 16:51 |
| jd__ | I thought the C API had no length | 16:51 |
| *** elemoine has joined #openstack-telemetry | 16:52 | |
| sileht | I was wrong it have a length but we use the default | 16:52 |
| jd__ | so 8k? | 16:52 |
| sileht | yes | 16:53 |
| jd__ | yeah your patch seems right now that I think about it | 16:53 |
| jd__ | it's late -_- | 16:53 |
| jd__ | sileht: alejandrito can try it | 16:53 |
| *** jaypipes has joined #openstack-telemetry | 16:53 | |
| *** fawadkhaliq has joined #openstack-telemetry | 16:54 | |
| *** fawadkhaliq has quit IRC | 16:55 | |
| *** fawadkhaliq has joined #openstack-telemetry | 16:55 | |
| openstackgerrit | Béla Vancsics proposed openstack/ceilometer: Reduced the complexity of the __init__ method https://review.openstack.org/247037 | 16:55 |
| *** khushbu_ has joined #openstack-telemetry | 16:57 | |
| *** prashantD has joined #openstack-telemetry | 16:58 | |
| *** belmoreira has joined #openstack-telemetry | 16:58 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: ceph: fix computation of read offset https://review.openstack.org/247031 | 17:01 |
| sileht | jd__, alejandrito : with a test: https://review.openstack.org/247031 | 17:01 |
| alejandrito | sileht, jd__ just got back, let me read everything :P | 17:02 |
| jd__ | sileht: 👍 | 17:03 |
| jd__ | i'll approve if alejandrito +1 | 17:03 |
| sileht | damn it, my unicode font is broken again on term ! | 17:04 |
| *** khushbu_ has quit IRC | 17:06 | |
| *** sileht has quit IRC | 17:10 | |
| *** sileht has joined #openstack-telemetry | 17:11 | |
| jd__ | alejandrito: got it? | 17:14 |
| * alejandrito is reading | 17:14 | |
| alejandrito | jd__, perfect, so .. want me to apply https://review.openstack.org/#/c/247031/ and re-try everything ? without deleting and recreating the ceph pool ? | 17:17 |
| *** khushbu_ has joined #openstack-telemetry | 17:19 | |
| *** rickyrem1 has quit IRC | 17:20 | |
| *** ViswaV has quit IRC | 17:20 | |
| *** khushbu_ has quit IRC | 17:21 | |
| jd__ | alejandrito: yeah | 17:21 |
| *** exploreshaifali has quit IRC | 17:22 | |
| *** ViswaV has joined #openstack-telemetry | 17:22 | |
| alejandrito | jd__, trying | 17:22 |
| *** exploreshaifali has joined #openstack-telemetry | 17:23 | |
| *** khushbu_ has joined #openstack-telemetry | 17:24 | |
| *** belmoreira has quit IRC | 17:25 | |
| *** changbl has joined #openstack-telemetry | 17:29 | |
| *** Ala has quit IRC | 17:29 | |
| alejandrito | jd__, sileht running | 17:30 |
| *** ildikov has joined #openstack-telemetry | 17:30 | |
| *** gordc has joined #openstack-telemetry | 17:33 | |
| alejandrito | jd__, sileht with the existing data should i not see "corrupted" messages anymore ? because im seeing them | 17:34 |
| jd__ | alejandrito: you should not, you're sure you applied it? :( | 17:35 |
| alejandrito | jd__, sileht let me double check ... cause maybe i didnt "setup installed" again :( | 17:35 |
| * jd__ crosses his fingers | 17:36 | |
| * alejandrito f*** again ... re-running ^_^ | 17:37 | |
| *** yassine__ has quit IRC | 17:38 | |
| *** khushbu_ has quit IRC | 17:38 | |
| *** elemoine has quit IRC | 17:39 | |
| alejandrito | jd__, sileht not seeing corrupted till now | 17:41 |
| alejandrito | jd__, should happened by now, i'm +1 to the fix to merge it | 17:43 |
| jd__ | kewl | 17:43 |
| jd__ | thanks alejandrito | 17:43 |
| alejandrito | jd__, i didnt quite got what the problem was even after reading everything | 17:44 |
| alejandrito | jd__, can you clarify to me please ? | 17:44 |
| jd__ | alejandrito: Gnocchi was reading the file from Ceph in a wrong manner :( | 17:45 |
| jd__ | so it was only reading the first few Kb | 17:45 |
| jd__ | and so that made the file appears corrupted | 17:45 |
| *** exploreshaifali has quit IRC | 17:47 | |
| *** exploreshaifali has joined #openstack-telemetry | 17:48 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: carbonara: deprecate TimeSerieArchive https://review.openstack.org/240905 | 17:49 |
| alejandrito | jd__, reading the fix to understand the difference between content being summarized as offset and data | 17:49 |
| alejandrito | jd__, sileht so .... do you think its because of THIS that my original message from today about not having yesterdays data ? | 17:58 |
| jd__ | alejandrito: yes | 17:59 |
| jd__ | alejandrito: you'll tell us if we were wrong tomorrow :)) | 18:00 |
| *** yprokule has quit IRC | 18:00 | |
| alejandrito | jd__, hope to tell you you were right ! what i dont understand its WHY happened from one day to another just miss-reading data ? | 18:00 |
| jd__ | alejandrito: I think it depends on the file size | 18:01 |
| *** khushbu_ has joined #openstack-telemetry | 18:02 | |
| jd__ | it's likely the way read() returned our data and how the function misbehaved was transparent for certain file sizes | 18:02 |
| *** belmoreira has joined #openstack-telemetry | 18:04 | |
| *** khushbu_ has quit IRC | 18:05 | |
| *** belmoreira has quit IRC | 18:05 | |
| alejandrito | jd__, i see, what i dont understand (and sorry again) is why the "data" being read at a specific moment and its len as an incremental of offset is any different of the content variable used ( and i know im not reading something, thats why i want to know :D ) | 18:06 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: use gnocchiclient instead of requests https://review.openstack.org/237538 | 18:08 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: Use keystoneauth1 instead of manual setup https://review.openstack.org/237537 | 18:08 |
| alejandrito | cc sileht ^^ | 18:12 |
| openstackgerrit | Merged openstack/ceilometer: Move the content of ReleaseNotes to README.rst https://review.openstack.org/246743 | 18:20 |
| *** fawadkhaliq has quit IRC | 18:21 | |
| *** jaypipes has quit IRC | 18:34 | |
| *** jwcroppe_ has joined #openstack-telemetry | 18:52 | |
| *** jwcroppe has quit IRC | 18:55 | |
| *** harlowja has quit IRC | 19:05 | |
| *** jwcroppe_ is now known as jwcroppe | 19:06 | |
| *** nadya_ has quit IRC | 19:06 | |
| *** harlowja has joined #openstack-telemetry | 19:08 | |
| *** prashantD has quit IRC | 19:20 | |
| *** prashantD has joined #openstack-telemetry | 19:23 | |
| *** bapalm has quit IRC | 19:23 | |
| *** elemoine has joined #openstack-telemetry | 19:24 | |
| *** bapalm has joined #openstack-telemetry | 19:26 | |
| *** vishwanathj has quit IRC | 19:29 | |
| *** nadya has joined #openstack-telemetry | 19:37 | |
| *** prashantD has quit IRC | 19:42 | |
| *** prashantD has joined #openstack-telemetry | 19:44 | |
| *** ddieterly has quit IRC | 19:45 | |
| gordc | alejandrito: http://docs.ceph.com/docs/v0.94/rados/api/python/#rados.Ioctx.read | 19:46 |
| *** pradk_ has quit IRC | 19:47 | |
| gordc | alejandrito: from what i can tell, by default, read only returns a limited chunk of data. | 19:47 |
| *** KrishR has joined #openstack-telemetry | 19:47 | |
| gordc | the offset uses data and not content because offset is a sum of all past data lengths. | 19:48 |
| gordc | if we take content it'll be too long as content is allready the sum of all data. i think to use len(content), offset shouldn't be adding the total each loop | 19:49 |
| *** onder has quit IRC | 19:50 | |
| *** onder has joined #openstack-telemetry | 19:53 | |
| alejandrito | gordc, let me read what you stated | 19:56 |
| gordc | it's probably a bit convoluted what i typed. basically 'content' is the aggregate of 'data' https://github.com/openstack/gnocchi/blob/master/gnocchi/storage/ceph.py#L200 | 19:59 |
| alejandrito | gordc, NOW i think i get it ^_^ | 20:00 |
| gordc | :) we could probably use len(content) as offset but if it works, it works. | 20:01 |
| alejandrito | gordc, you mean ... not offset += len(content) but offset = len(content) | 20:02 |
| gordc | alejandrito: right | 20:02 |
| alejandrito | gordc, totally understood | 20:03 |
| * gordc writes ceph expert on cv | 20:04 | |
| alejandrito | gordc, what i cant believe is that ... EVERY time till today, that a ceph file in gnocchi was bigger than 8K (the default BS on read) the second read was gonna be ok, but in THAT SECCOND LOOP, offset would have a wrong value, so ... any file bigger than 16K was (or IS, since the fix is not merged) gonna give CORRUPTED FILE | 20:09 |
| * alejandrito doesnt want to believe that, but seems so :O | 20:09 | |
| alejandrito | gordc, well ... its true ... since our metrics didnt last more than a day ( because they exceded the 16K) | 20:10 |
| gordc | yeah, that makes sense. 8k seems to be default read limit, so after first two reads, it should still make sense... and after that, the offset expotentially jumps rather than incremental. | 20:11 |
| alejandrito | gordc, exactly, that was it | 20:11 |
| gordc | good catch by sileht... subtle errors are always the biggest. | 20:12 |
| *** ddieterly has joined #openstack-telemetry | 20:13 | |
| alejandrito | gordc, yeah , thankfully we could keep the "corrupted" environment long enough to debug with jd__ and sileht , im happy ^_^ | 20:23 |
| gordc | alejandrito: appreciate you being guinea pig for gnocchi :) | 20:24 |
| alejandrito | gordc, (o.o) | 20:24 |
| jd__ | alejandrito: yeah I think we sucked at our testing, thanks to you we found a big bug :p | 20:30 |
| jd__ | we didn't test large archive enough | 20:30 |
| *** harlowja has quit IRC | 20:36 | |
| *** nadya has quit IRC | 20:36 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements https://review.openstack.org/247108 | 20:38 |
| *** harlowja has joined #openstack-telemetry | 20:41 | |
| *** thorst has quit IRC | 20:47 | |
| *** thorst has joined #openstack-telemetry | 20:48 | |
| *** thorst has quit IRC | 20:49 | |
| alejandrito | jd__, im really happy that we found it ... i remember the developers saying ... "that data disappears at 00hs" and me saying ... IMPOSIBLE hahahahahaha | 20:49 |
| *** thorst has joined #openstack-telemetry | 20:49 | |
| *** thorst has quit IRC | 20:49 | |
| *** thorst has joined #openstack-telemetry | 20:50 | |
| gordc | alejandrito: it's the cloud. anything can disappear. | 20:51 |
| alejandrito | gordc, anything but ceilometer data ! ^_^ | 20:52 |
| *** shardy_ has joined #openstack-telemetry | 20:52 | |
| gordc | alejandrito: :) | 20:52 |
| *** thorst_ has joined #openstack-telemetry | 20:53 | |
| *** thorst has quit IRC | 20:54 | |
| *** sergio__nubeliu has quit IRC | 20:55 | |
| *** thorst_ has quit IRC | 20:57 | |
| *** shardy_ has quit IRC | 21:00 | |
| *** openstack has joined #openstack-telemetry | 21:03 | |
| *** alejandrito has quit IRC | 21:04 | |
| *** thorst has joined #openstack-telemetry | 21:11 | |
| *** elemoine has quit IRC | 21:12 | |
| openstackgerrit | George Peristerakis proposed openstack/ceilometer: Load a directory of YAML event config files https://review.openstack.org/247177 | 21:13 |
| *** thorst has quit IRC | 21:15 | |
| openstackgerrit | George Peristerakis proposed openstack/ceilometer: Load a directory of YAML event config files https://review.openstack.org/247177 | 21:18 |
| *** changbl has joined #openstack-telemetry | 21:19 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements https://review.openstack.org/247108 | 21:22 |
| *** thorst has joined #openstack-telemetry | 21:24 | |
| *** thorst_ has joined #openstack-telemetry | 21:25 | |
| *** thorst has quit IRC | 21:28 | |
| *** rickyrem has joined #openstack-telemetry | 21:33 | |
| *** rickyrem has quit IRC | 21:43 | |
| *** julim has quit IRC | 21:51 | |
| *** changbl has quit IRC | 22:02 | |
| *** llu has quit IRC | 22:24 | |
| openstackgerrit | gordon chung proposed openstack/aodh: support queue based communication between evaluator and notifier https://review.openstack.org/247211 | 22:26 |
| openstackgerrit | Merged openstack/ceilometer: Fix an indent nit of enforce_limit method https://review.openstack.org/246744 | 22:29 |
| *** dan-t has quit IRC | 22:45 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements https://review.openstack.org/247108 | 22:47 |
| *** jaypipes has joined #openstack-telemetry | 22:47 | |
| *** edmondsw has quit IRC | 22:49 | |
| *** gordc has quit IRC | 22:50 | |
| *** rjaiswal has joined #openstack-telemetry | 22:52 | |
| *** safchain has quit IRC | 22:55 | |
| openstackgerrit | Merged openstack/gnocchi: Ensure file basepath exists https://review.openstack.org/245348 | 22:57 |
| openstackgerrit | Merged openstack/gnocchi: ceph: fix computation of read offset https://review.openstack.org/247031 | 22:57 |
| *** pradk has quit IRC | 23:19 | |
| *** ddieterly has quit IRC | 23:27 | |
| *** rbak has quit IRC | 23:35 | |
| *** thorst_ has quit IRC | 23:55 | |
| *** ddieterly has joined #openstack-telemetry | 23:56 | |
| openstackgerrit | Merged openstack/ceilometermiddleware: Updated from global requirements https://review.openstack.org/247125 | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!