| *** zhuangl_ has joined #openstack-telemetry | 00:14 | |
| *** zhuangl_ has quit IRC | 00:18 | |
| *** thorst_ has quit IRC | 00:28 | |
| *** thorst_ has joined #openstack-telemetry | 00:29 | |
| *** mnaser has quit IRC | 00:31 | |
| *** afazekas_ has quit IRC | 00:32 | |
| *** afazekas has joined #openstack-telemetry | 00:32 | |
| *** thorst_ has quit IRC | 00:37 | |
| *** mnaser has joined #openstack-telemetry | 00:42 | |
| *** thorst_ has joined #openstack-telemetry | 00:56 | |
| *** cheneydc has joined #openstack-telemetry | 01:03 | |
| *** thorst_ has quit IRC | 01:12 | |
| *** thorst_ has joined #openstack-telemetry | 01:13 | |
| *** lsmola has quit IRC | 01:21 | |
| *** thorst_ has quit IRC | 01:21 | |
| Kevin_Zheng | I thought about the name thing last night | 01:21 |
|---|---|---|
| Kevin_Zheng | alarm-history can also use name | 01:21 |
| *** yarkot_ has joined #openstack-telemetry | 01:24 | |
| *** chlong has quit IRC | 01:26 | |
| *** yarkot_ has quit IRC | 01:33 | |
| *** tongli has quit IRC | 01:34 | |
| *** Eris72 has joined #openstack-telemetry | 01:47 | |
| *** mdih has joined #openstack-telemetry | 01:53 | |
| *** ljxiash has joined #openstack-telemetry | 02:13 | |
| *** caisan has joined #openstack-telemetry | 02:22 | |
| *** nicodemus_ has quit IRC | 02:30 | |
| *** yuntongjin1 has joined #openstack-telemetry | 02:36 | |
| *** cheneydc has quit IRC | 02:54 | |
| *** yuntongjin1 has quit IRC | 03:13 | |
| *** dave-mccowan has quit IRC | 03:17 | |
| *** ljxiash has quit IRC | 03:22 | |
| *** ljxiash has joined #openstack-telemetry | 03:26 | |
| *** chlong has joined #openstack-telemetry | 03:30 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/python-aodhclient: Add name support for show and delete https://review.openstack.org/292305 | 03:32 |
| *** links has joined #openstack-telemetry | 03:38 | |
| *** ildikov has quit IRC | 03:39 | |
| *** jdandrea_ has quit IRC | 03:48 | |
| *** jdandrea has joined #openstack-telemetry | 03:51 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/python-aodhclient: Add name support for show, update and delete https://review.openstack.org/292305 | 04:04 |
| *** flwang1 has quit IRC | 04:05 | |
| *** jwcroppe has joined #openstack-telemetry | 04:08 | |
| *** jwcroppe has quit IRC | 04:08 | |
| *** jwcroppe has joined #openstack-telemetry | 04:09 | |
| *** yuntongjin has joined #openstack-telemetry | 04:26 | |
| *** yprokule has joined #openstack-telemetry | 04:47 | |
| *** Eris72 has quit IRC | 04:52 | |
| *** ljxiash has quit IRC | 04:56 | |
| *** ljxiash has joined #openstack-telemetry | 04:57 | |
| *** ljxiash has quit IRC | 05:01 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/python-aodhclient: Add name support for show, update and delete https://review.openstack.org/292305 | 05:02 |
| *** _nadya_ has joined #openstack-telemetry | 05:37 | |
| *** mwang14 has joined #openstack-telemetry | 05:37 | |
| mwang14 | Has anyone successfully deployed ceilometer with gnocchi? | 05:38 |
| mwang14 | In ceilometer.conf, [dispatch_gnocchi], what does the field "filter_project = " mean? The default setting is "gnocchi". Does it mean I have to create a tenant (project) named "gnocchi" in keystone? | 05:40 |
| *** ljxiash has joined #openstack-telemetry | 05:54 | |
| *** _nadya_ has quit IRC | 06:03 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer: Imported Translations from Zanata https://review.openstack.org/292715 | 06:07 |
| openstackgerrit | liusheng proposed openstack/python-aodhclient: Add --filter to "alarm list" https://review.openstack.org/292233 | 06:11 |
| openstackgerrit | liusheng proposed openstack/python-aodhclient: Add --query to alarm list https://review.openstack.org/291541 | 06:11 |
| openstackgerrit | liusheng proposed openstack/python-aodhclient: Don't support "alarm search" and mandatory --type in "alarm list" https://review.openstack.org/283958 | 06:11 |
| mwang14 | Hi liusheng, may I ask you a question about the setting of "ceilometer.conf"? | 06:13 |
| liusheng | mwang14: hi, you mean the question about ceilometer deployment with gnocchi ? | 06:16 |
| mwang14 | yes! | 06:16 |
| mwang14 | in [dispatcher_gnocchi], there is a field "filter_project = gnocchi". | 06:17 |
| mwang14 | I create a tenant/project named "gnocchi" | 06:17 |
| mwang14 | in keystone | 06:17 |
| liusheng | I just checked the config option, which is the gnocchi project as default and use to skip the sample generated by gnocchi itself | 06:24 |
| liusheng | mwang14: the samples may generated by gnocchi project itself, or the resource_id of samples is gnocchi project_id and the resource_type is swift count (gnocchi may use swift as storage backend), these two cases should be filtered out | 06:29 |
| mwang14 | is there any need to create project "gnocchi" in keystone? | 06:31 |
| liusheng | mwang14: if you want to use gnocchi as metric storage, I think it need | 06:33 |
| openstackgerrit | Zhenyu Zheng proposed openstack/python-aodhclient: Add name support for show, update and delete https://review.openstack.org/292305 | 06:34 |
| mwang14 | yes, I create a project in keystone. also attach a user with the project "gnocchi" with "admin" role. | 06:35 |
| mwang14 | after getting the token from keystone, using cURL, i can access the swift, v1/AUTH_<gnocchi_project_id> | 06:36 |
| mwang14 | but, when I start the ceilometer-collector, it prompts "ERROR ceilometer.dispatcher.gnocchi [-] fail to retrieve user of Gnocchi service". | 06:37 |
| mwang14 | This error is triggered by code: fetching the project id of "gnocchi" using swift-client. | 06:37 |
| liusheng | mwang14: Hmm, you have configured the filter_project as 'gnocchi' ? | 06:43 |
| mwang14 | yes | 06:43 |
| liusheng | mwang14: any error traceback raised? | 06:44 |
| mwang14 | 2016-03-15 14:18:32.767 22022 ERROR ceilometer.dispatcher.gnocchi [-] fail to retrieve user of Gnocchi service 2016-03-15 14:18:32.767 22022 ERROR ceilometer.dispatcher.gnocchi Traceback (most recent call last): 2016-03-15 14:18:32.767 22022 ERROR ceilometer.dispatcher.gnocchi File "/usr/local/lib/python2.7/dist-packages/ceilometer/dispatcher/gnocchi.py", line 240, in gnocchi_project_id 2016-03-15 14:18:32.767 22022 ERRO | 06:45 |
| mwang14 | sorry, the paste of log is completely a mess | 06:45 |
| mwang14 | ceilometer/dispatcher/gnocchi.py", line 240, in gnocchi_project_id | 06:45 |
| liusheng | mwang14: seems a bug, if you just want to run the collector service, you can try to config filter_service_activity as False to work aroud this | 06:45 |
| mwang14 | name=self.conf.dispatcher_gnocchi.filter_project) | 06:45 |
| liusheng | mwang14: and you can file a bug to describe this issue | 06:46 |
| mwang14 | project = self._ks_client.tenants.find(name=self.conf.dispatcher_gnocchi.filter_project) | 06:46 |
| mwang14 | I print self.conf.dispatcher_gnocchi.filter_project; it return "gnocchi" | 06:47 |
| liusheng | mwang14: as you described, it looks the keystoneclient cannot retrive the project of "gnocchi" | 06:48 |
| mwang14 | yes | 06:49 |
| mwang14 | but i am sure the keystone configuration in ceilometer.conf is fine; since when the storage dispatcher is "database", instead of "gnocchi", everything works fine. | 06:50 |
| *** ildikov_ has quit IRC | 06:53 | |
| liusheng | mwang14: yes, if you use database dispatcher, it won't go into the gnocchi dispatcher logic | 06:54 |
| mwang14 | liusheng: I create a project "gnocchi", a user "gnocchi", both under the "Default" domain. the user "gnocchi" is assigned the role "admin" to the project "gnocchi" | 06:55 |
| mwang14 | liusheng: in addition, I create the service "gnocchi", and endpoints as well in keystone. | 06:55 |
| *** vishwana_ has joined #openstack-telemetry | 07:07 | |
| *** vishwanathj_away has quit IRC | 07:10 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: devstack: allow uwsgi deployments https://review.openstack.org/292077 | 07:12 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: devstack: allow uwsgi deployments https://review.openstack.org/292077 | 07:13 |
| *** tesseract has joined #openstack-telemetry | 07:13 | |
| *** tesseract is now known as Guest97519 | 07:13 | |
| *** vishwana_ has quit IRC | 07:16 | |
| liusheng | mwang14: can your ceilometer-collector run with filter_service_activity configured as False ? | 07:18 |
| mwang14 | liusheng: let me try | 07:33 |
| mwang14 | liusheng: there is no problem, after setting "filter_service_activity=true". | 07:35 |
| mwang14 | liusheng: there is no problem, after setting "filter_service_activity=false" | 07:35 |
| liusheng | mwang14: ...I cannot reproduce your problem, I also added project "gnocchi" and user "gnocchi", and configured the filter_project as "gnocchi", it is OK | 07:44 |
| liusheng | mwang14: 2016-03-15 15:43:17.970 46343 DEBUG ceilometer.dispatcher.gnocchi [-] gnocchi project found: dfd4b11a7fc341f5834a92b56c191674 gnocchi_project_id /opt/stack/ceilometer/ceilometer/dispatcher/gnocchi.py:267 | 07:45 |
| mwang14 | I am using the gnocchi stable/2.0 version | 07:46 |
| mwang14 | the ceilometer is liberty | 07:46 |
| mwang14 | are you using keystone API v3 or v2.0 | 07:47 |
| mwang14 | i.e., auth_version | 07:47 |
| liusheng | mwang14: oh, I tested with lastet code with devstack | 07:52 |
| liusheng | mwang14: keystone v2 | 07:53 |
| mwang14 | after setting "filter_service_activity = false", the gnocchi keeps receiving the measurements. | 07:56 |
| mwang14 | that's exactly the expected behaviour. | 07:57 |
| *** belmoreira has joined #openstack-telemetry | 07:58 | |
| mwang14 | liusheng: by the way, could you please help check the version of python-keystoneclient? I install using the git source with the branch stable/liberty | 08:00 |
| *** Guest97519 has quit IRC | 08:00 | |
| liusheng | mwang14: 2.3.1 I used | 08:02 |
| *** tesseract has joined #openstack-telemetry | 08:05 | |
| *** tesseract is now known as Guest91794 | 08:06 | |
| *** _nadya_ has joined #openstack-telemetry | 08:06 | |
| mwang14 | liusheng: the version python-keystoneclient of branch stable/liberty is 1.7.5 | 08:08 |
| mwang14 | liusheng: i just configure the gnocchi to use "file" as the storage, rather than "swift"; the "ceilometer-collector" works fine, no errors throwing | 08:09 |
| openstackgerrit | Ilya Tyaptin proposed openstack/ceilometer: [ES] Implement event time to live https://review.openstack.org/200573 | 08:17 |
| openstackgerrit | Ilya Tyaptin proposed openstack/ceilometer: [ES] Implement event time to live https://review.openstack.org/200573 | 08:23 |
| *** pece has joined #openstack-telemetry | 08:28 | |
| *** chlong has quit IRC | 08:32 | |
| sileht | gerrit just died :( | 08:36 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: devstack: allow uwsgi deployments https://review.openstack.org/292077 | 08:36 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/aodh: devstack: allow uwsgi deployments https://review.openstack.org/292245 | 08:40 |
| mwang14 | liusheng: thanks very much! I will dig into the code, ceilometer/dispatcher/gnocchi.py. | 08:42 |
| liusheng | mwang14: np, I didn't help, I don't have an liberty environment ;( | 08:44 |
| *** Guest91794 has quit IRC | 08:45 | |
| *** flwang1 has joined #openstack-telemetry | 08:51 | |
| *** yuntongjin has quit IRC | 08:57 | |
| *** Guest91794 has joined #openstack-telemetry | 09:00 | |
| *** mwang14 has quit IRC | 09:08 | |
| *** safchain has joined #openstack-telemetry | 09:11 | |
| Kevin_Zheng | sileht: Hi | 09:17 |
| sileht | hi | 09:17 |
| Kevin_Zheng | I think your comment in https://review.openstack.org/#/c/291541 is very useful | 09:19 |
| Kevin_Zheng | I think utils.search_query_builder will make users easier to use | 09:19 |
| Kevin_Zheng | but Liusheng is working on add --filter for list | 09:20 |
| Kevin_Zheng | maybe it can be somehow overlapped? | 09:20 |
| Kevin_Zheng | because --filter works with parameters like xxx=xxx | 09:20 |
| sileht | Kevin_Zheng, filter and query are mutualy exclusive so this is not a problem | 09:21 |
| Kevin_Zheng | sileht: OK then, actually I like the method you suggested more | 09:22 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Add some resource types tests https://review.openstack.org/270419 | 09:25 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute number https://review.openstack.org/270091 | 09:25 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute bool https://review.openstack.org/270418 | 09:25 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute uuid https://review.openstack.org/270090 | 09:25 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Don't create Ceilometer resource types by default. https://review.openstack.org/270322 | 09:25 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Move legacy Ceilometer resource into indexer. https://review.openstack.org/270266 | 09:25 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource type CRUD. https://review.openstack.org/269844 | 09:25 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Implements resource attribute string https://review.openstack.org/269888 | 09:25 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Retry to delete resource types tables on deadlock https://review.openstack.org/292776 | 09:25 |
| *** rcernin has joined #openstack-telemetry | 09:31 | |
| *** ljxiash has quit IRC | 09:34 | |
| *** ljxiash has joined #openstack-telemetry | 09:34 | |
| *** yassine__ has joined #openstack-telemetry | 09:35 | |
| *** ljxiash has quit IRC | 09:39 | |
| *** ljxiash has joined #openstack-telemetry | 09:48 | |
| *** ljxiash has quit IRC | 10:11 | |
| *** ljxiash has joined #openstack-telemetry | 10:12 | |
| openstackgerrit | Julien Danjou proposed openstack/python-aodhclient: Clean flake8 paths and unused config options https://review.openstack.org/292791 | 10:15 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: use events to end Gnocchi resource https://review.openstack.org/217017 | 10:16 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: PoC: gnocchi: create/update from event https://review.openstack.org/292793 | 10:16 |
| *** ljxiash has quit IRC | 10:18 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: use events to end Gnocchi resource https://review.openstack.org/217017 | 10:19 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: PoC: gnocchi: create/update from event https://review.openstack.org/292793 | 10:19 |
| *** caisan has quit IRC | 10:19 | |
| *** cdent has joined #openstack-telemetry | 10:21 | |
| *** dave-mccowan has joined #openstack-telemetry | 10:24 | |
| *** fwdit has joined #openstack-telemetry | 10:37 | |
| *** shardy has joined #openstack-telemetry | 10:39 | |
| *** ljxiash has joined #openstack-telemetry | 10:47 | |
| *** cdent has quit IRC | 10:47 | |
| *** cdent has joined #openstack-telemetry | 10:48 | |
| *** ljxiash has quit IRC | 10:52 | |
| *** yassine__ has quit IRC | 10:58 | |
| *** jwcroppe has quit IRC | 11:12 | |
| *** ityaptin has quit IRC | 11:12 | |
| -openstackstatus- NOTICE: Gerrit is going to be restarted | 11:14 | |
| zigo | When running tempest, I get this: http://paste.openstack.org/show/490482/ | 11:27 |
| zigo | jd__: sileht: Any idea why? | 11:27 |
| zigo | Should I upgrade my tempest package? | 11:27 |
| sileht | zigo, Looks like it loads the new aodh tempest plugin (from aodh code tree), but your tempest package still provide the old one (from tempest code tree) | 11:32 |
| *** mdih has quit IRC | 11:32 | |
| -openstackstatus- NOTICE: Gerrit had to be restarted because was not responsive. As a consequence, some of the test results have been lost, from 08:30 UTC to 10:30 UTC approximately. Please recheck any affected jobs by this problem. | 11:34 | |
| sileht | after py day, recheck day ! | 11:35 |
| -openstackstatus- NOTICE: Gerrit had to be restarted because was not responsive. As a consequence, some of the test results have been lost, from 09:30 UTC to 11:30 UTC approximately. Please recheck any affected jobs by this problem. | 11:36 | |
| *** jwcroppe has joined #openstack-telemetry | 11:40 | |
| *** thorst_ has joined #openstack-telemetry | 11:41 | |
| zigo | sileht: Ok, so I guess I should upgrade tempest! :) | 11:54 |
| *** yassine__ has joined #openstack-telemetry | 12:02 | |
| *** yassine__ has quit IRC | 12:03 | |
| *** belmoreira has quit IRC | 12:07 | |
| *** shardy has quit IRC | 12:16 | |
| *** yassine__ has joined #openstack-telemetry | 12:37 | |
| openstackgerrit | litong01 proposed openstack/ceilometer: Add http publisher https://review.openstack.org/290624 | 12:42 |
| *** shardy has joined #openstack-telemetry | 12:44 | |
| *** shardy has quit IRC | 12:47 | |
| *** shardy has joined #openstack-telemetry | 12:48 | |
| *** gordc has joined #openstack-telemetry | 12:50 | |
| cdent | gordc: in my ever growing takeover of the world I'd like to add this channel to my little irc bot, described here: https://anticdent.org/purple-irc-bot.html | 12:52 |
| cdent | okay with you? I'm adding channels piecemeal to see how it deals with volume | 12:52 |
| gordc | cdent: sure go ahead. if anything breaks, i blame you | 13:00 |
| cdent | that's what you do anyway, so no real change | 13:00 |
| *** purplerbot has joined #openstack-telemetry | 13:01 | |
| cdent | p!logs | 13:01 |
| gordc | does it talk back? | 13:02 |
| cdent | in private messages | 13:02 |
| gordc | oh. you should consider building a chatbot. i hear it's big business. | 13:03 |
| cdent | meh | 13:04 |
| cdent | I'm going to make a chatbot that can play go | 13:04 |
| gordc | that talks smack while playing go? | 13:05 |
| *** peristeri has joined #openstack-telemetry | 13:11 | |
| cdent | gordc: duh! | 13:14 |
| *** ig0r__ has joined #openstack-telemetry | 13:19 | |
| *** ig0r_ has quit IRC | 13:22 | |
| *** chlong has joined #openstack-telemetry | 13:25 | |
| *** nicodemus_ has joined #openstack-telemetry | 13:40 | |
| *** pradk has joined #openstack-telemetry | 13:40 | |
| *** ljxiash has joined #openstack-telemetry | 13:56 | |
| *** rbak has joined #openstack-telemetry | 13:56 | |
| *** KrishR has joined #openstack-telemetry | 14:00 | |
| *** Guest41345 has quit IRC | 14:03 | |
| *** ildikov has joined #openstack-telemetry | 14:13 | |
| *** Guest91794 has quit IRC | 14:31 | |
| *** tasdomas has quit IRC | 14:56 | |
| *** tasdomas has joined #openstack-telemetry | 14:56 | |
| *** links has quit IRC | 14:59 | |
| *** ljxiash has quit IRC | 14:59 | |
| -openstackstatus- NOTICE: Launchpad OpenID SSO is currently experiencing issues preventing login. The Launchpad team is working on the issue | 14:59 | |
| *** ChanServ changes topic to "Launchpad OpenID SSO is currently experiencing issues preventing login. The Launchpad team is working on the issue" | 14:59 | |
| *** yprokule has quit IRC | 15:06 | |
| *** tesseract has joined #openstack-telemetry | 15:06 | |
| *** tesseract is now known as Guest71467 | 15:07 | |
| *** fwdit has quit IRC | 15:21 | |
| *** aswadr1 has joined #openstack-telemetry | 15:26 | |
| cdent | jd__++ excellent canidacy message | 15:27 |
| *** aswadr1 has quit IRC | 15:27 | |
| *** aswadr1 has joined #openstack-telemetry | 15:28 | |
| *** purplerbot has quit IRC | 15:30 | |
| *** purplerbot has joined #openstack-telemetry | 15:30 | |
| jd__ | cdent: thanks! | 15:30 |
| *** ChanServ changes topic to "OpenStack Telemetry | http://wiki.openstack.org/Telemetry" | 15:32 | |
| -openstackstatus- NOTICE: Launchpad SSO is back to normal - happy hacking | 15:32 | |
| *** chlong has quit IRC | 15:33 | |
| *** aswadr1 has left #openstack-telemetry | 15:38 | |
| *** vishwanathj has joined #openstack-telemetry | 15:43 | |
| *** tongli has joined #openstack-telemetry | 15:51 | |
| tongli | @gordc, u around? | 15:54 |
| krotscheck | Any cores around? dhellmann posted a couple of rather important bugfixes at https://review.openstack.org/#/c/291891/ and https://review.openstack.org/#/c/291890/ | 15:56 |
| krotscheck | (we already have +2 on them, and the're single-line fixes) | 15:56 |
| *** links has joined #openstack-telemetry | 15:58 | |
| gordc | jd__: sileht: ildikov: _nadya_: cdent: ^ | 16:04 |
| gordc | tongli: whatsup? | 16:04 |
| tongli | @gordc, finally got things running on latest code. now these cadf messages are all considered as events. | 16:06 |
| cdent | krotscheck: they both happy now | 16:06 |
| gordc | cdent: thanks | 16:06 |
| tongli | has to be defined in /etc/ceilometer/event_definition.yaml file. | 16:06 |
| tongli | there is no default traits for cadf messages. | 16:07 |
| gordc | tongli: right. i believe they are already? | 16:07 |
| tongli | if someone wants to see them, they have to be defined in /etc/ceilometer/event_definition.yaml. | 16:08 |
| gordc | tongli: does this not work? https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml#L364-L404 | 16:08 |
| krotscheck | cdent: Much obliged, thanks :) | 16:09 |
| tongli | that would work, the problem is that all the cadf info (fields) will be in the traits member. | 16:10 |
| tongli | the information will have to be extracted out of the what get passed into the publisher. | 16:10 |
| tongli | and I think the event_type actually is audit.http.* | 16:11 |
| gordc | tongli: that's how Events work. you can also store the raw if you want (it's just not indexed so not queryable) | 16:11 |
| tongli | @gordc, raw by default is empty. | 16:12 |
| gordc | yeah, you have to enable it | 16:12 |
| tongli | what flag enables it? | 16:12 |
| *** vishwana_ has joined #openstack-telemetry | 16:13 | |
| tongli | @gordc, the point I guess is that the message itself can not be used as is to be posted for cadf message, they have to be extracted out of the data passed to publisher. | 16:14 |
| gordc | https://github.com/openstack/ceilometer/blob/master/ceilometer/event/converter.py#L36-L40 | 16:14 |
| gordc | ok | 16:15 |
| tongli | https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml#L364-L404 | 16:16 |
| *** vishwanathj has quit IRC | 16:16 | |
| tongli | @gordc, these won't work since now cadf messages event_type actually is 'audit.http.*' | 16:16 |
| tongli | I guess I can update that file at least to make audit messages going somewhere. are you open for that? | 16:17 |
| gordc | ... *http.* should match with audit.http.* | 16:18 |
| tongli | @gordc, u r right. | 16:18 |
| tongli | that should work. | 16:18 |
| gordc | * is a wildcard... if it don't work, it's broken | 16:18 |
| *** vishwana_ is now known as vishwanathj | 16:20 | |
| *** permalac has joined #openstack-telemetry | 16:22 | |
| *** permalac has quit IRC | 16:22 | |
| *** permalac has joined #openstack-telemetry | 16:23 | |
| *** permalac has quit IRC | 16:23 | |
| *** permalac_ has joined #openstack-telemetry | 16:31 | |
| *** permalac_ has quit IRC | 16:31 | |
| *** rcernin has quit IRC | 16:35 | |
| *** purplerbot has quit IRC | 16:41 | |
| openstackgerrit | Merged openstack/aodh: register the config generator default hook with the right name https://review.openstack.org/291890 | 16:41 |
| *** Guest71467 has quit IRC | 16:42 | |
| *** purplerbot has joined #openstack-telemetry | 16:46 | |
| *** links has quit IRC | 16:47 | |
| openstackgerrit | Merged openstack/ceilometer: register the config generator default hook with the right name https://review.openstack.org/291891 | 17:00 |
| openstackgerrit | gordon chung proposed openstack/gnocchi: Revert "Log retrieve/store data speed in Carbonara based drivers" https://review.openstack.org/293045 | 17:01 |
| *** ljxiash has joined #openstack-telemetry | 17:11 | |
| openstackgerrit | gordon chung proposed openstack/gnocchi: Revert "Log retrieve/store data speed in Carbonara based drivers" https://review.openstack.org/293045 | 17:11 |
| *** openstackgerrit has quit IRC | 17:14 | |
| *** openstackgerrit_ is now known as openstackgerrit | 17:14 | |
| *** openstackgerrit has quit IRC | 17:15 | |
| *** ljxiash has quit IRC | 17:15 | |
| *** openstackgerrit_ has joined #openstack-telemetry | 17:15 | |
| *** openstackgerrit_ is now known as openstackgerrit | 17:16 | |
| *** openstackgerrit_ has joined #openstack-telemetry | 17:17 | |
| *** pcaruana has quit IRC | 17:24 | |
| *** yassine__ has quit IRC | 17:30 | |
| *** pcaruana has joined #openstack-telemetry | 17:36 | |
| *** _nadya_ has quit IRC | 18:03 | |
| openstackgerrit | gordon chung proposed openstack/gnocchi: Revert "Log retrieve/store data speed in Carbonara based drivers" https://review.openstack.org/293045 | 18:06 |
| *** terriyu has joined #openstack-telemetry | 18:11 | |
| *** flwang has quit IRC | 18:16 | |
| *** _nadya_ has joined #openstack-telemetry | 18:17 | |
| *** terriyu has quit IRC | 18:22 | |
| *** _nadya_ has quit IRC | 18:23 | |
| *** safchain has quit IRC | 18:26 | |
| *** terriyu has joined #openstack-telemetry | 18:28 | |
| *** flwang has joined #openstack-telemetry | 18:29 | |
| *** pece has quit IRC | 18:32 | |
| nicodemus_ | hello gordc, do you have a moment? | 18:42 |
| *** terriyu has quit IRC | 18:42 | |
| gordc | nicodemus_: sure | 18:48 |
| nicodemus_ | gordc, I've been seeing one gnocchi-metricd detail that think can cause delay in measures processing | 18:51 |
| nicodemus_ | all workers access the "measure" object in the ceph pool with a rw lock | 18:51 |
| nicodemus_ | the more workers you have, the more slow requests you see on the ceph cluster | 18:52 |
| *** PsionTheory has joined #openstack-telemetry | 18:52 | |
| nicodemus_ | I'm adding more workers trying to keep the backlog low | 18:53 |
| nicodemus_ | some slow requests take more than 30 seg, which could be the cause that some metrics take 30+ seconds to be processed | 18:54 |
| gordc | nicodemus_: hmm... i'll need to take a look. | 18:55 |
| gordc | sileht: ^ in case you want to look at this later (or if you're still working for some reason) | 18:55 |
| sileht | nicodemus_, that becase we have to be sure that the object is modified by only one worker at once | 18:56 |
| sileht | nicodemus_, but that looks a good place for performance improvement | 18:57 |
| nicodemus_ | let me know if you'd like me to open a bug report about it | 18:58 |
| gordc | nicodemus_: let's open it so we don't forget | 18:58 |
| nicodemus_ | from time to time I see some metric that takes an unusual long time | 18:58 |
| nicodemus_ | ok | 18:58 |
| sileht | nicodemus_, thx ! | 18:59 |
| gordc | nicodemus_: just curious, what is typical response time vs unusual response time? | 18:59 |
| nicodemus_ | sileht, np | 18:59 |
| nicodemus_ | gordc, it varies with the backlog. Right now with 20k measures waiting, it normally takes between 3 and 5 secs | 19:01 |
| nicodemus_ | a high time would be above 10 secs | 19:02 |
| nicodemus_ | with an 80k backlog, times go way up | 19:02 |
| nicodemus_ | just now one took 140.77 secs | 19:02 |
| *** flwang1 has quit IRC | 19:03 | |
| gordc | hmm... got it. are the numbers based on 'Computed new metric * with * new measures' log? | 19:05 |
| nicodemus_ | gordc, yes. I changed that log to 'info' in order not to have metricd on debug all the time (makes it easier to read and calculate) | 19:06 |
| nicodemus_ | gordc, this is from one HW node running 24 workers: http://paste.openstack.org/show/490586/ | 19:07 |
| gordc | nicodemus_: yeah i added a patch to remove the logs that constantly print 'retrieve, write, expire'... i found it to hard to filter | 19:08 |
| nicodemus_ | n_time is the sum of the reported time from all workers | 19:08 |
| nicodemus_ | and that 'pending' comes from the last 'metricd reporting' line in that 5-minute interval | 19:09 |
| gordc | n_measures is number of metrics? | 19:09 |
| nicodemus_ | yes, also the sum of all "computed new metric" log lines for that 5 minute itnerval | 19:10 |
| nicodemus_ | I meant: n_measures is the number of measures, not metrics | 19:11 |
| gordc | i see.. that's actually not that great. that's with 16 workers? | 19:13 |
| nicodemus_ | right now... I have two dedicated servers with 24 workers each, plus two instances with another 24 workers each | 19:14 |
| *** krotscheck is now known as kro_death_by_sin | 19:14 | |
| nicodemus_ | the servers have 24 cores, the instances 16 vcpus | 19:14 |
| *** kro_death_by_sin is now known as kro_sinus_hell | 19:14 | |
| nicodemus_ | still, the number of pending measures are slowly increasing | 19:14 |
| gordc | ... yikes. seems we need to partition a bit more. | 19:15 |
| gordc | how many metrics are pendings? as oppose to measures | 19:16 |
| gordc | does the pending measures increase a lot faster when you drop a maybe one of the instances or both? i wonder if just the number of workers is exhausting all the ceph connections. | 19:19 |
| nicodemus_ | let me give it a try stopping gnocchi-metricd on both instances | 19:25 |
| nicodemus_ | las report says "23267 measurements bundles across 1494 metrics" | 19:28 |
| nicodemus_ | I've stopped both instances, I'll give them 10 minutes to see if pending measures increase much more or not | 19:29 |
| gordc | kk | 19:30 |
| nicodemus_ | gordc, http://paste.openstack.org/show/490588/ | 19:36 |
| *** PsionTheory has quit IRC | 19:36 | |
| *** ctina has joined #openstack-telemetry | 19:36 | |
| nicodemus_ | last two reports are with both instances stopped (would be 48 workers instead of 96) | 19:36 |
| nicodemus_ | this is the report from the other HW node: http://paste.openstack.org/show/490589/ that shows the same trend | 19:38 |
| nicodemus_ | as if more workers cause them all to process less | 19:38 |
| gordc | yeah, i imagine it's a bunch of factors. i think they're all fighting for same resources/connections. | 19:40 |
| gordc | also, the load isn't actually partitioned across services, just workers. so each separate service you have is potentially clashing with other services (not a negative, just less effective) | 19:41 |
| openstackgerrit | Merged openstack/gnocchi: Revert "Log retrieve/store data speed in Carbonara based drivers" https://review.openstack.org/293045 | 19:41 |
| gordc | ^ that removes the crazy debug logging | 19:42 |
| gordc | i have no idea how to configure/check how many connections ceph allows at any one time. i need to look into that. probably notes we should add on how to configure workers for gnocchi and ceilometer | 19:44 |
| *** _nadya_ has joined #openstack-telemetry | 19:45 | |
| nicodemus_ | now we have 22 OSDs spread across 4 ceph nodes, will add another node later this week | 19:45 |
| nicodemus_ | btw, I created the bug report about the 'measure' object | 19:47 |
| gordc | cool cool. i'll triage it. | 19:50 |
| nicodemus_ | forgot to add gnocchi version, its 2.0.1.dev46 | 19:51 |
| *** ig0r_ has joined #openstack-telemetry | 20:05 | |
| *** ig0r__ has quit IRC | 20:06 | |
| *** eglynn has joined #openstack-telemetry | 20:07 | |
| *** MVenesio has joined #openstack-telemetry | 20:13 | |
| *** ctina has quit IRC | 20:20 | |
| *** flwang1 has joined #openstack-telemetry | 20:31 | |
| *** gordc has quit IRC | 20:34 | |
| *** nicodemus_ has quit IRC | 20:34 | |
| *** flwang1 has quit IRC | 20:37 | |
| openstackgerrit | litong01 proposed openstack/ceilometer: Add http publisher https://review.openstack.org/290624 | 20:38 |
| *** tongli has quit IRC | 20:46 | |
| *** _nadya_ has quit IRC | 20:53 | |
| *** flwang1 has joined #openstack-telemetry | 20:58 | |
| *** gordc has joined #openstack-telemetry | 21:03 | |
| *** thorst_ has quit IRC | 21:03 | |
| *** thorst_ has joined #openstack-telemetry | 21:09 | |
| *** shardy has quit IRC | 21:10 | |
| *** thorst_ has quit IRC | 21:13 | |
| *** vishwanathj has quit IRC | 21:14 | |
| *** thorst_ has joined #openstack-telemetry | 21:15 | |
| *** thorst_ has quit IRC | 21:20 | |
| *** vishwanathj has joined #openstack-telemetry | 21:26 | |
| *** MVenesio has quit IRC | 21:30 | |
| *** thorst_ has joined #openstack-telemetry | 21:33 | |
| *** thorst_ has quit IRC | 21:34 | |
| *** thorst_ has joined #openstack-telemetry | 21:34 | |
| *** eglynn has quit IRC | 21:36 | |
| *** pcaruana has quit IRC | 21:38 | |
| *** pradk has quit IRC | 21:38 | |
| *** pcaruana has joined #openstack-telemetry | 21:52 | |
| *** harlowja has joined #openstack-telemetry | 22:01 | |
| *** kbyrne has quit IRC | 22:01 | |
| *** vishwanathj has quit IRC | 22:02 | |
| *** flwang1 has quit IRC | 22:12 | |
| *** flwang1 has joined #openstack-telemetry | 22:12 | |
| *** david-lyle has quit IRC | 22:13 | |
| *** gordc has quit IRC | 22:15 | |
| *** cdent has quit IRC | 22:15 | |
| *** jwcroppe has quit IRC | 22:19 | |
| *** jwcroppe has joined #openstack-telemetry | 22:19 | |
| *** r-mibu has quit IRC | 22:20 | |
| *** r-mibu has joined #openstack-telemetry | 22:20 | |
| *** slogan has joined #openstack-telemetry | 22:25 | |
| *** shakamunyi has joined #openstack-telemetry | 22:32 | |
| *** hongbin has joined #openstack-telemetry | 22:44 | |
| *** hongbin has left #openstack-telemetry | 22:44 | |
| *** KrishR has quit IRC | 22:52 | |
| *** dave-mccowan has quit IRC | 22:54 | |
| *** rbak has quit IRC | 23:04 | |
| *** chlong has joined #openstack-telemetry | 23:27 | |
| *** peristeri has quit IRC | 23:30 | |
| *** harlowja has quit IRC | 23:47 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!