| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org is being quickly restarted to apply a new security patch | 00:57 | |
| *** rcernin has quit IRC | 01:44 | |
| *** rcernin has joined #openstack-telemetry | 02:09 | |
| *** irclogbot_0 has quit IRC | 03:27 | |
| *** irclogbot_3 has joined #openstack-telemetry | 04:42 | |
| *** irclogbot_3 has quit IRC | 04:54 | |
| *** irclogbot_0 has joined #openstack-telemetry | 04:59 | |
| *** e0ne has joined #openstack-telemetry | 07:03 | |
| *** rcernin has quit IRC | 07:37 | |
| *** bengates has joined #openstack-telemetry | 08:09 | |
| *** bengates has quit IRC | 08:09 | |
| *** bengates has joined #openstack-telemetry | 08:10 | |
| *** rcernin has joined #openstack-telemetry | 08:14 | |
| *** rcernin has quit IRC | 08:31 | |
| *** lxkong has quit IRC | 08:52 | |
| *** lxkong has joined #openstack-telemetry | 08:53 | |
| *** lxkong has quit IRC | 08:53 | |
| *** lxkong has joined #openstack-telemetry | 08:54 | |
| *** rcernin has joined #openstack-telemetry | 09:00 | |
| *** e0ne has quit IRC | 09:03 | |
| *** e0ne has joined #openstack-telemetry | 09:03 | |
| *** bengates_ has joined #openstack-telemetry | 09:03 | |
| *** bengates has quit IRC | 09:06 | |
| *** rcernin has quit IRC | 09:18 | |
| *** rcernin has joined #openstack-telemetry | 09:23 | |
| *** petrtuma has joined #openstack-telemetry | 10:03 | |
| *** e0ne has quit IRC | 10:07 | |
| *** rcernin has quit IRC | 10:08 | |
| *** rcernin has joined #openstack-telemetry | 10:19 | |
| petrtuma | Hi, I'm facing an issue with Telemetry in Train release. My environment is deployed by kolla-ansible (RHEL8 base container, binary build, containers run correct versions of packages) and consists of 2 controller nodes and 2 compute nodes. No custom metric definition and no custom gnocchi resources, all configs are defaults rendered by | 10:45 |
|---|---|---|
| petrtuma | kolla-ansible. Redis, memcached and rabbitmq seem to be working fine, I can see connections / traffic there. | 10:45 |
| petrtuma | I spawn 2 VMs in the stack and after couple of minutes I use 'gnocchi metric list', but it returns empty (opposite of what I see on my Rocky environment). I see no errors in both gnocchi and ceilometer logs. | 10:45 |
| petrtuma | As far as I can tell every configuration file (.conf) seems to be in line with Train documentation and all yaml definitions seem to be default values that come with the project. During my testing I narrowed the issue to pipeline.yaml, that contains just one wildcard 'sources' definition pointing to one sink that should be published to gnocchi (i.e. | 10:45 |
| petrtuma | the default). | 10:45 |
| petrtuma | Strangely enough, if I use pipeline.yaml from Rocky release (i.e. file that contains transformer definitions) I start getting metrics almost immediately. Although only the metrics that has tranformers specified and not all of those (cpu and memory.* are still missing). | 10:45 |
| petrtuma | I'm uncertain what is wrong at this point. Transformers are supposed to be deprecated in Train, but I receive some of the metrics only if I define them. | 10:45 |
| petrtuma | Any ideas what might be wrong or where to look? Is it a bug? | 10:45 |
| mrunge | petrtuma, hi, openstack metric status shows you a status? | 11:05 |
| mrunge | at some point, the gnocchi command is/was deprecated | 11:06 |
| petrtuma | hmmm, so it seems there is some issue: | 11:07 |
| petrtuma | # openstack metric status | 11:07 |
| petrtuma | <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> | 11:07 |
| petrtuma | <html><head> | 11:07 |
| petrtuma | <title>500 Internal Server Error</title> | 11:07 |
| petrtuma | </head><body> | 11:07 |
| petrtuma | <h1>Internal Server Error</h1> | 11:07 |
| petrtuma | <p>The server encountered an internal error or | 11:07 |
| petrtuma | misconfiguration and was unable to complete | 11:07 |
| petrtuma | your request.</p> | 11:07 |
| petrtuma | <p>Please contact the server administrator at | 11:07 |
| petrtuma | root@localhost to inform them of the time this error occurred, | 11:07 |
| petrtuma | and the actions you performed just before this error.</p> | 11:07 |
| petrtuma | <p>More information about this error may be available | 11:07 |
| petrtuma | in the server error log.</p> | 11:07 |
| petrtuma | </body></html> | 11:07 |
| petrtuma | (HTTP 500) | 11:07 |
| mrunge | okay, that's what I expected | 11:07 |
| mrunge | there is a known issue with gnocchi on python3 | 11:08 |
| petrtuma | I can now see a fresh error from wsgi in gnocchi-api-error.log | 11:08 |
| mrunge | iirc, you'll need https://github.com/gnocchixyz/gnocchi/commit/bd2cafadd0961231748cf4b67efd28c6db277b04 | 11:09 |
| mrunge | where are you getting your containers from? | 11:09 |
| mrunge | ... they should probably get fixed | 11:09 |
| mrunge | or maybe there are already? | 11:10 |
| petrtuma | We are building them locally (kolla used is iirc ~2 months old) and we use local mirrors of RDO RPMs to do the binary build. | 11:11 |
| mrunge | I know I have seen that issue, and I know it is fixed downstream | 11:12 |
| mrunge | it will be fixed in the next minor release of gnocchi | 11:12 |
| *** rcernin has quit IRC | 11:13 | |
| mrunge | so, we are talking about https://github.com/gnocchixyz/gnocchi/issues/1083 | 11:14 |
| petrtuma | ah yes, it is exactly same strack trace I have in the log | 11:15 |
| mrunge | I see the fixes are also merged in https://github.com/gnocchixyz/gnocchi/commits/stable/4.3 | 11:15 |
| petrtuma | hmmm, it seems it's not yet build into RPMs, http://mirror.centos.org/centos-8/8/cloud/x86_64/openstack-train/Packages/g/ is I think what corresponds to RDO packages right? I can see only gnocchi rpm there, which was built in 1st half of 2020 | 11:18 |
| mrunge | yes | 11:19 |
| mrunge | gnocchi 4.3.4 is also the latest tag for the 4.3 branch | 11:20 |
| mrunge | next step would be to get a new 4.3.5 tag created in upstream gnocchi | 11:20 |
| mrunge | petrtuma, I've created https://github.com/gnocchixyz/gnocchi/issues/1117 | 11:23 |
| mrunge | feel free to chime in :) | 11:23 |
| petrtuma | Thanks for the tip :) We will build the package locally until fixed version is released. | 11:26 |
| mrunge | you're welcome. | 11:27 |
| *** rcernin has joined #openstack-telemetry | 11:35 | |
| *** hoonetorg has quit IRC | 11:38 | |
| *** rcernin has quit IRC | 12:04 | |
| openstackgerrit | Matthias Runge proposed openstack/ceilometer master: Using Iterable was deprecated in python 3.3 https://review.opendev.org/c/openstack/ceilometer/+/773652 | 12:34 |
| *** hoonetorg has joined #openstack-telemetry | 13:03 | |
| *** bengates has joined #openstack-telemetry | 13:08 | |
| *** bengates has quit IRC | 13:09 | |
| *** bengates_ has quit IRC | 13:09 | |
| *** bengates has joined #openstack-telemetry | 13:09 | |
| *** bengates_ has joined #openstack-telemetry | 13:11 | |
| *** bengates has quit IRC | 13:14 | |
| *** bengates_ has quit IRC | 13:41 | |
| *** bengates has joined #openstack-telemetry | 13:42 | |
| *** petrtuma has quit IRC | 14:05 | |
| *** petrtuma has joined #openstack-telemetry | 14:14 | |
| petrtuma | mrunge I updated my containers with the fix we talked about. It fixed the issue with 'openstack metric status', but not my original problem, I still don't have any metrics in the gnocchi. | 14:22 |
| petrtuma | Here's what I get with metric status: | 14:22 |
| petrtuma | # openstack metric status -f json --noindent | 14:22 |
| petrtuma | {"metricd/processors": [[112, 101, 116, 114, 45, 114, 104, 56, 45, 111, 115, 99, 49, 46, 50, 46, 48, 50, 98, 53, 57, 54, 48, 97, 45, 51, 99, 102, 57, 45, 52, 56, 99, 50, 45, 57, 100, 55, 57, 45, 56, 52, 50, 52, 102, 48, 50, 57, 48, 98, 50, 50], [112, 101, 116, 114, 45, 114, 104, 56, 45, 111, 115, 99, 49, 46, 51, 46, 52, 54, 54, 98, 100, 53, 101, | 14:22 |
| petrtuma | 102, 45, 57, 100, 55, 56, 45, 52, 54, 50, 97, 45, 57, 101, 98, 57, 45, 51, 51, 52, 54, 99, 102, 55, 53, 98, 56, 97, 49], [112, 101, 116, 114, 45, 114, 104, 56, 45, 111, 115, 99, 49, 46, 49, 46, 51, 52, 52, 98, 54, 51, 57, 100, 45, 100, 57, 52, 57, 45, 52, 49, 51, 50, 45, 97, 97, 50, 54, 45, 56, 50, 51, 53, 50, 52, 56, 56, 54, 56, 50, 102], [112, | 14:22 |
| petrtuma | 101, 116, 114, 45, 114, 104, 56, 45, 111, 115, 99, 49, 46, 48, 46, 100, 54, 100, 51, 57, 50, 51, 55, 45, 56, 48, 54, 102, 45, 52, 48, 52, 53, 45, 97, 99, 98, 102, 45, 53, 53, 53, 102, 56, 56, 56, 54, 97, 51, 53, 97], [112, 101, 116, 114, 45, 114, 104, 56, 45, 111, 115, 99, 49, 46, 52, 46, 98, 51, 48, 51, 98, 51, 56, 100, 45, 97, 56, 100, 98, 45, | 14:22 |
| petrtuma | 52, 48, 97, 55, 45, 56, 53, 57, 98, 45, 49, 53, 98, 97, 57, 51, 100, 54, 49, 99, 100, 55]], "storage/number of metric having measures to process": 0, "storage/total number of measures to process": 0} | 14:22 |
| *** bengates has quit IRC | 14:23 | |
| *** bengates has joined #openstack-telemetry | 14:23 | |
| mrunge | how are you sending metrics to gnocchi then? | 14:26 |
| petrtuma | I assume it's the ceilometer agents that are on machines, my assumption is based on pipeline.yaml in the configs: | 14:32 |
| petrtuma | --- | 14:32 |
| petrtuma | sources: | 14:32 |
| petrtuma | - name: meter_source | 14:32 |
| petrtuma | meters: | 14:32 |
| petrtuma | - "*" | 14:32 |
| petrtuma | sinks: | 14:32 |
| petrtuma | - meter_sink | 14:32 |
| petrtuma | sinks: | 14:32 |
| petrtuma | - name: meter_sink | 14:32 |
| petrtuma | publishers: | 14:32 |
| petrtuma | - gnocchi:// | 14:32 |
| petrtuma | This does not seem to be working at all. I tried with pipeline.yaml from Rocky release, only two differences between the two files are transformers are being specified in Rocky and Rocky one has filter in gnocchi:// definition. To my suprise it started to work, it seems because of the transformers, I tried with just the filter, didn't work. | 14:42 |
| petrtuma | I find it very strange, because the agents do have this patch in them: https://review.opendev.org/c/openstack/ceilometer/+/580709 | 14:42 |
| *** petrtuma has quit IRC | 14:51 | |
| *** petrtuma has joined #openstack-telemetry | 14:52 | |
| mrunge | I assume you already have a vm running? | 15:14 |
| mrunge | otherwise you won't see any metrics | 15:15 |
| petrtuma | yes | 15:20 |
| petrtuma | (clients)# openstack server list -f json | 15:28 |
| petrtuma | [ | 15:28 |
| petrtuma | { | 15:28 |
| petrtuma | "ID": "759a297f-9ed0-4211-be67-ab98a589a783", | 15:28 |
| petrtuma | "Name": "test-2", | 15:28 |
| petrtuma | "Status": "ACTIVE", | 15:28 |
| petrtuma | "Networks": "test-net=10.10.10.236", | 15:28 |
| petrtuma | "Image": "cirros-0.3.5", | 15:28 |
| petrtuma | "Flavor": "m1.small" | 15:28 |
| petrtuma | }, | 15:28 |
| petrtuma | { | 15:28 |
| petrtuma | "ID": "d2a69bee-483e-471f-9f0d-9a20ab4250c9", | 15:28 |
| petrtuma | "Name": "test-1", | 15:28 |
| petrtuma | "Status": "ACTIVE", | 15:28 |
| petrtuma | "Networks": "test-net=10.10.10.93", | 15:28 |
| petrtuma | "Image": "cirros-0.3.5", | 15:28 |
| petrtuma | "Flavor": "m1.small" | 15:28 |
| petrtuma | } | 15:28 |
| mrunge | hmmm. transformers should not be necessary | 15:29 |
| petrtuma | (clients)# openstack metric list -f json | 15:29 |
| petrtuma | [ | 15:29 |
| petrtuma | { | 15:29 |
| petrtuma | "id": "a4979061-8d06-4fae-a74f-9fede967334e", | 15:29 |
| petrtuma | "archive_policy/name": "ceilometer-low", | 15:29 |
| petrtuma | "name": "image.size", | 15:29 |
| petrtuma | "unit": "B", | 15:29 |
| petrtuma | "resource_id": "1268d5d0-63d6-454c-953a-ba68f463857b" | 15:29 |
| petrtuma | } | 15:29 |
| petrtuma | ] | 15:29 |
| petrtuma | Only metric I ever saw in Train, image that is uploaded right after kolla-ansible is done. | 15:29 |
| *** nijaba has left #openstack-telemetry | 15:30 | |
| mrunge | strange, my pipeline config looks like this: http://paste.openstack.org/show/802246/ | 15:32 |
| mrunge | including transformers | 15:32 |
| mrunge | (empty) | 15:32 |
| petrtuma | hmmm, I think I already tried with empty transformer definition as well, but let me try with yours | 15:37 |
| petrtuma | 5 minutes after I restarted containers: | 15:46 |
| petrtuma | (clients)# openstack metric list -f json | 15:46 |
| petrtuma | [ | 15:46 |
| petrtuma | { | 15:46 |
| petrtuma | "id": "a4979061-8d06-4fae-a74f-9fede967334e", | 15:46 |
| petrtuma | "archive_policy/name": "ceilometer-low", | 15:46 |
| petrtuma | "name": "image.size", | 15:46 |
| petrtuma | "unit": "B", | 15:46 |
| petrtuma | "resource_id": "1268d5d0-63d6-454c-953a-ba68f463857b" | 15:46 |
| petrtuma | } | 15:46 |
| petrtuma | ] | 15:46 |
| petrtuma | I will enable debug logs everywhere I can and leave it running in this state over night, perhaps it will yield some further insight. I'll let you know if I find something. | 16:07 |
| mrunge | alright, thank you | 16:22 |
| mrunge | meanwhile, I'll look into mine here. Also puppet-ceilometer needs fixing, since it still deploys transformers | 16:22 |
| mrunge | petrtuma, I see this change in puppet-ceilometer: https://github.com/openstack/puppet-ceilometer/commit/e73ba70b8ab6768a62c7e21a6dc013b95a5bff13 | 16:47 |
| mrunge | which removed the use of transformers there; apparently this was just recently and not backported to train (yet) | 16:47 |
| mrunge | but: https://review.opendev.org/c/openstack/puppet-ceilometer/+/773212 | 16:48 |
| *** bengates has quit IRC | 17:31 | |
| *** rcernin has joined #openstack-telemetry | 17:59 | |
| *** e0ne has joined #openstack-telemetry | 18:00 | |
| *** rcernin has quit IRC | 18:04 | |
| *** bengates has joined #openstack-telemetry | 18:06 | |
| *** bengates has quit IRC | 18:10 | |
| *** bengates has joined #openstack-telemetry | 18:44 | |
| *** bengates has quit IRC | 18:48 | |
| *** petrtuma has quit IRC | 18:53 | |
| *** e0ne has quit IRC | 19:03 | |
| *** e0ne has joined #openstack-telemetry | 19:57 | |
| *** rcernin has joined #openstack-telemetry | 20:00 | |
| *** rcernin has quit IRC | 20:04 | |
| *** rcernin has joined #openstack-telemetry | 20:25 | |
| *** e0ne has quit IRC | 20:38 | |
| *** hoonetorg has quit IRC | 20:38 | |
| *** e0ne has joined #openstack-telemetry | 20:41 | |
| *** hoonetorg has joined #openstack-telemetry | 20:51 | |
| *** rcernin has quit IRC | 21:22 | |
| *** e0ne has quit IRC | 21:38 | |
| *** e0ne has joined #openstack-telemetry | 21:48 | |
| *** e0ne has quit IRC | 21:48 | |
| *** rcernin has joined #openstack-telemetry | 21:52 | |
| *** rcernin has quit IRC | 22:08 | |
| *** rcernin has joined #openstack-telemetry | 22:09 | |
| *** hoonetorg has quit IRC | 22:40 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!