| *** craigbr has quit IRC | 00:01 | |
| *** craigbr has joined #openstack-monasca | 00:04 | |
| *** craigbr has quit IRC | 00:09 | |
| *** craigbr has joined #openstack-monasca | 00:10 | |
| *** craigbr has quit IRC | 00:14 | |
| *** dschroeder has quit IRC | 00:18 | |
| *** craigbr has joined #openstack-monasca | 01:21 | |
| *** craigbr has quit IRC | 01:32 | |
| *** craigbr has joined #openstack-monasca | 01:33 | |
| *** craigbr has quit IRC | 01:41 | |
| *** th_ has joined #openstack-monasca | 03:02 | |
| *** theresa_ has quit IRC | 03:06 | |
| *** th_ has quit IRC | 03:14 | |
| *** amalagon has quit IRC | 03:39 | |
| *** amalagon has joined #openstack-monasca | 04:31 | |
| *** sanjayu has joined #openstack-monasca | 04:50 | |
| *** sanjayu has quit IRC | 04:55 | |
| *** garyh has quit IRC | 05:09 | |
| *** Guest59555 has joined #openstack-monasca | 07:02 | |
| *** boris-42 has quit IRC | 08:33 | |
| *** boris-42 has joined #openstack-monasca | 11:05 | |
| *** Guest59555 has quit IRC | 12:29 | |
| *** sanjayu has joined #openstack-monasca | 12:45 | |
| *** MVenesio has joined #openstack-monasca | 13:14 | |
| *** MVenesio_ has joined #openstack-monasca | 13:15 | |
| *** MVenesio_ has quit IRC | 13:17 | |
| *** MVenesio_ has joined #openstack-monasca | 13:17 | |
| *** MVenesio has quit IRC | 13:18 | |
| *** LeanReox has joined #openstack-monasca | 13:24 | |
| LeanReox | Hi everyone, getting some issues with the libvirt plugin for monasca agent, im getting {"forbidden":{"code":403,"message":"Project d2b33fed2be34e7d8b5388107c3eaf16 cannot POST cross tenant metrics","details":"","internal_code":"4d74a071f83f544f"}} | 13:25 |
|---|---|---|
| LeanReox | the user has monitoring-delegate role configured, and its admin | 13:25 |
| LeanReox | reading the code i saw that this exception is thrown when crossTenantId is not found | 13:25 |
| LeanReox | if (!Strings.isNullOrEmpty(crossTenantId)) | 13:25 |
| LeanReox | but its actually there ... http://20.20.20.195:8080/v2.0/metrics?tenant_id=10d575812bf74c2086f549846f1687a6 | 13:25 |
| LeanReox | any clues ? | 13:25 |
| LeanReox | sandywalsh, maybe you have a pinpoint here :) | 13:27 |
| sandywalsh | LeanReox: sorry, I'm not a monasca expert :) But it sounds like your agent is trying to post to a different tenant ID than the credentials | 13:28 |
| LeanReox | yes it is, but it should, the libvirt plugin should poss cross-tenants metrics | 13:29 |
| sandywalsh | any way to see what the actual POST is sending? | 13:30 |
| LeanReox | yes let me dump on the monasca-api server | 13:31 |
| LeanReox | sandywalsh, http://pastebin.com/8QPZgmKU | 13:36 |
| sandywalsh | seems fine ... definitely something wonky server-side / permissions. Anything in the logs? | 13:38 |
| LeanReox | sandywalsh, besides the 403 forbidden nothing at all, but debuggin the code, the 403 is cached by an exception if (!Strings.isNullOrEmpty(crossTenantId)) | 13:42 |
| LeanReox | and that variable is defined like this @QueryParam("tenant_id") String crossTenantId, | 13:43 |
| LeanReox | and the querystring param, its actually there ... | 13:43 |
| LeanReox | i can see it on the POST | 13:43 |
| LeanReox | http://20.20.20.195:8080/v2.0/metrics?tenant_id=10d575812bf74c2086f549846f1687a6 | 13:43 |
| LeanReox | so thats odd, sandywalsh | 13:43 |
| LeanReox | here sandywalsh , https://github.com/stackforge/monasca-api/blob/84dfcb98a1c2365a547c21f73a85e16ba8733930/src/main/java/monasca/api/resource/MetricResource.java | 13:44 |
| sandywalsh | <looking> | 13:45 |
| *** Miouge has quit IRC | 13:45 | |
| sandywalsh | and there's the X-Tenant-Id expected in the header, which I don't see | 13:46 |
| LeanReox | yes sandywalsh , but for that one it should crash on the service.create line+ | 13:48 |
| *** Miouge has joined #openstack-monasca | 13:49 | |
| LeanReox | if its missing, should go throug the 403 one | 13:49 |
| sandywalsh | ah, hang on | 13:49 |
| sandywalsh | Project d2b33fed2be34e7d8b5388107c3eaf16 cannot POST cross tenant | 13:50 |
| sandywalsh | but the tenant_id on the POST is 10d575812bf74c2086f549846f1687a6 | 13:50 |
| LeanReox | yes, cause its a cross-tenant post, a user is postingm etrics from another tenant vm | 13:50 |
| LeanReox | tenant_id = @QueryParam("tenant_id") | 13:51 |
| sandywalsh | oh, right ... trying to find out where it got the tenantID since the header isn't present | 13:51 |
| LeanReox | and tenantId = @HeaderParam("X-Tenant-Id") | 13:51 |
| LeanReox | sandywalsh, if you see the line that logs the 403 | 13:52 |
| LeanReox | it uses tenantId | 13:52 |
| LeanReox | so @HeaderParam("X-Tenant-Id") its there | 13:52 |
| sandywalsh | right, but X-Tenant-Id isn't in the curl | 13:52 |
| sandywalsh | so it's defaulting it | 13:52 |
| LeanReox | weird cause its actually the tenantiD from the user trying to do the post, but im going to create a new tenant and assign monitoring-delegate role to it, to see if the logs changes | 13:54 |
| LeanReox | better ill try to add the header to the request | 13:54 |
| *** htruta has quit IRC | 13:55 | |
| sandywalsh | and there's no X-Roles, so isDelegate will be False, which gets it into that code block at line 77 | 13:55 |
| sandywalsh | no idea why crossTenantId should be Null | 13:56 |
| sandywalsh | oh ... wait | 13:57 |
| sandywalsh | if ! empty | 13:57 |
| sandywalsh | and crossTenantID is not empty | 13:57 |
| sandywalsh | so, I think the problem is that X-Roles is not being included | 13:58 |
| sandywalsh | if you include X-Roles to the post it should work | 13:58 |
| LeanReox | ill try that one | 13:58 |
| sandywalsh | (and it has the MONITORING_DELEGATE_ROLE) | 13:58 |
| LeanReox | yup the user has the role | 13:59 |
| sandywalsh | onitoring-delegate | 13:59 |
| sandywalsh | monitoring-delegate | 13:59 |
| sandywalsh | not in the curl | 13:59 |
| LeanReox | yep, thats monasca-agent | 13:59 |
| LeanReox | so maybe its a bug | 13:59 |
| sandywalsh | @HeaderParam("X-Roles") String roles, | 13:59 |
| sandywalsh | not seeing that in the paste | 13:59 |
| LeanReox | if its not there, the pyton monasca client its not included | 14:00 |
| LeanReox | including it | 14:00 |
| LeanReox | sorry | 14:00 |
| sandywalsh | right, and that's the problem | 14:00 |
| sandywalsh | (I think :) | 14:00 |
| *** Miouge has quit IRC | 14:04 | |
| *** htruta has joined #openstack-monasca | 14:06 | |
| LeanReox | sandywalsh, http://pastebin.com/b9DrwqBT | 14:13 |
| LeanReox | weird, still the same, added both headers | 14:13 |
| sandywalsh | ok, that's just weird | 14:13 |
| sandywalsh | ah, wrong tenant_id | 14:14 |
| sandywalsh | oh, nm | 14:14 |
| sandywalsh | no | 14:15 |
| sandywalsh | should be ?tenant_id=10d575812bf74c2086f549846f1687a6 | 14:15 |
| LeanReox | sandywalsh, yes should be that one | 14:16 |
| LeanReox | | 10d575812bf74c2086f549846f1687a6 | tour_tenant | True | | 14:17 |
| LeanReox | thats the owner of the vm | 14:17 |
| LeanReox | and the other tenant is from the user that its posting the cross-tenant metric | 14:17 |
| sandywalsh | ok, but it doesn't match the first paste | 14:17 |
| sandywalsh | first POST POST /v2.0/metrics?tenant_id=10d575812bf74c2086f549846f1687a6 | 14:18 |
| sandywalsh | second http://20.20.20.195:8080/v2.0/metrics?tenant_id=d2b33fed2be34e7d8b5388107c3eaf16 | 14:18 |
| LeanReox | true i grabbed antoher curl, but thats worst because its the same tenant owner of the vm | 14:19 |
| LeanReox | so in that case its not cross-tenant metric | 14:19 |
| LeanReox | ill try the first one with the x-roles header | 14:19 |
| *** garyh has joined #openstack-monasca | 14:23 | |
| *** rbak_ has quit IRC | 14:25 | |
| sandywalsh | I don't see any actual role checks ... just the presence or lack of the parameters | 14:25 |
| sandywalsh | exists or not | 14:25 |
| LeanReox | sandywalsh, tried adding the headers to the original curl, same thing | 14:27 |
| sandywalsh | hmm, I think we're in the right church, but the wrong pew :) | 14:28 |
| sandywalsh | better wait until the proper monasca crew get in | 14:28 |
| sandywalsh | there's some magic going on | 14:28 |
| LeanReox | haha ok sandywalsh thanks for the support anyway | 14:29 |
| sandywalsh | np, I'll have another peek and will let you know if anything jumps out | 14:29 |
| LeanReox | thanks, i appreciate that, ill wait for Tim or craig to jump in | 14:30 |
| *** theresa_ has joined #openstack-monasca | 14:30 | |
| sandywalsh | with 'X-Roles: monitoring-delegate' all that code should be bypassed, so I don't know what's going on | 14:31 |
| *** sanjayu has quit IRC | 14:35 | |
| LeanReox | yup, i thought the same, but http://pastebin.com/Q8GwgHEf | 14:38 |
| sandywalsh | voodoo | 14:47 |
| *** craigbr has joined #openstack-monasca | 14:57 | |
| *** dlfryar has joined #openstack-monasca | 14:59 | |
| *** rbak has joined #openstack-monasca | 15:06 | |
| *** Miouge has joined #openstack-monasca | 15:10 | |
| *** rhochmuth has joined #openstack-monasca | 15:16 | |
| *** ericpeterson has joined #openstack-monasca | 15:53 | |
| openstackgerrit | Craig Bryant proposed stackforge/monasca-api: Distribute messages better among the Kafka partitions https://review.openstack.org/148967 | 15:56 |
| *** dschroeder has joined #openstack-monasca | 16:02 | |
| *** theresa_ has quit IRC | 16:02 | |
| stackedsax | rhochmuth: wait, wait, come back! | 16:33 |
| sandywalsh | rhochmuth: fujitsu is online ... can you rejoin? | 16:34 |
| stackedsax | rhochmuth: we're going to chat with them for a bit, join if you can | 16:36 |
| *** amalagon has quit IRC | 16:50 | |
| *** amalagon has joined #openstack-monasca | 16:50 | |
| *** amalagon has quit IRC | 16:54 | |
| *** tongli has joined #openstack-monasca | 17:03 | |
| openstackgerrit | Joe Keen proposed stackforge/monasca-notification: Refactored notification engine types https://review.openstack.org/147664 | 17:10 |
| openstackgerrit | Gary Hessler proposed stackforge/monasca-agent: Updated dimension processing for service and component https://review.openstack.org/147346 | 17:18 |
| *** robbybb has joined #openstack-monasca | 17:28 | |
| robbybb | Could anyone provide a pointer to info on the mid-cycle meet-up for Monasca? | 17:28 |
| LeanReox | craigbr, hi Craig can i bother with a couple of questions about cross-tenant metrics, and ceilometer-monasca integration ? | 17:29 |
| craigbr | I know some stuff about cross-tenant metrics. What do you need? | 17:30 |
| LeanReox | craigbr, great | 17:30 |
| LeanReox | first of all i found a bug on monasca-ceilometer, iss missing from "ceilometer.i18n import _ " on monclient.py | 17:31 |
| LeanReox | causing the LOG.debug(_("somethin") to fail , saying that "_" is not defined | 17:31 |
| craigbr | Ok. I'll take a look at that | 17:32 |
| LeanReox | added the missing import fixed the issue but dont know where to report it | 17:32 |
| LeanReox | thats the first one | 17:32 |
| craigbr | The easy thing is just to submit the fix | 17:32 |
| craigbr | It will get reviewed and merged | 17:32 |
| LeanReox | great thanks | 17:33 |
| LeanReox | im keep getting "403" forbidden when libvirt module for monasca agent tries to put metrics | 17:33 |
| LeanReox | doesnt matter if its the same tenant | 17:33 |
| LeanReox | or on behalf of another tenant | 17:33 |
| LeanReox | the user has the role "monitoring-delegate" | 17:34 |
| *** amalagon has joined #openstack-monasca | 17:34 | |
| craigbr | Let me get somebody else on to help with that | 17:34 |
| LeanReox | ok | 17:34 |
| craigbr | It might be a little while, many people are in a meeting right now | 17:34 |
| LeanReox | and, the second one its , on the monasca-ceilometer integration, on the monclient.py | 17:34 |
| LeanReox | no matter what i put on the kwargs dict to authtenticate to keystone | 17:35 |
| LeanReox | that just simply doesnt work and the publisher gets always an unathorized | 17:35 |
| *** ericpeterson has quit IRC | 17:35 | |
| *** ericpeterson has joined #openstack-monasca | 17:36 | |
| craigbr | I will have to do some research on that one | 17:36 |
| LeanReox | i tried adding the keystone admin token + admin user and pass combination | 17:36 |
| rhochmuth | robbybb We haven't decided on a Monasca mid-cycle meetup | 17:37 |
| *** ericpete_ has joined #openstack-monasca | 17:37 | |
| *** ddieterly has joined #openstack-monasca | 17:37 | |
| *** ericpeterson has quit IRC | 17:37 | |
| rhochmuth | We had wanted to do one after the operators meetup at that location | 17:37 |
| rhochmuth | but, as that is hosted by comcast, logistics are difficult | 17:37 |
| rhochmuth | I could easily host it in Fort Collins | 17:37 |
| rhochmuth | but hosting at other locations where no one is involved with Monasca is a problem | 17:38 |
| rhochmuth | We have lot's of involvement with TWC though, but the Comcast/TWC merger hasnt' happened | 17:38 |
| LeanReox | craigbr, several combinations, and the python-monasca-agent always tries to auth with "curl -i -X POST -H 'X-Auth-Key: xxxx' -H 'Accept: application/json' -H 'Content-Type: application/json' -H 'X-Auth-Url: http://20.20.20.76:5000/v3' -H 'X-Auth-User: xxx' -H 'User-Agent: python-monascaclient'" | 17:38 |
| LeanReox | but fails with Bad Request: Token not found in the request | 17:39 |
| craigbr | Ok. I will take a look when our meeting is done | 17:39 |
| LeanReox | thanks craigbr let me know when you have a couple of minutes | 17:39 |
| robbybb | rhochmuth Thanks, I will look for updates | 17:39 |
| LeanReox | sorry to interrupt _) | 17:39 |
| craigbr | No problem. I just have to actually talk now | 17:40 |
| LeanReox | craigbr, haha great, let me know, being couple of days trying to figure it out and its getting frustrating :) | 17:42 |
| *** rbak has quit IRC | 17:49 | |
| *** boris-42 has quit IRC | 17:53 | |
| *** MVenesio has joined #openstack-monasca | 17:55 | |
| *** LeanReox has quit IRC | 17:55 | |
| *** MVenesio_ has quit IRC | 17:55 | |
| *** LeanReox has joined #openstack-monasca | 17:56 | |
| *** jkrappa has joined #openstack-monasca | 17:56 | |
| *** tongli has quit IRC | 18:02 | |
| *** rhochmuth has quit IRC | 18:11 | |
| *** rbak has joined #openstack-monasca | 18:12 | |
| *** robbybb has quit IRC | 18:14 | |
| dschroeder | Hi LeanReox, I'm looking into this problem now. | 18:24 |
| craigbr | LeanReox: Looks like monclient.py needs to be updated to work with keystone authentication. I'll take a look at it but may take a while | 18:25 |
| *** boris-42 has joined #openstack-monasca | 18:28 | |
| LeanReox | dschroeder, thanks | 18:30 |
| LeanReox | craigbr, hmmm i was afraid of that | 18:31 |
| craigbr | I could be wrong, though. Just need to look at it | 18:31 |
| LeanReox | dschroeder, youre taking a look to the cross-tenant issue ? | 18:32 |
| dschroeder | Yes. | 18:32 |
| LeanReox | thanks dschroeder let me know if you need more info | 18:32 |
| dschroeder | LeanReox: Please help me understand your test environment. Are you using monasca-vagrant with the mini-mon and devstack VMs, or have you installed the Agent elsewhere (like on baremetal)? | 18:34 |
| LeanReox | baremetal | 18:34 |
| LeanReox | a distributed baremetal install 1 controller 2 computes | 18:35 |
| LeanReox | on the controller the monasca-api is running | 18:35 |
| LeanReox | then the agent on the computes | 18:35 |
| LeanReox | that run machines from several tenants | 18:35 |
| LeanReox | the metrics for the physical host are getting posted just fine | 18:35 |
| LeanReox | but the libvirt vm metrics | 18:35 |
| LeanReox | are getting 403 | 18:36 |
| LeanReox | craigbr, its weird that has the "auth_url" as part of the kwargs dictionary | 18:36 |
| LeanReox | craigbr, the monclient.py | 18:36 |
| LeanReox | craigbr, but it seems that is not doing anything with it | 18:37 |
| dschroeder | LeonReox: On a compute note, please run this command: sudo -u monasca-agent monasca-collector check libvirt | 18:37 |
| LeanReox | dschroeder, roger | 18:40 |
| *** MVenesio has quit IRC | 18:42 | |
| LeanReox | dschroeder, http://pastebin.com/ALVQHiFq | 18:42 |
| *** MVenesio has joined #openstack-monasca | 18:42 | |
| *** craigbr has quit IRC | 18:43 | |
| *** craigbr has joined #openstack-monasca | 18:45 | |
| dschroeder | LeonReox, great, that looks good. Now, you said you get the 403 error when the agent is posting those metrics, do you see that in /var/log/monsaca/agent/forwarder.log? | 18:49 |
| LeanReox | dschroeder, thats correct | 18:50 |
| LeanReox | dschroeder, here you have the request, responses, tenants users and roles | 18:51 |
| LeanReox | dschroeder, http://pastebin.com/QwuaygTG | 18:51 |
| dschroeder | LeanReox, very helpful, thanks. Can you please confirm the username and project_name specified in /etc/monasca/agent/agent.conf ? | 19:18 |
| *** theresa_ has joined #openstack-monasca | 19:18 | |
| *** ericpeterson has joined #openstack-monasca | 19:29 | |
| *** ericpete_ has quit IRC | 19:33 | |
| LeanReox | yes | 19:41 |
| LeanReox | user = monascauser | 19:41 |
| LeanReox | pass = 123456 | 19:42 |
| LeanReox | im seeing that the configured in the livirt plugin | 19:42 |
| LeanReox | its not used to post the metrics | 19:42 |
| LeanReox | should work like that ? | 19:42 |
| LeanReox | dschroeder, ? | 19:42 |
| LeanReox | dschroeder, i fix it ... | 19:57 |
| LeanReox | dschroeder, added monitoring-delegate role to the user that its on the agent | 19:57 |
| LeanReox | but im wondering why its not using the user that its on libvirt.yaml | 19:57 |
| LeanReox | to post the metrics (that already has monitoring-delegate) role | 19:57 |
| *** ddieterly has quit IRC | 19:57 | |
| LeanReox | cause if the user its not admin, it cannot sort all the instances from nova | 19:58 |
| *** ericpeterson has quit IRC | 20:03 | |
| *** ericpeterson has joined #openstack-monasca | 20:03 | |
| *** ddieterly has joined #openstack-monasca | 20:17 | |
| *** ddieterly has quit IRC | 20:18 | |
| dschroeder | LeanReox, yes, that's the fix. The simplest explanation for this behavior is that the user specified in the libvirt plugin's config is responsible for gathering the metrics, and the user in the Agent's config is responsible for posting the metrics, and both processes are fairly separate. I see where the documentation is insufficient/misleading in this regard, and will update it now. | 20:34 |
| LeanReox | dschroeder, yes thats why i get it working when assigned monitoring-delegate all over the world, and when i did i fresh install "supposing" all that, it failed again | 20:36 |
| LeanReox | craigbr, ill try to get a version of monclient.py that supports keystone, and i get back to you | 20:37 |
| LeanReox | dschroeder, i got an existencial question, the thing that its keeping me with ceilometer its the heat integration for autoscaling purposes, it will be an extension to do this? | 20:40 |
| LeanReox | cause i see a lot of interesting features comming in on monasca (abnormaly detectrion, log correlation, etc) | 20:41 |
| *** theresa_ has quit IRC | 20:44 | |
| *** craigbr has quit IRC | 20:48 | |
| dschroeder | LeonReox, yes, Heat integration is something we are planning on adding to Monasca. I don't have a firm timeframe on this yet, though. | 20:56 |
| *** craigbr has joined #openstack-monasca | 20:57 | |
| *** seeg has quit IRC | 20:57 | |
| *** seeg has joined #openstack-monasca | 20:57 | |
| *** theresa_ has joined #openstack-monasca | 21:01 | |
| *** craigbr has quit IRC | 21:10 | |
| *** MVenesio has quit IRC | 21:11 | |
| *** LeanReox has quit IRC | 21:11 | |
| *** jkrappa has quit IRC | 21:16 | |
| *** cindyo has joined #openstack-monasca | 21:17 | |
| *** rbak has quit IRC | 21:48 | |
| *** dlfryar has quit IRC | 21:49 | |
| *** ericpeterson has quit IRC | 21:51 | |
| *** dlfryar has joined #openstack-monasca | 21:52 | |
| *** craigbr has joined #openstack-monasca | 22:01 | |
| *** craigbr has quit IRC | 22:12 | |
| *** dlfryar has quit IRC | 22:47 | |
| openstackgerrit | David Schroeder proposed stackforge/monasca-agent: Clarify Libvirt documentation https://review.openstack.org/149070 | 22:51 |
| *** dlfryar has joined #openstack-monasca | 23:00 | |
| *** dlfryar has quit IRC | 23:31 | |
| *** theresa_ has quit IRC | 23:40 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!