| *** kevinc_ has quit IRC | 00:12 | |
| *** haigang has joined #openstack-ceilometer | 00:26 | |
| *** ankita_w_ has joined #openstack-ceilometer | 00:28 | |
| *** _cjones_ has quit IRC | 00:31 | |
| *** ankita_wagh has quit IRC | 00:32 | |
| *** ankita_w_ has quit IRC | 00:32 | |
| *** thorst has quit IRC | 00:37 | |
| *** haigang has quit IRC | 00:50 | |
| *** hhuang has joined #openstack-ceilometer | 01:11 | |
| *** liusheng has joined #openstack-ceilometer | 01:14 | |
| *** ankita_wagh has joined #openstack-ceilometer | 01:27 | |
| *** haigang has joined #openstack-ceilometer | 01:39 | |
| *** haigang has quit IRC | 01:56 | |
| *** haigang has joined #openstack-ceilometer | 01:59 | |
| *** ankita_w_ has joined #openstack-ceilometer | 02:09 | |
| *** Ephur has quit IRC | 02:11 | |
| *** ankita_wagh has quit IRC | 02:13 | |
| openstackgerrit | liusheng proposed openstack/ceilometer-specs: Improve Nova Instance metering https://review.openstack.org/209799 | 02:15 |
|---|---|---|
| *** boris-42 has quit IRC | 02:20 | |
| *** julim has quit IRC | 03:03 | |
| *** jwcroppe has joined #openstack-ceilometer | 03:29 | |
| *** ankita_w_ has quit IRC | 03:57 | |
| *** ankita_wagh has joined #openstack-ceilometer | 03:58 | |
| *** ankita_wagh has quit IRC | 04:02 | |
| *** ankita_wagh has joined #openstack-ceilometer | 04:09 | |
| *** dobson has quit IRC | 04:42 | |
| *** dobson has joined #openstack-ceilometer | 04:48 | |
| openstackgerrit | Merged openstack/aodh: devstack: use $API_WORKERS to set the number of WSGI workers in Apache https://review.openstack.org/210064 | 04:58 |
| *** ankita_wagh has quit IRC | 05:02 | |
| *** ankita_wagh has joined #openstack-ceilometer | 05:02 | |
| *** yprokule has joined #openstack-ceilometer | 05:04 | |
| *** ankita_w_ has joined #openstack-ceilometer | 05:04 | |
| *** ankita_wagh has quit IRC | 05:05 | |
| *** boris-42 has joined #openstack-ceilometer | 05:06 | |
| *** ankita_w_ has quit IRC | 05:07 | |
| *** ankita_wagh has joined #openstack-ceilometer | 05:08 | |
| *** ankita_wagh has quit IRC | 05:12 | |
| *** yprokule has quit IRC | 05:18 | |
| *** yprokule has joined #openstack-ceilometer | 05:20 | |
| *** rjaiswal has quit IRC | 05:20 | |
| *** ankita_wagh has joined #openstack-ceilometer | 05:45 | |
| *** haigang has quit IRC | 05:45 | |
| *** haigang has joined #openstack-ceilometer | 05:46 | |
| *** haigang has quit IRC | 05:47 | |
| *** haigang has joined #openstack-ceilometer | 05:47 | |
| *** fawadkhaliq has joined #openstack-ceilometer | 05:48 | |
| *** yprokule has quit IRC | 06:14 | |
| *** nadya has joined #openstack-ceilometer | 06:18 | |
| *** yprokule has joined #openstack-ceilometer | 06:20 | |
| *** ildikov has quit IRC | 06:25 | |
| *** ildikov has joined #openstack-ceilometer | 07:02 | |
| *** haigang has quit IRC | 07:21 | |
| *** afazekas_ has joined #openstack-ceilometer | 07:24 | |
| openstackgerrit | Nadya Shakhat proposed openstack/ceilometer: Make it possible to run postgresql functional job https://review.openstack.org/211651 | 07:26 |
| openstackgerrit | Nadya Shakhat proposed openstack/ceilometer: Make it possible to run postgresql functional job https://review.openstack.org/211651 | 07:27 |
| *** jwcroppe has quit IRC | 07:33 | |
| *** ankita_wagh has quit IRC | 07:42 | |
| *** ankita_wagh has joined #openstack-ceilometer | 07:43 | |
| *** ankita_wagh has quit IRC | 07:47 | |
| *** IBerezovskiy has quit IRC | 07:52 | |
| *** ityaptin has quit IRC | 07:52 | |
| *** cdent has joined #openstack-ceilometer | 07:55 | |
| *** jwcroppe has joined #openstack-ceilometer | 07:59 | |
| *** exploreshaifali has joined #openstack-ceilometer | 08:14 | |
| *** yassine__ has joined #openstack-ceilometer | 08:18 | |
| *** exploreshaifali has quit IRC | 08:23 | |
| *** IBerezovskiy has joined #openstack-ceilometer | 08:27 | |
| *** ityaptin has joined #openstack-ceilometer | 08:28 | |
| openstackgerrit | Merged openstack/aodh: Refactor api tests (_update_alarm) https://review.openstack.org/212604 | 08:34 |
| *** yassine__ has quit IRC | 08:38 | |
| *** yassine__ has joined #openstack-ceilometer | 08:39 | |
| *** nadya has quit IRC | 08:41 | |
| *** nadya_ has joined #openstack-ceilometer | 08:41 | |
| openstackgerrit | Nadya Shakhat proposed openstack/ceilometer: Make it possible to run postgresql functional job https://review.openstack.org/211651 | 08:59 |
| *** jwcroppe has quit IRC | 09:00 | |
| openstackgerrit | Nadya Shakhat proposed openstack/ceilometer: Make it possible to run postgresql functional job https://review.openstack.org/211651 | 09:01 |
| *** jwcroppe has joined #openstack-ceilometer | 09:01 | |
| *** jwcroppe has quit IRC | 09:03 | |
| jd__ | cdent: I've put a comment on the bug report related to the orphaned metrics, I'm still struggling to see where it comes from | 09:03 |
| *** jwcroppe has joined #openstack-ceilometer | 09:03 | |
| jd__ | I need a hint :( | 09:03 |
| *** jwcroppe has quit IRC | 09:05 | |
| *** jwcroppe has joined #openstack-ceilometer | 09:05 | |
| cdent | jd__: uh, hmmm | 09:05 |
| * cdent looks | 09:05 | |
| cdent | jd__: have you actually tried it? spinning up a devstack etc, or are you still working at it from an inspecting the code standpoint | 09:09 |
| jd__ | cdent: no I didn't try | 09:11 |
| jd__ | cdent: is there any particular setup I should use or the default one does that? | 09:11 |
| jd__ | I was hoping not to have to do that but if you don't have any hint I'll give it a run | 09:11 |
| cdent | In my tests I just turn on the gnocchi plugin and set file as the backend (in local.conf) | 09:12 |
| cdent | let me look at the code a sec | 09:12 |
| cdent | you should probably run ceilo+gnocchi devstack and just state at the logs for a while | 09:12 |
| cdent | stare | 09:13 |
| cdent | it is rather revealing/interesting/mesmerizing | 09:13 |
| jd__ | ack | 09:13 |
| * jd__ running ./stack.sh | 09:13 | |
| jd__ | I miss dope'n stack | 09:13 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements https://review.openstack.org/212736 | 09:14 |
| cdent | jd__: my read of the code is: | 09:18 |
| cdent | group stuff by resource | 09:18 |
| cdent | post a measure for a resource | 09:18 |
| cdent | post the resource | 09:18 |
| cdent | if there is a "workflow error" (whatever that is) it is ignored | 09:18 |
| jd__ | that's also what I read, so no metric are created | 09:19 |
| cdent | no sorry, I got it slightly wrong | 09:19 |
| cdent | actually, I don't know | 09:20 |
| * cdent needs more coffee | 09:20 | |
| cdent | it should be this hard | 09:20 |
| cdent | shouldn't | 09:20 |
| cdent | i'm going to make that coffee and will come back and make a devstack of my own and we can compare notes | 09:21 |
| * jd__ nods | 09:22 | |
| openstackgerrit | liusheng proposed openstack/ceilometer: Updated from global requirements https://review.openstack.org/213061 | 09:27 |
| cdent | i'm on a different machine now, I hope this doesn't turn out to be "that other machine is just weird" | 09:28 |
| cdent | jd__: now I have a different problem: no metrics or resources are being saved at all | 09:46 |
| * cdent cleans his configs | 09:46 | |
| *** ildikov has quit IRC | 09:49 | |
| *** haigang has joined #openstack-ceilometer | 09:56 | |
| cdent | and now keystone is not working | 10:00 |
| *** haigang has quit IRC | 10:03 | |
| jd__ | cdent: you're doomed | 10:08 |
| cdent | yeah, it seems that today is going to be one of those days where nothing works | 10:08 |
| cdent | you making any progress? | 10:08 |
| openstackgerrit | Chris Dent proposed openstack/ceilometer: Revert "Remove version from os_auth_url in service_credentials" https://review.openstack.org/213072 | 10:15 |
| cdent | jd__: ^ | 10:16 |
| openstackgerrit | Chris Dent proposed openstack/ceilometer: Revert "Remove version from os_auth_url in service_credentials" https://review.openstack.org/213072 | 10:18 |
| jd__ | cdent: slowly stacking | 10:20 |
| cdent | when i put that fix in place I finally start getting some metrics, and I'm seeing orphans | 10:21 |
| jd__ | so I just stack and ceilometer-collector gets 500 from gnocchi-api, why I'm looking for | 10:21 |
| jd__ | where the hell are those gnocchi api logs I wonder | 10:26 |
| cdent | wherever you apache logs live | 10:27 |
| jd__ | lol | 10:28 |
| jd__ | yeah except that it's actually overriden by Horizon | 10:29 |
| jd__ | anyway the problem is that Horizon was broken I just disabled it | 10:29 |
| cdent | u88iiiiiiiiiiiiiiiiiiiiioooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo | 10:29 |
| cdent | oooooooooooooooooooooooooooo8 | 10:29 |
| cdent | that was my cat | 10:29 |
| openstackgerrit | Nadya Shakhat proposed openstack/ceilometer: Make it possible to run postgresql functional job https://review.openstack.org/211651 | 10:29 |
| jd__ | now I got 400 \o/ | 10:29 |
| jd__ | Unknown archive policy low !? grr | 10:30 |
| cdent | jd__: see: https://review.openstack.org/#/c/206263/ | 10:30 |
| jd__ | I guess horizon being broken broke the creation of the AP at stacking time | 10:30 |
| jd__ | damn it | 10:30 |
| cdent | just disable horizon in local.conf, do you ever actually use it | 10:30 |
| jd__ | right, I'll do that for the next stack | 10:31 |
| * jd__ restacks | 10:31 | |
| cdent | also make sure you set the keystone stuff (a la my revision above) | 10:32 |
| cdent | or the services won't be able to query for meters etc | 10:32 |
| jd__ | everyday I'm devstacking https://www.youtube.com/watch?v=54lmikcFbAo | 10:33 |
| *** exploreshaifali has joined #openstack-ceilometer | 10:33 | |
| jd__ | cdent: gni didn't this keystone stuff got merged in ceilo? | 10:33 |
| jd__ | ah I thought aodh was wrong, it's actually Ceilo | 10:34 |
| jd__ | actually I might not be impacted I've enabled ceilo from devstack | 10:35 |
| jd__ | not from the plugin | 10:35 |
| jd__ | old school | 10:35 |
| cdent | yeah, it is just the plugin where things go boom | 10:41 |
| cdent | jd__: I've found a trigger for the NULL stuff: | 10:43 |
| cdent | basically each time a new something (such as an instance) there is a burst of activity which creates the orphaned metrics | 10:44 |
| cdent | so I ought to be able to instrument the collector to see what's going on, and then pull the trigger | 10:44 |
| *** haigang has joined #openstack-ceilometer | 10:55 | |
| *** ildikov has joined #openstack-ceilometer | 10:56 | |
| cdent | jd__: I've narrowed it down a little bit | 11:11 |
| cdent | nope, no i haven't | 11:25 |
| *** Kennan2 has joined #openstack-ceilometer | 11:36 | |
| *** exploreshaifali has quit IRC | 11:36 | |
| *** Kennan has quit IRC | 11:37 | |
| jd__ | :( | 11:38 |
| jd__ | I'm afk a while for lunch and I'll be back | 11:38 |
| *** thorst has joined #openstack-ceilometer | 11:43 | |
| *** afazekas_ has quit IRC | 11:47 | |
| *** gordc has joined #openstack-ceilometer | 12:01 | |
| *** haigang has quit IRC | 12:05 | |
| *** liusheng has quit IRC | 12:16 | |
| openstackgerrit | Merged openstack/ceilometer: Revert "Remove version from os_auth_url in service_credentials" https://review.openstack.org/213072 | 12:19 |
| gordc | cdent: curious but do you know why our functional gate tests spin up a devstack too? | 12:19 |
| gordc | nadya_: ^ | 12:19 |
| *** edmondsw has joined #openstack-ceilometer | 12:29 | |
| nadya_ | gordc: hmm, we only need vm with needed backend installed indeed | 12:30 |
| gordc | nadya_: so i guess we can probably disable nova and everything else. | 12:33 |
| gordc | i wonder why we didn't need to do it originally when unit/functional tests were all together | 12:33 |
| nadya_ | gordc: illustrates the importance of question "why we are here" :) | 12:35 |
| gordc | lol | 12:36 |
| gordc | i'll just go back to sleep | 12:36 |
| *** exploreshaifali has joined #openstack-ceilometer | 12:44 | |
| cdent | gordc: I don't really understand your question | 12:45 |
| gordc | cdent: ... why does this exist. http://logs.openstack.org/51/211651/9/check/gate-ceilometer-dsvm-functional-mongodb/fc020cc/logs/ | 12:46 |
| gordc | nadya_ already told me it's because we need vm with our backend. | 12:46 |
| gordc | i just asked because it's strange to see functional tests testing less, but taking longer now (after we split unit and functional) | 12:48 |
| cdent | the job definition presumably sets up a default devstack + ceilometer, thus the nova etc, as a way of providing future functionality for intergration tests too? | 12:48 |
| cdent | they take less if you run them locally, which is kind of the point | 12:48 |
| cdent | you can tox -epy27,functional | 12:49 |
| cdent | or tox -epy27,py-pgsql | 12:49 |
| gordc | cdent: yeah, just asked out of curiosity... it's fine when i run locally | 12:49 |
| cdent | I don't know who set up the jenkins defintion for those tests | 12:49 |
| cdent | on the aodh and gnocchi versions of the same thing we run what would presumably be called integration tests of sorts | 12:50 |
| gordc | ah, got it. i was going to say i saw it in aodh too so i assumed it was something required. | 12:50 |
| openstackgerrit | Igor Degtiarov proposed openstack/ceilometer: Avoid from storing samples with empty or not numerical volumes https://review.openstack.org/212610 | 13:05 |
| *** yassine__ has quit IRC | 13:07 | |
| *** rodrigods has quit IRC | 13:13 | |
| *** julim has joined #openstack-ceilometer | 13:16 | |
| *** yassine__ has joined #openstack-ceilometer | 13:17 | |
| openstackgerrit | Merged openstack/ceilometer: Updated from global requirements https://review.openstack.org/212736 | 13:21 |
| openstackgerrit | Merged openstack/ceilometer: Make it possible to run postgresql functional job https://review.openstack.org/211651 | 13:21 |
| *** gordc has quit IRC | 13:32 | |
| *** yassine__ has quit IRC | 13:36 | |
| *** yassine__ has joined #openstack-ceilometer | 13:38 | |
| *** exploreshaifali has quit IRC | 13:53 | |
| *** exploreshaifali has joined #openstack-ceilometer | 13:55 | |
| *** rbak_ has joined #openstack-ceilometer | 13:59 | |
| *** gordc has joined #openstack-ceilometer | 14:05 | |
| *** prad_ has joined #openstack-ceilometer | 14:16 | |
| *** prad_ is now known as pradk | 14:17 | |
| openstackgerrit | Rohit Jaiswal proposed openstack/ceilometer: Fixed event requeuing/ack on publisher failure https://review.openstack.org/211779 | 14:20 |
| *** fawadkhaliq has quit IRC | 14:22 | |
| jd__ | cdent: so no news on that bug? | 14:22 |
| cdent | jd__: not really | 14:23 |
| jd__ | cdent: do you know what configures [keystone_authtoken] in gnocchi? | 14:26 |
| cdent | there's a function call hosted by devstack itself that the plugin uses | 14:27 |
| jd__ | hm | 14:27 |
| cdent | what's going on? | 14:27 |
| jd__ | looks like it's using a deprecated alias for an option | 14:27 |
| cdent | yes, that's a devstack buglet | 14:28 |
| jd__ | ok then I'll ignore it's probably already in a patch waiting to be merged | 14:28 |
| jd__ | or not? cannot find anything that looks like that | 14:29 |
| cdent | as far as I can tell, jd__, unless stuff is actively broken it rarely gets fixed :( | 14:30 |
| jd__ | yeah :/ | 14:30 |
| jd__ | I sent https://review.openstack.org/#/c/213173/ we'll see | 14:31 |
| jd__ | back to our Gnocchi business now… | 14:32 |
| cdent | I have no clue what's going on with the dispatcher and I've given myself a delightful headache trying to unwind things. I'm going to need to take a bit of a break soon. | 14:32 |
| cdent | The theory I was operating with was that somehow a create_metric is eventually called on the gnocchi side, without a resource | 14:32 |
| cdent | but actually tracing back to something that does that is not working out | 14:32 |
| jd__ | I thought about that but Gnocchi is raising the 400 error very soon | 14:33 |
| jd__ | way before creating any metric | 14:33 |
| cdent | the 400 may be a red herring | 14:33 |
| cdent | it is supposed to happen | 14:33 |
| cdent | and may have nothing to do with the problm | 14:33 |
| cdent | that's just gnocchi saying: that id is no good, I reject you | 14:33 |
| jd__ | I lol at MySQL printing me binary strings in my terminal when doing SELECT * FROM metric; :( | 14:34 |
| cdent | at that point that whole set of metrics should be dropped | 14:34 |
| cdent | and maybe it is | 14:34 |
| cdent | I don't know _what_ is actually creating the metrics because they have no name | 14:34 |
| jd__ | I only polled images and I have orphaned metrics on my brand new devstack | 14:35 |
| jd__ | and I don't have any 400 in logs | 14:35 |
| cdent | yeah that's what I was trying to say above | 14:35 |
| cdent | the 400 is not relevant | 14:35 |
| jd__ | understood indeed | 14:35 |
| jd__ | it was a wrong lead! | 14:36 |
| jd__ | I think Ceilometer is trying to lose us | 14:36 |
| jd__ | it might be listening what we're saying… shhhht | 14:36 |
| cdent | not that there was really any doubt, but the orphans are created by the ceilometer keystone user and the service project | 14:37 |
| *** alejandrito has joined #openstack-ceilometer | 14:44 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/ceilometer: Update install docs with gnocchi dispatcher info https://review.openstack.org/206110 | 14:53 |
| pradk | nadya_, ^^ can you see if this helps.. I added a gnocchi lines two lines below where I talked about the backend. Also extended the note about url to say its gnocchi endpoint | 14:54 |
| pradk | s/lines/links | 14:54 |
| *** r-daneel has joined #openstack-ceilometer | 14:55 | |
| nadya_ | pradk: yes, thank you! | 14:55 |
| pradk | cool | 14:55 |
| jd__ | I'm considering restacking with PG http://stackoverflow.com/questions/9842250/does-the-mysql-cli-tool-provide-a-way-to-display-binary-data-in-a-console-friend | 14:59 |
| jd__ | grrrr | 15:00 |
| cdent | he | 15:00 |
| cdent | I just did the HEX trick in my selects | 15:00 |
| cdent | but as far as I can tell there's not much the database can tell us other than the fact the data is there | 15:00 |
| cdent | s/data is/orphans are/ | 15:01 |
| *** exploreshaifali has quit IRC | 15:01 | |
| *** rjaiswal has joined #openstack-ceilometer | 15:02 | |
| *** lsmola has quit IRC | 15:13 | |
| cdent | gordc: I'm as off the net as possible for the next 4 days. I've got some grenade-related infra stuff pending. Do you want to keep an eye on that, or should I -workflow it? | 15:14 |
| jd__ | I'm making progress cdent | 15:14 |
| cdent | huzzah! | 15:14 |
| cdent | what do you see jd__ | 15:14 |
| jd__ | it's too soon to tell but I know that the problem is triggered by _ensure_resource_and_metric() in the dispatcher | 15:15 |
| jd__ | I'm still going down that rabbit hole | 15:15 |
| jd__ | I'll report later | 15:15 |
| jd__ | jd__ over | 15:15 |
| cdent | for a little while I though that the decorator was causing issues, so you might try removing that. I was convinced that not having it there was removing the problem, but then when I tried to replicate it reliably, it didn't. Later I realized that I had twiddled some other knobs at the same time, so it may still be an issue | 15:16 |
| gordc | cdent: which grenade stuff? | 15:17 |
| cdent | gordc: https://review.openstack.org/#/c/213036/ | 15:17 |
| gordc | cdent: what do i need to do? revert it if it breaks? | 15:18 |
| cdent | it has a dependency on removing ceilometer from grenade itself | 15:18 |
| cdent | so you'd have to revert two things | 15:18 |
| cdent | but yeah, basically | 15:19 |
| gordc | err maybe -workflow... i don't think i'll be tracking much things this weekend. | 15:19 |
| cdent | no prob, that was kind of what I was thinking too | 15:19 |
| gordc | cool. thanks. | 15:20 |
| gordc | enjoy your escape from technology | 15:20 |
| cdent | I hope so. I've lost any capacity for thought | 15:21 |
| *** prad_ has joined #openstack-ceilometer | 15:21 | |
| gordc | "things devs say after working on openstack" | 15:21 |
| gordc | i'll start a tumblr... that still exists right? | 15:22 |
| *** prad_ has quit IRC | 15:29 | |
| jd__ | ooohhh I smell a race condition | 15:30 |
| openstackgerrit | gordon chung proposed openstack/ceilometer: handle list payloads in notifications https://review.openstack.org/213202 | 15:31 |
| cdent | oh dear | 15:31 |
| *** openstackgerrit has quit IRC | 15:31 | |
| *** openstackgerrit has joined #openstack-ceilometer | 15:31 | |
| *** nadya_ has quit IRC | 15:39 | |
| *** wayneeseguin has quit IRC | 15:40 | |
| jd__ | cdent: definitely a race :) I set executor_thread_pool_size = 1 in ceilometer.conf and the problem disapear entirely | 15:45 |
| jd__ | now I need to wire my brain in concurrency mode to understand where the race is | 15:46 |
| jd__ | but it should be pretty obvious | 15:46 |
| jd__ | haha found it I guess | 15:50 |
| cdent | i would translate this as: eventlet is not actually your friend | 15:50 |
| cdent | (but maybe it has nothing to do with eventlet, I just felt like saying that) | 15:51 |
| jd__ | it never has been | 15:51 |
| jd__ | but it's not a problem with eventlet itself, just with concurrency and the way the code is written | 15:51 |
| jd__ | so the following happen | 15:51 |
| jd__ | (rubber duck mode) | 15:51 |
| jd__ | the problem is very visible if you start the collector with 64 threads (the default) and a lot of samples in the queue | 15:52 |
| jd__ | because you have suddenly a bunch of threads starting to handle the sample | 15:52 |
| openstackgerrit | Igor Degtiarov proposed openstack/ceilometer: Avoid from storing samples with empty or not numerical volumes https://review.openstack.org/212610 | 15:52 |
| jd__ | what they do, pretty much all at the same time is try to post the mesure to <resource_id>/metric/<metric_name> | 15:52 |
| jd__ | they all get 404 | 15:53 |
| jd__ | ok so they all try to create the resource | 15:53 |
| jd__ | only one will succeed (highlander mode) | 15:53 |
| jd__ | the resource is created with ALL the metrics (e.g. image, image.size, image.send, etc) | 15:53 |
| openstackgerrit | Igor Degtiarov proposed openstack/ceilometer: [WIP]Avoid from storing samples with empty or not numerical volumes https://review.openstack.org/212610 | 15:53 |
| jd__ | the others threads got a 409 Conflict | 15:53 |
| jd__ | they think that well, the resource exists, but probably not the metric, I'll create it | 15:53 |
| jd__ | but wait… the first thread already created the resource and ALL the metrics! | 15:54 |
| jd__ | anyway they go ahead and create the metric, which oprhans the previous one | 15:54 |
| jd__ | tada (sound of victory) | 15:54 |
| cdent | ah ha | 15:54 |
| jd__ | now, how to fix that, missa think… | 15:55 |
| cdent | so that never crossed my mind because I hadn't realized that a subsequent association would break a previous one. I thought there would be some kind of error. I was assume that something, somehow, must be doing create_metric (whether client or server) and I just could not find it | 15:55 |
| cdent | no wonder | 15:55 |
| jd__ | maybe we should not allow to replace a metric if it already exists | 15:57 |
| jd__ | the call POST /v1/resource/generic/<id>/metric does not do any check if the metrics exist or not, it just does what is said | 15:57 |
| jd__ | it if would refuse the call, that'd fix the issue | 15:58 |
| jd__ | saying "nah that already exist" because it has been created in the meantime | 15:58 |
| jd__ | yeah we could return 409 | 15:59 |
| jd__ | let's try that | 15:59 |
| cdent | is it gnocchi code that is causing the association to break, and if so, where? | 15:59 |
| gordc | does it make sense to hide all this in a client? wondering if it's easier to manage all the connections that way. | 16:02 |
| cdent | gordc: there's a client growing next to the dispatcher which will presumably be extracted when it is shown to be a proper thing | 16:03 |
| cdent | clearly it is not quite there yet | 16:03 |
| jd__ | ok so we actually already return 409 where I thought we didn't | 16:03 |
| gordc | cdent: yeah. that's what i noticed... i guess we need a few more features to justify breaking it out. | 16:03 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: gabbi: fix a typo in name of a test https://review.openstack.org/213222 | 16:04 |
| *** ankita_wagh has joined #openstack-ceilometer | 16:10 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: handle list payloads in notifications https://review.openstack.org/213202 | 16:13 |
| jd__ | cdent: ok I found the bug I can reproduce it with curl finally | 16:20 |
| jd__ | it's in Gnocchi | 16:20 |
| jd__ | if you create a resource a first time => 201 | 16:20 |
| jd__ | if you create a resource a second time => 409 | 16:21 |
| jd__ | well 409… but it actually changed the metric :-> | 16:21 |
| cdent | oh noes! | 16:22 |
| cdent | I looked for that but I could not for the life of me trace the code | 16:23 |
| cdent | which is why I'm making gordc give me half his salary | 16:23 |
| gordc | jokes on you. | 16:24 |
| cdent | oh yeah, communism | 16:24 |
| gordc | i have a gambling problem. it's all gone. | 16:24 |
| cdent | how very canadian of you | 16:25 |
| gordc | enjoy your probing when you travel | 16:25 |
| jd__ | I'm writing a unit test | 16:26 |
| cdent | again with the inscrutability, gordc, the therapy is not working | 16:26 |
| gordc | hmmm... there's only four countries according to wikipedia... this don't seem right | 16:26 |
| jd__ | then I push it then I'm off drinking mojitoS | 16:26 |
| cdent | jd__++ | 16:26 |
| gordc | to a onlooker it seems we have massive addiction problems in Ceilometer. | 16:27 |
| cdent | i blame society! | 16:27 |
| cdent | it's the system man | 16:27 |
| jd__ | ok so the 409 does not change the metric | 16:28 |
| * cdent is ever more confused | 16:28 | |
| jd__ | it just created new metrics that will stay orphan due to the call convert_metric_list() | 16:28 |
| jd__ | I'll show you the code | 16:28 |
| jd__ | it's just a plain dumb bug lol | 16:29 |
| cdent | i see it now | 16:29 |
| cdent | convert_mertric is before update_resource | 16:29 |
| jd__ | https://git.openstack.org/cgit/openstack/gnocchi/tree/gnocchi/rest/__init__.py#n890 | 16:29 |
| jd__ | so that creates brand new metric | 16:29 |
| jd__ | with no parent | 16:29 |
| jd__ | and since 409 will be returned, well, the metric stays orphan | 16:30 |
| jd__ | but we never saw that because well it does not break anything | 16:30 |
| jd__ | it just creates orphan metrics | 16:30 |
| jd__ | pollution! | 16:30 |
| cdent | weird, I swear I looked at that code multiple times | 16:30 |
| jd__ | so we need to either 1. delete those orphan metrics before abort()ing 2. not create them in the first place | 16:30 |
| gordc | coding is harder when you boozing | 16:31 |
| jd__ | and it's hard to see that bug with a unit test since it's just well, pollution | 16:31 |
| cdent | is there a reason convert_metric_list is called before update_resource? | 16:31 |
| cdent | (and create_resource) | 16:31 |
| jd__ | cdent: convert_metric_list() is used to create the resource in the storage and metric layers first if you pass not a metric uuid but a { | 16:31 |
| cdent | each time it is called, it is called before the associated indexer resource method | 16:31 |
| jd__ | but a {"archive_policy_name": foobar} to create one dynamically | 16:32 |
| jd__ | it's a 2 pass process, 1. create the metrics in storage+indexer 2. create/update the resource with those new metrics | 16:32 |
| jd__ | but if 2. fails we never rollback 1. | 16:32 |
| * cdent boggles | 16:32 | |
| cdent | no wonder I didn't get it | 16:32 |
| jd__ | I'll add that as a note in the bug report and I'll fix it tomorrow-monday | 16:33 |
| openstackgerrit | gordon chung proposed openstack/ceilometer: full multi-meter support https://review.openstack.org/212815 | 16:33 |
| cdent | cool | 16:33 |
| cdent | it will be nice to have that one out of the way: it was confusing other digging around quite a lot | 16:34 |
| jd__ | comment added, taking off :) | 16:38 |
| cdent | me too | 16:38 |
| cdent | thanks for the digging | 16:39 |
| *** cdent has quit IRC | 16:40 | |
| *** nadya has joined #openstack-ceilometer | 16:41 | |
| *** ankita_wagh has quit IRC | 16:45 | |
| *** exploreshaifali has joined #openstack-ceilometer | 16:45 | |
| *** ankita_wagh has joined #openstack-ceilometer | 16:46 | |
| *** nadya has quit IRC | 16:46 | |
| *** yprokule has quit IRC | 16:49 | |
| *** ankita_wagh has quit IRC | 16:50 | |
| *** yassine__ has quit IRC | 16:59 | |
| *** _cjones_ has joined #openstack-ceilometer | 17:00 | |
| *** Kennan2 has quit IRC | 17:05 | |
| *** ankita_wagh has joined #openstack-ceilometer | 17:14 | |
| *** david-lyle is now known as cbrown | 17:17 | |
| *** Kennan has joined #openstack-ceilometer | 17:21 | |
| *** rbak_ has quit IRC | 17:39 | |
| *** rbak_ has joined #openstack-ceilometer | 17:39 | |
| *** nadya has joined #openstack-ceilometer | 17:41 | |
| *** rbak has joined #openstack-ceilometer | 17:48 | |
| *** rbak_ has quit IRC | 17:49 | |
| *** boris-42 has quit IRC | 17:50 | |
| *** Ephur has joined #openstack-ceilometer | 17:51 | |
| openstackgerrit | Clément Contini proposed openstack/ceilometermiddleware: HTTP_X_TENANT_ID is deprecated and replaced by HTTP_X_PROJECT_ID https://review.openstack.org/213276 | 17:55 |
| *** nadya has quit IRC | 17:57 | |
| pradk | gordc, did you have any luck with jsonpath matching static values? one way i'm able to get it working is if i explicitly define names as a list for each unit | 18:07 |
| pradk | i dint really find a way to lookup what unit is for meter as thats not in the notification its part of the definition | 18:08 |
| pradk | which wont work for us | 18:08 |
| openstackgerrit | Clément Contini proposed openstack/ceilometermiddleware: HTTP_X_TENANT_ID is deprecated and replaced by HTTP_X_PROJECT_ID https://review.openstack.org/213276 | 18:09 |
| openstackgerrit | Clément Contini proposed openstack/ceilometermiddleware: HTTP_X_TENANT_ID is deprecated and replaced by HTTP_X_PROJECT_ID https://review.openstack.org/213276 | 18:11 |
| *** ankita_w_ has joined #openstack-ceilometer | 18:17 | |
| *** nadya has joined #openstack-ceilometer | 18:20 | |
| *** ankita_wagh has quit IRC | 18:21 | |
| *** nadya has quit IRC | 18:25 | |
| *** fawadkhaliq has joined #openstack-ceilometer | 18:30 | |
| gordc | pradk: hmm. i'll look now | 18:36 |
| *** exploreshaifali has quit IRC | 18:41 | |
| gordc | pradk: hmm. it's doable i think but it requires a regex to match .*\.percent meters | 18:50 |
| gordc | pradk: i think the easier way is to have a specific definition for each. | 18:51 |
| pradk | gordc, yea i originally had the definitions separate, but the volume still needs to be acquired from the payload list | 18:52 |
| pradk | one way i was thinking is we have the multi meter logic handle list of names and look up corresponding attrs | 18:53 |
| gordc | http://paste.openstack.org/show/414681/ | 18:53 |
| gordc | that should work | 18:54 |
| pradk | gordc, yea i was hoping to leverage multi meters here, but i guess thats too complex | 18:55 |
| gordc | pradk: yeah... we'd have to regex on names that end with percent | 18:55 |
| gordc | and then assume those that don't are have ns units... it'll get messy. | 18:55 |
| gordc | i think this is actually a cleaner/better way... to tackle each one individually. | 18:56 |
| pradk | yea my motivation was to reduce verbosity but seems like it isnt worth the complexity | 18:56 |
| pradk | yea i'll revert back to individual | 18:56 |
| pradk | i'll poke some more at multi meter approach and see if i can get regex working | 18:57 |
| gordc | i wonder if it's worth changing my other multi-meter back. | 18:57 |
| pradk | which one? | 18:57 |
| gordc | the ceilometermiddleware one | 18:57 |
| *** ankita_wagh has joined #openstack-ceilometer | 18:57 | |
| gordc | i actually like being explicit... it's less guessing from reader pov | 18:58 |
| pradk | you mean the swift middleware one | 18:58 |
| gordc | yeah. well swift in ceilometermiddleware | 18:59 |
| pradk | yea from readability standpoint it does help.. but the file is growing huge | 18:59 |
| pradk | hopefully in M we can split is into projects | 18:59 |
| gordc | that'll convince ildikov to accept our idea to split and let projects own ;) | 18:59 |
| gordc | we'll make it really massive. | 18:59 |
| pradk | hehe yea, good idea | 19:00 |
| openstackgerrit | Rohit Jaiswal proposed openstack/ceilometer: Fixed event requeuing/ack on publisher failure https://review.openstack.org/211779 | 19:00 |
| *** ankita_w_ has quit IRC | 19:01 | |
| gordc | the resource stuff is still super weird though... is the 'separator' attr for concatenating the resource_id? | 19:02 |
| pradk | gordc, yea it is, i thought about this and discussed with cdent as well.. the question is do we want to make the yaml look like how its represented or do we want yaml to achieve what we want it to look like | 19:05 |
| pradk | gordc, we leaned toward the latter .. if we start making the yaml match how it looks, its going to compromise the simplicity | 19:06 |
| pradk | gordc, we could move the separator under resource id if that makes it cleaner, something like: | 19:12 |
| pradk | resource_id: | 19:12 |
| pradk | attr: - foo -bar | 19:12 |
| pradk | separator: '_' | 19:12 |
| gordc | yeah, that'd be better from grouping pov | 19:13 |
| gordc | i agree with what you and cdent reached... some of the stuff we have probably ain't meant for declarative meters. | 19:13 |
| *** fawadkhaliq has quit IRC | 19:14 | |
| gordc | it'll help teach ppl to make simple notifications if they want simple meters. | 19:14 |
| pradk | yea agreed | 19:14 |
| *** nadya has joined #openstack-ceilometer | 19:19 | |
| *** boris-42 has joined #openstack-ceilometer | 19:24 | |
| *** nadya has quit IRC | 19:31 | |
| *** ankita_wagh has quit IRC | 19:34 | |
| *** ankita_wagh has joined #openstack-ceilometer | 19:34 | |
| *** ankita_wagh has quit IRC | 19:38 | |
| pradk | gordc, argh bummer jsonpath-rw doesnt support ? | 19:42 |
| gordc | ...well that blows | 19:44 |
| *** nadya has joined #openstack-ceilometer | 19:58 | |
| *** alejandrito has quit IRC | 20:04 | |
| gordc | pradk: that's very frustrating jsonpath-rw doesn't work... we can try adding it... or we can switch to objectpath (which has this fancy webpage). | 20:25 |
| pradk | gordc, yea thats what i'm trying .. objectpath has different format so something like: | 20:25 |
| gordc | f* never mind objectpath has bs license | 20:26 |
| *** ankita_wagh has joined #openstack-ceilometer | 20:26 | |
| pradk | d.execute('payload.metrics.*[@['name'] is 'cpu.frequency'].value') | 20:26 |
| pradk | ah bs wont work wrt licensing for openstack? | 20:26 |
| gordc | it won't pass openstack licensing. | 20:26 |
| pradk | gotcha | 20:26 |
| *** kevinc_ has joined #openstack-ceilometer | 20:26 | |
| pradk | gordc, there is a enhancement request in place https://github.com/kennknowles/python-jsonpath-rw/issues/8 | 20:27 |
| pradk | i guess they never got to it | 20:27 |
| gordc | or i don't think it's kosher... it's got same license as mongo | 20:27 |
| *** nadya has quit IRC | 20:28 | |
| gordc | i'll look at it over weekend... going to pack and move boxes... office moving day | 20:28 |
| gordc | have a good weekend. | 20:28 |
| *** gordc has quit IRC | 20:28 | |
| pradk | laters | 20:28 |
| kevinc_ | Is there an example of a sample list query using the python-ceilometerclient (in python, not on the command line) that shows a query using a filter together with orderby? | 20:30 |
| openstackgerrit | Merged openstack/gnocchi: gabbi: fix a typo in name of a test https://review.openstack.org/213222 | 20:48 |
| *** pradk has quit IRC | 21:04 | |
| *** jwcroppe_ has joined #openstack-ceilometer | 21:11 | |
| *** jwcroppe has quit IRC | 21:13 | |
| *** jwcroppe has joined #openstack-ceilometer | 21:15 | |
| *** jwcroppe_ has quit IRC | 21:16 | |
| *** cbrown is now known as david-lyle | 21:20 | |
| *** thorst has quit IRC | 21:42 | |
| *** ankita_w_ has joined #openstack-ceilometer | 21:46 | |
| *** ankita_wagh has quit IRC | 21:49 | |
| *** jwcroppe_ has joined #openstack-ceilometer | 21:55 | |
| *** jwcroppe has quit IRC | 21:58 | |
| *** jwcroppe has joined #openstack-ceilometer | 21:59 | |
| *** jwcroppe_ has quit IRC | 22:01 | |
| openstackgerrit | Merged openstack/ceilometer: Update install docs with gnocchi dispatcher info https://review.openstack.org/206110 | 22:04 |
| *** jwcroppe_ has joined #openstack-ceilometer | 22:14 | |
| *** jwcroppe has quit IRC | 22:16 | |
| *** nadya has joined #openstack-ceilometer | 22:20 | |
| *** nadya has quit IRC | 22:25 | |
| *** ankita_wagh has joined #openstack-ceilometer | 22:27 | |
| *** ankita_w_ has quit IRC | 22:30 | |
| *** ankita_w_ has joined #openstack-ceilometer | 22:42 | |
| *** ankita_wagh has quit IRC | 22:42 | |
| *** ankita_wagh has joined #openstack-ceilometer | 22:44 | |
| *** ankita_w_ has quit IRC | 22:44 | |
| *** edmondsw has quit IRC | 22:51 | |
| *** nadya has joined #openstack-ceilometer | 23:01 | |
| *** r-daneel has quit IRC | 23:01 | |
| *** nadya has quit IRC | 23:06 | |
| *** jwcroppe has joined #openstack-ceilometer | 23:19 | |
| *** jwcroppe_ has quit IRC | 23:21 | |
| *** ankita_wagh has quit IRC | 23:22 | |
| *** ankita_wagh has joined #openstack-ceilometer | 23:23 | |
| *** gordc has joined #openstack-ceilometer | 23:24 | |
| *** gordc has quit IRC | 23:24 | |
| *** ankita_wagh has quit IRC | 23:27 | |
| *** julim has quit IRC | 23:30 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!