| *** harshs has quit IRC | 00:17 | |
| *** harshs has joined #openstack-performance | 00:31 | |
| *** harshs_ has joined #openstack-performance | 00:34 | |
| *** harshs has quit IRC | 00:36 | |
| *** harshs_ is now known as harshs | 00:36 | |
| *** harshs has quit IRC | 00:37 | |
| *** mdorman has quit IRC | 00:40 | |
| *** harshs has joined #openstack-performance | 00:44 | |
| *** harshs has quit IRC | 00:44 | |
| *** mriedem has joined #openstack-performance | 01:05 | |
| *** paco20151113 has joined #openstack-performance | 01:40 | |
| *** boris-42 has joined #openstack-performance | 01:50 | |
| *** harshs has joined #openstack-performance | 01:52 | |
| *** paco20151113 has quit IRC | 01:58 | |
| *** harshs has quit IRC | 02:02 | |
| *** dims_ has quit IRC | 02:30 | |
| *** arnoldje has joined #openstack-performance | 02:55 | |
| *** mriedem has quit IRC | 03:55 | |
| *** markvoelker_ has quit IRC | 04:40 | |
| *** paco20151113 has joined #openstack-performance | 04:41 | |
| *** harshs has joined #openstack-performance | 04:52 | |
| *** harshs_ has joined #openstack-performance | 04:53 | |
| *** harshs has quit IRC | 04:56 | |
| *** harshs_ is now known as harshs | 04:56 | |
| *** rpodolyaka1 has joined #openstack-performance | 05:33 | |
| *** rpodolyaka1 has quit IRC | 05:37 | |
| *** rpodolyaka1 has joined #openstack-performance | 05:38 | |
| *** harshs_ has joined #openstack-performance | 05:44 | |
| *** harshs has quit IRC | 05:44 | |
| *** harshs_ is now known as harshs | 05:44 | |
| *** rpodolyaka1 has quit IRC | 05:46 | |
| *** rpodolyaka1 has joined #openstack-performance | 05:51 | |
| *** rpodolyaka1 has quit IRC | 06:06 | |
| *** harlowja_at_home has joined #openstack-performance | 06:18 | |
| *** rpodolyaka1 has joined #openstack-performance | 06:25 | |
| *** rpodolyaka1 has quit IRC | 06:28 | |
| *** rpodolyaka1 has joined #openstack-performance | 06:30 | |
| *** lpetrut has joined #openstack-performance | 06:33 | |
| *** harlowja_at_home has quit IRC | 06:35 | |
| *** markvoelker has joined #openstack-performance | 06:40 | |
| *** markvoelker has quit IRC | 06:45 | |
| *** rpodolyaka1 has quit IRC | 06:59 | |
| *** rpodolyaka1 has joined #openstack-performance | 07:01 | |
| *** arnoldje has quit IRC | 07:02 | |
| *** ikhudoshyn has left #openstack-performance | 07:02 | |
| *** harshs has quit IRC | 07:36 | |
| *** markvoelker has joined #openstack-performance | 07:41 | |
| *** markvoelker has quit IRC | 07:46 | |
| *** lpetrut has quit IRC | 08:07 | |
| *** lpetrut has joined #openstack-performance | 08:07 | |
| *** rmart04 has joined #openstack-performance | 08:11 | |
| *** sandeepr has joined #openstack-performance | 08:12 | |
| *** nihilifer has quit IRC | 08:36 | |
| *** nihilifer has joined #openstack-performance | 08:36 | |
| *** lpetrut has quit IRC | 08:45 | |
| *** rpodolyaka1 has quit IRC | 09:08 | |
| *** amaretskiy has joined #openstack-performance | 09:25 | |
| *** linuxaddicts has joined #openstack-performance | 09:29 | |
| *** sandeepr has quit IRC | 09:29 | |
| *** xek has joined #openstack-performance | 09:37 | |
| *** rpodolyaka1 has joined #openstack-performance | 09:41 | |
| *** markvoelker has joined #openstack-performance | 09:42 | |
| *** linuxaddicts has quit IRC | 09:43 | |
| *** sandeepr has joined #openstack-performance | 09:43 | |
| *** markvoelker has quit IRC | 09:46 | |
| *** redixin has joined #openstack-performance | 09:50 | |
| *** lpetrut has joined #openstack-performance | 09:52 | |
| *** itsuugo has joined #openstack-performance | 09:53 | |
| *** paco20151113 has quit IRC | 10:00 | |
| *** sandeepr has quit IRC | 10:24 | |
| *** sandeepr has joined #openstack-performance | 10:25 | |
| *** rpodolyaka1 has quit IRC | 10:52 | |
| *** rpodolyaka1 has joined #openstack-performance | 10:53 | |
| *** dims has joined #openstack-performance | 11:07 | |
| *** sandeepr has quit IRC | 11:08 | |
| *** dims has quit IRC | 11:12 | |
| *** dims has joined #openstack-performance | 11:18 | |
| *** aswadr has joined #openstack-performance | 11:32 | |
| *** mwagner has quit IRC | 11:34 | |
| *** markvoelker has joined #openstack-performance | 11:43 | |
| *** markvoelker has quit IRC | 11:47 | |
| *** rpodolyaka1 has quit IRC | 12:12 | |
| *** aswadr has quit IRC | 12:15 | |
| *** markvoelker has joined #openstack-performance | 12:43 | |
| *** markvoelker has quit IRC | 12:48 | |
| *** markvoelker has joined #openstack-performance | 13:09 | |
| *** nihilifer has quit IRC | 13:14 | |
| *** nihilifer has joined #openstack-performance | 13:15 | |
| *** nihilifer has quit IRC | 13:15 | |
| *** nihilifer has joined #openstack-performance | 13:16 | |
| *** rpodolyaka1 has joined #openstack-performance | 13:26 | |
| *** mwagner has joined #openstack-performance | 13:42 | |
| *** bapalm has joined #openstack-performance | 14:08 | |
| *** boris-42 has quit IRC | 14:08 | |
| *** bapalm has quit IRC | 14:43 | |
| *** bapalm has joined #openstack-performance | 14:43 | |
| *** mriedem has joined #openstack-performance | 14:45 | |
| *** kashyap has left #openstack-performance | 14:46 | |
| *** harshs has joined #openstack-performance | 14:58 | |
| *** arnoldje has joined #openstack-performance | 15:07 | |
| *** harshs has quit IRC | 15:10 | |
| *** rpodolyaka1 has quit IRC | 15:20 | |
| *** rpodolyaka1 has joined #openstack-performance | 15:22 | |
| *** rmart04 has quit IRC | 15:46 | |
| *** mdorman has joined #openstack-performance | 15:49 | |
| *** mriedem is now known as mriedem_meeting | 15:56 | |
| DinaBelova | boris-42 - I've created a draft multi backend change https://review.openstack.org/#/c/247005/1 | 16:11 |
|---|---|---|
| DinaBelova | I'm still testing it so it's WIP | 16:12 |
| *** dims_ has joined #openstack-performance | 16:22 | |
| *** dims has quit IRC | 16:24 | |
| *** rpodolyaka1 has quit IRC | 16:25 | |
| *** rpodolyaka1 has joined #openstack-performance | 16:26 | |
| klindgren | Si I updated metadata_cache_expiration across all of our metadata nodes from the default of 15 seconds to 180 seconds (they are configured to use a centralized memcache). This dropped load down from ~2% per core across all 3 servers. | 16:38 |
| klindgren | Not the magic bullet that I was hoping it would be | 16:38 |
| *** rpodolyaka1 has quit IRC | 16:45 | |
| *** rpodolyaka1 has joined #openstack-performance | 16:52 | |
| *** rpodolyaka1 has quit IRC | 16:53 | |
| *** rpodolyaka1 has joined #openstack-performance | 16:56 | |
| *** itsuugo has quit IRC | 17:04 | |
| *** mriedem_meeting is now known as mriedem | 17:08 | |
| *** lpetrut has quit IRC | 17:25 | |
| *** mwagner has quit IRC | 17:38 | |
| *** amaretskiy has quit IRC | 17:42 | |
| *** regXboi has joined #openstack-performance | 18:20 | |
| *** regXboi has quit IRC | 18:21 | |
| *** itsuugo has joined #openstack-performance | 18:25 | |
| *** mwagner has joined #openstack-performance | 18:27 | |
| *** rpodolyaka1 has quit IRC | 18:28 | |
| *** itsuugo has quit IRC | 18:29 | |
| *** rpodolyaka1 has joined #openstack-performance | 18:31 | |
| *** itsuugo has joined #openstack-performance | 18:31 | |
| *** rpodolyaka1 has quit IRC | 18:31 | |
| *** rpodolyaka1 has joined #openstack-performance | 18:32 | |
| harlowja | klindgren :( | 18:39 |
| harlowja | klindgren are u recording all that in https://etherpad.openstack.org/p/remote-conductor-performance ? | 18:40 |
| harlowja | or https://etherpad.openstack.org/p/perf-zoom-zoom | 18:40 |
| harlowja | (just so people know) | 18:40 |
| klindgren | remote-conductor-performance | 18:40 |
| harlowja | k | 18:40 |
| klindgren | line 58 | 18:40 |
| harlowja | thx | 18:40 |
| klindgren | ;-) | 18:40 |
| * harlowja saw chat with dan on operators | 18:40 | |
| * harlowja agree with comparison to existing system and 1/200 of the load .... | 18:41 | |
| klindgren | it also uses python | 18:41 |
| harlowja | ya | 18:41 |
| harlowja | thats good to know also | 18:41 |
| harlowja | so its really architectural things... | 18:41 |
| harlowja | not language blah blah speed | 18:41 |
| harlowja | (nothing really unexpected imho) | 18:42 |
| klindgren | I dunno how much of the system was pushed out to each compute node though | 18:42 |
| harlowja | when u establish funnel points in an architecture, such things aren't usually a good idear | 18:42 |
| klindgren | its not really a good apples-to-apples comparison | 18:42 |
| harlowja | fair enough | 18:42 |
| klindgren | but I was just trying to provide a counter point to the "throw more hardware at it" | 18:43 |
| harlowja | ya | 18:43 |
| harlowja | agreed | 18:43 |
| harlowja | "throw more hardware at it" imho is a copout solution | 18:43 |
| *** harshs has joined #openstack-performance | 18:43 | |
| klindgren | agreed | 18:43 |
| harlowja | DinaBelova sooo i was thinking with some folks that we might want to setup a plan (maybe during next meeting) of how we can get people that are interested in the performance work to actually get assigned real work that they will be responsible for | 18:44 |
| harlowja | vs letting those folks more organically show-up (someday...) | 18:45 |
| harlowja | and instead try to seize the people we know, and get them to commit to doing stuff ;) | 18:45 |
| harlowja | like i'll assign everything to klindgren | 18:45 |
| harlowja | :) | 18:45 |
| klindgren | and I will find a new job | 18:47 |
| klindgren | lolz | 18:47 |
| DinaBelova | harlowja - that, that will be great idea | 18:48 |
| klindgren | I am going to be interested in the Cern stuff | 18:48 |
| DinaBelova | let's do that next meeting | 18:48 |
| klindgren | to see if they notice an up-tick- in conductor usage | 18:48 |
| klindgren | I think they are nova-network though - so unsure if they are going to have the same issues | 18:48 |
| harlowja | DinaBelova sweet | 18:49 |
| * harlowja just thinking if we have people that showup, and they may be looking for something to do, we can actually get them to do soemthing (get them to do stuff while they are around) | 18:50 | |
| harlowja | more peopel that do things, the more people will be engaged ... (and the cycle repeats) | 18:50 |
| DinaBelova | that will be very cool, I agree. we don't want to be powerpoint team :) | 18:51 |
| harlowja | :) | 18:51 |
| harlowja | DinaBelova +200 | 18:51 |
| harlowja | lol | 18:51 |
| harlowja | klindgren will fix it all | 18:52 |
| harlowja | i just know it | 18:52 |
| harlowja | lol | 18:52 |
| DinaBelova | :d | 18:52 |
| DinaBelova | :D | 18:52 |
| klindgren | so as someone who has never used one of the python/openstack profilers - What can I do to start gettign some actionable results? | 18:52 |
| klindgren | I want to start making decisions from data, not guesses :-) | 18:52 |
| DinaBelova | running rally is quite simple, they have great instuctions, for instance | 18:53 |
| DinaBelova | I hope to do the same for osprofiler | 18:53 |
| DinaBelova | so that will be intuitive enough to just run tests and collect results | 18:54 |
| DinaBelova | the main thing with be to analyse it then | 18:54 |
| klindgren | well I don't need to add load to the system - I already have an issue with load. I just want to see whats causing the current load | 18:54 |
| *** rpodolyaka1 has quit IRC | 18:54 | |
| DinaBelova | well, there is no good and intuative profiling / tracing now in openstack | 18:55 |
| DinaBelova | the only way is CProfiler | 18:55 |
| DinaBelova | now | 18:55 |
| DinaBelova | but that's hard to use obviously | 18:55 |
| DinaBelova | and not very easy to analyse | 18:55 |
| DinaBelova | :( | 18:55 |
| *** badari has joined #openstack-performance | 18:56 | |
| DinaBelova | klindgren - so right now that won't be very easy to do that without some experience of python profiling background | 18:57 |
| *** itsuugo has quit IRC | 19:00 | |
| *** Guest12059 is now known as med_ | 19:04 | |
| *** med_ has quit IRC | 19:04 | |
| *** med_ has joined #openstack-performance | 19:04 | |
| *** harlowja has quit IRC | 19:05 | |
| *** itsuugo has joined #openstack-performance | 19:08 | |
| *** harlowja has joined #openstack-performance | 19:08 | |
| *** itsuugo has quit IRC | 19:12 | |
| *** itsuugo has joined #openstack-performance | 19:19 | |
| *** mriedem has quit IRC | 19:20 | |
| *** bapalm has quit IRC | 19:23 | |
| *** itsuugo has quit IRC | 19:24 | |
| *** bapalm has joined #openstack-performance | 19:26 | |
| *** itsuugo has joined #openstack-performance | 19:26 | |
| *** itsuugo has quit IRC | 19:28 | |
| *** itsuugo has joined #openstack-performance | 19:28 | |
| *** mriedem has joined #openstack-performance | 19:30 | |
| *** itsuugo has quit IRC | 19:36 | |
| *** itsuugo has joined #openstack-performance | 19:37 | |
| *** rpodolyaka1 has joined #openstack-performance | 19:55 | |
| klindgren | So I ran cprofile on conductor with 1 worker so everything stayed in the main process. I created a call graph using gprof2dot | 20:16 |
| klindgren | I can put the .svg some where | 20:16 |
| * klindgren has no clue how to follow it :-D | 20:17 | |
| harlowja | klindgren i use http://tempsend.com/ for that stuff | 20:22 |
| harlowja | if u want | 20:23 |
| harlowja | seems to work :-P | 20:23 |
| klindgren | http://tempsend.com/5340867576 | 20:24 |
| harlowja | pretty | 20:24 |
| harlowja | ha | 20:24 |
| harlowja | bb | 20:25 |
| harlowja | weird, i wonder what is doing 'utils:44:build_ffi_for_binding' | 20:26 |
| harlowja | odd | 20:26 |
| harlowja | didn't know that was happening, lol | 20:26 |
| harlowja | anyways, bbl | 20:27 |
| *** harlowja has quit IRC | 20:36 | |
| *** harlowja has joined #openstack-performance | 20:41 | |
| *** rpodolyaka1 has quit IRC | 20:53 | |
| *** pglbutt has joined #openstack-performance | 20:54 | |
| *** pglass has joined #openstack-performance | 21:01 | |
| *** openstack has joined #openstack-performance | 21:03 | |
| *** pglbutt has quit IRC | 21:04 | |
| *** rpodolyaka1 has joined #openstack-performance | 21:16 | |
| *** rpodolyaka1 has quit IRC | 21:19 | |
| *** itsuugo has quit IRC | 21:30 | |
| *** itsuugo has joined #openstack-performance | 21:30 | |
| *** arnoldje has quit IRC | 21:50 | |
| *** msemenov has quit IRC | 21:56 | |
| *** harshs has quit IRC | 22:26 | |
| *** itsuugo has quit IRC | 22:27 | |
| *** itsuugo has joined #openstack-performance | 22:40 | |
| *** itsuugo has quit IRC | 22:45 | |
| *** arnoldje has joined #openstack-performance | 22:47 | |
| *** itsuugo has joined #openstack-performance | 22:48 | |
| klindgren | DinaBelova, ping | 22:50 |
| klindgren | when you guys do your tests - do you have ssl enabled on rabbitmq? | 22:52 |
| *** itsuugo has quit IRC | 22:52 | |
| klindgren | We disabled ssl on a single rabbitmq host and forced a conductor node to connect to that - CPU usage dropped from 7-30% average to a max usage of 1.3% withe average usage of .7% | 22:53 |
| *** mriedem has quit IRC | 22:54 | |
| *** redixin has quit IRC | 22:56 | |
| *** itsuugo has joined #openstack-performance | 23:00 | |
| *** itsuugo has quit IRC | 23:04 | |
| *** itsuugo has joined #openstack-performance | 23:09 | |
| klindgren | so turns out htat was because I screwed up the change - and didn't notice that conductor was logging: 2015-11-18 16:02:13.374 16155 ERROR oslo_messaging._drivers.impl_rabbit [req-dfe0c216-9f36-4a57-970d-0475dc80a01c - - - - -] AMQP server p3plcldcap001-03.prod.phx3.secureserver.net:5671 closed the connection. Check login credentials: Socket closed | 23:10 |
| klindgren | 2015-11-18 16:02:13.393 16180 ERROR oslo_messaging._drivers.impl_rabbit [req-292d6537-681e-4146-b018-098853470841 - - - - -] AMQP server p3plcldcap001-003 | 23:10 |
| *** itsuugo has quit IRC | 23:13 | |
| *** itsuugo has joined #openstack-performance | 23:18 | |
| *** itsuugo has quit IRC | 23:22 | |
| *** harshs has joined #openstack-performance | 23:23 | |
| *** arnoldje has quit IRC | 23:23 | |
| *** itsuugo has joined #openstack-performance | 23:29 | |
| *** rmart04 has joined #openstack-performance | 23:31 | |
| *** itsuugo has quit IRC | 23:34 | |
| *** pglass has quit IRC | 23:39 | |
| *** itsuugo has joined #openstack-performance | 23:40 | |
| *** itsuugo has quit IRC | 23:44 | |
| *** redixin has joined #openstack-performance | 23:46 | |
| *** itsuugo has joined #openstack-performance | 23:48 | |
| *** itsuugo has quit IRC | 23:52 | |
| *** rmart04 has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!