Monday, 2015-11-16

« Sunday, 2015-11-15 Index Tuesday, 2015-11-17 »
*** mriedem has quit IRC00:30
*** mwagner has quit IRC01:00
*** markvoelker has joined #openstack-performance01:06
*** markvoelker has quit IRC01:10
*** bapalm has quit IRC01:29
*** kun_huang has quit IRC01:32
*** serverascode has quit IRC01:32
*** mgagne has quit IRC01:33
*** paco20151113 has joined #openstack-performance01:35
*** serverascode has joined #openstack-performance01:37
*** mgagne has joined #openstack-performance01:38
*** mgagne is now known as Guest9104301:38
*** kun_huang has joined #openstack-performance01:39
*** bapalm has joined #openstack-performance01:56
*** bapalm has quit IRC02:04
*** harshs has joined #openstack-performance03:01
*** markvoelker has joined #openstack-performance03:07
*** markvoelker has quit IRC03:11
*** harshs has quit IRC05:04
*** mwagner has joined #openstack-performance05:39
*** arnoldje has quit IRC06:45
*** nihilifer has quit IRC07:02
*** nihilifer has joined #openstack-performance07:03
*** markvoelker has joined #openstack-performance07:08
*** markvoelker has quit IRC07:13
*** boris-42 has joined #openstack-performance07:22
*** rmart04 has joined #openstack-performance07:32
*** rmart04 has quit IRC07:37
*** rmart04 has joined #openstack-performance08:09
*** markvoelker has joined #openstack-performance09:09
*** markvoelker has quit IRC09:13
*** rmart04 has quit IRC09:16
*** xek_ is now known as xek09:22
*** itsuugo has joined #openstack-performance10:00
*** itsuugo has quit IRC10:16
*** itsuugo has joined #openstack-performance10:16
*** paco20151113 has quit IRC10:19
*** itsuugo has quit IRC10:26
*** itsuugo has joined #openstack-performance10:28
*** itsuugo has quit IRC10:33
*** itsuugo has joined #openstack-performance10:34
*** itsuugo has quit IRC10:39
*** itsuugo has joined #openstack-performance10:52
*** aojea has joined #openstack-performance10:54
*** itsuugo has quit IRC10:56
*** aojea has quit IRC10:59
*** markvoelker has joined #openstack-performance11:10
*** markvoelker has quit IRC11:14
*** itsuugo has joined #openstack-performance11:46
*** xek has quit IRC11:47
*** itsuugo has quit IRC11:51
*** dims has joined #openstack-performance11:58
*** markvoelker has joined #openstack-performance12:40
*** markvoelker has quit IRC12:45
*** openstack has joined #openstack-performance12:48
-cameron.freenode.net- [freenode-info] channel trolls and no channel staff around to help? please check with freenode support: http://freenode.net/faq.shtml#gettinghelp12:48
*** itsuugo has joined #openstack-performance13:04
*** itsuugo has quit IRC13:08
*** markvoelker has joined #openstack-performance13:13
DinaBelovaSpamapS - hello! any news about conductor profiling?13:30
DinaBelovaSpamapS - btw, I've read you were able to reproduce the issue with only instances showing/listing.. but afair nova-conductor is not used by nova-api, so the load on the conductor was most probably generated by creation of these VMs?13:33
DinaBelovakun_huang - here is some number of osprofiler-related changes - https://review.openstack.org/#/c/245130/ https://review.openstack.org/#/c/245136/ and https://review.openstack.org/#/q/status:open+branch:master+topic:osprofiler-optional-api-paste-ini,n,z13:35
kun_huangDinaBelova: good start ;)13:38
DinaBelovaso the issue with Ceilometer events usage right now is that they switched to the declarative events and meters definition, so that leads to no opportunity to say "save all notification payload" now. Ceilo dev will work on this, so current version of https://review.openstack.org/#/c/245136/ is not final one - it's just to make all the stuff somehow workable now13:39
*** boris-42 has quit IRC13:48
*** itsuugo has joined #openstack-performance13:59
*** boris-42 has joined #openstack-performance14:01
*** regXboi has joined #openstack-performance14:20
*** mriedem has joined #openstack-performance14:30
*** arnoldje has joined #openstack-performance14:30
*** itsuugo has quit IRC14:33
*** regXboi has quit IRC14:46
*** regXboi has joined #openstack-performance14:48
*** itsuugo has joined #openstack-performance14:49
*** bapalm has joined #openstack-performance14:51
*** itsuugo has quit IRC14:52
*** itsuugo has joined #openstack-performance14:52
*** bapalm has quit IRC14:53
*** bapalm has joined #openstack-performance15:04
*** itsuugo has quit IRC15:11
*** regXboi has quit IRC15:13
*** regXboi has joined #openstack-performance15:31
*** itsuugo has joined #openstack-performance15:34
*** arnoldje has quit IRC15:38
*** itsuugo has quit IRC15:38
*** arnoldje has joined #openstack-performance15:49
*** harlowja_at_home has joined #openstack-performance15:53
*** rpodolyaka1 has joined #openstack-performance15:58
*** klindgren__ has joined #openstack-performance16:20
*** klindgren__ is now known as klindgren16:21
klindgrenmorning all16:21
harlowja_at_homesup16:23
*** klindgren_ has quit IRC16:23
*** harlowja_at_home has quit IRC16:39
*** mriedem has quit IRC17:22
*** rpodolyaka1 has quit IRC17:24
*** itsuugo has joined #openstack-performance17:27
*** Guest91043 is now known as mgagne17:28
*** mgagne has joined #openstack-performance17:28
*** mgagne is now known as Guest9104317:29
*** Guest91043 is now known as mgagne17:29
*** itsuugo has quit IRC17:32
*** harshs has joined #openstack-performance17:42
*** mriedem has joined #openstack-performance18:03
*** harlowja has joined #openstack-performance18:06
*** itsuugo has joined #openstack-performance18:06
*** itsuugo has quit IRC18:08
*** rpodolyaka1 has joined #openstack-performance18:32
*** rpodolyaka1 has quit IRC18:34
*** rpodolyaka1 has joined #openstack-performance18:34
*** rpodolyaka1 has quit IRC18:41
*** rpodolyaka1 has joined #openstack-performance18:49
*** Guest17963 is now known as med_19:05
*** med_ has joined #openstack-performance19:05
DinaBelovaklindgren, harlowja, good morning :)19:07
harlowjamorning?19:07
harlowjai think its night where u are DinaBelova  ;)19:07
DinaBelovawell, I really use to use (funny words combination) time zone of people I'm talking to )19:07
DinaBelovaso yeah, a bit late here :)19:07
harlowja;)19:08
harlowjai'm actually on the moon19:08
harlowjaso its moontime here19:08
harlowjalol19:08
DinaBelovaharlowja - I have a bunch of changes for yo to review :)19:08
DinaBelova:D19:08
harlowjaoh man19:08
harlowjalol19:08
harlowjaDinaBelova can u shove all those on an etherpad?19:08
DinaBelovahttps://review.openstack.org/#/c/245130/  https://review.openstack.org/#/c/245136/ and https://review.openstack.org/#/q/status:open++branch:master+topic:osprofiler-optional-api-paste-ini,n,z19:08
DinaBelova:D19:08
harlowjaoh19:08
harlowjathat works to19:08
harlowjamaybe we should have perf-review etherpad anyway?19:09
harlowja*something like what i did @ https://etherpad.openstack.org/p/mitaka-taskflow-2.019:09
DinaBelovayep, it'll be very nice if you'll create it19:09
harlowjabut for perf19:09
harlowjak19:09
DinaBelovathanks sir!19:09
DinaBelovaharlowja - I managed to fix current osprofiler to work with current ceilo version - https://drive.google.com/file/d/0ByRtVrZu5ifzWlQtWkVlUnBIYzQ/view?usp=sharing19:10
harlowjaDinaBelova https://etherpad.openstack.org/p/perf-zoom-zoom19:10
harlowjathere u go19:10
harlowjaat your service maddam19:10
harlowja*naming may or may not == mazda zoom zoom, lol19:10
harlowja(i think thats mazda anyway)19:11
harlowjaDinaBelova want to include https://etherpad.openstack.org/p/perf-zoom-zoom in channel topic?19:13
harlowjai think u can modify that :-P19:13
harlowja*being an admin19:13
harlowjaDinaBelova i think we (yahoo) really want to switch osprofiler to not ceilometer19:14
harlowjabecause, ummm, ya, we don't use celiometer19:14
harlowjabecause ya, ummm, other systems existed before it...19:14
harlowjaso i'll see if i can do the osprofiler 'pluggable writers' or whatever19:14
DinaBelovaharlowja - funny naming :D19:17
*** DinaBelova changes topic to "OpenStack performance discussions - testing, bottlenecks, etc. Performance-related changes on review - https://etherpad.openstack.org/p/perf-zoom-zoom (tools, bottlenecks, etc.)"19:18
DinaBelovaharlowja - about switching to something else - that will be next thing I'll be working on19:18
DinaBelovastep #1 was to make current profiler working with current ceilometer - to have something to compare with :D19:19
DinaBelova============================================19:23
harlowjaDinaBelova far enough19:23
DinaBelovafolks, just a reminder to add your items to the tomorrows agenda19:23
harlowja*fair19:23
DinaBelova============================================19:23
DinaBelova:D19:23
harlowjadamn, early meeting again19:23
harlowjalol19:23
harlowjai'll see if i'll be there19:24
harlowjaha19:24
harlowjamoontime its early19:24
harlowjalol19:24
DinaBelovain half-humanish way will beok as well :D19:24
harlowja:)19:24
harlowjahalf-awake,half-human on moontime19:25
harlowjathats me19:25
DinaBelovalol :D19:27
SpamapSDinaBelova: regarding your question earlier: yes I confirmed, the booting was what was eating up resources on the conductor.19:28
SpamapSDinaBelova: I had to step back from the work. It does look like metadata lookups are going through conductor because they have to fetch info from neutron.19:29
DinaBelovaSpamapS - oh, interesting19:29
SpamapSklindgren: ^^ I was able to see that right before I ran out of time and moved on to other stuff.19:29
SpamapSklindgren: your hunch, I think, is right.19:29
DinaBelovaSpamapS, klindgren - most probably I'll try to investigate that tomorrow in parallel with first patch to make osprofiler multi-backend19:30
SpamapSDinaBelova: I wasn't able to draw hard conclusions, but my theory is still that a lot of that CPU time is spent in json.19:31
DinaBelovaSpamapS - that might be the issue, yep. Although I19:32
DinaBelovawill need some additional info to ensure :)19:32
DinaBelovaand first of all to reproduce on my env :D19:32
SpamapSthe only other significant source I saw in the (lame) python profiler was str.join19:35
SpamapSbut absent a callgraph there was no way to know where that joining was happening. :-P19:36
klindgrenAll the connections go through conductor because they need neutron - but they also need to access the DB to get the metadata information for the vm's as well19:36
DinaBelovaklindgren - thanks for the clarification19:37
harlowjaklindgren do u know if u are using the c boosted json?19:38
harlowjai think u posted a pip freeze, but not sure19:38
klindgrenI posted a pip freeze19:38
harlowjakk19:38
* harlowja locates link19:38
harlowjalol19:38
klindgrenis simplejson c boosted?19:39
harlowjafrom what i remember yes19:39
harlowja'simplejson is a simple, fast, complete, correct and extensible JSON <http://json.org> encoder and decoder for Python 2.5+ and Python 3.3+. It is pure Python code with no dependencies, but includes an optional C extension for a serious speed boost.'19:40
harlowjasooo then the next question becomes are u using the optional C extension :-P19:40
harlowjaanyway u can see if your simplejson package has so files in it?19:40
harlowjalet me see if there is another way to tell if its uisng the boosted one19:41
harlowjaif u try19:42
harlowja>>> import simplejson19:42
harlowja>>> simplejson._import_c_make_encoder()19:42
harlowja>>>19:42
harlowjawhat does that result in19:42
harlowjaseems without the .so file it returns None vs <type 'simplejson._speedups.Encoder'> (when .so exists)19:42
harlowjaso might be worthwhile to check that19:43
SpamapSsimplejson is not preferred though19:43
SpamapSlook again at oslo.serialization19:43
SpamapSit only uses it _if you can't import json_19:43
harlowja* https://github.com/openstack/oslo.serialization/blob/master/oslo_serialization/jsonutils.py#L4119:43
klindgrenType "help", "copyright", "credits" or "license" for more information.19:43
klindgren>>> import simplejson19:43
klindgren>>> simplejson.import_c_make_encoder()19:43
klindgrenTraceback (most recent call last):19:43
klindgren  File "<stdin>", line 1, in <module>19:43
klindgrenAttributeError: 'module' object has no attribute 'import_c_make_enco19:43
SpamapSoh wait, no, ok, it actually does doe simplejson first19:43
SpamapSharlowja: we should add in ujson19:44
harlowjaklindgren  simplejson._import_c_make_encoder()19:44
harlowja^ note underscore19:44
klindgrenAttributeError: 'module' object has no attribute 'import_c_make_encoder'19:44
harlowjaya, its a hidden method i think, but seems to show u if it can load so19:44
klindgren>>> simplejson._import_c_make_encoder()19:44
klindgren<type 'simplejson._speedups.Encoder'>19:44
harlowjasooo ok, thats good i think19:44
harlowjaand klindgren 2.6 right?19:44
harlowja*py2.619:44
harlowja2.7 i thhink has native c json built-in19:44
SpamapSwait no19:44
SpamapS2.7's native C json is still slower than simplejson19:45
harlowja:-/19:45
harlowjaSpamapS hmmm19:45
SpamapShttps://gist.github.com/lightcatcher/113641519:45
SpamapSyeah that code is just wrong19:46
harlowjahmmm, SpamapS durn19:46
SpamapSwe should basically always try simplejson19:46
SpamapSand ujson19:46
harlowjaSpamapS https://github.com/openstack/oslo.serialization/blob/master/oslo_serialization/jsonutils.py#L266 also sucks imho19:46
harlowjabut thats a differnt story ^19:46
harlowja    anyjson._modules.append((__name__, 'dumps', TypeError,19:46
harlowja                                       'loads', ValueError, 'load'))19:46
harlowja    anyjson.force_implementation(__name__)19:46
harlowjai hate that so much19:46
harlowjalol19:46
SpamapSanyjson does not help19:46
harlowjamakes me feel firty19:46
SpamapSanyjson just tries a bunch of them but does not prefer faster ones19:47
harlowjawell not just that, anyone using anyjson after that call happens gets oslo.utils special implementation of json encoding/decoding :(19:47
harlowja*any library that is further imported after oslo.utils jsonutils ...19:47
harlowjaso ya, its shit imho19:47
SpamapSoh19:47
harlowjabut i tried fighting that one a while ago...19:47
harlowjai failed. lol19:47
SpamapSmonkeypatched anyjson, ossum19:47
harlowjaanyjson.force_implementation(__name__) ...19:47
harlowja^19:47
harlowjathat line19:47
harlowjaand what anyjson._modules.append does ...19:47
harlowja*and whatever19:47
harlowjaso ya, super stuff19:48
harlowja:(19:48
klindgrenfrom the pip freeze grepping for json19:48
klindgrenanyjson==0.3.319:48
klindgrenjsonpatch==1.219:48
klindgrenjsonpath-rw==1.2.319:48
klindgrenjsonpointer==1.019:48
klindgrenjsonschema==2.3.019:48
klindgrensimplejson==3.6.519:48
harlowjaklindgren py2.6?19:48
harlowjaor 2.6?19:48
harlowja*or 2.719:48
klindgrenits a cent7 box - py 2.719:48
harlowjak19:48
harlowjasoooo it might be interesting to tweak a few lines and always import simplejson19:48
harlowjabasically remove https://github.com/openstack/oslo.serialization/blob/master/oslo_serialization/jsonutils.py#L4119:48
harlowjaremove that line, always let simplejson get imported as json19:49
harlowjaand see how that affects your boxes...19:49
SpamapSharlowja: I'd go further. Lets get ujson is as the requirement19:49
SpamapSharlowja: and just skip all the madness.19:49
harlowjaSpamapS fair nuff, i have to look at ujson19:49
SpamapSimport ujson19:49
SpamapSdone19:49
harlowjahaven't heard of that one tiill u mentioned it :-P19:49
SpamapShttps://review.openstack.org/24594419:50
harlowjacool19:50
harlowjaklindgren if u feel adventours might be interesting to try that as well ;)19:50
harlowjaimport ujson as json (vs import simplejson as json)19:50
harlowja*destroy anyjson crap if u want19:50
harlowjai lost that battle, lol19:50
harlowjaSpamapS https://review.openstack.org/#/c/117844/19:51
harlowja^ the history of anyjson stuffs19:51
harlowja*my prior failed removal, ha19:51
* harlowja never was sure why oslo-incubator (before oslo.serilization) even tried to force itself into anyjson19:52
harlowjasuch weird historical things that nobody probabaly knows19:52
SpamapSpre-2.7 was ugly19:52
SpamapSanyjson should probably just go away19:52
harlowjadef19:52
harlowjai might try to revive https://review.openstack.org/#/c/117844/19:53
harlowjaidk19:53
harlowjamaybe SpamapS  u want to :-P19:53
harlowjai beat that horse enough ha19:53
harlowjaneeds new horse beater19:53
harlowjalol19:53
harlowjabb19:53
SpamapShrm19:54
SpamapSso ujson does not actually implement the whole json API19:55
SpamapSit doesn't have JSONEncoder or JSONDecoder19:55
SpamapSwhich, unfortunately, several things actually depend on19:55
*** sw3_ has joined #openstack-performance19:57
*** sw3 has quit IRC19:57
*** sw3_ is now known as sw319:58
SpamapSsince in the benchmarks simplejson comes in second, I'll just try with that19:58
SpamapStests seem to pass19:58
*** itsuugo has joined #openstack-performance19:59
SpamapShttps://review.openstack.org/24595720:04
SpamapSwe'll see how the tests do here20:04
SpamapSif they pass... I think thats a relatively easy performance win.20:05
SpamapSujson would still be better20:06
dimsharlowja : mriedem may remember details on why we did that20:07
*** jproulx has left #openstack-performance20:08
mriedemoh man, json vs simplejson, i do remember a perf reason...20:08
mriedemhttp://stackoverflow.com/questions/712791/what-are-the-differences-between-json-and-simplejson-python-modules20:09
dimsthanks mriedem :)20:10
mriedemC optimizations20:10
*** mwagner has quit IRC20:19
*** mwagner has joined #openstack-performance20:21
SpamapSindeed20:31
SpamapSin fact..20:31
SpamapSIt appears that simplejson3 is the one that is actually faster20:31
SpamapShttps://gist.github.com/lightcatcher/1136415 <-- try with various versions20:31
SpamapSoddly enough20:34
SpamapSit appears that they made _encoding_ faster, but _decoding_ slower20:34
SpamapSDOH20:34
SpamapSand it gets worse and worse20:35
SpamapS(the decoding)20:35
SpamapShttp://paste.openstack.org/show/479039/ <-- with 3.8.120:36
SpamapShttp://paste.openstack.org/show/479040/ <-- with 2.2.020:36
SpamapSnotice ujson _destroying_ everybody20:37
klindgrenSpamapS, curious what 3.6.5 version renders?20:38
klindgrensince thats the version I am running20:38
SpamapShttp://paste.openstack.org/show/479041/ <-- with 3.6.520:39
SpamapSklindgren: but you don't have simplejson20:39
SpamapSklindgren: you are always using built in json20:39
SpamapSwhich is _the worst_ for encoding, but 3rd best for decoding20:39
SpamapSNote that this is a rather large document20:39
SpamapSBigger than I'd expect openstack to be sending around.20:40
klindgrenlol - you never know :-)20:40
SpamapSI mean, probably at the upper bounds of what happens20:40
klindgrenif https://review.openstack.org/#/c/245957/1/oslo_serialization/jsonutils.py20:40
SpamapSbut I'm actually more curious about smaller docs20:40
SpamapSklindgren: oh are you on python 2.6??20:40
klindgrenis the change that needs to happen - I can locally modify that guy and give a restart20:40
klindgrenpython 2.720:41
SpamapSklindgren: perhaps wait until the integration tests on that don't explode? Just.. because ? ;)20:41
SpamapSSo maybe we need to make a frankenjson that uses simplejson for encoding, and built in json for decoding20:42
*** rpodolyaka1 has quit IRC20:47
*** rpodolyaka1 has joined #openstack-performance20:48
klindgrenI assume conductor is mainly doing encoding vs's decoding20:49
*** rpodolyaka1 has quit IRC21:02
*** rpodolyaka1 has joined #openstack-performance21:07
*** rpodolyaka1 has quit IRC21:09
SpamapSklindgren: would be both21:15
SpamapSklindgren: but yeah, more large docs encoded and sent to whoever is requesting them seems logical21:15
*** rpodolyaka1 has joined #openstack-performance21:30
*** markvoelker has quit IRC21:37
*** rpodolyaka1 has quit IRC22:00
* harlowja starts to wonder if we should re-think the chatty nature of conductor 22:45
harlowjaand/or use something like service discovery to determine if things should be sent to the conductor for conversion in the first place (instead of always sending stuff....)22:45
*** itsuugo has quit IRC22:45
harlowjaand/or inverse the workflow (so that conductor does most things locally, and tells nova-compute to act upon things it declares)22:45
*** regXboi has quit IRC22:47
harlowjaand/or a bunch of other ways to reduce chatty behavior22:47
harlowjacause eventually the gains by simplejson/ujson... will wear off22:48
harlowjaboris-42 and i talked about this years ago, i think the nova folks talked about it years ago also, klindgren and i talked about recently also ...22:48
harlowjamaybe we just need to create a realistic plan to do it...22:48
klindgrenyea - I would really like to see the clients move to being "dumb"  - where when a change happens it gets told what to do and given all the information it needs in format it understands.22:50
*** mwagner has quit IRC22:52
*** beisner has quit IRC22:54
*** med_ has quit IRC22:58
*** beisner has joined #openstack-performance22:59
*** mriedem is now known as mriedem_away23:01
*** med_ has joined #openstack-performance23:02
*** med_ is now known as Guest1205923:02
*** arnoldje has quit IRC23:06
*** arnoldje has joined #openstack-performance23:12
*** arnoldje has quit IRC23:19
« Sunday, 2015-11-15 Index Tuesday, 2015-11-17 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!