Friday, 2016-02-05

openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.log: Remove bandit.yaml in favor of defaults  https://review.openstack.org/27650800:05
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.messaging: Remove bandit.yaml in favor of defaults  https://review.openstack.org/27650700:11
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.service: Remove bandit.yaml in favor of defaults  https://review.openstack.org/27650900:13
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.messaging: Remove bandit.yaml in favor of defaults  https://review.openstack.org/27650700:14
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.utils: Remove bandit.yaml in favor of defaults  https://review.openstack.org/27651000:16
openstackgerritMerged openstack/oslo.utils: Updated from global requirements  https://review.openstack.org/27640200:33
openstackgerritMerged openstack/oslo.service: Updated from global requirements  https://review.openstack.org/27640100:35
*** mss-4 has quit IRC00:37
*** dims_ has joined #openstack-oslo00:39
openstackgerritMerged openstack/oslo.log: Updated from global requirements  https://review.openstack.org/27639900:40
*** dims has quit IRC00:40
*** sputnik13 has quit IRC00:41
openstackgerritMerged openstack/oslo.messaging: Updated from global requirements  https://review.openstack.org/27640000:42
*** salv-orl_ has quit IRC00:54
*** Jokke__ has joined #openstack-oslo01:06
*** Jokke__ has quit IRC01:06
*** mtanino has quit IRC01:34
openstackgerritMerged openstack/oslo.config: Updated from global requirements  https://review.openstack.org/27639802:04
*** edmondsw has quit IRC02:10
openstackgerritJoshua Harlow proposed openstack/taskflow: Work on fix for timeout of tasks (WIP)  https://review.openstack.org/27653602:16
openstackgerritMerged openstack/oslosphinx: Add version and download badges to README  https://review.openstack.org/27596502:17
*** davidlenwell has quit IRC02:18
*** davidlenwell has joined #openstack-oslo02:19
*** vilobhmm11 has quit IRC02:21
*** yamamoto_ has joined #openstack-oslo02:21
*** zz_dimtruck is now known as dimtruck02:22
*** davidlenwell has quit IRC02:23
*** davidlenwell has joined #openstack-oslo02:27
*** achanda has joined #openstack-oslo02:42
dims_gus : ping about oslo.privsep..is there a TODO list?02:44
*** browne has quit IRC03:00
openstackgerritMerged openstack/oslo.versionedobjects: Make sure functions in child object registries work  https://review.openstack.org/27011703:07
openstackgerritMerged openstack/oslo.privsep: Add version and download badges to README  https://review.openstack.org/27596403:09
*** dimtruck is now known as zz_dimtruck03:13
openstackgerritMerged openstack/oslo.messaging: [zmq] RPC timeout for CAST  https://review.openstack.org/27401303:14
*** eliqiao has quit IRC03:15
*** zz_dimtruck is now known as dimtruck03:21
*** dims_ has quit IRC03:33
*** browne has joined #openstack-oslo03:37
*** dimtruck is now known as zz_dimtruck03:37
*** browne has quit IRC03:38
*** yamahata has quit IRC03:44
openstackgerritMerged openstack/oslo.service: Fix test execution on CentOS 7  https://review.openstack.org/26533803:47
*** achanda has quit IRC03:59
*** vilobhmm11 has joined #openstack-oslo04:32
*** vilobhmm11 has quit IRC04:33
*** links has joined #openstack-oslo04:43
*** jeckersb is now known as jeckersb_gone04:53
*** achanda has joined #openstack-oslo04:58
*** vilobhmm11 has joined #openstack-oslo05:01
*** dkehn has quit IRC05:05
*** dkehn has joined #openstack-oslo05:05
*** salv-orlando has joined #openstack-oslo05:08
*** amotoki has joined #openstack-oslo05:11
*** salv-orlando has quit IRC05:18
openstackgerritMerged openstack/oslo.db: exception: fix InvalidSortKey message  https://review.openstack.org/27486705:44
openstackgerritMerged openstack/oslo.db: exceptions: enhance InvalidSortKey to carry the invalid key  https://review.openstack.org/27486805:44
*** yamamoto_ has quit IRC06:09
openstackgerritOpenStack Proposal Bot proposed openstack/oslo.utils: Imported Translations from Zanata  https://review.openstack.org/27502306:14
*** achanda has quit IRC06:19
*** nkrinner has joined #openstack-oslo06:19
*** achanda has joined #openstack-oslo06:20
openstackgerritMerged openstack/futurist: Allow PeriodicWorker to skip executor shutdown in case of a preexisting executor  https://review.openstack.org/27532606:38
openstackgerritJoshua Harlow proposed openstack/taskflow: Fix for WBE sporadic timeout of tasks  https://review.openstack.org/27653606:39
*** yamamoto_ has joined #openstack-oslo06:44
openstackgerritJoshua Harlow proposed openstack/taskflow: Fix for WBE sporadic timeout of tasks  https://review.openstack.org/27653606:48
*** links has quit IRC07:04
*** stevemar has joined #openstack-oslo07:06
*** links has joined #openstack-oslo07:14
openstackgerritDmitriy Ukhlov proposed openstack/oslo.messaging: Switches pika driver to eager connection to RabbitMQ  https://review.openstack.org/27425507:21
*** e0ne has joined #openstack-oslo07:30
openstackgerritMerged openstack/oslo.vmware: Remove bandit.yaml in favor of defaults  https://review.openstack.org/27647207:45
*** eliqiao has joined #openstack-oslo07:53
*** zakora has joined #openstack-oslo08:21
*** dtardivel has joined #openstack-oslo08:30
*** ihrachys has joined #openstack-oslo08:43
*** salv-orlando has joined #openstack-oslo08:45
*** salv-orlando has quit IRC08:49
*** ihrachys has quit IRC08:53
*** salv-orlando has joined #openstack-oslo09:00
*** EinstCrazy has quit IRC09:01
*** EinstCrazy has joined #openstack-oslo09:02
haypomorning09:04
*** EinstCrazy has quit IRC09:07
*** vilobhmm11 has quit IRC09:08
*** openstackgerrit has quit IRC09:17
*** openstackgerrit has joined #openstack-oslo09:17
*** yassine has joined #openstack-oslo09:17
*** lucas-dinner is now known as lucasagomes09:22
*** achanda has quit IRC09:31
*** ihrachys has joined #openstack-oslo09:33
*** ihrachys has quit IRC09:35
*** stevemar has quit IRC09:35
*** stevemar has joined #openstack-oslo09:36
*** mkoderer has joined #openstack-oslo09:36
*** mkoderer has left #openstack-oslo09:36
*** mkoderer has joined #openstack-oslo09:37
*** stevemar has quit IRC09:40
*** ihrachys has joined #openstack-oslo09:44
*** ihrachys has quit IRC09:51
*** mhickey has joined #openstack-oslo09:51
*** boris-42 has joined #openstack-oslo09:52
*** ihrachys has joined #openstack-oslo09:53
*** ihrachys has quit IRC10:02
*** ihrachys has joined #openstack-oslo10:05
*** ihrachys has quit IRC10:08
*** ihrachys has joined #openstack-oslo10:08
*** yamamoto_ has quit IRC10:15
*** yamamoto_ has joined #openstack-oslo10:17
*** EinstCrazy has joined #openstack-oslo10:22
*** ihrachys has quit IRC10:24
*** bnemec has joined #openstack-oslo10:25
*** ihrachys has joined #openstack-oslo10:25
*** ihrachys has quit IRC10:30
*** ihrachys has joined #openstack-oslo10:31
haypojd__: it looks like your forgot to fix DBInvalidUnicodeParameter https://review.openstack.org/#/c/274867/1/oslo_db/exception.py10:33
*** EinstCrazy has quit IRC10:35
*** ihrachys has quit IRC10:36
*** EinstCrazy has joined #openstack-oslo10:36
*** stevemar has joined #openstack-oslo10:37
*** salv-orl_ has joined #openstack-oslo10:39
*** stevemar has quit IRC10:40
*** EinstCrazy has quit IRC10:40
*** salv-orlando has quit IRC10:42
*** e0ne has quit IRC10:57
*** jeckersb_gone has quit IRC11:11
*** jokke_ has quit IRC11:12
*** toabctl_ has quit IRC11:12
*** jokke_ has joined #openstack-oslo11:13
*** jeckersb_gone has joined #openstack-oslo11:16
*** toabctl has joined #openstack-oslo11:17
*** yamamoto_ has quit IRC11:18
*** bnemec has quit IRC11:31
*** mss-4 has joined #openstack-oslo11:34
*** mss-4 has quit IRC11:38
*** dims has joined #openstack-oslo11:46
*** e0ne has joined #openstack-oslo11:55
*** salv-orl_ has quit IRC12:01
*** mkoderer__ has joined #openstack-oslo12:04
mkoderer__dims: I am thinking about a CLI tool for stevedore. IMHO it would be useful to list all installed plugins12:05
mkoderer__dims: what do you think?12:05
dimsmkoderer__ : sure, would be very handy12:05
mkoderer__dims: do you think it can live in stevedore directly? So this would add a cliff dependency12:06
dimsmkoderer__ : we could add it as an optional dependency so it's not installed always.12:12
mkoderer__dims: aha ok didn't know that this is possible. I will have a look12:13
dimsmkoderer : example http://git.openstack.org/cgit/openstack/glance_store/tree/setup.cfg#n5312:16
*** pkholkin has quit IRC12:18
*** yamamoto has joined #openstack-oslo12:19
*** yamamoto has quit IRC12:25
*** EinstCrazy has joined #openstack-oslo12:29
*** salv-orlando has joined #openstack-oslo12:29
*** gordc has joined #openstack-oslo12:33
*** stevemar has joined #openstack-oslo12:35
*** lucasagomes is now known as lucas-hungry12:35
*** EinstCrazy has quit IRC12:36
*** stevemar has quit IRC12:38
openstackgerritOleksii Zamiatin proposed openstack/oslo.messaging: WIP: [zmq] Test linger default value  https://review.openstack.org/27671012:43
*** links has quit IRC12:43
*** mtanino has joined #openstack-oslo12:50
*** jaypipes has quit IRC13:00
*** krotscheck has joined #openstack-oslo13:01
*** jaypipes has joined #openstack-oslo13:04
*** tcammann has joined #openstack-oslo13:12
*** salv-orlando has quit IRC13:13
*** yassine_ has joined #openstack-oslo13:18
*** tcammann has quit IRC13:19
*** tcammann has joined #openstack-oslo13:20
*** yassine has quit IRC13:20
jd__haypo: https://review.openstack.org/#/c/276253/113:21
*** boris-42 has quit IRC13:23
*** tcammann has quit IRC13:25
*** kgiusti has joined #openstack-oslo13:27
*** jeckersb_gone is now known as jeckersb13:31
*** pradk has joined #openstack-oslo13:32
*** edmondsw has joined #openstack-oslo13:34
*** takedakn has joined #openstack-oslo13:34
*** cdent has joined #openstack-oslo13:40
*** ihrachys has joined #openstack-oslo13:50
*** ihrachys has quit IRC13:54
haypojd__: haha, unit test on exception message. i like that13:55
*** takedakn has quit IRC14:00
*** lucas-hungry is now known as lucasagomes14:00
*** rlrossit has joined #openstack-oslo14:11
*** takedakn has joined #openstack-oslo14:12
*** salv-orlando has joined #openstack-oslo14:13
*** mkoderer__ has quit IRC14:16
*** mkoderer__ has joined #openstack-oslo14:17
*** yamamoto has joined #openstack-oslo14:24
*** nkrinner has quit IRC14:24
*** salv-orlando has quit IRC14:30
*** takedakn has quit IRC14:31
*** amotoki has quit IRC14:37
*** regXboi has joined #openstack-oslo14:47
openstackgerritOleksii Zamiatin proposed openstack/oslo.messaging: WIP: [zmq] Test linger default value  https://review.openstack.org/27671014:49
*** mriedem has joined #openstack-oslo14:57
*** yamamoto has quit IRC15:01
*** pradk has quit IRC15:03
*** pradk has joined #openstack-oslo15:03
*** sigmavirus24_awa is now known as sigmavirus2415:10
*** zz_dimtruck is now known as dimtruck15:12
*** amotoki has joined #openstack-oslo15:14
*** mhickey has quit IRC15:14
*** mhickey has joined #openstack-oslo15:15
*** nikhil_k is now known as nikhil15:17
*** yamamoto has joined #openstack-oslo15:29
*** yamamoto has quit IRC15:31
*** yamamoto has joined #openstack-oslo15:32
*** yamamoto has quit IRC15:37
*** bnemec has joined #openstack-oslo15:39
*** amotoki has quit IRC15:40
*** yamamoto has joined #openstack-oslo15:42
*** yamamoto has quit IRC15:44
*** david-lyle has joined #openstack-oslo15:47
openstackgerritIvan Kolodyazhny proposed openstack/oslo.reports: Use file modification events instead of signal handler  https://review.openstack.org/26097615:48
*** amotoki has joined #openstack-oslo15:54
*** mgarza has joined #openstack-oslo15:55
*** nkrinner has joined #openstack-oslo15:57
*** bnemec has quit IRC15:59
*** bnemec has joined #openstack-oslo16:00
*** cburgess_ has joined #openstack-oslo16:05
*** mhickey has quit IRC16:11
*** bnemec has quit IRC16:13
*** cburgess has quit IRC16:14
*** ozamiatin has quit IRC16:14
*** ozamiatin has joined #openstack-oslo16:17
*** david-lyle has quit IRC16:23
*** dimtruck is now known as zz_dimtruck16:23
*** yassine_ has quit IRC16:25
openstackgerritMerged openstack/oslo.db: exception: fix DBInvalidUnicodeParameter error message  https://review.openstack.org/27625316:26
*** mss-4 has joined #openstack-oslo16:27
*** zakora has quit IRC16:28
*** david-lyle has joined #openstack-oslo16:29
*** zz_dimtruck is now known as dimtruck16:33
*** harlowja_at_home has joined #openstack-oslo16:34
*** dimtruck is now known as zz_dimtruck16:35
*** e0ne has quit IRC16:37
*** shardy has joined #openstack-oslo16:43
*** yamamoto has joined #openstack-oslo16:45
sdaguedims: oslo.cache is .... quite verbose16:45
sdaguehttp://logs.openstack.org/54/276554/1/check/gate-tempest-dsvm-full/3de1525/logs/apache/keystone.txt.gz#_2016-02-05_04_50_21_88588016:45
dimssdague : ack we merged a oslo.log change to make it less chatty16:45
sdagueit looks like we're getting 20 - 30 debug lines out of oslo.cache for every keystone request16:45
sdaguedims: ok, but should those be there at all?16:45
dimslooking16:46
sdagueit seems like just turning up the log level isn't the right fix16:46
sdaguelocally I'm seeing this kind of spike apache load16:46
harlowja_at_homeother option is to switch some of those to TRACE and leave the log level as is...16:46
sdaguesure, though, I guess is it really useful now that the lib is released?16:47
sdaguecan we not just check for an aberant behavior and only log on that16:47
dimsbknudson_ : ^^16:47
harlowja_at_homei thought the other change was to make dogpiple also less chatty16:48
harlowja_at_homeso even if we make oslo.cache dogpile will be chatty still :-/16:48
openstackgerritJoshua Harlow proposed openstack/taskflow: Fix for WBE sporadic timeout of tasks  https://review.openstack.org/27653616:49
dimsharlowja_at_home : some of these are from here http://git.openstack.org/cgit/openstack/oslo.cache/tree/oslo_cache/_memcache_pool.py#n11216:49
harlowja_at_homeright16:50
dimsharlowja_at_home : guess we can switch those to trace instead of debug16:50
harlowja_at_homei get why they exist, helpfully when debugging16:50
harlowja_at_homebut maybe trace more appropriate?16:50
bknudson_caching was enabled in devstack recently.16:50
*** yamamoto has quit IRC16:51
bknudson_I agree these don't look useful16:51
*** e0ne has joined #openstack-oslo16:52
bknudson_oslo.log change: https://review.openstack.org/#/c/274510/16:54
bknudson_set oslo.cache and dogpile to INFO16:54
*** sputnik13 has joined #openstack-oslo17:00
*** sputnik13 has quit IRC17:02
*** stevemar has joined #openstack-oslo17:03
openstackgerritJoshua Harlow proposed openstack/oslo.cache: Move some of these noisy logs to trace level  https://review.openstack.org/27683517:09
harlowja_at_homebknudson_, dims sdague ^17:09
*** david-lyle has quit IRC17:14
*** achanda has joined #openstack-oslo17:15
*** harlowja_at_home has quit IRC17:17
*** stevemar has quit IRC17:17
*** stevemar has joined #openstack-oslo17:18
*** david-lyle has joined #openstack-oslo17:21
*** stevemar has quit IRC17:22
*** david-lyle has quit IRC17:25
*** lucasagomes is now known as lucas-dinner17:27
*** mriedem is now known as mriedem_afk17:31
dimsdhellmann : whoa! just got the packet, the oslo jacket is kick-ass! was expecting the t-shirt :)17:33
dimsdhellmann : thanks!17:33
dhellmannI thought you'd enjoy that, it seemed nice and warm. :-)17:34
dimsdhellmann : just in time, we got about 6-8 inches today :)17:36
dhellmann:-)17:37
*** sigmavirus24 is now known as sigmavirus24_awa17:39
*** amotoki has quit IRC17:45
*** zz_dimtruck is now known as dimtruck17:49
*** e0ne has quit IRC17:51
*** salv-orlando has joined #openstack-oslo17:55
sdagueharlowja: is there a reason these are actually useful messages outside of developing the library?17:57
sdaguethat's the part I'm not sure I understand, like why they don't just get deleted17:58
dimssdague : talking to memcached has been a big thorn in some of the production environments, so i'd support having traces on these18:01
sdaguedims: and it wedges in a way that no error condition is triggered18:02
sdague?18:02
sdagueI just want to challenge the notion that the right approach for finding production issues is log every step of the possitive path, vs. asserting when things don't look right, and logging the errors.18:04
sdagueespecially as this amount of output basically cripples the apache servers. I tripped over this because apache was running flat out on 3 cores just processing output18:06
*** achanda has quit IRC18:06
sdaguefor logs18:06
sdagueso turning on these traces is going to change the performance profile quite a lot18:06
*** dtardivel has quit IRC18:08
*** nikhil_k has joined #openstack-oslo18:11
*** nikhil has quit IRC18:14
*** dansmith is now known as superdan18:16
*** achanda has joined #openstack-oslo18:18
harlowjai imagine if there is a bug somewhere in the library and there is misunderstanding as to what's going on then well knowing when a connection is in the pool or not might be quite useful to know18:19
harlowjabut i do get that challenge, and the balance isn't always easy to find18:20
sdagueright, but if we go with that theory we're basically going to put line by line tracing into all our libraries, which seems excessive.18:22
harlowjaimho sdague  i do start to wonder if we need to rethink our logging guidelines (especially around DEBUG/TRACE) because it seems somewhat like we (openstack devs and users) are fighting an uphill battle in doing what u say18:22
sdagueharlowja: in what way?18:22
harlowjachanging the world to our ideas on DEBUG/TRACE likely will never happen, so perhaps we should reevaluate how the bigger world uses DEBUG/TRACE18:22
sdagueI mean, there are many uphill battles, so mostly I wanted to be specific about it18:22
harlowjaDEBUG/TRACE being used by most 3rd party libraries and all in different ways than i think we view it to be used18:23
harlowjajust a thought i was thinking on the way to work :-P18:23
sdagueharlowja: yes, 3rd party libraries are all over the map here18:23
harlowjaright18:23
sdaguebut it's not like they all have a different idea than us18:23
sdaguethey all have different ideas from each other as well18:24
harlowjaya, which makes me think its a larger python community issue/battle18:24
*** sputnik13 has joined #openstack-oslo18:25
sdagueok, that is a battle, all on it's own. And is a problem because oslo.log isn't a log engine, it's a setup tool for python logging18:25
*** stevemar has joined #openstack-oslo18:25
harlowjaright18:25
sdaguebut that's kind of different from the stuff we have under our control18:25
sdaguefor the stuff under our control, we should be confident that it does the right thing in the main path and not step log everything18:25
sdagueif we are not confident in that, we should add more tests18:26
sdagueand we should clearly log any exceptional behavior18:26
harlowjaya, i guess the weird part becomes when interacting with other systems, where a log message (at debug or trace) that shows main path will actually be useful in figuring out why the interaction is messed up18:27
harlowjabut maybe we should recommend more pdb for that18:27
harlowja(except ya, eventlet makes that hard)18:27
sdaguethat feels to me like the library contract wasn't defined18:27
sdagueif everything works down here, but goes nutters at a higher level18:27
sdagueI get that if there is a specific bug people are trying to sort out, adding stuff like this is fine in the short run18:28
sdaguebut it really shouldn't be part of normal main path code18:28
harlowjahow would u turn it on then if its not in the code?18:29
harlowja__debug__ ?18:29
harlowjaie, deleting it would mean person fixing bug would have to patch it back in18:29
harlowja* https://docs.python.org/2/library/constants.html#__debug__ (this could do it i guess)18:29
*** dims is now known as dimsum__18:30
harlowjaalthough idk if most people know about that, lol18:30
harlowjaits also what causes the assert statement to be a noop18:30
sdagueI'm not saying it's there to turn on. I'm saying that code only gets instrumented like this when chasing an active bug.18:30
harlowjaright, so maybe __debug__ then18:31
harlowjawhich is as close as i know to something like #ifdef debug, lol18:31
sdagueI think we're talking past each other18:32
harlowjamaybe18:32
harlowjalol18:32
sdagueI'm not saying put this behind __debug__18:32
sdagueI'm saying delete all this logging18:32
harlowjaright, which then someone who wants it has to put it all back in18:33
sdagueat some future date, if there is a registered bug people are working on, put in appropriate trace logging if required to solve that18:33
harlowjaand re-release lib?18:33
harlowjaor just temporarily?18:33
sdagueand once the bug is addressed, pull it out and go forward18:33
sdaguetemporarily18:33
*** lxsli_web has quit IRC18:34
sdaguein a release if you really need to18:34
sdaguebut don't make that the norm18:34
harlowjaya, i guess that depends on if people can actually change code that easy to mess with bugs18:34
harlowjasometimes u can't change code, but can change a log level, and changing the log level can help u figure out the bug18:35
harlowja*in an older version that is18:35
harlowjai know i can't go on a production hypervisor, and alter code, but can request logging levels to change temporarily if i suspect some memcache bug is happening18:36
harlowja(hypothetical situation)18:36
sdagueand kill your keystone in the process?18:36
harlowjahopefully one of X keystones goes down, where X is not all of them18:37
sdaguebecause that seems highly on the hypothetical based on the loads I was seeing here. It makes the assumption that flipping on this giant amount of verbose logging means you get the same code executing18:37
sdaguewhich you don't18:37
sdagueevery log message means a new eventlet greenlet context switch point, because it's IO18:38
harlowjayup18:38
sdagueso what you are saying is "I might have a bug. I'll turn on this option. Now I won't be running any of the same flows, but I'll debug those."18:38
sdaguewhich doesn't seem like a good approach18:39
harlowjadepends on the situation i think, it might be good enough18:39
sdagueeverything might be good enough.18:39
harlowjalol18:39
sdagueok, I'm kind of exhausted by this. But massively frustrated that the new model here seems to be step logging every function.18:40
harlowjatake a deep breath18:40
harlowjaeverything will be ok18:40
sdaguehonestly, it won't, we released a library that is supposed to increase performance that instead made it worse18:41
sdagueand the fix is tweak oslo.log, but hope that no one does custom python logging rules, because if so, they won't get that, and they'll take the perf hit18:41
dimsum__sdague : here's an example https://bugs.launchpad.net/mos/+bug/1516737 with keystone log paste - http://paste.openstack.org/show/479017/18:41
openstackLaunchpad bug 1516737 in Mirantis OpenStack "Keystone returns 500 deque index out of range" [High,Invalid] - Assigned to MOS Keystone (mos-keystone)18:41
dimsum__sdague : what's the perf hit when folks are supposed to be at DEBUG level and we are doing this at TRACE level?18:42
sdaguedimsum__: that should be covered by logging the exceptional cases18:42
dimsum__clearly we have no clue where the problem is coming from18:43
dimsum__if we did we would have :)18:43
*** browne has joined #openstack-oslo18:44
sdaguedimsum__: because I think that if we default to step trace logging, then people will just assume that's good enough and not think about having the code figure out it's in an aberant state, someone can just read 100 GB of logs, build and analysis engine, and handle it there18:45
sdaguelike I said, if we don't test the code, build test cases for it18:45
sdaguesorry, if we don't trust the code18:45
sdaguecatching this at a full stack level is going to take forever and be error prone18:45
sdaguesee: gate debugging18:45
dimsum__sdague : yep, problem is this is occuring under really heavy load :(18:45
sdaguedimsum__: right, which is *way* easier to trigger at the library than in a full stack18:46
sdaguestart up oslo.cache against a memcache18:46
dimsum__sdague : i wish :( been struggling with this for a long while18:47
sdaguefork 100 workers, beat the crap out of it18:47
*** sputnik13 has quit IRC18:47
dimsum__sdague : we haven't done this with oslo.cache yet18:47
dimsum__this was the older code in keystone18:47
dimsum__sdague : going to try some stress tests with keystone and nova both hitting memcached soon-ish18:48
sdaguecool, that should be a much better approach18:49
*** e0ne has joined #openstack-oslo18:50
*** shardy has quit IRC18:59
openstackgerritJoshua Harlow proposed openstack/taskflow: Don't bother scanning for workers if no new messages arrived  https://review.openstack.org/27687219:00
*** sputnik13 has joined #openstack-oslo19:03
dimsum__harlowja : here's tricky one since you like osprofiler :) https://bugs.launchpad.net/osprofiler/+bug/144002719:04
openstackLaunchpad bug 1440027 in osprofiler "osprofiler is Incompatibe with staticmethod within a class" [Undecided,New]19:04
harlowjaeck19:04
harlowjadon't use @profiler.trace_cls imho19:04
harlowjalol19:04
dimsum__apparently jobs are failing https://bugs.launchpad.net/cinder/+bug/154199619:05
openstackLaunchpad bug 1440027 in osprofiler "duplicate for #1541996 osprofiler is Incompatibe with staticmethod within a class" [Undecided,New]19:05
harlowjaDinaBelova was working on some of that stuff19:05
harlowjai wonder if her fix fixed up that19:05
dimsum__DinaBelova : ping19:05
harlowjahttps://review.openstack.org/#/c/269860/ dimsum__19:06
harlowjamaybe just needs a release?19:06
dimsum__we gotta talk to boris about +2A's19:06
DinaBelovaharlowja dimsum__ yep, this included to the release19:07
DinaBelova1.0.0.19:07
DinaBelovaof osprofiler19:07
harlowjaboris not around, lol19:07
harlowjaboris in hiding?19:07
DinaBelova@harlowja @dimsum__ https://review.openstack.org/#/c/269860/19:07
harlowjaright19:07
dimsum__DinaBelova : is that in the release?19:08
harlowjaDinaBelova https://bugs.launchpad.net/osprofiler/+bug/1440027/comments/2 :-/19:08
openstackLaunchpad bug 1440027 in osprofiler "osprofiler is Incompatibe with staticmethod within a class" [Undecided,New]19:08
harlowjamaybe still an issue?19:08
DinaBelovayes, it's included to 1.0.019:08
harlowja'I've submitted a requirements patch to prevent 1.0.0'19:08
dimsum__DinaBelova : please make sure reviews get 2 +2's19:08
harlowjaprevent there19:08
harlowjahttps://review.openstack.org/#/c/276761/ DinaBelova19:08
harlowja:-/19:08
DinaBelovavery strange19:09
DinaBelovaNova gates were failing without this commit19:09
DinaBelovaand now - nope19:09
harlowjaweird19:09
harlowjamakes me wonder if they are using some other version19:11
DinaBelovaor probably it's other issue19:12
DinaBelovathat looks similar19:12
harlowjamaybe19:13
*** sigmavirus24_awa is now known as sigmavirus2419:14
*** nkrinner has quit IRC19:14
dimsum__DinaBelova : osprofiler===1.0.0 http://openstack-logs.purestorage.com/19/275919/1/check/PureFCDriver-tempest-dsvm-volume-multipath/21e6a5e/logs/pip-freeze.txt.gz19:14
DinaBelovadimsum__ I see19:15
dimsum__line number matches too19:15
DinaBelovathat's really strange, nova gates are feeling ok with this osprofiler version19:17
dimsum__i know19:17
*** stevemar has quit IRC19:20
*** vilobhmm11 has joined #openstack-oslo19:20
*** stevemar has joined #openstack-oslo19:20
*** salv-orlando has quit IRC19:24
*** stevemar has quit IRC19:24
*** pradk has quit IRC19:27
*** bauzas is now known as bauwser19:27
harlowjaDinaBelova dimsum__ ok i see it19:28
harlowjau guys figure it out yet :-P19:28
dimsum__he is calling a static method with self?19:28
dimsum__he/she i mean19:28
harlowjanope19:28
harlowjalol19:28
dimsum__:)19:29
dimsum__that was a wild guess19:29
dimsum__what's the verdict? :)19:29
harlowjalol19:29
harlowjahttps://gist.github.com/harlowja/b3f9c920c3a6951128b519:29
harlowjahttps://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L24119:29
harlowja^ that isn't a class19:29
DinaBelovaif so error about args number will be called19:29
harlowjabut its applying the trace_cls decorator to a driver object....19:29
*** rlrossit has quit IRC19:29
harlowjagist causes same thing to happen19:29
harlowjaidk whats up with #L241 there19:30
harlowjalol19:30
harlowjabb19:30
DinaBelovaharlowja for this purpose there is this line https://github.com/openstack/osprofiler/blob/master/osprofiler/profiler.py#L15119:33
harlowjahmmm19:33
*** rlrossit_ has joined #openstack-oslo19:34
harlowjawhy people using trace_cls on things not classes, lol19:34
harlowjaweird19:34
dimsum__harlowja : like this? http://paste.openstack.org/show/486132/19:34
DinaBelovaeven if they are, it should not be an issue19:34
harlowjahttps://gist.github.com/harlowja/b3f9c920c3a6951128b5 should work then right?19:34
harlowja:-/19:34
harlowjafor me it does https://gist.github.com/harlowja/f43f8a14a0d0c501b7b319:35
harlowjaand boom19:35
*** stevemar has joined #openstack-oslo19:35
harlowjadimsum__ probably like yours also, if people have references to the wrong static method after @trace_cls (for a lot of reasons) then it will get werid19:35
harlowjastaticmethods are weird, because people can be referenceing them from all over19:37
DinaBelovaalso I cannot understand my experimental job with tempest did not catch it when we ran it against that patch19:37
dimsum__DinaBelova : because that does not exercise the PureFC driver19:37
harlowjalike in the module that the class is in, after defintion it could be making local variable reference to staticmethod, and then later @trace_cls changes it, but the module has reference to old one :-/19:37
dimsum__it's a 3rd party job19:37
DinaBelovadimsum__ ah, ok19:38
harlowja(that's also why i don't like tracecls, its not really connected into the python class creation process, and therefore gets weird)19:38
harlowjadimsum__ in your example http://paste.openstack.org/show/486132/19:38
harlowjaA.e() which e() is that calling :)19:39
dimsum__harlowja : i would vote to fail with better error message pointing them to fix their own broken code19:39
harlowjabecause in your paste, there is really 2 e()s19:39
*** krotscheck is now known as krotscheck_dcm19:39
*** vilobhmm111 has joined #openstack-oslo19:39
harlowjaone in the class defintion19:39
harlowjaand one new function created by trace_cls19:39
harlowja*wrapper function19:39
harlowjai don't like messing with python class stuff19:40
harlowjalol19:40
harlowja*except via normal metaclass processes, lol19:40
harlowja(and even that thing is hairy), lol19:40
*** vilobhmm11 has quit IRC19:40
dimsum__ok back to the problem at hand :)19:41
harlowjalol19:41
dimsum__how do we get them going DinaBelova :)19:41
dimsum__see what i did there harlowja :)19:41
harlowjaredirection ftw19:41
harlowjalol19:41
dimsum__yay :)19:41
harlowjau could be a web browser :-P19:42
harlowjahaha19:42
harlowjaif u ever needed a side-job19:42
DinaBelovadimsum__ I'm looking now, but the issue is I will need to go away from a laptop in 20 minutes19:42
dimsum__harlowja : easy thing would be to not decorate the static methods no?19:43
dimsum__DinaBelova ack19:43
harlowjathats one way19:43
harlowjaits still is weird19:43
harlowjabut might be acceptable19:44
harlowjamight be good to have more staticmethod and classmethod tests19:44
harlowjabunch of weirdo cases u can have19:44
dimsum__y, so we get DinaBelova and boris-42 past the revert and they can do that :)19:44
harlowjak19:45
DinaBelovadimsum__ probably the best way right now will be not to trace staticmethods... if we'll revert this stuff at all, nova change won't be able to land19:46
dimsum__right DinaBelova19:47
harlowjadimsum__ DinaBelova  https://gist.github.com/harlowja/0dc708a09e535632e6f819:47
harlowjabunch of weird cases there :-P19:47
harlowjato see when something is traced/not , https://gist.github.com/harlowja/09ea124f33e9d6b39585 (tiny little print patch)19:48
harlowjaactually, nm, a few less cases in there than i thought19:49
harlowjahttps://gist.github.com/harlowja/6dc4eb90bf150eaeca29 (some cases)19:50
dimsum__harlowja : this would do the trick? http://paste.openstack.org/show/486136/19:51
harlowjashould do it, unsure what else that changes, but sure :)19:51
harlowjamaybe wrapped_method is None, just incase some method for some reason has __bool__ or something on it (who knows)19:52
*** sigmavirus24 is now known as sigmavirus24_awa19:52
harlowjai think classmethods will be ok, but probably some stuff that might be off19:53
DinaBelovadimsum__ I even used http://paste.openstack.org/show/Cw0X6lNw5dbVcGhjC0s1/19:55
DinaBelovawithout any wrapping at all19:56
dimsum__DinaBelova we haven't see a problem with classmethod, so we should not skip that19:56
DinaBelovayeah, indeed19:56
dimsum__let's call continue with just the staticmethod19:56
DinaBelovaok19:56
DinaBelovaone moment sir19:57
harlowjaclass methods i think will be ok, although if modules reference class method before trace_cls() gets applied, they will bypass tracing19:57
dimsum__DinaBelova : one test has to be fixed "def test_static(self, mock_start, mock_stop)"19:57
DinaBelovayes, I know that19:57
dimsum__DinaBelova : harlowja means we have to doc the edge cases :)19:57
harlowjadef19:57
harlowjatranslator applied19:57
harlowjaha19:57
dimsum__:)19:58
harlowjasputnik13 also does that prettty well to :)19:58
harlowjalol19:58
dimsum__DinaBelova : so two of us are here, we can get that change in and you can test nova et. al19:58
dimsum__DinaBelova : then find boris-41 and open up a openstack/releases review please?19:59
DinaBelovadimsum__ I need to go very soon :( so do not know when I'll be able to do it...19:59
dimsum__sorry 4219:59
dimsum__ah ok19:59
DinaBelovaI'll upload staticmethod ignorance change in next few minutes20:00
harlowjafire the missles20:00
dimsum__DinaBelova : blast boris-42's phone to show up here too please? :)20:00
harlowjai can do that to, i think i got his number20:01
harlowjalol20:01
* harlowja if needed20:02
dimsum__worst case we miss the boat on Nova20:02
openstackgerritDina Belova proposed openstack/osprofiler: Disable staticmethods tracing  https://review.openstack.org/27689720:05
DinaBelovadimsum__ ^^20:05
dimsum__ack DinaBelova : will wait for CI20:05
DinaBelovaыure20:05
DinaBelovasure*20:05
harlowjarussian mode toggle20:05
harlowja:)20:05
DinaBelovadimsum__ harlowja - I quickly checked nova with this osprofiler modification against my dev env. Boot request seems to feel ok20:08
harlowjacool20:08
harlowjagood feelings are happy feelings20:08
harlowja(or some other quote), lol20:09
DinaBelovaharlowja dimsum__ http://dinabelova.github.io/nova-no-static-traces.html here it is20:10
harlowjacool20:10
harlowjaprobably not missing to much by not tracing staticmethods20:10
DinaBelovait's with the modified profiler and does not seem to have any differences (or at least visually)20:10
DinaBelovaindeed20:11
harlowjacool20:11
*** dimtruck is now known as zz_dimtruck20:14
*** zz_dimtruck is now known as dimtruck20:15
*** pballand has quit IRC20:20
*** salv-orlando has joined #openstack-oslo20:24
*** stevemar has quit IRC20:26
*** pballand has joined #openstack-oslo20:26
*** stevemar has joined #openstack-oslo20:26
sputnik13huh?20:34
dimsum__sputnik13 : there you are! :)20:36
sputnik13here I am20:36
sputnik13uhhh...  what did I do? :)20:36
dimsum__:) all's well sputnik1320:36
sputnik13whew20:37
sputnik13thought I was in trouble20:37
dimsum__well we are waiting for boris-42 to show up :) he's in trouble20:37
sputnik13uh oh20:37
*** david-lyle has joined #openstack-oslo20:39
harlowjalol20:41
*** achanda has quit IRC20:45
openstackgerritJoshua Harlow proposed openstack/taskflow: Remove need for separate notify thread  https://review.openstack.org/27691520:54
*** david-lyle has quit IRC20:55
*** david-lyle_ has joined #openstack-oslo20:55
*** achanda has joined #openstack-oslo20:59
*** david-lyle_ is now known as david-lyle21:00
*** achanda has quit IRC21:06
*** jeckersb is now known as jeckersb_gone21:08
*** gordc has quit IRC21:09
*** achanda has joined #openstack-oslo21:12
*** e0ne has quit IRC21:20
*** boris-42 has joined #openstack-oslo21:23
boris-42dims I approved that patch21:24
dimsum__boris-42 : can you please file a openstack/releases review?21:25
*** stevemar has quit IRC21:27
boris-42dimsum__: yep one second21:28
*** stevemar has joined #openstack-oslo21:28
*** kgiusti has left #openstack-oslo21:29
*** achanda has quit IRC21:31
*** stevemar_ has joined #openstack-oslo21:32
*** stevemar has quit IRC21:33
*** achanda has joined #openstack-oslo21:41
openstackgerritJoshua Harlow proposed openstack/taskflow: Remove need for separate notify thread  https://review.openstack.org/27691521:52
openstackgerritDoug Hellmann proposed openstack/oslo.config: refactor sphinxext to make reformatting easier  https://review.openstack.org/27692721:55
openstackgerritDoug Hellmann proposed openstack/oslo.config: support multiple namespaces in sphinxext  https://review.openstack.org/27692821:56
openstackgerritDoug Hellmann proposed openstack/oslo.config: have show-options load the generator config file  https://review.openstack.org/27692921:56
dhellmannrbradfor : ^^21:56
rbradfordhellmann, will take a look21:56
dhellmannrbradfor : I'm working on a patch to glance as an example, too21:57
rbradfordhellmann, fighting the PXE tftp daemons at present21:57
dhellmannrbradfor : god speed21:57
rbradfordhellmann, at whits end, I get get my PXE menu, but I'm then stuck,21:58
dhellmannrbradfor : https://review.openstack.org/27693122:03
*** dims has joined #openstack-oslo22:04
*** dimsum__ has quit IRC22:04
rbradfordhellmann, so here is a downside to that idea.22:05
rbradforyou now are effectively duplicating common options (aka the logging options) per page.  It's an unwinable situation either way, and I see the issues with the operating manuals where the same options are duplicated so many times.  in isolation, i.e. I want to look at one config file (now one page) I would see everything I need to know, we get exactly what we want22:06
dhellmannnormally one wouldn't want to duplicate the information because you would be afraid of missing an update if it changes. this is all automatically generated, though.22:07
rbradfordhellmann, that is true, it is generated, not humans22:08
*** cdent is now known as dentures22:10
*** jeckersb_gone is now known as jeckersb22:22
*** stevemar_ has quit IRC22:24
*** stevemar has joined #openstack-oslo22:25
dimsboris-42 : don't see it yet22:34
openstackgerritJoshua Harlow proposed openstack/taskflow: Some WBE protocol/executor cleanups  https://review.openstack.org/27694122:34
*** dentures has quit IRC22:39
*** salv-orl_ has joined #openstack-oslo22:40
openstackgerritJoshua Harlow proposed openstack/taskflow: Don't bother scanning for workers if no new messages arrived  https://review.openstack.org/27687222:41
openstackgerritJoshua Harlow proposed openstack/taskflow: Some WBE protocol/executor cleanups  https://review.openstack.org/27694122:41
openstackgerritJoshua Harlow proposed openstack/taskflow: Remove need for separate notify thread  https://review.openstack.org/27691522:41
openstackgerritJoshua Harlow proposed openstack/taskflow: Fix for WBE sporadic timeout of tasks  https://review.openstack.org/27653622:41
*** salv-orlando has quit IRC22:43
*** dims_ has joined #openstack-oslo23:06
*** rlrossit_ has quit IRC23:06
openstackgerritDoug Hellmann proposed openstack/oslo.config: always show coverage output from tests  https://review.openstack.org/27694623:08
openstackgerritDoug Hellmann proposed openstack/oslo.config: refactor sphinxext and add unit tests  https://review.openstack.org/27694723:08
*** dims has quit IRC23:10
*** edmondsw has quit IRC23:13
*** dimtruck is now known as zz_dimtruck23:14
openstackgerritJoshua Harlow proposed openstack/osprofiler: Disable staticmethods tracing  https://review.openstack.org/27689723:18
*** vilobhmm111 has quit IRC23:19
openstackgerritBoris Pavlovic proposed openstack/osprofiler: Disable staticmethods tracing  https://review.openstack.org/27689723:22
*** bauwser is now known as bauzas23:23
harlowjaboris-42 overwriting my change are u23:23
harlowjalol23:23
boris-42harlowja: ?23:23
boris-42harlowja: ahaha lol23:23
harlowja:)23:23
boris-42ok23:24
harlowjaok ok23:25
harlowjalol23:25
*** regXboi has quit IRC23:25
openstackgerritDoug Hellmann proposed openstack/oslo.log: always run coverage report  https://review.openstack.org/27695323:28
*** mgarza has quit IRC23:37
dims_boris-42 : it's still not right. the FIXME that dina added should be removed as you are skipping the test now23:59
boris-42dims_: =(23:59
boris-42dims_: ok23:59
openstackgerritBoris Pavlovic proposed openstack/osprofiler: Disable staticmethods tracing  https://review.openstack.org/27689723:59
boris-42dims_: ^ fixed23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!