Friday, 2015-04-17

« Thursday, 2015-04-16 Index Saturday, 2015-04-18 »
*** dims__ has joined #openstack-oslo00:04
*** dims_ has joined #openstack-oslo00:05
*** dims__ has quit IRC00:09
*** bknudson has joined #openstack-oslo00:12
*** sputnik13 has quit IRC00:13
*** sdake has joined #openstack-oslo00:20
*** sdake_ has quit IRC00:24
*** thingee has joined #openstack-oslo00:44
openstackgerritJoshua Harlow proposed openstack/taskflow: Add working eventlet py3.3 and py3.4 version (WIP)  https://review.openstack.org/17466500:44
*** achanda has quit IRC00:51
*** achanda has joined #openstack-oslo00:59
openstackgerritSachi King proposed openstack/oslo-specs: graduate-oslo-service  https://review.openstack.org/14265901:05
*** jecarey has joined #openstack-oslo01:25
openstackgerritTiantian Gao proposed openstack/oslo.messaging: Rabbit: Fix message ttl not work  https://review.openstack.org/17424001:36
openstackgerritMerged openstack-dev/pbr: Ignore non-release related tags  https://review.openstack.org/14214401:43
*** harlowja is now known as harlowja_away01:47
*** dims_ has quit IRC01:54
*** achanda has quit IRC02:20
*** achanda has joined #openstack-oslo02:22
*** liusheng has joined #openstack-oslo02:46
*** jecarey has quit IRC02:47
*** achanda has quit IRC02:49
*** sputnik13 has joined #openstack-oslo02:53
*** yamahata has quit IRC02:56
openstackgerritMerged openstack/cliff: Uncap library requirements for liberty  https://review.openstack.org/17448803:05
*** liusheng has quit IRC03:07
*** liusheng has joined #openstack-oslo03:07
*** crc32 has joined #openstack-oslo03:20
*** achanda has joined #openstack-oslo03:20
*** sdake_ has joined #openstack-oslo03:34
*** sdake has quit IRC03:38
*** achanda has quit IRC03:52
*** sputnik13 has quit IRC03:53
*** jungleboyj has joined #openstack-oslo04:04
*** achanda has joined #openstack-oslo04:07
*** achanda has quit IRC04:11
*** arnaud___ has joined #openstack-oslo04:15
*** thingee has quit IRC04:15
*** achanda has joined #openstack-oslo04:16
*** achanda has quit IRC04:18
*** achanda has joined #openstack-oslo04:21
*** achanda has quit IRC04:29
*** BrianShang_ has quit IRC04:38
*** crc32 has quit IRC04:38
*** crc32 has joined #openstack-oslo04:38
*** BrianShang has joined #openstack-oslo04:39
*** achanda has joined #openstack-oslo04:42
*** crc32 has quit IRC04:45
*** sputnik13 has joined #openstack-oslo05:09
*** sputnik13 has quit IRC05:11
*** crc32 has joined #openstack-oslo05:15
*** arnaud___ has quit IRC05:20
*** openstackgerrit has quit IRC05:21
*** openstackgerrit has joined #openstack-oslo05:21
*** stevemar has joined #openstack-oslo05:21
*** harlowja_at_home has joined #openstack-oslo05:31
*** stevemar has quit IRC05:32
*** stevemar has joined #openstack-oslo05:33
*** rushiagr_away is now known as rushiagr05:38
*** inc0 has joined #openstack-oslo05:42
*** stevemar has quit IRC06:12
*** harlowja_at_home has quit IRC06:18
*** sreshetnyak has joined #openstack-oslo06:30
*** bnemec has quit IRC06:32
openstackgerritMerged openstack/taskflow: Have reset state handlers go through a shared list  https://review.openstack.org/16738906:33
*** yamahata has joined #openstack-oslo06:38
*** Kennan2 has quit IRC06:41
*** Kennan has joined #openstack-oslo06:42
*** dulek has joined #openstack-oslo06:48
*** achanda has quit IRC06:53
*** achanda has joined #openstack-oslo06:56
*** sdake has joined #openstack-oslo07:06
*** e0ne has joined #openstack-oslo07:08
*** jaosorior has joined #openstack-oslo07:09
*** sdake_ has quit IRC07:09
*** amotoki has quit IRC07:18
*** haypo has joined #openstack-oslo07:18
*** svasheka has quit IRC07:21
*** e0ne has quit IRC07:23
*** e0ne has joined #openstack-oslo07:28
*** viktors has quit IRC07:28
*** achanda has quit IRC07:29
*** alexpilotti has joined #openstack-oslo07:30
*** andreykurilin__ has joined #openstack-oslo07:32
*** achanda_ has joined #openstack-oslo07:32
*** dims__ has joined #openstack-oslo07:33
haypohello07:34
*** yamahata has quit IRC07:34
*** dims__ has quit IRC07:38
*** achanda_ has quit IRC07:40
haypois it possible to add gates to projects on stackforge?07:42
haypoi'm working on sqlachemy-migrate and i don't see any gate at07:42
haypo<mriedem> so i guess i'm not sure where instance_extra is used later for formatting07:42
haypooops07:43
haypoat https://review.openstack.org/#/c/174735/07:43
haypo(copy/paste failure)07:43
*** ndipanov has quit IRC07:44
haypohum, i don't understand the complain about sqlachemy-migrate: "tox -e py34" pass07:47
*** BrianShang has quit IRC07:51
*** andreykurilin___ has joined #openstack-oslo07:53
*** andreykurilin__ has quit IRC07:53
*** BrianShang has joined #openstack-oslo07:53
*** e0ne has quit IRC07:57
*** e0ne has joined #openstack-oslo08:01
*** rushiagr is now known as rushiagr_away08:04
*** achanda has joined #openstack-oslo08:06
*** e0ne has quit IRC08:09
*** e0ne has joined #openstack-oslo08:14
*** e0ne has quit IRC08:18
*** achanda has quit IRC08:20
*** achanda_ has joined #openstack-oslo08:22
*** e0ne has joined #openstack-oslo08:22
*** crc32 has quit IRC08:23
*** sdake has quit IRC08:28
*** achanda_ has quit IRC08:35
*** ndipanov has joined #openstack-oslo08:40
*** e0ne has quit IRC08:42
*** andreykurilin___ has quit IRC08:43
*** e0ne has joined #openstack-oslo08:46
*** pcaruana has joined #openstack-oslo08:50
*** e0ne has quit IRC08:51
*** e0ne has joined #openstack-oslo08:53
*** e0ne has quit IRC09:00
*** arnaud___ has joined #openstack-oslo09:03
*** e0ne has joined #openstack-oslo09:04
haypo"is it possible to add gates to projects on stackforge?" nevermind, there are already gates :)09:06
*** e0ne has quit IRC09:09
*** rushiagr_away is now known as rushiagr09:10
*** ihrachyshka has joined #openstack-oslo09:16
*** sreshetnyak has quit IRC09:21
*** arnaud___ has quit IRC09:33
*** jamielennox is now known as jamielennox|away09:49
*** e0ne has joined #openstack-oslo09:50
*** inc0_ has joined #openstack-oslo09:57
*** inc0 has quit IRC09:59
*** cdent has joined #openstack-oslo10:01
*** bogdando has quit IRC10:07
*** ozamiatin has quit IRC10:08
*** sreshetnyak has joined #openstack-oslo10:08
*** enikanorov_ has quit IRC10:09
*** sputnik13 has joined #openstack-oslo10:12
*** cdent has quit IRC10:12
*** cdent has joined #openstack-oslo10:13
*** enikanorov_ has joined #openstack-oslo10:15
*** ozamiatin_ has joined #openstack-oslo10:15
*** sputnik13 has quit IRC10:15
*** bogdando has joined #openstack-oslo10:30
*** inc0_ has quit IRC10:31
*** inc0_ has joined #openstack-oslo10:31
*** e0ne is now known as e0ne_10:38
*** e0ne_ is now known as e0ne10:42
*** ozamiatin_ has quit IRC10:43
openstackgerritMerged openstack/oslotest: Uncap library requirements for liberty  https://review.openstack.org/17452010:57
*** dulek_ has joined #openstack-oslo10:59
*** inc0_ has quit IRC11:01
*** inc0 has joined #openstack-oslo11:01
*** dulek has quit IRC11:02
*** dims__ has joined #openstack-oslo11:02
*** inc0 has quit IRC11:04
*** inc0 has joined #openstack-oslo11:04
*** sreshetnyak has quit IRC11:09
*** inc0 has quit IRC11:11
*** inc0 has joined #openstack-oslo11:12
*** inc0 has quit IRC11:19
*** inc0_ has joined #openstack-oslo11:19
*** arnaud___ has joined #openstack-oslo11:22
*** inc0_ has quit IRC11:28
*** arnaud___ has quit IRC11:29
*** inc0 has joined #openstack-oslo11:33
*** prad has quit IRC11:36
*** dulek_ has quit IRC11:46
*** dulek has joined #openstack-oslo11:49
*** inc0 has quit IRC11:51
*** prad has joined #openstack-oslo11:51
*** inc0 has joined #openstack-oslo11:51
*** prad has quit IRC11:54
*** cdent has quit IRC11:55
sdaguehmmm... with a fresh devstack, oslo.messaging 1.8.1 I'm getting this heartbeat error over and over and over again11:55
*** e0ne is now known as e0ne_11:58
sdague2015-04-17 07:58:50.030 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds.11:58
*** inc0 has quit IRC12:00
*** cdent has joined #openstack-oslo12:01
sdaguesileht: you about? because something has happened, and now 0 services work because they go into heartbeat loops and fail12:03
*** ozamiatin_ has joined #openstack-oslo12:06
silehtsdague, you are the second person to report that12:06
sdaguesileht: is there already a bug?12:06
silehta bug report have been opened12:06
sdaguesileht: ok, yeh this seems to be happening 100% of the time on my new devstack12:07
silehtsdague, https://bugs.launchpad.net/oslo.messaging/+bug/143676912:07
openstackLaunchpad bug 1436769 in oslo.messaging "ConnectionError: Too many heartbeats missed" [Undecided,Confirmed]12:07
sdagueso in my case, this is basically happening every second on every API service12:07
sdagueand effectively 0 messages are getting across the rpc bus12:08
sdaguepossibly because this is storming it12:08
*** e0ne_ has quit IRC12:08
silehtsdague, ubuntu or fedora ?12:09
sdagueubuntu12:09
sdague14.0412:09
*** e0ne has joined #openstack-oslo12:10
* sileht tries to find a difference with its devstack12:10
silehtsdague, this occurs just after the stack have been started or you have started to boot or something12:11
sdaguesileht: I was trying to figure out why the stack was taking so long12:11
sdaguelots of commands were hanging12:11
sdagueand it turns out this was why, all the api services were in permaloops trying to heartbeat/fail with rabbit12:12
sdaguerabbit looks up, and was burned down to the ground before I started12:12
*** bknudson has quit IRC12:15
sdagueI'm trying to force downgrade to 1.8.0 and see if that helps12:15
silehtsdague, that should help the latest heartbeat patch that enable it come in 1.8.112:18
silehtsdague, or you can disable it by using a timeout of 012:18
*** kgiusti has joined #openstack-oslo12:19
*** dulek has quit IRC12:19
sdagueok, so... https://bugs.launchpad.net/oslo.messaging/+bug/1436769 isn't actually my bug, but I see what's happening there. There was another use of the n-cert service that was not anticipated during devstack start.12:19
openstackLaunchpad bug 1436769 in oslo.messaging "ConnectionError: Too many heartbeats missed" [Undecided,Confirmed]12:19
*** dulek has joined #openstack-oslo12:22
sdagueso, I don't have a narrow reproduce case for the heartbeat loop12:24
sdaguebut it looks like if it gets into a loop, you're done12:24
sdagueit just starts storming12:25
sdagueand not reconnecting correctly12:25
*** sreshetnyak has joined #openstack-oslo12:25
*** arnaud___ has joined #openstack-oslo12:26
sdagueoh, never mind https://bugs.launchpad.net/oslo.messaging/+bug/143676 is my bug, there is a different one on the mailing list that I'm fixing now though12:26
openstackLaunchpad bug 143676 in Zope 2 "RestrictedPython sensitive to line endings in Python 2.4" [Medium,Fix released] - Assigned to ChrisW (chris-simplistix)12:26
sdaguehttps://bugs.launchpad.net/oslo.messaging/+bug/143676912:26
openstackLaunchpad bug 1436769 in oslo.messaging "ConnectionError: Too many heartbeats missed" [Undecided,Confirmed]12:26
sdaguestupid truncation of paste :)12:26
sdagueit looks like quite a few people are hitting this in various ways12:27
sdaguemaybe heartbeats should be off by default until the reason why is figured out?12:28
openstackgerritMerged openstack/oslo.log: Uncap library requirements for liberty  https://review.openstack.org/17451012:28
openstackgerritMerged openstack/oslo.context: Uncap library requirements for liberty  https://review.openstack.org/17450212:30
*** arnaud___ has quit IRC12:30
silehtsdague, Perhaps we should do that, the most annoying bug I have discovered is: https://bugs.launchpad.net/oslo.messaging/+bug/143678812:30
openstackLaunchpad bug 1436788 in oslo.messaging "rabbitmq heartbeat failures don't reset connections" [Undecided,Triaged]12:31
openstackgerritMerged openstack/oslo.policy: Uncap library requirements for liberty  https://review.openstack.org/17451512:31
silehtsdague, this is a socket timeout that kombu doesn't allow us to set12:31
sdagueright, there was something at the ops summit about the fact that pika is the prefered library from the rabbit folks12:31
silehtsdague, yeah pika allow to set all timeouts12:32
silehtsdague, I have not yet taken the times to discus with kombu folks, I have just opened the bug for now12:32
openstackgerritMerged openstack/oslo.messaging: Uncap library requirements for liberty  https://review.openstack.org/17451212:33
openstackgerritMerged openstack/oslo.i18n: Uncap library requirements for liberty  https://review.openstack.org/17450912:33
sdagueyeh, 1.8.1 is reproducing this for me 100% of the time now12:34
*** cdent has quit IRC12:34
openstackgerritMerged openstack/oslo.utils: Uncap library requirements for liberty  https://review.openstack.org/17452112:34
sdagueI've got a system looping like this atm, anything useful I can dump out of it for you?12:34
sdagueto help narrow the bug12:34
openstackgerritMerged openstack/oslo.db: Uncap library requirements for liberty  https://review.openstack.org/17450412:35
*** cdent has joined #openstack-oslo12:35
openstackgerritMerged openstack/oslo.versionedobjects: Uncap library requirements for liberty  https://review.openstack.org/17452212:35
*** bknudson has joined #openstack-oslo12:36
silehtsdague, you run devstack in kvm/virtual/.... ?12:36
sdaguethis is on real hardware12:36
silehtsdague, ok12:36
openstackgerritMerged openstack/oslo.middleware: Uncap library requirements for liberty  https://review.openstack.org/17451312:36
silehtsdague, perhaps a pip freeze output to ensure I have the same deps to start12:36
silehtI just recreated a devstack and everything works :(12:37
sdagueyeh, I think it's not a 100% fail, but I think if you find yourself in this loop for timing reasons, then you are done12:37
sdagueand never can recover12:38
*** jaypipes has joined #openstack-oslo12:38
sdaguesileht: http://paste.openstack.org/show/204454/ is my pip freeze12:39
silehtsdague, thanks12:40
openstackgerritMerged openstack/debtcollector: Uncap library requirements for liberty  https://review.openstack.org/17449112:41
*** cdent has quit IRC12:41
sdaguegetting another cup of coffee, back in 5. Let me know if there is anything else that would be useful to dump here to get to the bottom of this, happy to spend the next hour extracting things from this environment before I force back to 1.8.0 to verify a nova fix12:41
*** cdent has joined #openstack-oslo12:41
openstackgerritMerged openstack/oslo.config: Uncap library requirements for liberty  https://review.openstack.org/17449412:43
silehtsdague, I have found the issue: https://github.com/celery/py-amqp/issues/612:43
silehtsdague, I have amqp 1.4 and you 1.312:43
*** gordc has joined #openstack-oslo12:44
*** stpierre has joined #openstack-oslo12:44
silehtsdague, so disable heartbeat ? or raise requirement ?12:45
sdaguesileht: let me see if forcing that up fixes it here12:47
sdagueok, nova-conductor remains in the loop after I kill and start, so I should be able to try an inline upgrade and see if it recovers12:48
sdagueforcing in 1.4.0 seems to have made it recover12:50
sdagueI'd do another ./clean.sh && ./stack.sh12:50
sdaguebut it seems like lifting up that requirement is probably in order then12:51
sdagueas that seems needed for the heartbeat code to be reliable12:51
sdagueI don't know, what's your opinion of the risk of finding other issues here?12:52
*** sreshetnyak has quit IRC12:52
silehtsdague, heartbeat or not, the really complains that if something bad occurs on the communication between oslo.msg and rabbit, we never want to wait the the system tcp timeout to discover it12:55
silehtsdague, many people think that implementing heartbeat was fixing the issue12:56
silehtsdague, but that not true, to really fix that we have to be able the set timeout on socket12:56
silehtsdague, and THEN use heartbeat to use very long timeout12:56
sdagueok, that's fair12:57
sdagueso what's your opinion on the safest option for the kilo release to keep regressions to a minimum?12:57
silehtsdague, but kombu doesn't allow us to do that on the socket used for write12:57
sdagueshould we bump the requirement?12:57
sdagueor turn off hearbeats by default?12:57
silehtperhaps turn off hearbeats because the read timeout is still forced to a maximun of 1sec to ensure we detect connection lost on read12:58
sdaguesileht: ok, so that would be a 1.8.2 with the default behavior changed?12:59
silehton write, heartbeat or not, we can still have issue, if the packet that fail to transfert is the heartbeat one12:59
*** inc0 has joined #openstack-oslo12:59
silehtsdague, but on the other side ops really really want this feature13:00
silehtsdague, even it's not yet perfect yet13:00
sdaguewell, they can turn it back on13:00
silehttrue13:00
sdaguecall it experimental because there are known issues, which is why it defaults off13:00
sdaguebut explain how it can be enabled if people want to try it13:00
silehtsdague, that soon reasonable13:01
silehtsdague, I will proposed a patch and see other core dev opinion13:01
silehts/soon/sound13:01
sdaguesileht: ok, great, thank you13:02
silehtsdague, thanks for your help and testing13:03
sdagueno problem, happy to help13:04
*** sreshetnyak has joined #openstack-oslo13:08
*** cdent has quit IRC13:23
*** rushiagr is now known as rushiagr_away13:23
*** jungleboyj has quit IRC13:25
*** ozamiatin_ has quit IRC13:25
*** mriedem has joined #openstack-oslo13:26
*** jecarey has joined #openstack-oslo13:32
*** dulek has quit IRC13:33
sdaguesileht: ok, just to final confirm, after upgrading to amqp 1.4.0 ... ./clean.sh && ./stack.sh, and I'm not seeing the heartbeating errors in my environment any more. So it might be worth honestly doing *both* things, both turning off the heartbeating by default, and raising the requirement, so that if you turn it on it's more likely to work13:34
*** prad has joined #openstack-oslo13:35
silehtsdague, of course the requirement need to be raised at least on master13:35
*** dansmith is now known as superdan13:35
silehtls13:35
silehtoups13:35
sdagueI'd raise it on stable/kilo as well so that 1.8.2 could have it raised13:36
*** pblaho_ has quit IRC13:37
silehtsdague, I'm thinking of something else, amqp is not a oslo.messaging requirement but a kombu one13:38
silehtsdague, that ok to add requirement just to fix bug ?13:38
sdaguesileht: yeh, I think so13:39
sdaguewe should try to rope ttx into this conversation13:39
sdaguebecause this impacts the whole release, and I'd like his opinions13:40
*** pblaho has joined #openstack-oslo13:41
*** dulek has joined #openstack-oslo13:42
*** sigmavirus24_awa is now known as sigmavirus2413:45
*** inc0 has quit IRC13:47
*** pblaho has quit IRC13:49
*** pblaho has joined #openstack-oslo13:51
*** jaypipes is now known as leakypipes13:59
openstackgerritIhar Hrachyshka proposed openstack/oslo.config: Enforce alphabetic loading order for --config-dir contents  https://review.openstack.org/17488314:02
*** bnemec has joined #openstack-oslo14:03
*** mriedem has quit IRC14:03
*** mriedem has joined #openstack-oslo14:06
*** rushiagr_away is now known as rushiagr14:08
*** cdent has joined #openstack-oslo14:09
*** e0ne is now known as e0ne_14:11
*** sdake has joined #openstack-oslo14:12
*** e0ne_ is now known as e0ne14:13
dims__sileht: ping me please when you file the oslo.messaging and requirements reviews?14:15
*** arnaud___ has joined #openstack-oslo14:15
dims__sileht: let me know if you want me to do those14:17
*** inc0 has joined #openstack-oslo14:19
*** arnaud___ has quit IRC14:20
*** ozamiatin has joined #openstack-oslo14:24
*** sreshetnyak has quit IRC14:24
*** alexpilotti has quit IRC14:27
*** sreshetnyak has joined #openstack-oslo14:29
*** dims__ has quit IRC14:33
*** sreshetnyak has quit IRC14:33
*** zz_jgrimm is now known as jgrimm14:34
*** stevemar has joined #openstack-oslo14:37
*** sreshetnyak has joined #openstack-oslo14:38
*** mtanino has joined #openstack-oslo14:41
*** sdake_ has joined #openstack-oslo14:47
*** sdake has quit IRC14:50
*** sreshetnyak has quit IRC14:51
openstackgerritMerged openstack/oslo.concurrency: Uncap library requirements for liberty  https://review.openstack.org/17448614:51
openstackgerritMerged openstack/oslo.rootwrap: Uncap library requirements for liberty  https://review.openstack.org/17451614:52
*** sreshetnyak has joined #openstack-oslo14:55
*** prad has quit IRC14:57
*** prad has joined #openstack-oslo15:01
*** yamahata has joined #openstack-oslo15:01
*** dims__ has joined #openstack-oslo15:04
*** zzzeek has joined #openstack-oslo15:08
*** inc0 has quit IRC15:19
*** harlowja_at_home has joined #openstack-oslo15:19
*** dims__ has quit IRC15:22
harlowja_at_homehaypo, yt15:22
harlowja_at_homehave u seen any kind of weird issue in 3.4 where not even using eventlet, but importing it, causes thread locks to not work?15:23
harlowja_at_homeone of the taskflow examples with 3.4 + eventlet (although not even using eventlet) triggers it15:24
harlowja_at_homehttp://logs.openstack.org/65/174665/1/check/gate-taskflow-python34/70bf225/console.html mainly15:24
*** ozamiatin has quit IRC15:26
harlowja_at_home*its weird cause that example doesn't even use eventlet ...15:26
haypoharlowja_at_home, yes, i saw a lot of bugs with threading, but they *should* be fixed in eventlet 0.17.315:28
harlowja_at_homehmmm15:28
harlowja_at_homethats the version i was using15:28
haypoharlowja_at_home, do you have a same script to reproduce your issue?15:28
harlowja_at_homenot yet, gonna try to get one there15:29
*** sreshetnyak has quit IRC15:29
harlowja_at_homehaypo, an example out from that example http://paste.openstack.org/show/204479/15:30
harlowja_at_homeu'll notice https://github.com/openstack/taskflow/blob/master/taskflow/examples/jobboard_produce_consume_colors.py#L128 and https://github.com/openstack/taskflow/blob/master/taskflow/examples/jobboard_produce_consume_colors.py#L149 occur15:30
haypoharlowja_at_home, i spent a lot of time trying to understand _wait_for_tstate_lock() especially how to handle it in eventlet15:31
*** arnaud___ has joined #openstack-oslo15:31
harlowja_at_home:-/15:31
harlowja_at_homein that output u'll see15:31
harlowja_at_homeP-1: started15:31
harlowja_at_homeP-2: started15:31
harlowja_at_homeP-3: started15:31
harlowja_at_homeW-1: started15:31
harlowja_at_homeW-2: started15:31
harlowja_at_homeW-3: started15:31
harlowja_at_homeW-4: started15:31
harlowja_at_homeW-5: started15:31
haypoharlowja_at_home, _wait_for_tstate_lock() is a new lock introduced in python 3.4 to fix a corner case bug15:31
haypoharlowja_at_home, the lock is released by a C function which is not called in eventlet, since eventlet uses green threads not native threads15:31
harlowja_at_homeand correspnding finished 'output' for all of those, but they still don't correctly join -/15:31
haypoharlowja_at_home, my fix is to hack each Thread instance to release this lock15:32
harlowja_at_homehttps://github.com/openstack/taskflow/blob/master/taskflow/examples/jobboard_produce_consume_colors.py#L164 (the join part)15:32
harlowja_at_homehmmm15:32
harlowja_at_homehaypo, the weird part is that the example doesn't even use eventlet (although it might get imported but not used)15:32
harlowja_at_homeso if i pip uninstall eventlet, problem goes away15:33
harlowja_at_homeotherwise its ok15:33
harlowja_at_home*otherwise its not ok i mean15:33
haypoharlowja_at_home, be careful, your eyes will bleed https://github.com/eventlet/eventlet/blob/master/eventlet/green/thread.py#L4615:33
haypoharlowja_at_home, you may check that you enter in the hack (that thread._bootstrap_inner is patched)15:33
harlowja_at_homewould it be patched even if eventlet isn't getting used?15:34
haypoharlowja_at_home, call eventlet.is_monkey_patched('thread') to check if monkey-patch is enabled15:34
harlowja_at_homechecking15:34
haypoharlowja_at_home, or put a breaking in eventlet.patcher.monkey_patch()15:34
*** arnaud___ has quit IRC15:35
harlowja_at_homeso nope, doesn't seem patched15:36
harlowja_at_homeweird stuff, lol15:37
haypoharlowja_at_home, weird, yes15:37
haypoharlowja_at_home, maybe it's a real bug in your code :)15:37
haypoharlowja_at_home, as i said, python 3.4 changed thread.join()15:38
harlowja_at_homei'm not so sure about that, seeing as what gets printed out to stdout showing my code is finishing, but then everything is locking up around the join() call15:38
harlowja_at_homemaybe join is broken, lol15:39
haypoharlowja_at_home, but if you only get the bug when eventlet is installed, it's more likely an eventlet bug15:39
harlowja_at_homeya, but that's weird, what in eventlet has those kind of side-effects, when things aren't even monkey patching themself in15:40
harlowja_at_homepretty easy to try it also; checkout taskflow, pip install the existing requirements, install eventlet, then run15:42
harlowja_at_home$ python taskflow/examples/jobboard_produce_consume_colors.py15:42
harlowja_at_homereliably doesn't ever join15:42
harlowja_at_homeremove eventlet, then reliably does join...15:42
harlowja_at_home^ weird15:42
*** joesavak has joined #openstack-oslo15:43
openstackgerritMehdi Abaakouk proposed openstack/oslo.messaging: Disable and mark heartbeat as experimental  https://review.openstack.org/17492915:43
openstackgerritMerged openstack/tooz: Uncap library requirements for liberty  https://review.openstack.org/17454915:45
openstackgerritMerged openstack/stevedore: Uncap library requirements for liberty  https://review.openstack.org/17454715:45
*** sreshetnyak has joined #openstack-oslo15:47
*** enikanorov__ has joined #openstack-oslo15:48
openstackgerritMerged openstack/oslo.serialization: Uncap library requirements for liberty  https://review.openstack.org/17451815:50
openstackgerritMerged openstack/taskflow: Uncap library requirements for liberty  https://review.openstack.org/17454815:50
*** tsekiyama has joined #openstack-oslo15:51
*** enikanorov_ has quit IRC15:51
openstackgerritMehdi Abaakouk proposed openstack/oslo.messaging: Disable and mark heartbeat as experimental  https://review.openstack.org/17492915:51
openstackgerritMerged openstack/oslo.vmware: Uncap library requirements for liberty  https://review.openstack.org/17452315:52
openstackgerritMehdi Abaakouk proposed openstack/oslo.messaging: Disable and mark heartbeat as experimental  https://review.openstack.org/17492915:54
*** jsavak has joined #openstack-oslo15:55
*** browne has joined #openstack-oslo15:55
*** dulek has quit IRC15:58
*** joesavak has quit IRC15:59
*** kbyrne has quit IRC16:00
*** arnaud___ has joined #openstack-oslo16:04
*** yamahata has quit IRC16:08
*** harlowja_at_home has quit IRC16:09
*** pcaruana has quit IRC16:10
*** e0ne has quit IRC16:12
*** salv-orl_ has quit IRC16:19
*** mriedem1 has joined #openstack-oslo16:19
*** subscope has joined #openstack-oslo16:20
*** mriedem has quit IRC16:21
*** haypo has quit IRC16:23
*** salv-orlando has joined #openstack-oslo16:24
*** dims__ has joined #openstack-oslo16:24
*** yamahata has joined #openstack-oslo16:31
*** arnaud___ has quit IRC16:34
*** sdake has joined #openstack-oslo16:35
*** mdbooth has quit IRC16:37
*** sdake_ has quit IRC16:39
*** mdbooth has joined #openstack-oslo16:44
*** dims__ is now known as dimsum__16:50
*** pblaho has quit IRC16:54
*** ihrachyshka has quit IRC17:00
*** sreshetnyak has quit IRC17:00
*** prad has quit IRC17:02
*** sreshetnyak has joined #openstack-oslo17:02
*** prad has joined #openstack-oslo17:05
*** salv-orlando has quit IRC17:06
*** prad has quit IRC17:06
*** sreshetnyak has quit IRC17:09
*** prad has joined #openstack-oslo17:09
*** sreshetnyak has joined #openstack-oslo17:11
*** russellb has quit IRC17:14
*** harlowja_away is now known as harlowja17:17
*** russellb has joined #openstack-oslo17:17
*** sreshetnyak has quit IRC17:22
*** inc0 has joined #openstack-oslo17:24
*** russellb has quit IRC17:25
*** inc0 has quit IRC17:25
*** russellb has joined #openstack-oslo17:29
*** salv-orlando has joined #openstack-oslo17:29
*** arnaud___ has joined #openstack-oslo17:35
*** e0ne has joined #openstack-oslo17:37
*** mriedem1 is now known as mriedem17:38
*** arnaud___ has quit IRC17:40
*** prad has quit IRC17:45
*** e0ne is now known as e0ne_17:45
*** prad has joined #openstack-oslo17:46
*** achanda has joined #openstack-oslo17:48
*** e0ne_ is now known as e0ne17:52
*** jaosorior has quit IRC17:52
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.utils: Allow deprecated decorator to specify no plan for removal  https://review.openstack.org/17046117:55
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.utils: Add Kilo release name to versionutils  https://review.openstack.org/17046217:55
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.utils: Enhance versionutils.deprecated to work with classes  https://review.openstack.org/17046317:55
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.utils: Remove code that moved to oslo.i18n  https://review.openstack.org/17046417:56
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.utils: Remove oslo.log code and clean up versionutils API  https://review.openstack.org/17046517:56
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.utils: Switch from oslo.config to oslo_config  https://review.openstack.org/17046617:56
*** AAzza_afk has joined #openstack-oslo17:56
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.utils: Expose opts entry point for version_utils  https://review.openstack.org/17046717:56
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.utils: Add liberty release name to versionutils  https://review.openstack.org/17046817:57
*** e0ne is now known as e0ne_17:57
*** AAzzal has joined #openstack-oslo17:58
*** AAzza has quit IRC17:59
*** AAzzal is now known as AAzza17:59
*** AAzza_afk has quit IRC18:02
openstackgerritMerged openstack/oslo-incubator: Revert "Revert "Optimization of waiting subprocesses in ProcessLauncher""  https://review.openstack.org/17436518:07
harlowjarevert revert revert revert of revert18:08
harlowjalol18:08
*** leakypipes has quit IRC18:25
*** rushiagr is now known as rushiagr_away18:26
*** andreykurilin__ has joined #openstack-oslo18:29
*** jungleboyj has joined #openstack-oslo18:29
*** e0ne_ is now known as e0ne18:30
*** jungleboyj has quit IRC18:36
*** cdent has quit IRC18:36
harlowjaanyone around with a 3.4 environment18:39
harlowjawanna see if this locks up for u18:39
*** achanda has quit IRC18:46
*** jungleboyj has joined #openstack-oslo18:51
harlowjadimsum__ can u try https://github.com/eventlet/eventlet/issues/23018:52
harlowjaif u get some time18:52
harlowjaits something that imho we're gonna hit more :-P18:52
harlowjajust want to make sure that freezes up for others :-P18:52
harlowja^ nothing special ...18:52
*** achanda has joined #openstack-oslo18:56
*** salv-orlando has quit IRC18:57
*** e0ne has quit IRC18:58
*** jungleboyj has quit IRC19:00
openstackgerritMerged openstack/oslo.serialization: Expose base msgpack exceptions so users don't need to import  https://review.openstack.org/17455319:04
openstackgerritMerged openstack/oslo.serialization: Add docstring(s) to handler registry(s)  https://review.openstack.org/17308919:10
openstackgerritMerged openstack/oslo.serialization: More docstring cleanups/tweaks  https://review.openstack.org/17309519:10
*** jungleboyj has joined #openstack-oslo19:13
bnemecharlowja: Also seeing that hang here.19:18
harlowjadurn19:19
harlowjasomeone here on a mac tried it, didn't see it19:19
harlowjasuck19:19
harlowjasomething weird19:19
*** jungleboyj has quit IRC19:19
bnemecI'm on f21, with eventlet installed in a 3.4 venv.19:19
harlowjakk19:19
harlowjai'll note that in PR (trying on a few systems)19:20
bnemecMaybe worth noting that I did not see the hang in a py2 venv.19:20
dimsum__harlowja: y give me a few19:20
harlowjaya19:20
harlowjabnemec something weird with the import machinery that eventlet maybe(?) is messing with19:20
harlowjaeven though eventlet isn't being used anywhere...19:21
bnemecharlowja: I wouldn't even want to speculate. :-)19:21
harlowjahaha19:21
harlowjaya19:21
harlowjaaccept eventlet into your heart bnemec19:21
harlowjalol19:21
bnemecI've accepted that sometimes I just don't understand eventlet. ;-)19:22
*** bnemec is now known as beekneemech19:22
* beekneemech notices that it's Friday19:22
harlowjaha19:23
*** arnaud___ has joined #openstack-oslo19:24
harlowjaalright, added note/comment to PR about systems that have seen this19:27
harlowjalet's see what happens, no idea whats going on there, eventlet is weird (and I totally didn't expect weird import time side-effects)19:27
*** ajo has quit IRC19:28
beekneemechAnd here we thought all the hidden landmines were in the monkey patching.19:28
*** ajo has joined #openstack-oslo19:28
*** ajo has quit IRC19:28
harlowja:-/19:28
dimsum__harlowja: so Ubuntu 14.04.2 LTS + python3.4 i can see the same issue19:28
dimsum__trying on mac now19:28
harlowjadimsum__ thx, tried it on my 14.04 box i have laying around also19:28
*** arnaud___ has quit IRC19:29
harlowjaweird stuff, ha19:29
dimsum__harlowja: i can see it on mac as well19:30
dimsum__Python 3.4.319:30
harlowjakk, can u add that to PR, coworker here said he didn't see it on his mac (oddly)19:30
harlowjaguess i could try it on my mac, ha19:30
*** yamahata has quit IRC19:31
harlowjathe only maybe weird thing is the following19:33
harlowjaimporting eventlet imports eventlet.convenience which imports eventlet.green.socket which imports eventlet.support.greendns which starts to monkey around with things...19:34
harlowjabut not sure19:34
harlowjahttps://github.com/eventlet/eventlet/blob/master/eventlet/green/socket.py#L9 (whatever all this stuff is...)19:34
harlowjabut who knows, could be something else, lol19:34
* harlowja was hoping eventlet didn't patch things until u asked it to, lol19:36
*** stevemar has quit IRC19:40
*** stevemar2 has joined #openstack-oslo19:40
*** subscope has quit IRC19:40
*** ozialien has joined #openstack-oslo19:41
openstackgerritMerged openstack/oslo.utils: Allow deprecated decorator to specify no plan for removal  https://review.openstack.org/17046119:44
*** salv-orlando has joined #openstack-oslo19:49
*** achanda has quit IRC19:50
dimsum__harlowja: fix this line19:52
dimsum__harlowja: https://github.com/eventlet/eventlet/blob/master/eventlet/hubs/__init__.py#L919:52
dimsum__harlowja: instead of patcher.original('threading') do patcher.original('threadingxxxxxxxx')19:52
dimsum__and it works fine :)19:52
dimsum__so the original method is doing something funny19:53
dimsum__if already_patched in patcher.py is empty, it should just bail out with sys.modules.get(modname)19:55
dimsum__if you ask me19:55
openstackgerritMerged openstack/oslo.utils: Add Kilo release name to versionutils  https://review.openstack.org/17046220:03
*** ozamiatin has joined #openstack-oslo20:08
*** ozamiatin has quit IRC20:09
openstackgerritMerged openstack/oslo.utils: Enhance versionutils.deprecated to work with classes  https://review.openstack.org/17046320:11
openstackgerritMerged openstack/oslo.utils: Remove code that moved to oslo.i18n  https://review.openstack.org/17046420:12
openstackgerritMerged openstack/oslo.utils: Remove oslo.log code and clean up versionutils API  https://review.openstack.org/17046520:12
*** ndipanov has quit IRC20:13
*** stevemar2 is now known as stevemar20:15
openstackgerritMerged openstack/oslo.utils: Switch from oslo.config to oslo_config  https://review.openstack.org/17046620:18
openstackgerritMerged openstack/oslo.utils: Expose opts entry point for version_utils  https://review.openstack.org/17046720:18
openstackgerritMerged openstack/oslo.utils: Add liberty release name to versionutils  https://review.openstack.org/17046820:18
*** kgiusti has left #openstack-oslo20:21
*** openstackgerrit has quit IRC20:22
*** openstackgerrit has joined #openstack-oslo20:22
harlowjadimsum__ weird20:24
dimsum__saw my diff?20:24
harlowjaya, seems to make sense20:25
*** achanda has joined #openstack-oslo20:27
*** andreykurilin___ has joined #openstack-oslo20:28
*** andreykurilin__ has quit IRC20:29
harlowjalet's see what eventlet devs think i guess20:29
*** kgiusti has joined #openstack-oslo20:31
*** kgiusti has left #openstack-oslo20:39
openstackgerritVictor Stinner proposed openstack/oslo.concurrency: Add binary parameter to execute and ssh_execute  https://review.openstack.org/17171020:43
openstackgerritVictor Stinner proposed openstack/oslo.concurrency: Port processutils to Python 3  https://review.openstack.org/14698420:43
*** stpierre has quit IRC20:44
*** sdake_ has joined #openstack-oslo20:56
openstackgerritJoshua Harlow proposed openstack/tooz: Use oslo.serialization msgpackutils  https://review.openstack.org/17305820:59
*** sdake has quit IRC20:59
-openstackstatus- NOTICE: Gerrit will be unavailable between 22:00 and 23:59 UTC for project renames and a database update.21:05
*** mriedem has quit IRC21:07
openstackgerritJoshua Harlow proposed openstack/taskflow: Add ability to use tooz to find and advertise workers  https://review.openstack.org/15149521:10
*** arnaud___ has joined #openstack-oslo21:13
*** ozialien has quit IRC21:16
*** arnaud___ has quit IRC21:17
*** shardy has quit IRC21:20
*** openstackgerrit has quit IRC21:23
*** openstackgerrit has joined #openstack-oslo21:23
*** sdake has joined #openstack-oslo21:25
*** stevemar has quit IRC21:29
*** sdake_ has quit IRC21:29
*** ozialien has joined #openstack-oslo21:37
*** bknudson has quit IRC21:44
*** jamielennox|away is now known as jamielennox21:48
*** gordc has quit IRC21:54
openstackgerritVictor Stinner proposed openstack/oslo.concurrency: Add binary parameter to execute and ssh_execute  https://review.openstack.org/17171022:02
openstackgerritVictor Stinner proposed openstack/oslo.concurrency: Port processutils to Python 3  https://review.openstack.org/14698422:02
*** jsavak has quit IRC22:02
*** jecarey has quit IRC22:04
-openstackstatus- NOTICE: Gerrit is unavailable until 23:59 UTC for project renames and a database update.22:05
-openstackstatus- NOTICE: Gerrit is unavailable until 23:59 UTC for project renames and a database update.22:08
*** ChanServ changes topic to "Gerrit is unavailable until 23:59 UTC for project renames and a database update."22:08
*** sigmavirus24 is now known as sigmavirus24_awa22:13
*** andreykurilin___ has quit IRC22:13
*** jgrimm is now known as zz_jgrimm22:20
*** sdake_ has joined #openstack-oslo22:24
*** sdake has quit IRC22:27
*** sdake has joined #openstack-oslo22:46
*** sdake_ has quit IRC22:50
openstackgerritVictor Stinner proposed openstack/oslo.concurrency: Add binary parameter to execute and ssh_execute  https://review.openstack.org/17171022:51
openstackgerritVictor Stinner proposed openstack/oslo.concurrency: Port processutils to Python 3  https://review.openstack.org/14698422:51
*** achanda has quit IRC22:54
*** achanda has joined #openstack-oslo22:56
*** arnaud___ has joined #openstack-oslo23:02
*** ChanServ changes topic to "#openstack-oslo"23:05
-openstackstatus- NOTICE: Gerrit is available again.23:05
*** arnaud___ has quit IRC23:06
*** achanda has quit IRC23:11
*** sdake_ has joined #openstack-oslo23:14
*** achanda has joined #openstack-oslo23:15
*** sdake has quit IRC23:17
*** zzzeek has quit IRC23:23
*** jamielennox is now known as jamielennox|away23:30
*** sdake has joined #openstack-oslo23:32
*** zzzeek has joined #openstack-oslo23:33
*** sdake_ has quit IRC23:36
*** flaper87 has quit IRC23:38
*** zzzeek has quit IRC23:55
« Thursday, 2015-04-16 Index Saturday, 2015-04-18 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!