| openstackgerrit | Takashi NATSUME proposed openstack/oslo.config master: Fix string interpolation in ValueError https://review.openstack.org/458502 | 00:07 |
|---|---|---|
| *** dkehn has quit IRC | 00:12 | |
| *** salv-orlando has quit IRC | 00:22 | |
| *** jerrygb has joined #openstack-oslo | 00:45 | |
| *** jerrygb has quit IRC | 00:52 | |
| *** dimtruck is now known as zz_dimtruck | 00:56 | |
| *** dkehn has joined #openstack-oslo | 00:57 | |
| *** amrith has quit IRC | 01:04 | |
| *** jerrygb has joined #openstack-oslo | 01:04 | |
| *** zz_dimtruck is now known as dimtruck | 01:14 | |
| *** eck`gone is now known as eck` | 01:18 | |
| *** salv-orlando has joined #openstack-oslo | 01:19 | |
| *** dimtruck is now known as zz_dimtruck | 01:22 | |
| *** gcb has joined #openstack-oslo | 01:31 | |
| *** zhangguoqing has joined #openstack-oslo | 01:33 | |
| *** zhangguoqing has quit IRC | 01:38 | |
| *** zhangguoqing has joined #openstack-oslo | 01:39 | |
| *** salv-orlando has quit IRC | 01:48 | |
| *** zz_dimtruck is now known as dimtruck | 01:50 | |
| *** amrith has joined #openstack-oslo | 01:58 | |
| *** amrith has quit IRC | 02:00 | |
| *** yamahata has quit IRC | 02:03 | |
| *** amrith has joined #openstack-oslo | 02:04 | |
| *** sdague has quit IRC | 02:16 | |
| *** gcb has quit IRC | 02:31 | |
| *** gcb has joined #openstack-oslo | 02:44 | |
| *** gcb has quit IRC | 02:44 | |
| *** salv-orlando has joined #openstack-oslo | 02:45 | |
| *** amotoki has joined #openstack-oslo | 02:58 | |
| *** salv-orl_ has joined #openstack-oslo | 03:03 | |
| *** salv-orlando has quit IRC | 03:03 | |
| *** ihrachys has quit IRC | 03:03 | |
| *** amotoki has quit IRC | 03:03 | |
| *** zhangguoqing has quit IRC | 03:03 | |
| *** zhangguo_ has joined #openstack-oslo | 03:04 | |
| *** ihrachys has joined #openstack-oslo | 03:04 | |
| *** david-lyle has joined #openstack-oslo | 03:11 | |
| *** salv-orl_ has quit IRC | 03:16 | |
| *** david-lyle has quit IRC | 03:19 | |
| *** nicolasbock has quit IRC | 03:28 | |
| *** dave-mcc_ has quit IRC | 04:02 | |
| *** amotoki has joined #openstack-oslo | 04:09 | |
| *** salv-orlando has joined #openstack-oslo | 04:13 | |
| *** zhangguo_ has quit IRC | 04:24 | |
| *** yamahata has joined #openstack-oslo | 04:24 | |
| *** salv-orlando has quit IRC | 04:41 | |
| *** shyama has joined #openstack-oslo | 04:45 | |
| *** salv-orlando has joined #openstack-oslo | 04:46 | |
| *** Rockyg has joined #openstack-oslo | 05:04 | |
| *** zhangguoqing has joined #openstack-oslo | 05:06 | |
| *** dimtruck is now known as zz_dimtruck | 05:09 | |
| *** gcb has joined #openstack-oslo | 05:18 | |
| *** zhangguoqing has quit IRC | 05:42 | |
| *** lpetrut has joined #openstack-oslo | 05:47 | |
| *** Rockyg has quit IRC | 05:48 | |
| *** jbadiapa has joined #openstack-oslo | 05:59 | |
| *** Rockyg has joined #openstack-oslo | 06:00 | |
| *** jaosorior has joined #openstack-oslo | 06:06 | |
| *** zhangguoqing has joined #openstack-oslo | 06:09 | |
| openstackgerrit | Merged openstack/oslo.log master: Add oslo_messaging to the list of log levels https://review.openstack.org/458069 | 06:11 |
| *** salv-orlando has quit IRC | 06:11 | |
| *** salv-orlando has joined #openstack-oslo | 06:11 | |
| *** zhangguoqing has quit IRC | 06:13 | |
| *** rcernin has joined #openstack-oslo | 06:15 | |
| *** salv-orlando has quit IRC | 06:16 | |
| *** pcaruana has joined #openstack-oslo | 06:18 | |
| *** hoonetorg has quit IRC | 06:19 | |
| *** hoonetorg has joined #openstack-oslo | 06:32 | |
| *** lpetrut has quit IRC | 06:37 | |
| *** gcb has quit IRC | 06:42 | |
| *** e0ne has joined #openstack-oslo | 06:47 | |
| *** e0ne has quit IRC | 06:54 | |
| *** gcb has joined #openstack-oslo | 06:56 | |
| *** salv-orlando has joined #openstack-oslo | 07:01 | |
| *** tesseract has joined #openstack-oslo | 07:11 | |
| *** Rockyg has quit IRC | 07:13 | |
| *** shardy has joined #openstack-oslo | 07:24 | |
| *** e0ne has joined #openstack-oslo | 07:37 | |
| openstackgerrit | Merged openstack-dev/pbr master: trivial: Add note about multiple builders support https://review.openstack.org/439545 | 07:46 |
| *** shyama has quit IRC | 07:57 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-oslo | 08:00 | |
| *** lucas-afk is now known as lucasagomes | 08:07 | |
| *** jerrygb_ has joined #openstack-oslo | 08:11 | |
| *** jerrygb has quit IRC | 08:11 | |
| *** rcernin has quit IRC | 08:13 | |
| *** jerrygb_ has quit IRC | 08:21 | |
| *** jerrygb has joined #openstack-oslo | 08:22 | |
| *** mhickey has joined #openstack-oslo | 08:22 | |
| *** mhickey has quit IRC | 08:26 | |
| *** mhickey has joined #openstack-oslo | 08:26 | |
| *** rcernin has joined #openstack-oslo | 08:27 | |
| *** kani__ has joined #openstack-oslo | 08:28 | |
| kani__ | Hi, I have created RPC server(using oslo_messaging) in my service to process the rpc cast request. Once it receives the request, the processing takes minimum of 4 minutes to complete. But in the meantime the rabbitmq-server closes the connection and reconnecting to the client. Error logged in Rabbitmq-server log : =ERROR REPORT==== 17-Apr-2017::16:25:26 === closing AMQP connection <0.28633.6> (192.18.17.3:54762 -> 192.18.17.1 | 08:29 |
| therve | sileht, Should I open a bug in oslo.messaging for that log weirdness? | 08:32 |
| sileht | therve, most of the times this is because time or thread is not monkeypatched so heartbeat thread can't do heartbeat and connection is closed | 08:33 |
| therve | sileht, Hum... Are you sure about that? | 08:33 |
| *** shardy has quit IRC | 08:34 | |
| kani__ | Error logged in my service log: 2017-04-13 16:26:05 ERROR [oslo.messaging._drivers.impl_rabbit] impl_rabbit:impl_rabbit.py:676 AMQP server myamqp-m2-clm:5672 closed the connection. Check login credentials: Socket closed | 08:34 |
| sileht | therve, if the connection is closed by rabbit you have more detail on the reason, do you have the complete rabbit log ? so I can confirm | 08:34 |
| kani__ | Error logged in Rabbitmq-server log : =ERROR REPORT==== 17-Apr-2017::16:25:26 === closing AMQP connection <0.28633.6> (192.18.17.3:54762 -> 192.18.17.10:5672): missed heartbeats from client, timeout: 60s | 08:35 |
| *** shardy has joined #openstack-oslo | 08:35 | |
| therve | sileht, Are you talking to kani__ ? :) | 08:35 |
| sileht | therve, not you ? | 08:35 |
| therve | I was talking about https://review.openstack.org/458069 myself | 08:35 |
| sileht | oh I mix both :) | 08:36 |
| kani__ | :) | 08:36 |
| kani__ | Hi sileht, is this error message from rabbitmq log enough? Or do u want something more? | 08:37 |
| sileht | kani__, something hold the heartbeat thread, usualy this is because the application does not monkeypatched time and/or threading correctly | 08:37 |
| sileht | therve, it's already merged | 08:38 |
| sileht | therve, if you want to backport it you need a bug, otherwise I don't care | 08:38 |
| kani__ | sileht, do u have any example of how to do it? | 08:38 |
| therve | sileht, Sure, but it's mostly a workaround in oslo.log. Shouldn't it be fixed in oslo.messaging? | 08:38 |
| sileht | therve, I don't see the issue | 08:40 |
| therve | sileht, Well it logs with 2 different namespaces | 08:40 |
| sileht | therve, all logs should come from oslo_messaging since a while now | 08:40 |
| therve | sileht, Yeah it should, but it doesn't :) | 08:40 |
| *** shyama has joined #openstack-oslo | 08:42 | |
| therve | sileht, http://logs.openstack.org/86/405586/21/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/daeb997/logs/screen-ir-cond.txt.gz#_2017-04-20_17_36_01_349 for example | 08:42 |
| therve | I think it's because of stevedore or something | 08:42 |
| sileht | therve, interresting, I think you can open a bug then | 08:42 |
| therve | OK will do | 08:43 |
| sileht | therve, it's wierd the logger use __name__ | 08:44 |
| *** cdent has joined #openstack-oslo | 08:44 | |
| therve | sileht, Isn't that overridable? | 08:47 |
| sileht | therve, I don't find where we do that, maybe a stevedore or pbr magic | 08:49 |
| therve | Yeah that's what I think too | 08:50 |
| *** e0ne has quit IRC | 08:54 | |
| *** amotoki has quit IRC | 08:57 | |
| *** aarefiev_afk is now known as aarefiev | 09:02 | |
| *** e0ne has joined #openstack-oslo | 09:07 | |
| *** amotoki has joined #openstack-oslo | 09:08 | |
| *** sambetts|afk is now known as sambetts | 09:08 | |
| *** geguileo has left #openstack-oslo | 09:11 | |
| *** e0ne has quit IRC | 09:23 | |
| therve | sileht, Ah: https://github.com/openstack/oslo.log/blob/master/oslo_log/log.py#L435 | 09:24 |
| sileht | therve, so if we use oslo_log we have oslo.msg, if we use logging we have oslo_msg | 09:25 |
| therve | Yeah | 09:26 |
| therve | sileht, Probably should only use one of them though? | 09:26 |
| sileht | therve, only a couple of place use oslo_log, and lib you rely on logging I think | 09:26 |
| therve | Yeah a bunch of drivers, not even all | 09:27 |
| sileht | therve, I remember we remove oslo_log from oslo messaging, but it comes back | 09:28 |
| sileht | therve, oslo_log have been reintroduced to be able to use LOG.trace | 09:30 |
| *** amotoki has quit IRC | 09:30 | |
| therve | Ah ok | 09:31 |
| *** amotoki has joined #openstack-oslo | 09:33 | |
| kani__ | Hi sileht, I have added log messages in _heartbeat_thread_job method, I could see those messages in my log. Does this mean the heartbeat is not blocked? | 09:39 |
| sileht | kani__, no | 09:40 |
| sileht | kani__, the method have to called very often, if it can't because of something block/freeze the application, you're disconnected from rabbit | 09:41 |
| sileht | kani__, did you call eventlet.monkeypatch() before any modules is loaded in your application ? | 09:42 |
| *** salv-orl_ has joined #openstack-oslo | 09:48 | |
| *** salv-orlando has quit IRC | 09:51 | |
| kani__ | sileht, yes monkeypatch is called: eventlet.monkeypatch(all=True) | 09:56 |
| sileht | kani__, before anything have been loaded like in yourapp/__init__.py ? | 09:56 |
| kani__ | its in our api/_init__.py | 09:58 |
| kani__ | sileht, this is in our script to start the service: | 10:02 |
| kani__ | import eventlet if os.name == 'nt': # eventlet monkey patching the os module causes subprocess.Popen to fail # on Windows when using pipes due to missing non-blocking IO support. eventlet.monkey_patch(os=False) else: eventlet.monkey_patch() | 10:02 |
| *** Guest86170 has quit IRC | 10:02 | |
| sileht | kani__, what the app ? | 10:03 |
| sileht | cinder ? | 10:04 |
| kani__ | this app helps for configuration in cinder | 10:04 |
| kani__ | To configure block drivers in cinder | 10:05 |
| *** amotoki has quit IRC | 10:05 | |
| *** e0ne has joined #openstack-oslo | 10:07 | |
| kani__ | sileht: eventlet.monkey_patch() - does this block/ hold the heartbeat thread? But every 15 seconds I could see messages from _heartbeat_thread_job method | 10:10 |
| sileht | kani__, you have to find what in you app is blocking the application more than 60s | 10:12 |
| sileht | is/what is | 10:12 |
| *** Guest86170 has joined #openstack-oslo | 10:14 | |
| *** nicolasbock has joined #openstack-oslo | 10:15 | |
| kani__ | sileht, We have spawned one rpc_server which takes minimum of 210seconds to complete the request. During that time only the rabbitmq-server closes the connection with this RPC server due to missing heartbeats | 10:15 |
| sileht | kani__, during this 210 seconds, if IO occurs eventlet should switch to the heartbeat thread, but it looks like in your case it doesn't, maybe you are using library that not compatible with eventlet | 10:17 |
| sileht | kani__, if no IO occurs (100% cpu processing), you have to call eventlet.sleep() or doing you cpu intensive task into another real thread/process | 10:18 |
| kani__ | sileht, every 15 seconds I could see messages from _heartbeat_thread_job method. How is it possible? | 10:21 |
| sileht | kani__, oslo.messaging opens many connections maybe it works for some of them, but not all | 10:22 |
| *** yamamoto has quit IRC | 10:25 | |
| *** cdent has quit IRC | 10:45 | |
| *** yamahata has quit IRC | 11:00 | |
| *** cdent has joined #openstack-oslo | 11:16 | |
| *** lucasagomes is now known as lucas-afk | 11:25 | |
| *** shardy is now known as shardy_lunch | 11:32 | |
| *** yamamoto has joined #openstack-oslo | 11:33 | |
| *** yamamoto has quit IRC | 11:35 | |
| *** yamamoto has joined #openstack-oslo | 11:39 | |
| *** salv-orl_ has quit IRC | 11:40 | |
| *** yamamoto has quit IRC | 11:52 | |
| *** sdague has joined #openstack-oslo | 11:54 | |
| *** lpetrut has joined #openstack-oslo | 11:55 | |
| *** jaosorior is now known as jaosorior_away | 12:12 | |
| *** salv-orlando has joined #openstack-oslo | 12:17 | |
| *** shardy_lunch is now known as shardy | 12:25 | |
| *** dmellado has quit IRC | 12:26 | |
| *** lucas-afk is now known as lucasagomes | 12:26 | |
| *** shyama_ has joined #openstack-oslo | 12:28 | |
| *** shyama has quit IRC | 12:29 | |
| *** shyama_ is now known as shyama | 12:30 | |
| *** gordc has joined #openstack-oslo | 12:36 | |
| *** gcb has quit IRC | 12:45 | |
| *** kani__ has quit IRC | 12:47 | |
| *** dmellado has joined #openstack-oslo | 12:47 | |
| *** efried has quit IRC | 12:57 | |
| *** yamamoto has joined #openstack-oslo | 12:57 | |
| *** yamamoto has quit IRC | 12:58 | |
| *** yamamoto has joined #openstack-oslo | 12:58 | |
| *** efried has joined #openstack-oslo | 13:09 | |
| *** yamamoto has quit IRC | 13:15 | |
| *** yamamoto has joined #openstack-oslo | 13:16 | |
| *** yamamoto has quit IRC | 13:16 | |
| *** yamamoto has joined #openstack-oslo | 13:16 | |
| *** ctp has quit IRC | 13:19 | |
| *** ctp has joined #openstack-oslo | 13:21 | |
| *** yamamoto has quit IRC | 13:25 | |
| *** yamamoto has joined #openstack-oslo | 13:26 | |
| *** dave-mccowan has joined #openstack-oslo | 13:27 | |
| *** yamamoto has quit IRC | 13:30 | |
| *** Dinesh_Bhor has quit IRC | 13:37 | |
| *** shardy is now known as shardy_afk | 13:44 | |
| *** pooja_jadhav has quit IRC | 13:55 | |
| *** jaosorior_away is now known as jaosorior_mtg | 14:05 | |
| *** cdent has quit IRC | 14:09 | |
| *** shardy_afk is now known as shardy | 14:17 | |
| *** bnemec is now known as beekneemech | 14:20 | |
| *** yamahata has joined #openstack-oslo | 14:21 | |
| *** rcernin has quit IRC | 14:24 | |
| *** yamamoto has joined #openstack-oslo | 14:27 | |
| *** salv-orlando has quit IRC | 14:28 | |
| *** zz_dimtruck is now known as dimtruck | 14:29 | |
| *** amotoki has joined #openstack-oslo | 14:33 | |
| *** amotoki has quit IRC | 14:36 | |
| *** amotoki has joined #openstack-oslo | 14:39 | |
| *** yamamoto has quit IRC | 14:53 | |
| *** jaosorior_mtg is now known as jaosorior | 14:58 | |
| *** jaosorior has quit IRC | 14:59 | |
| *** yamahata has quit IRC | 15:03 | |
| *** yamahata_ has joined #openstack-oslo | 15:03 | |
| *** cdent has joined #openstack-oslo | 15:03 | |
| breton | how are cache backends tested in oslo? | 15:07 |
| breton | for example, memcache_pool | 15:07 |
| breton | ok, i see | 15:09 |
| *** amotoki has quit IRC | 15:09 | |
| *** aarefiev is now known as aarefiev_afk | 15:11 | |
| *** amotoki has joined #openstack-oslo | 15:26 | |
| *** amotoki has quit IRC | 15:30 | |
| *** amotoki has joined #openstack-oslo | 15:32 | |
| *** e0ne has quit IRC | 15:32 | |
| *** amotoki has quit IRC | 15:35 | |
| *** mhickey has quit IRC | 15:42 | |
| *** lpetrut has quit IRC | 15:44 | |
| *** ildikov is now known as coffee_cat | 15:45 | |
| *** amotoki has joined #openstack-oslo | 15:48 | |
| *** yamamoto has joined #openstack-oslo | 15:51 | |
| *** dansmith is now known as superdan | 15:55 | |
| *** Guest86170 has quit IRC | 15:56 | |
| efried | harlowja Perhaps execute exception handler could use save_and_reraise_exception to ensure all of 'em get logged... | 15:57 |
| *** Guest86170 has joined #openstack-oslo | 15:58 | |
| *** david-lyle has joined #openstack-oslo | 15:58 | |
| *** salv-orlando has joined #openstack-oslo | 16:00 | |
| *** salv-orlando has quit IRC | 16:04 | |
| *** salv-orlando has joined #openstack-oslo | 16:05 | |
| *** cdent has quit IRC | 16:09 | |
| *** john51 has quit IRC | 16:27 | |
| *** ihrachys has quit IRC | 16:27 | |
| *** ihrachys has joined #openstack-oslo | 16:27 | |
| *** john51_ has joined #openstack-oslo | 16:28 | |
| *** beekneemech has quit IRC | 16:30 | |
| *** lucasagomes is now known as lucas-afk | 16:36 | |
| *** amotoki has quit IRC | 16:39 | |
| *** bnemec has joined #openstack-oslo | 16:44 | |
| *** bnemec is now known as beekneemech | 16:45 | |
| harlowja | efried ya, that shouldn't be to hard | 16:48 |
| harlowja | at least to get them logged | 16:49 |
| *** shardy has quit IRC | 16:50 | |
| *** dmellado has quit IRC | 16:52 | |
| *** yamamoto has quit IRC | 16:55 | |
| *** salv-orlando has quit IRC | 17:00 | |
| *** dmellado has joined #openstack-oslo | 17:06 | |
| *** ctp has quit IRC | 17:08 | |
| *** ctp has joined #openstack-oslo | 17:10 | |
| *** Rockyg has joined #openstack-oslo | 17:13 | |
| *** tesseract has quit IRC | 17:19 | |
| *** ig0r_ has quit IRC | 17:24 | |
| *** ig0r_ has joined #openstack-oslo | 17:27 | |
| *** salv-orlando has joined #openstack-oslo | 17:34 | |
| *** ctp has quit IRC | 18:02 | |
| *** ctp has joined #openstack-oslo | 18:04 | |
| *** yamahata_ has quit IRC | 18:04 | |
| *** sambetts is now known as sambetts|afk | 18:15 | |
| *** yamahata_ has joined #openstack-oslo | 18:22 | |
| *** salv-orlando has quit IRC | 18:29 | |
| *** amotoki has joined #openstack-oslo | 18:39 | |
| *** jerrygb has quit IRC | 18:40 | |
| *** amotoki has quit IRC | 18:44 | |
| *** jerrygb has joined #openstack-oslo | 18:49 | |
| *** ansmith has quit IRC | 18:59 | |
| *** pcaruana has quit IRC | 19:02 | |
| *** e0ne has joined #openstack-oslo | 19:06 | |
| *** cdent has joined #openstack-oslo | 19:36 | |
| -openstackstatus- NOTICE: Gerrit will be offline briefly starting at 20:00 for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html | 19:42 | |
| *** ig0r_ has quit IRC | 19:55 | |
| *** e0ne has quit IRC | 19:56 | |
| -openstackstatus- NOTICE: Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html | 20:03 | |
| *** ChanServ changes topic to "Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html" | 20:03 | |
| *** jerrygb has quit IRC | 20:10 | |
| *** jerrygb has joined #openstack-oslo | 20:25 | |
| *** ChanServ changes topic to "#openstack-oslo" | 20:36 | |
| -openstackstatus- NOTICE: Gerrit is back in service and generally usable, though remote Git replicas (git.openstack.org and github.com) will be stale for the next few hours until online reindexing completes | 20:36 | |
| *** amotoki has joined #openstack-oslo | 20:40 | |
| *** amotoki has quit IRC | 20:46 | |
| *** openstackgerrit has quit IRC | 20:48 | |
| *** yamamoto has joined #openstack-oslo | 20:54 | |
| *** yamamoto has quit IRC | 20:58 | |
| *** gordc has quit IRC | 21:13 | |
| *** salv-orlando has joined #openstack-oslo | 21:38 | |
| *** amotoki has joined #openstack-oslo | 21:42 | |
| *** amotoki has quit IRC | 21:47 | |
| *** dimtruck is now known as zz_dimtruck | 21:47 | |
| *** salv-orl_ has joined #openstack-oslo | 21:49 | |
| *** salv-orlando has quit IRC | 21:52 | |
| *** zz_dimtruck is now known as dimtruck | 22:47 | |
| *** cdent has quit IRC | 22:47 | |
| *** salv-orl_ has quit IRC | 22:48 | |
| *** sdague has quit IRC | 23:24 | |
| *** yamahata_ has quit IRC | 23:40 | |
| *** yamamoto has joined #openstack-oslo | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!