| *** bobh has joined #openstack-oslo | 00:00 | |
| *** yamamoto has quit IRC | 00:03 | |
| *** salv-orlando has joined #openstack-oslo | 00:11 | |
| *** salv-orlando has quit IRC | 00:16 | |
| *** edmondsw has quit IRC | 00:37 | |
| *** namnh has joined #openstack-oslo | 00:58 | |
| *** yamamoto has joined #openstack-oslo | 00:59 | |
| *** njohnston has quit IRC | 01:04 | |
| *** yamamoto has quit IRC | 01:05 | |
| *** salv-orlando has joined #openstack-oslo | 01:11 | |
| *** salv-orlando has quit IRC | 01:15 | |
| *** bobh has quit IRC | 01:37 | |
| *** harlowja has quit IRC | 01:46 | |
| *** mordred has quit IRC | 01:49 | |
| *** Guest92236 has quit IRC | 01:53 | |
| *** mordred has joined #openstack-oslo | 02:07 | |
| *** Guest92236 has joined #openstack-oslo | 02:10 | |
| *** yamamoto has joined #openstack-oslo | 03:01 | |
| *** yamamoto has quit IRC | 03:07 | |
| openstackgerrit | Steve Kowalik proposed openstack/oslo.messaging master: Remove fake_rabbit configuration option https://review.openstack.org/577964 | 03:13 |
|---|---|---|
| openstackgerrit | Merged openstack/osprofiler master: Add minimum version and fix dulwich issue https://review.openstack.org/577791 | 03:42 |
| *** AlexeyAbashkin has joined #openstack-oslo | 03:47 | |
| *** lpetrut has joined #openstack-oslo | 03:50 | |
| *** AlexeyAbashkin has quit IRC | 03:54 | |
| *** AlexeyAbashkin has joined #openstack-oslo | 03:54 | |
| openstackgerrit | Tovin Seven proposed openstack/osprofiler master: Put 'db' parameter back and provide a deprecation warning before remove https://review.openstack.org/574207 | 04:01 |
| openstackgerrit | Tovin Seven proposed openstack/osprofiler master: opts: Fix invalid rST formatting https://review.openstack.org/577423 | 04:02 |
| openstackgerrit | Tovin Seven proposed openstack/osprofiler master: [WIP] Add Zuul Tempest job https://review.openstack.org/515387 | 04:02 |
| *** yamamoto has joined #openstack-oslo | 04:04 | |
| *** AlexeyAbashkin has quit IRC | 04:04 | |
| *** yamamoto has quit IRC | 04:09 | |
| *** spilla has joined #openstack-oslo | 04:23 | |
| *** lpetrut has quit IRC | 04:28 | |
| *** dims has quit IRC | 04:30 | |
| *** dims has joined #openstack-oslo | 04:35 | |
| *** yamamoto has joined #openstack-oslo | 05:05 | |
| *** CrayZee has joined #openstack-oslo | 05:07 | |
| *** yamamoto has quit IRC | 05:12 | |
| *** e0ne has joined #openstack-oslo | 05:13 | |
| *** lifeless has joined #openstack-oslo | 05:16 | |
| *** spilla has quit IRC | 05:41 | |
| openstackgerrit | Steve Kowalik proposed openstack/oslo.messaging master: Remove fake_rabbit configuration option https://review.openstack.org/577964 | 05:49 |
| *** snapiri- has joined #openstack-oslo | 05:53 | |
| *** CrayZee has quit IRC | 05:56 | |
| *** openstackgerrit has quit IRC | 06:04 | |
| *** e0ne has quit IRC | 06:08 | |
| *** yamamoto has joined #openstack-oslo | 06:09 | |
| *** AlexeyAbashkin has joined #openstack-oslo | 06:13 | |
| *** yamamoto has quit IRC | 06:14 | |
| *** Alexey_Abashkin has joined #openstack-oslo | 06:15 | |
| *** Alexey_Abashkin1 has joined #openstack-oslo | 06:16 | |
| *** AlexeyAbashkin has quit IRC | 06:18 | |
| *** Alexey_Abashkin1 is now known as AlexeyAbashkin | 06:18 | |
| *** Alexey_Abashkin has quit IRC | 06:19 | |
| *** nicolasbock has joined #openstack-oslo | 06:40 | |
| *** peereb has joined #openstack-oslo | 06:48 | |
| *** peereb has quit IRC | 06:49 | |
| *** peereb has joined #openstack-oslo | 06:49 | |
| *** Guest92236 has quit IRC | 06:50 | |
| *** peereb has quit IRC | 06:50 | |
| *** openstackgerrit has joined #openstack-oslo | 06:53 | |
| openstackgerrit | Merged openstack/osprofiler master: Put 'db' parameter back and provide a deprecation warning before remove https://review.openstack.org/574207 | 06:53 |
| *** yamamoto has joined #openstack-oslo | 06:56 | |
| *** AlexeyAbashkin has quit IRC | 07:05 | |
| *** namnh has quit IRC | 07:06 | |
| *** AlexeyAbashkin has joined #openstack-oslo | 07:07 | |
| *** lpetrut has joined #openstack-oslo | 07:07 | |
| *** rcernin has quit IRC | 07:08 | |
| *** shardy has joined #openstack-oslo | 07:10 | |
| *** tesseract has joined #openstack-oslo | 07:14 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/oslo.messaging master: rabbit: acknowledge message sooner https://review.openstack.org/577912 | 07:19 |
| *** pcaruana has joined #openstack-oslo | 07:20 | |
| *** tosky has joined #openstack-oslo | 07:41 | |
| *** e0ne has joined #openstack-oslo | 08:05 | |
| *** pblaho has joined #openstack-oslo | 08:21 | |
| *** salv-orlando has joined #openstack-oslo | 08:22 | |
| *** salv-orlando has quit IRC | 08:25 | |
| *** salv-orlando has joined #openstack-oslo | 08:26 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/oslo.policy master: Implement base for pluggable policy drivers https://review.openstack.org/577807 | 08:26 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/oslo.policy master: Implement base for pluggable policy drivers https://review.openstack.org/577807 | 08:30 |
| *** rpittau has joined #openstack-oslo | 08:37 | |
| *** shardy has quit IRC | 08:38 | |
| *** jaosorior has quit IRC | 08:39 | |
| *** salv-orlando has quit IRC | 08:40 | |
| *** shardy has joined #openstack-oslo | 08:43 | |
| *** mago_ has joined #openstack-oslo | 08:54 | |
| *** shardy has quit IRC | 09:10 | |
| *** shardy has joined #openstack-oslo | 09:13 | |
| openstackgerrit | Stephen Finucane proposed openstack/sphinx-feature-classification master: doc: Rework everything https://review.openstack.org/577862 | 09:16 |
| openstackgerrit | Stephen Finucane proposed openstack/sphinx-feature-classification master: Parse 'driver-notes-XXX' values https://review.openstack.org/577863 | 09:16 |
| *** lifeless has quit IRC | 09:16 | |
| stephenfin | dhellmann: Awesome. Thanks for sorting that out (y) | 09:22 |
| *** salv-orlando has joined #openstack-oslo | 09:41 | |
| *** salv-orlando has quit IRC | 09:46 | |
| *** jaosorior has joined #openstack-oslo | 10:29 | |
| *** salv-orlando has joined #openstack-oslo | 10:42 | |
| *** salv-orlando has quit IRC | 10:47 | |
| *** yamamoto has quit IRC | 10:58 | |
| *** yamamoto has joined #openstack-oslo | 11:07 | |
| openstackgerrit | Michal Arbet proposed openstack/taskflow master: Fix code to support networkx > 2.0 https://review.openstack.org/577833 | 11:17 |
| *** yamamoto has quit IRC | 11:30 | |
| *** namnh has joined #openstack-oslo | 11:30 | |
| *** sambetts|afk is now known as sambetts | 11:31 | |
| *** yamamoto has joined #openstack-oslo | 11:33 | |
| *** mago_ has quit IRC | 11:33 | |
| *** ansmith has quit IRC | 11:48 | |
| *** mago_ has joined #openstack-oslo | 11:51 | |
| *** mago_ has quit IRC | 12:08 | |
| *** raildo has joined #openstack-oslo | 12:12 | |
| *** shardy_ has joined #openstack-oslo | 12:12 | |
| *** bobh has joined #openstack-oslo | 12:13 | |
| *** shardy has quit IRC | 12:15 | |
| *** shardy_ is now known as shardy | 12:15 | |
| openstackgerrit | Merged openstack/osprofiler master: opts: Fix invalid rST formatting https://review.openstack.org/577423 | 12:20 |
| *** mago_ has joined #openstack-oslo | 12:25 | |
| dhellmann | stephenfin : when you're ready, I can do a release, or give you permission on PyPI to do it if you want | 12:55 |
| openstackgerrit | Moisés Guimarães de Medeiros proposed openstack/oslo.config master: User guide documentation for backend drivers for oslo.config https://review.openstack.org/576947 | 12:56 |
| stephenfin | dhellmann: Whatever suits. I've a PR opened that you should probably review, given that I'm changing your email address https://github.com/sphinx-contrib/pecanwsme/pull/20 | 12:56 |
| *** edmondsw has joined #openstack-oslo | 12:58 | |
| openstackgerrit | Moisés Guimarães de Medeiros proposed openstack/oslo.config master: User guide documentation for backend drivers for oslo.config https://review.openstack.org/576947 | 12:59 |
| *** yamamoto has quit IRC | 13:01 | |
| *** r-mibu has joined #openstack-oslo | 13:01 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/oslo.policy master: Implement base for pluggable policy drivers https://review.openstack.org/577807 | 13:03 |
| dhellmann | stephenfin : oh, nice, is project_urls a new thing? | 13:05 |
| stephenfin | dhellmann: Indeed. I think fungi added support for it in pbr a while back | 13:05 |
| *** kgiusti has joined #openstack-oslo | 13:05 | |
| dhellmann | yay, another source of trivial patches | 13:05 |
| dhellmann | :-) | 13:05 |
| *** namnh has quit IRC | 13:06 | |
| stephenfin | Shhh, don't tell anyone and we'll be fine :) | 13:06 |
| dhellmann | we need to figure out how to list the maintainer of these packages as the collective | 13:06 |
| *** ansmith has joined #openstack-oslo | 13:10 | |
| *** bobh has quit IRC | 13:29 | |
| fungi | i added it mainly because it was being touted as the "solution" for people who had been using (then deprecated, now gone) pythonhosted for their docs but there was no support for it in setuptools so seemed like a cop-out answer | 13:29 |
| *** yamamoto has joined #openstack-oslo | 13:29 | |
| fungi | once i got support added in setuptools, i realized if i wanted to do that with my pbr-using personal projects, pbr needed support for it too | 13:30 |
| fungi | and while it's basically just cosmetic, it at least gives you a nice linkbox on the pypi project page where people can easily figure out where you host your documentation, source code, et cetera | 13:32 |
| dhellmann | fungi : yeah, I like the idea, I was just being a bit of an early-morning cranky-pants | 13:34 |
| fungi | heh, you're entitled | 13:35 |
| fungi | it was really just my personal battle with ocd and noticing setuptools (and so pbr) lacked support for some types of metadata from accepted peps | 13:37 |
| *** e0ne has quit IRC | 13:39 | |
| *** cdent has joined #openstack-oslo | 13:40 | |
| * cdent waves | 13:40 | |
| *** snapiri- has quit IRC | 13:41 | |
| dhellmann | cdent : I think we're getting close to a point where we can merge the driver work that the environment variable spec will rely on. Do you want to try to write that driver this cycle? Or were you planning to leave it for Stein? either is fine, I just want to coordinate plans | 13:42 |
| cdent | I'd like to do it sooner than later and I think it would be a bit of vacation (in a twisted sort of way) | 13:42 |
| cdent | I wasn't clear on the state of the driver stuff, but if it is coming RSN, then thats great | 13:43 |
| dhellmann | we're using the series of patches currently ending with https://review.openstack.org/#/c/576947/ as a sort of feature branch | 13:43 |
| dhellmann | and just putting some finishing touches on things like the config generator and docs | 13:43 |
| dhellmann | so if you wanted to base a patch on that series it ought to be pretty safe. maybe base it on https://review.openstack.org/#/c/554316/16 since I don't expect significant changes below that point | 13:44 |
| dhellmann | bnemec : ^^ | 13:44 |
| *** e0ne has joined #openstack-oslo | 13:44 | |
| * cdent makes a note of that stack | 13:46 | |
| dhellmann | oh, except https://review.openstack.org/#/c/575107/5 changes the driver API so you should start there | 13:46 |
| dhellmann | and that one isn't quite fully baked yet | 13:46 |
| dhellmann | but yeah, it's close enough that if you want to start you could and you'll just want to keep an eye on the need to rebase | 13:46 |
| cdent | seems fair. at least gives me something to read and understand what's going on | 13:47 |
| dhellmann | the usual disclaimers about landing late features apply, but we tend to be pretty lenient | 13:47 |
| dhellmann | in your case we might want to have the behavior disabled by default for this cycle and enable it by default next cycle, but that might be overly cautious | 13:48 |
| cdent | we can see how it goes | 13:50 |
| dhellmann | ++ | 13:50 |
| cdent | i'll try to get at least a spike pretty soon so I can get some redirects if I'm off in the weeds | 13:52 |
| dhellmann | cdent : good plan | 13:55 |
| *** jaosorior has quit IRC | 13:59 | |
| *** njohnston has joined #openstack-oslo | 14:02 | |
| *** chhagarw has joined #openstack-oslo | 14:19 | |
| openstackgerrit | Stephen Finucane proposed openstack/oslo.log master: Migrate to stestr https://review.openstack.org/571205 | 14:23 |
| openstackgerrit | Stephen Finucane proposed openstack/oslo.log master: Follow the new PTI for document build https://review.openstack.org/549338 | 14:23 |
| *** e0ne has quit IRC | 14:23 | |
| *** e0ne has joined #openstack-oslo | 14:25 | |
| openstackgerrit | Moisés Guimarães de Medeiros proposed openstack/oslo.config master: User guide documentation for backend drivers for oslo.config https://review.openstack.org/576947 | 14:36 |
| *** bobh has joined #openstack-oslo | 14:40 | |
| *** chhavi__ has joined #openstack-oslo | 14:41 | |
| *** chhagarw has quit IRC | 14:41 | |
| *** namnh has joined #openstack-oslo | 14:53 | |
| *** AlexeyAbashkin has quit IRC | 15:01 | |
| *** spilla has joined #openstack-oslo | 15:01 | |
| *** e0ne has quit IRC | 15:07 | |
| *** e0ne has joined #openstack-oslo | 15:11 | |
| openstackgerrit | Michal Arbet proposed openstack/taskflow master: Fix code to support networkx > 2.0 https://review.openstack.org/577833 | 15:18 |
| *** pcaruana has quit IRC | 15:42 | |
| *** pbourke has quit IRC | 15:47 | |
| *** pbourke has joined #openstack-oslo | 15:49 | |
| *** mago_ has quit IRC | 15:58 | |
| *** e0ne has quit IRC | 16:09 | |
| *** harlowja has joined #openstack-oslo | 16:12 | |
| *** e0ne has joined #openstack-oslo | 16:13 | |
| *** r-mibu has quit IRC | 16:14 | |
| openstackgerrit | Raildo Mascena proposed openstack/oslo.config master: New cache layer for external sources https://review.openstack.org/578144 | 16:15 |
| *** namnh has quit IRC | 16:16 | |
| *** mago_ has joined #openstack-oslo | 16:31 | |
| *** e0ne has quit IRC | 16:35 | |
| *** tesseract has quit IRC | 16:40 | |
| *** njohnston has quit IRC | 16:48 | |
| *** njohnston has joined #openstack-oslo | 16:48 | |
| *** CrayZee has joined #openstack-oslo | 16:49 | |
| *** lpetrut has quit IRC | 16:50 | |
| *** CrayZee has quit IRC | 16:51 | |
| *** sambetts is now known as sambetts|afk | 17:04 | |
| *** harlowja has quit IRC | 17:10 | |
| *** e0ne has joined #openstack-oslo | 17:11 | |
| *** yamamoto has quit IRC | 17:12 | |
| *** yamamoto has joined #openstack-oslo | 17:14 | |
| *** e0ne has quit IRC | 17:18 | |
| *** yamamoto has quit IRC | 17:20 | |
| *** imacdonn has joined #openstack-oslo | 18:07 | |
| *** jaosorior has joined #openstack-oslo | 18:13 | |
| *** yamamoto has joined #openstack-oslo | 18:16 | |
| *** yamamoto has quit IRC | 18:20 | |
| *** yamamoto has joined #openstack-oslo | 18:21 | |
| *** AlexeyAbashkin has joined #openstack-oslo | 18:25 | |
| *** dmellado has quit IRC | 18:32 | |
| *** chhavi__ has quit IRC | 18:37 | |
| *** AlexeyAbashkin has quit IRC | 19:13 | |
| *** AlexeyAbashkin has joined #openstack-oslo | 19:13 | |
| *** AlexeyAbashkin has quit IRC | 19:14 | |
| *** lpetrut has joined #openstack-oslo | 19:46 | |
| *** Kvisle has joined #openstack-oslo | 19:48 | |
| *** lpetrut has quit IRC | 19:50 | |
| *** e0ne has joined #openstack-oslo | 19:52 | |
| Kvisle | Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" _raise_timeout /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:1037 ... I'm troubleshooting neutron looping over a bunch of RPC-messages, and this showed up when turning debug on --- does this error make sense in any way? | 20:04 |
| *** ys has joined #openstack-oslo | 20:13 | |
| *** e0ne has quit IRC | 20:15 | |
| ys | hi not sure if this is the correct place to ask. I am running openstack pike and trying to do revert_from_snapshot. it failed and in cinder log it throws error, oslo_messaging.rpc.server AttributeError: 'Snapshot' object has no attribute '_obj_metadata' | 20:16 |
| ys | Not sure what to do now... | 20:17 |
| bnemec | ys: I think that's just oslo.messaging reporting that the RPC call failed with that message. You probably want to talk to the Cinder folks about why. | 20:17 |
| smcginnis | ys: #openstack-cinder might be a better place. Though I've seen something similar to that error before and it might actually be from an oslo library, but we can figure that out there. | 20:18 |
| kgiusti | Kvisle: looks like whatever raised that didn't properly supply the topic/method/etc. | 20:19 |
| ys | thanks | 20:19 |
| ys | Ok thanks. | 20:19 |
| bnemec | I merely _think_ about pinging kgiusti and he appears. :-) | 20:19 |
| kgiusti | Kvisle: the log message is bad, but I suspect the timeout is due to the rpc call not receiving the reply back in time | 20:19 |
| kgiusti | Kvisle: do you have a stack trace? | 20:20 |
| *** spilla has quit IRC | 20:20 | |
| *** raildo has quit IRC | 20:20 | |
| Kvisle | kgiusti: I have a stack trace, https://gist.github.com/kvisle/1404998a129ef65e59f17a3d739369af ... that I made by adding traceback.print() to the _raise_timeout()-method | 20:20 |
| kgiusti | Kvisle: thanks | 20:20 |
| * kgiusti looks.... | 20:21 | |
| Kvisle | running python2-oslo-messaging-5.30.2-1.el7.noarch (RDO) --- this problem started appearing in pike | 20:22 |
| *** cdent has quit IRC | 20:28 | |
| kgiusti | Kvisle: that timeout appears to be the periodic timeout of a listener thread | 20:32 |
| kgiusti | Kvisle: which is supposed to be ok.... one sec... | 20:33 |
| Kvisle | I can see the same request id's being logged for days, on _ALL_ nodes (and there are like 50 of them) --- so I'm suspecting that even though the timeouts themselves do not cause any direct harm --- processing the messages in a loop everywhere causes some unwanted load on the cluster (?) | 20:34 |
| kgiusti | Kvisle: I would expect that timeout exception to be caught on line 286 of amqpdriver.py | 20:35 |
| Kvisle | I'm using rabbit, though? | 20:35 |
| kgiusti | Kvisle: e.g. periodic timeout of the consumer. | 20:35 |
| kgiusti | Kvisle: yep - that's the rabbitmq driver | 20:35 |
| Kvisle | oh yeah my bad | 20:35 |
| kgiusti | Kvisle: Are these messages appearing in the context of a Traceback? or appearing in a log w/o traceback? | 20:37 |
| Kvisle | no traceback anywhere - the one I gave you came by me adding traceback.print() in the source code | 20:37 |
| kgiusti | Kvisle: good - an exception traceback would be bad, mmmmkay? Now where did you find these messages originally - in a log? | 20:38 |
| Kvisle | I have loops in neutron-linuxbridge-agent that goes over the same request ids repeatedly on all nodes (level INFO) - and on neutron-server | 20:38 |
| kgiusti | Kvisle: I ask because they *should* be silently discarded - not logged. | 20:39 |
| kgiusti | Kvisle: same request ids? Are you seeing duplicate RPC messages? | 20:39 |
| Kvisle | # grep req-141d3163-6832-4567-a30f-ebc7ddf9f9cc /var/log/neutron/server.log | wc -l | 20:40 |
| Kvisle | 1411 | 20:40 |
| smcginnis | bnemec: Does this look familiar? http://paste.openstack.org/show/724339/ | 20:40 |
| smcginnis | I seem to remember an OVO issue, but it was a long time ago. | 20:40 |
| smcginnis | This is pike code though. | 20:40 |
| *** ansmith has quit IRC | 20:41 | |
| Kvisle | kgiusti: I THINK that's a clear yes? | 20:41 |
| kgiusti | Kvisle: sorry - peeling myself off the ceiling... | 20:41 |
| kgiusti | Kvisle: one sec... | 20:42 |
| *** spilla has joined #openstack-oslo | 20:43 | |
| bnemec | smcginnis: It doesn't to me, but you probably want dansmith for that question. | 20:44 |
| kgiusti | Kvisle: are there any logs with "Found duplicate message" appearing? | 20:45 |
| dansmith | smcginnis: that would be a Snapshot.metadata having been manually deleted from the object or something very wrong | 20:45 |
| smcginnis | bnemec: OK, thanks. | 20:45 |
| kgiusti | Kvisle: rabbit should check for duplication | 20:46 |
| dansmith | smcginnis: it's ovo related, but not from any issue other than user error that I know of | 20:46 |
| kgiusti | Kvisle: rabbit driver, that is | 20:46 |
| smcginnis | dansmith: It's just a feeling, but I seem to remember that error being an issue for awhile until something was fixed on the lib side. | 20:46 |
| smcginnis | dansmith: Or some other dependency that impact ova. | 20:46 |
| smcginnis | ovo | 20:46 |
| dansmith | smcginnis: don't think so | 20:46 |
| smcginnis | dansmith: OK, thanks. I'll keep digging. | 20:47 |
| Kvisle | kgiusti: let me grep around | 20:48 |
| Kvisle | kgiusti: no | 20:49 |
| Kvisle | kgiusti: I can see some old entries in cinder scheduler logs, but that was rarely, and before we upgraded to pike | 20:49 |
| kgiusti | Kvisle: Alright, I was wondering if you were hitting this: https://bugzilla.redhat.com/show_bug.cgi?id=1592528 | 20:50 |
| openstack | bugzilla.redhat.com bug 1592528 in rabbitmq-server "[OSP13] cannot launch instance(cannot bind port) and ping existing instance using floating IP after reboot of OC nodes" [Urgent,On_dev] - Assigned to jeckersb | 20:50 |
| kgiusti | Kvisle: rabbitmq driver adds a uuid to each outgoing message. That uuid is used to check for a duplicate message. | 20:55 |
| kgiusti | Kvisle: but we're not seeing duplicate message ids | 20:56 |
| kgiusti | Kvisle: and oslo.messaging does not retransmit the same message (or it isn't designed to :) | 20:56 |
| kgiusti | Kvisle: I'm beginning to suspect that some client is sending the same req- over and over again. | 20:57 |
| kgiusti | Kvisle: which would generate a new message id - thus no duplicates found by oslo.messaging, but duplicates at the application level | 20:57 |
| kgiusti | Kvisle: at least that's my current guess (subject to change at any moment). | 20:58 |
| kgiusti | Kvisle: can you create a launchpad bug for this: https://launchpad.net/oslo.messaging | 20:59 |
| Kvisle | of course | 20:59 |
| kgiusti | Kvisle: and attach the neutron server log to it? | 21:00 |
| Kvisle | hm, one thing | 21:00 |
| Kvisle | neutron-linuxbridge-agent does NOT have a backtrace in it | 21:01 |
| Kvisle | but the neutron server log has; https://gist.github.com/kvisle/6edbab6ce7aed8bd4b0b2846d56c576f | 21:02 |
| kgiusti | Kvisle: ah, so the server is raising an error during RPC call processing. | 21:03 |
| kgiusti | Kvisle: that exception should be sent back to the RPC caller and raised again there. | 21:03 |
| kgiusti | Kvisle: forgive my ignorance - does the n-linuxbridge-agent make RPC calls to neutron server? | 21:05 |
| Kvisle | kgiusti: I think you're referring to this https://gist.github.com/kvisle/b5b9bcb0a5946cb40e366ace9f6feb81 | 21:05 |
| kgiusti | Kvisle: bingo | 21:06 |
| Kvisle | all nodes were doing the loop, not all nodes were throwing the backtrace (but a handful are, I think) | 21:06 |
| kgiusti | Kvisle: those backtraces indicate a failure at the application layer above oslo.messaging. | 21:08 |
| Kvisle | kgiusti: I should probably move over to troubleshooting neutron directly, then? | 21:08 |
| kgiusti | Kvisle: yeah neutron doesn't look very happy. | 21:09 |
| Kvisle | my assumption that this thing propagates and induces extra load on the system is valid, right? | 21:09 |
| Kvisle | (maybe?) | 21:09 |
| Kvisle | kgiusti: thank you very much for your time! | 21:10 |
| kgiusti | Kvisle: exceptions that occur on servers that are servicing RPC calls essentially send the stacktrace back to the client as a reply | 21:10 |
| kgiusti | Kvisle: so it's not really more expensive than a non-error call/reply | 21:10 |
| Kvisle | hm, ok | 21:10 |
| kgiusti | Kvisle: HOWEVER, if the client doesn't handle that error and just keeps sending the same request.... that's naughty. | 21:11 |
| kgiusti | Kvisle: Keep us posted. I need to leave for the evening now. Good luck. | 21:11 |
| Kvisle | kgiusti: good evening =) | 21:12 |
| Kvisle | and thank you very much for your time | 21:12 |
| kgiusti | Kvisle: yw and good hunting.... | 21:12 |
| *** kgiusti has left #openstack-oslo | 21:12 | |
| Kvisle | looks like a fix is on its way https://review.openstack.org/#/c/564755/ | 21:14 |
| *** mago_ has quit IRC | 21:33 | |
| *** lifeless has joined #openstack-oslo | 21:38 | |
| *** rcernin has joined #openstack-oslo | 21:47 | |
| *** edmondsw has quit IRC | 22:04 | |
| *** edmondsw has joined #openstack-oslo | 22:07 | |
| *** bobh has quit IRC | 22:07 | |
| *** edmondsw has quit IRC | 22:11 | |
| *** bnemec has quit IRC | 22:29 | |
| *** bnemec has joined #openstack-oslo | 22:35 | |
| *** ansmith has joined #openstack-oslo | 22:35 | |
| *** threestrands has joined #openstack-oslo | 22:38 | |
| *** yassine has joined #openstack-oslo | 22:46 | |
| *** yassine is now known as Guest35635 | 22:46 | |
| *** nicolasbock has quit IRC | 23:06 | |
| *** tosky has quit IRC | 23:08 | |
| *** bobh has joined #openstack-oslo | 23:08 | |
| *** spilla has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!