| *** rajathagasthya has quit IRC | 00:20 | |
| *** kiennt has joined #openstack-oslo | 00:37 | |
| *** rajathagasthya has joined #openstack-oslo | 00:43 | |
| openstackgerrit | Vipin Balachandran proposed openstack/osprofiler master: Improve unit test coverage https://review.openstack.org/489739 | 01:11 |
|---|---|---|
| *** https_GK1wmSU has joined #openstack-oslo | 01:32 | |
| *** https_GK1wmSU has left #openstack-oslo | 01:34 | |
| *** rajathagasthya has quit IRC | 01:55 | |
| *** xiaoyang has joined #openstack-oslo | 02:19 | |
| *** yamamoto has quit IRC | 02:24 | |
| *** yamamoto has joined #openstack-oslo | 02:30 | |
| *** yamamoto has quit IRC | 02:35 | |
| *** yamamoto has joined #openstack-oslo | 02:36 | |
| *** yamamoto has quit IRC | 02:41 | |
| *** dave-mcc_ has joined #openstack-oslo | 02:41 | |
| *** dave-mccowan has quit IRC | 02:42 | |
| *** yamamoto has joined #openstack-oslo | 02:45 | |
| *** nicolasbock has joined #openstack-oslo | 02:57 | |
| *** dave-mccowan has joined #openstack-oslo | 03:11 | |
| *** dave-mcc_ has quit IRC | 03:14 | |
| *** gtrxcb has joined #openstack-oslo | 03:21 | |
| *** rajathagasthya has joined #openstack-oslo | 03:22 | |
| *** yamamoto has quit IRC | 03:33 | |
| *** dave-mccowan has quit IRC | 03:33 | |
| *** carolina1 has joined #openstack-oslo | 03:34 | |
| *** carolina1 has quit IRC | 03:43 | |
| *** links has joined #openstack-oslo | 03:48 | |
| *** yamamoto has joined #openstack-oslo | 03:49 | |
| *** rajathagasthya has quit IRC | 04:03 | |
| *** yamamoto has quit IRC | 04:05 | |
| *** yamamoto has joined #openstack-oslo | 04:10 | |
| *** kornicameister has quit IRC | 04:31 | |
| *** kornicameister has joined #openstack-oslo | 04:32 | |
| *** yamamoto has quit IRC | 04:50 | |
| *** yamamoto has joined #openstack-oslo | 05:07 | |
| *** kiennt has quit IRC | 05:36 | |
| *** kiennt has joined #openstack-oslo | 05:56 | |
| *** e0ne has joined #openstack-oslo | 06:01 | |
| *** hoonetorg has quit IRC | 06:02 | |
| *** e0ne has quit IRC | 06:03 | |
| *** kiennt has quit IRC | 06:10 | |
| *** kiennt has joined #openstack-oslo | 06:12 | |
| *** hoonetorg has joined #openstack-oslo | 06:15 | |
| *** jaosorior has quit IRC | 06:25 | |
| openstackgerrit | Abhishek Sharma M proposed openstack/pycadf master: Adding panko_api_audit_map.conf to pycadf https://review.openstack.org/489680 | 06:28 |
| openstackgerrit | Abhishek Sharma M proposed openstack/pycadf master: Adding panko_api_audit_map.conf to pycadf https://review.openstack.org/489680 | 06:30 |
| *** jaosorior has joined #openstack-oslo | 06:34 | |
| *** pcaruana has joined #openstack-oslo | 06:34 | |
| *** rcernin has joined #openstack-oslo | 06:46 | |
| *** aarefiev_afk is now known as aarefiev | 07:07 | |
| openstackgerrit | Abhishek Sharma M proposed openstack/pycadf master: Removed references to events api https://review.openstack.org/489892 | 07:21 |
| *** lpetrut has joined #openstack-oslo | 07:37 | |
| *** gtrxcb has quit IRC | 07:44 | |
| *** ralonsoh has joined #openstack-oslo | 07:46 | |
| *** gcb_ has quit IRC | 07:48 | |
| *** tesseract has joined #openstack-oslo | 07:48 | |
| *** pblaho has joined #openstack-oslo | 07:49 | |
| *** ralonsoh has quit IRC | 08:12 | |
| *** xiaoyang has quit IRC | 08:13 | |
| *** ralonsoh has joined #openstack-oslo | 08:13 | |
| *** xiaoyang has joined #openstack-oslo | 08:13 | |
| *** ralonsoh has quit IRC | 08:15 | |
| *** ralonsoh has joined #openstack-oslo | 08:18 | |
| *** https_GK1wmSU has joined #openstack-oslo | 08:24 | |
| *** https_GK1wmSU has left #openstack-oslo | 08:25 | |
| *** lucas-afk is now known as lucasagomes | 08:32 | |
| *** openstackgerrit has quit IRC | 08:33 | |
| *** ralonsoh_ has joined #openstack-oslo | 08:40 | |
| *** ralonsoh has quit IRC | 08:40 | |
| *** openstackgerrit has joined #openstack-oslo | 08:46 | |
| openstackgerrit | Tovin Seven proposed openstack/osprofiler master: Add function/sql results to trace info https://review.openstack.org/450072 | 08:46 |
| openstackgerrit | Tovin Seven proposed openstack/osprofiler master: [WiP] OSprofiler with OpenTracing compatible https://review.openstack.org/480018 | 08:47 |
| *** gcb has joined #openstack-oslo | 09:28 | |
| *** sdague has joined #openstack-oslo | 09:32 | |
| *** e0ne has joined #openstack-oslo | 09:53 | |
| *** yassine has quit IRC | 09:59 | |
| *** kiennt has quit IRC | 10:01 | |
| *** sambetts|afk is now known as sambetts | 10:09 | |
| *** jaosorior has quit IRC | 10:10 | |
| *** ansmith has joined #openstack-oslo | 10:40 | |
| *** ansmith_ has joined #openstack-oslo | 10:41 | |
| *** ansmith_ has quit IRC | 10:53 | |
| *** ansmith has quit IRC | 10:53 | |
| *** jaosorior has joined #openstack-oslo | 11:02 | |
| *** lucasagomes is now known as lucas-hungry | 11:37 | |
| *** nicolasbock_ has joined #openstack-oslo | 11:41 | |
| *** nicolasbock__ has joined #openstack-oslo | 11:42 | |
| *** nicolasbock has quit IRC | 11:44 | |
| *** nicolasbock has joined #openstack-oslo | 11:45 | |
| *** nicolasbock_ has quit IRC | 11:46 | |
| *** nicolasbock__ has quit IRC | 11:47 | |
| *** raildo has joined #openstack-oslo | 12:00 | |
| *** yassine has joined #openstack-oslo | 12:01 | |
| *** lucas-hungry is now known as lucasagomes | 12:39 | |
| *** sdague_ has joined #openstack-oslo | 12:43 | |
| *** HenryG has quit IRC | 12:45 | |
| *** gordc has joined #openstack-oslo | 12:45 | |
| *** HenryG has joined #openstack-oslo | 12:47 | |
| *** yamamoto has quit IRC | 13:02 | |
| *** gordc has quit IRC | 13:09 | |
| *** yamamoto has joined #openstack-oslo | 13:18 | |
| *** kgiusti has joined #openstack-oslo | 13:21 | |
| *** ansmith_ has joined #openstack-oslo | 13:38 | |
| *** ansmith has joined #openstack-oslo | 13:39 | |
| *** ansmith__ has joined #openstack-oslo | 13:40 | |
| *** ansmith has quit IRC | 13:40 | |
| *** ansmith_ has quit IRC | 13:41 | |
| *** ansmith__ has quit IRC | 13:41 | |
| *** sdague has quit IRC | 13:43 | |
| *** ansmith has joined #openstack-oslo | 13:44 | |
| *** aarefiev is now known as aarefiev_afk | 13:51 | |
| openstackgerrit | Abhishek Sharma M proposed openstack/pycadf master: Removed references to events api https://review.openstack.org/489892 | 13:54 |
| *** yamamoto has quit IRC | 13:55 | |
| *** sdague has joined #openstack-oslo | 13:56 | |
| -openstackstatus- NOTICE: We have disable infracloud-vanilla due to the compute host running mirror.regionone.infracloud-vanilla.o.o being offline. Please recheck your failed jobs to schedule them to another cloud. | 13:57 | |
| *** nicolasbock_ has joined #openstack-oslo | 14:04 | |
| *** nicolasbock__ has joined #openstack-oslo | 14:07 | |
| *** nicolasbock has quit IRC | 14:07 | |
| *** nicolasbock_ has quit IRC | 14:10 | |
| *** xiaoyang has quit IRC | 14:20 | |
| *** xiaoyang has joined #openstack-oslo | 14:20 | |
| *** nicolasbock_ has joined #openstack-oslo | 14:21 | |
| *** nicolasbock__ has quit IRC | 14:25 | |
| *** seba has joined #openstack-oslo | 14:26 | |
| *** janzian has joined #openstack-oslo | 14:33 | |
| *** ralonsoh_ has quit IRC | 14:39 | |
| *** gordc has joined #openstack-oslo | 14:41 | |
| *** yamamoto has joined #openstack-oslo | 14:55 | |
| openstackgerrit | Abhishek Sharma M proposed openstack/pycadf master: Removed references to events api https://review.openstack.org/489892 | 14:59 |
| *** yamamoto has quit IRC | 15:00 | |
| *** links has quit IRC | 15:01 | |
| *** e0ne has quit IRC | 15:16 | |
| *** nicolasbock__ has joined #openstack-oslo | 15:21 | |
| *** nicolasbock_ has quit IRC | 15:24 | |
| *** ralonsoh has joined #openstack-oslo | 15:31 | |
| *** sdague_ has quit IRC | 15:43 | |
| *** e0ne has joined #openstack-oslo | 15:46 | |
| *** ralonsoh has quit IRC | 15:47 | |
| *** ralonsoh has joined #openstack-oslo | 15:47 | |
| *** aselius has joined #openstack-oslo | 15:58 | |
| *** links has joined #openstack-oslo | 15:58 | |
| *** janzian has quit IRC | 15:59 | |
| *** ihrachys has joined #openstack-oslo | 16:00 | |
| *** sdague has quit IRC | 16:03 | |
| *** lpetrut has quit IRC | 16:07 | |
| *** rcernin has quit IRC | 16:08 | |
| *** Rockyg has joined #openstack-oslo | 16:09 | |
| *** pcaruana has quit IRC | 16:14 | |
| *** janzian has joined #openstack-oslo | 16:15 | |
| *** yassine has quit IRC | 16:20 | |
| *** lucasagomes is now known as lucas-afk | 16:28 | |
| *** hoonetorg has quit IRC | 16:30 | |
| *** sdague has joined #openstack-oslo | 16:40 | |
| *** hoonetorg has joined #openstack-oslo | 16:42 | |
| *** e0ne has quit IRC | 16:57 | |
| *** e0ne has joined #openstack-oslo | 16:58 | |
| *** rajathagasthya has joined #openstack-oslo | 16:58 | |
| *** e0ne has quit IRC | 16:58 | |
| *** ralonsoh has quit IRC | 17:00 | |
| *** tesseract has quit IRC | 17:07 | |
| *** harlowja has joined #openstack-oslo | 17:13 | |
| *** rajathagasthya has quit IRC | 17:30 | |
| *** sambetts is now known as sambetts|afk | 17:30 | |
| *** rajathagasthya has joined #openstack-oslo | 17:48 | |
| openstackgerrit | Merged openstack/pycadf master: Removed references to events api https://review.openstack.org/489892 | 18:02 |
| openstackgerrit | Merged openstack/pycadf master: Adding panko_api_audit_map.conf to pycadf https://review.openstack.org/489680 | 18:16 |
| *** Rockyg has quit IRC | 18:25 | |
| *** links has quit IRC | 18:31 | |
| *** ihrachys has quit IRC | 18:42 | |
| *** ihrachys has joined #openstack-oslo | 18:42 | |
| *** nicolasbock__ has quit IRC | 19:10 | |
| ihrachys | kgiusti, I know you were looking at handling NotFound raised on missing fanout exchange in oslo.messaging | 19:18 |
| ihrachys | please have a look at https://bugs.launchpad.net/nova/+bug/1707160 comment 16 | 19:18 |
| openstack | Launchpad bug 1707160 in OpenStack Compute (nova) "test_create_port_in_allowed_allocation_pools test fails on ironic grenade" [Undecided,New] | 19:18 |
| ihrachys | I think we may have more serious problems with NotFound than just a programmatic exception raised | 19:18 |
| kgiusti | ihrachys: thanks for the analysis - I'm blocked getting any patches through oslo.messaging CI due to https://review.openstack.org/#/c/488563/ | 19:22 |
| ihrachys | kgiusti, I am not saying you should fix it. I am more interested in someone just looking whether what we see suggests a oslo/rabbitmq issue and not ironic/nova/neutron | 19:23 |
| ihrachys | in neutron, we now catch NotFound for all cast calls | 19:23 |
| ihrachys | but it does seem to leave some issues with timeouts of channels used when NotFound was raised | 19:24 |
| kgiusti | ihrachys: let me take a look into oslo - see what if anything is done to the channel on NotFound... | 19:25 |
| *** e0ne has joined #openstack-oslo | 19:27 | |
| *** e0ne has quit IRC | 19:34 | |
| *** rajathagasthya has quit IRC | 19:35 | |
| *** rajathagasthya has joined #openstack-oslo | 19:38 | |
| *** sileht has quit IRC | 19:55 | |
| *** rajathagasthya has quit IRC | 20:00 | |
| *** e0ne has joined #openstack-oslo | 20:01 | |
| *** rajathagasthya has joined #openstack-oslo | 20:02 | |
| *** hoonetorg has quit IRC | 20:03 | |
| *** e0ne has quit IRC | 20:15 | |
| *** hoonetorg has joined #openstack-oslo | 20:20 | |
| kgiusti | ihrachys: ping - looks like the channel is not cleaned up for the generic exception path https://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo_messaging/_drivers/impl_rabbit.py#n810 | 20:23 |
| kgiusti | ihrachys: perhaps that needs to be done for NotFound? | 20:23 |
| *** ansmith has quit IRC | 20:25 | |
| ihrachys | kgiusti, is it the self._set_current_channel(None) call? | 20:26 |
| kgiusti | ihrachys: yep. But another question: when did these NotFound errors start manifesting? Kombu 4.1.0 recently rolled out.... | 20:27 |
| *** e0ne has joined #openstack-oslo | 20:27 | |
| ihrachys | kgiusti, in neutron, it started manifesting when we migrated our agents from one RPC fanout exchange to another | 20:27 |
| ihrachys | so no component now listens to some exchanges | 20:27 |
| ihrachys | in which case you get NotFound | 20:27 |
| *** raildo has quit IRC | 20:28 | |
| kgiusti | ihrachys: so the exchanges don't exist, and the fanout attempts to send to them, correct? | 20:28 |
| * kgiusti is looking for an easy reproducer.... | 20:29 | |
| ihrachys | yes | 20:29 |
| ihrachys | reproducer should be just send to a random topic | 20:29 |
| ihrachys | then you get NotFound | 20:29 |
| ihrachys | catch it and try to send something proper with the same connection | 20:29 |
| kgiusti | ihrachys: let me give that a try.... | 20:30 |
| ihrachys | kgiusti, having an RPCClient/_Context, is it possible to get to the Connection object to call that internal method resetting it? | 20:31 |
| *** https_GK1wmSU has joined #openstack-oslo | 20:35 | |
| kgiusti | ihrachys: it's a bit of a mess since the connection is selected from a pool on each call and not referenced from the transport | 20:36 |
| ihrachys | kevinbenton, fyi we may have broken api calls that trigger NotFound after which consequent attempts to use amqp by the same thread result in timeouts ^ | 20:37 |
| *** https_GK1wmSU has left #openstack-oslo | 20:37 | |
| ihrachys | kgiusti, looks like we can't work that around from neutron side while you are fixing it | 20:37 |
| kgiusti | ihrachys: probably not easily, no | 20:38 |
| ihrachys | kgiusti, I assume that even if we have a fix, we would need to release a new library meaning FFE for the new release? | 20:40 |
| kgiusti | ihrachys: Running a quick test here - sending a fanout to a non existent exchange/topic actually causes the exchange to be created on rabbitmq | 20:41 |
| kgiusti | ihrachys: so there must be some race where the exchange is deleted while the fanout is in-progress... | 20:42 |
| ihrachys | kgiusti, the trace http://logs.openstack.org/01/489701/1/check/gate-grenade-dsvm-ironic-ubuntu-xenial/50785e5/logs/new/screen-q-svc.txt.gz#_2017-08-01_22_48_58_434 | 20:42 |
| ihrachys | kgiusti, note it's grenade | 20:43 |
| ihrachys | and old agents used the topic | 20:43 |
| ihrachys | so MAYBE it was created by older agent; then it expired (?) and was removed in the middle of fanout? | 20:43 |
| ihrachys | rather rare scenario I would assume | 20:43 |
| ihrachys | because we would need to catch the exact moment of expiration | 20:44 |
| kgiusti | ihrachys: rare yes, but it sounds like you're hitting quite often. | 20:44 |
| ihrachys | yep, they say it's ~50% runs | 20:44 |
| ihrachys | ofc there are multiple tests to fail on | 20:44 |
| ihrachys | so it shouldn't necessarily be every second api request :) | 20:45 |
| ihrachys | kgiusti, also it looks like we have same topic used before as well as after the failure with no issue | 20:48 |
| ihrachys | so maybe we indeed hit some race between cleanup after old agents and creating a new one before fanout? | 20:49 |
| kgiusti | ihrachys: perhaps - I'm trying to hack out the creation of the exchange to see if it fails in a similar manner.... | 20:50 |
| *** e0ne has quit IRC | 20:52 | |
| ihrachys | kgiusti, I checked, and this happens ~30 minutes after upgrade of neutron-server | 20:54 |
| ihrachys | ~26 minutes in new neutron-server run | 20:54 |
| ihrachys | it does correlate with rabbit_transient_queues_ttl = 30 minutes | 20:54 |
| ihrachys | (it was 10 minutes but then bumped to 30 minutes in N) | 20:56 |
| ihrachys | I updated the bug https://bugs.launchpad.net/nova/+bug/1707160 with gist of the discussion above | 21:02 |
| openstack | Launchpad bug 1707160 in neutron "test_create_port_in_allowed_allocation_pools test fails on ironic grenade" [Critical,Confirmed] - Assigned to Ihar Hrachyshka (ihar-hrachyshka) | 21:02 |
| ihrachys | I will need to drop in 5 mins | 21:02 |
| kgiusti | ihrachys: yeah me also, but to confirm I get a NOT FOUND if I hack out the exchange creation during the fanout send... | 21:04 |
| ihrachys | where is the code creating the missing topic? | 21:04 |
| kgiusti | https://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo_messaging/_drivers/impl_rabbit.py#n1279 | 21:05 |
| kgiusti | ihrachys: that creates the exchange, then it is bound to the channel here: https://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo_messaging/_drivers/impl_rabbit.py#n1172 | 21:06 |
| kgiusti | ihrachys: removing that .declare() and I can simulate the not found | 21:06 |
| ihrachys | I see. this declare() probably doesn't bump expiration window for existing topic | 21:07 |
| ihrachys | sorry, exchange | 21:07 |
| kgiusti | ihrachys: it's possible also that the list of _declared_exchanges is stale | 21:08 |
| *** yamamoto_ has joined #openstack-oslo | 21:08 | |
| kgiusti | ihrachys: which would certainly be the case if the exchange was deleted at some point | 21:08 |
| kgiusti | ihrachys: and clearing the channel does reset that cache, more evidence! | 21:09 |
| kgiusti | ihrachys: we're not setting channel to None when NotFound hits.... | 21:09 |
| ihrachys | kgiusti, you can push a change; I only wonder if we can test ironic grenade gate before committing it | 21:10 |
| ihrachys | kgiusti, since oslo.msg is consumed via reqs... | 21:10 |
| kgiusti | ihrachys: never done that before - perhaps infra can help us here? | 21:11 |
| ihrachys | I will need to drop now; if you don't reach them till tomorrow, I will chase that. if you can push a WIP patch to test against till then, that would be great. | 21:12 |
| ihrachys | that's probably one of the blocker issues we will need to solve somehow for neutron this cycle | 21:12 |
| ihrachys | and reverting the changes that triggered it on neutron side is a PITA | 21:12 |
| ihrachys | a huge PITA | 21:13 |
| kgiusti | ihrachys: Let's avoid the huge PITA | 21:13 |
| ihrachys | it will result in like 5-10 patches reverted :) | 21:13 |
| ihrachys | yeah, that's why I really want the oslo.msg fix | 21:13 |
| ihrachys | chasing a FFE should be easier | 21:13 |
| kgiusti | ihrachys: I can push to a branch when I have a simple candidate | 21:13 |
| ihrachys | ok, thanks | 21:14 |
| ihrachys | let's sync tomorrow | 21:14 |
| kgiusti | ihrachys: and the WIP patch too | 21:14 |
| kgiusti | ihrachys: see you tomorrow | 21:14 |
| ihrachys | kevinbenton, I highly suggest you to read that ^; it's release affecting. | 21:14 |
| *** yamamoto_ has quit IRC | 21:15 | |
| *** yamamoto_ has joined #openstack-oslo | 21:17 | |
| *** lpetrut has joined #openstack-oslo | 21:21 | |
| *** janzian has quit IRC | 21:34 | |
| kevinbenton | ihrachys: doesn't your general patch cover us if that doesn't make it? | 21:49 |
| openstackgerrit | Ken Giusti proposed openstack/oslo.messaging master: Prevent rabbit from raising kombu or py-amqp exceptions https://review.openstack.org/486706 | 21:49 |
| *** kgiusti has left #openstack-oslo | 21:53 | |
| *** lpetrut has quit IRC | 22:14 | |
| *** ansmith has joined #openstack-oslo | 22:15 | |
| *** https_GK1wmSU has joined #openstack-oslo | 22:19 | |
| *** https_GK1wmSU has left #openstack-oslo | 22:21 | |
| *** gordc has quit IRC | 22:28 | |
| *** yamamoto_ has quit IRC | 22:29 | |
| *** yamamoto has joined #openstack-oslo | 22:32 | |
| *** yamamoto has quit IRC | 22:36 | |
| *** yamamoto has joined #openstack-oslo | 22:41 | |
| *** yamamoto has quit IRC | 22:44 | |
| *** sdague has quit IRC | 22:50 | |
| *** yamamoto has joined #openstack-oslo | 22:56 | |
| *** yassine has joined #openstack-oslo | 23:02 | |
| *** gtrxcb has joined #openstack-oslo | 23:06 | |
| *** yamamoto has quit IRC | 23:11 | |
| *** yamamoto has joined #openstack-oslo | 23:14 | |
| *** rajathagasthya has quit IRC | 23:29 | |
| *** ansmith has quit IRC | 23:54 | |
| *** rajathagasthya has joined #openstack-oslo | 23:54 | |
| *** ansmith has joined #openstack-oslo | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!