| rm_work | well had to loose carlos' review again for a rebase | 00:29 |
|---|---|---|
| rm_work | but https://review.opendev.org/#/c/740815/ is good johnsom :) | 00:29 |
| rm_work | If you +2 I might merge it because it's really mostly their work not mine, and carlos definitely +2'd it enough lol | 00:29 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Refactoring amphora stats driver interface https://review.opendev.org/737111 | 00:47 |
| *** gmann has quit IRC | 00:51 | |
| *** gmann has joined #openstack-lbaas | 00:51 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Refactoring amphora stats driver interface https://review.opendev.org/737111 | 01:31 |
| *** ramishra has joined #openstack-lbaas | 01:53 | |
| *** sapd1 has joined #openstack-lbaas | 02:33 | |
| *** rcernin has quit IRC | 02:59 | |
| *** hongbin has joined #openstack-lbaas | 03:00 | |
| *** rcernin has joined #openstack-lbaas | 03:03 | |
| *** armax has quit IRC | 03:33 | |
| sorrison | johnsom: OK I'll put up a story, any tips on how I can get this LB back to a healthy state? | 04:19 |
| *** hongbin has quit IRC | 04:20 | |
| *** vishalmanchanda has joined #openstack-lbaas | 04:22 | |
| sorrison | dam, the user has deleted the LB so I can't replicate this | 04:26 |
| *** gcheresh has joined #openstack-lbaas | 04:56 | |
| *** sapd1 has quit IRC | 05:04 | |
| *** sapd1 has joined #openstack-lbaas | 05:47 | |
| *** lxkong has quit IRC | 05:55 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 06:10 | |
| *** sapd1 has quit IRC | 06:28 | |
| *** sapd1 has joined #openstack-lbaas | 07:22 | |
| *** ccamposr__ has joined #openstack-lbaas | 07:26 | |
| *** ccamposr has quit IRC | 07:29 | |
| *** rcernin has quit IRC | 07:32 | |
| openstackgerrit | zhufl proposed openstack/octavia master: [Trivial]Add missing space between words https://review.opendev.org/743476 | 07:35 |
| *** rcernin has joined #openstack-lbaas | 08:48 | |
| *** rcernin has quit IRC | 08:54 | |
| *** ataraday_ has joined #openstack-lbaas | 08:58 | |
| *** wuchunyang has joined #openstack-lbaas | 08:59 | |
| *** born2bake has joined #openstack-lbaas | 09:15 | |
| *** ramishra has quit IRC | 10:15 | |
| *** rcernin has joined #openstack-lbaas | 10:19 | |
| *** wuchunyang has quit IRC | 10:21 | |
| *** rcernin has quit IRC | 10:43 | |
| *** wuchunyang has joined #openstack-lbaas | 10:53 | |
| *** ramishra has joined #openstack-lbaas | 10:58 | |
| *** wuchunyang has quit IRC | 11:05 | |
| *** sapd1 has quit IRC | 11:07 | |
| *** wuchunyang has joined #openstack-lbaas | 11:12 | |
| *** wuchunyang has quit IRC | 11:23 | |
| *** wuchunyang has joined #openstack-lbaas | 11:23 | |
| *** spatel has joined #openstack-lbaas | 11:50 | |
| *** spatel has quit IRC | 11:54 | |
| *** tkajinam has quit IRC | 12:02 | |
| *** wuchunyang has quit IRC | 12:04 | |
| *** sapd1 has joined #openstack-lbaas | 12:20 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Fix amphora image build jobs https://review.opendev.org/743520 | 12:27 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Fix amphora image build jobs https://review.opendev.org/743520 | 12:42 |
| *** sapd1 has quit IRC | 12:53 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Fix amphora image build jobs https://review.opendev.org/743520 | 12:58 |
| *** rcernin has joined #openstack-lbaas | 12:59 | |
| *** rcernin has quit IRC | 13:03 | |
| *** ramishra has quit IRC | 13:27 | |
| *** TrevorV has joined #openstack-lbaas | 13:36 | |
| *** ramishra has joined #openstack-lbaas | 13:39 | |
| *** ataraday_ has quit IRC | 13:43 | |
| *** ccamposr__ has quit IRC | 13:44 | |
| *** ccamposr has joined #openstack-lbaas | 13:56 | |
| *** ramishra has quit IRC | 13:59 | |
| *** ramishra has joined #openstack-lbaas | 14:00 | |
| *** maciejjozefczyk has quit IRC | 14:03 | |
| *** sapd1 has joined #openstack-lbaas | 14:12 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 14:46 | |
| *** armax has joined #openstack-lbaas | 14:47 | |
| *** TrevorV has quit IRC | 15:06 | |
| *** TrevorV has joined #openstack-lbaas | 15:13 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Fix amphora image build jobs https://review.opendev.org/743520 | 15:14 |
| johnsom | Great, looks like the gates might be broken this morning: Unable to find interface with MAC: fa:16:3e:37:09:cd | 15:17 |
| cgoncalves | try finding with a PC *ba dum tss* | 15:18 |
| * johnsom groans | 15:18 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Fix amphora image build jobs https://review.opendev.org/743520 | 15:21 |
| xgerman | johnsom: cgoncalves rm_work Just a reminder that we are on with the Happy hour tomorrow after the meeting... | 15:26 |
| *** maciejjozefczyk has quit IRC | 15:27 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Fix amphora image build jobs https://review.opendev.org/743520 | 15:42 |
| cgoncalves | xgerman, confirmed | 15:43 |
| *** wuchunyang has joined #openstack-lbaas | 16:05 | |
| *** wuchunyang has quit IRC | 16:10 | |
| *** gcheresh has quit IRC | 16:18 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 16:22 | |
| *** maciejjozefczyk has quit IRC | 16:36 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 16:41 | |
| *** shtepanie has joined #openstack-lbaas | 16:43 | |
| *** maciejjozefczyk has quit IRC | 17:01 | |
| *** ccamposr has quit IRC | 17:12 | |
| johnsom | sigh | 17:15 |
| johnsom | os_vif nova.exception.Resched | 17:15 |
| johnsom | uledException: Build of instance 0343da1a-e765-492b-a887-78af223379dc was re-sch | 17:15 |
| johnsom | eduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFOpenVSw | 17:15 |
| johnsom | itch(active=False,address=fa:16:3e:43:a0:e9,bridge_name='br-int',has_traffic_fil | 17:15 |
| johnsom | tering=True,id=16776f34-a9e6-4c82-98ae-d77ae1fac525,network=Network(e77c1e43-9dd | 17:15 |
| johnsom | f-49b5-b432-edbe73660cda),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,pr | 17:15 |
| johnsom | eserve_on_delete=False,vif_name='tap16776f34-a9'). Got error: a bytes-like objec | 17:15 |
| johnsom | t is required, not 'str' | 17:15 |
| johnsom | I think this old VM didn't have python3 in localrc, so maybe that is why it's blowing up. | 17:17 |
| *** redrobot has quit IRC | 17:41 | |
| *** sapd1 has quit IRC | 17:42 | |
| *** jamesden_ has joined #openstack-lbaas | 18:27 | |
| *** jamesdenton has quit IRC | 18:30 | |
| *** wuchunyang has joined #openstack-lbaas | 19:30 | |
| *** wuchunyang has quit IRC | 19:34 | |
| johnsom | So it looks like the latest pyroute2 broke ipr.link_lookup | 19:37 |
| johnsom | Digging into it | 19:37 |
| johnsom | yeah, so the 0.5.13 pyroute2 release broke us. Neutron needed a fix in it. I have opened a bug: https://github.com/svinota/pyroute2/issues/724 | 20:17 |
| johnsom | I plan to wait until tomorrow to see if they push a fix before proposing the revert in requirements, etc. | 20:19 |
| *** TrevorV has quit IRC | 20:20 | |
| *** gcheresh has joined #openstack-lbaas | 20:20 | |
| *** vishalmanchanda has quit IRC | 20:41 | |
| rm_work | kk | 20:43 |
| johnsom | If you need to work around it in an image, revert this change: https://github.com/svinota/pyroute2/commit/3e0b8bd2a34a188f41c465d5313a5dc9fee8db2f | 20:47 |
| *** lxkong has joined #openstack-lbaas | 21:03 | |
| *** shtepanie has quit IRC | 21:03 | |
| *** gcheresh has quit IRC | 21:05 | |
| *** ccamposr has joined #openstack-lbaas | 21:28 | |
| mchlumsky | Hello! While reading this comment: https://storyboard.openstack.org/#!/story/2007022#comment-148786 I am wondering what is meant by "manual recovery" in "ERROR states will not block deletes and manual recovery" ? | 21:42 |
| johnsom | Objects in ERROR can be deleted and/or failed over. ERROR does not block those actions | 21:43 |
| *** redrobot has joined #openstack-lbaas | 21:43 | |
| rm_work | johnsom: have a sec? | 21:46 |
| johnsom | Yeah | 21:47 |
| johnsom | Just futzing with tempest stuff | 21:47 |
| mchlumsky | Yes, that part I got, thanks :) but the sentence makes it sound like there is something else called a "manual recovery" (I think I was thinking too hard while reading that). Deletes and failovers should be sufficient for most cases. | 21:47 |
| johnsom | Yeah, manual in this case is running a failover via the API as opposed to the controllers automatic failover functions, which evidently gave up if the object is in ERROR | 21:48 |
| rm_work | johnsom: so in our stats driver we want to use the loadbalancer_id and project_id as well | 21:49 |
| rm_work | but I don't know if we're going to want to look up the listener from the DB every event and pull that | 21:49 |
| rm_work | doesn't it exist on the amphora somewhere? | 21:49 |
| rm_work | could we ... push that back in the UDP message? :/ | 21:49 |
| rm_work | or am i missing something obvious | 21:49 |
| johnsom | Umm, so, the listener IDs are already in the message, so are you specifically talking about lb and project ID or listener? | 21:51 |
| mchlumsky | I'm glad I found that story. Tweaking the lengthy timeouts will no doubt save me a lot of pain in the future. Thanks johnsom ! | 21:52 |
| rm_work | right we have listener_id, we would have to look up the listener object from the DB and pull lb_id and project_id off it | 21:54 |
| rm_work | but that's an additional trip to the DB for every health message <_< | 21:55 |
| johnsom | Yeah. That is probably the best option. We do store some of that in the amp, but adding those to the message will bulk it up unnecessarily. | 21:55 |
| rm_work | hmm k | 21:55 |
| johnsom | Yeah, which is what every 10 seconds and reading from a RAM cache? | 21:55 |
| rm_work | i guess maybe we could cache it, since project_id and loadbalancer_id for a listener are immutable | 21:55 |
| rm_work | so you'd just let mysql cache it and not worry about a local cache? | 21:56 |
| rm_work | I worry about scale when we add another SQL query for every UDP health message | 21:56 |
| johnsom | Personally, yes. It's a very simple read only query against an indexed column. You wouldn't even need a join. | 21:56 |
| rm_work | ok | 22:02 |
| rm_work | I mean we'd be using the ListenerRepository | 22:02 |
| rm_work | so ...... who knows what will actually happen | 22:02 |
| rm_work | (you might know) | 22:02 |
| rm_work | (I certainly do not) | 22:03 |
| johnsom | Yeah, a get there should be fine | 22:03 |
| rm_work | k | 22:03 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Refactoring amphora stats driver interface https://review.opendev.org/737111 | 22:05 |
| *** rcernin has joined #openstack-lbaas | 22:09 | |
| *** rcernin has quit IRC | 22:22 | |
| *** rcernin has joined #openstack-lbaas | 22:37 | |
| *** rcernin has quit IRC | 22:50 | |
| *** rcernin has joined #openstack-lbaas | 22:50 | |
| *** tkajinam has joined #openstack-lbaas | 22:52 | |
| *** born2bake has quit IRC | 23:11 | |
| *** wuchunyang has joined #openstack-lbaas | 23:30 | |
| *** wuchunyang has quit IRC | 23:35 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!