| *** fnaval has joined #openstack-lbaas | 00:03 | |
| *** fnaval has quit IRC | 00:04 | |
| rm_work | err | 00:15 |
|---|---|---|
| *** Emine has joined #openstack-lbaas | 00:15 | |
| rm_work | Hmm actually not sure, I had my levels set very explicitly for literally every log source | 00:16 |
| rm_work | But I think that's not normal | 00:16 |
| *** Emine has quit IRC | 00:27 | |
| colin- | be happy to check which params just not sure what to look at | 00:48 |
| colin- | we aren't overriding very much in config for log settings | 00:49 |
| *** yamamoto has joined #openstack-lbaas | 00:51 | |
| *** yamamoto has quit IRC | 00:56 | |
| *** fyx has joined #openstack-lbaas | 00:59 | |
| *** icey has joined #openstack-lbaas | 00:59 | |
| *** coreycb has joined #openstack-lbaas | 00:59 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Refactor the pluggin of the VIP https://review.openstack.org/604479 | 01:15 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add amphora agent configuration update admin API https://review.openstack.org/632842 | 01:30 |
| *** Dinesh_Bhor has joined #openstack-lbaas | 01:49 | |
| *** yamamoto has joined #openstack-lbaas | 01:53 | |
| *** hongbin has joined #openstack-lbaas | 02:29 | |
| *** sapd1 has joined #openstack-lbaas | 02:47 | |
| *** yamamoto has quit IRC | 02:48 | |
| *** yamamoto has joined #openstack-lbaas | 03:14 | |
| *** sapd1 has quit IRC | 03:59 | |
| *** sapd1 has joined #openstack-lbaas | 03:59 | |
| *** ramishra has joined #openstack-lbaas | 04:06 | |
| *** AlexStaf has joined #openstack-lbaas | 04:56 | |
| *** hongbin has quit IRC | 05:30 | |
| *** Dinesh_Bhor has quit IRC | 06:05 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 06:09 | |
| *** AlexStaf has quit IRC | 06:12 | |
| *** gcheresh has joined #openstack-lbaas | 06:17 | |
| *** AlexStaf has joined #openstack-lbaas | 06:33 | |
| *** ccamposr has joined #openstack-lbaas | 07:04 | |
| *** Emine has joined #openstack-lbaas | 07:32 | |
| *** ramishra_ has joined #openstack-lbaas | 07:37 | |
| *** ramishra has quit IRC | 07:38 | |
| *** Emine has quit IRC | 07:49 | |
| *** pckizer has quit IRC | 07:51 | |
| openstackgerrit | Reedip proposed openstack/octavia-tempest-plugin master: Check Monitor in Member Scenario Tests https://review.openstack.org/634891 | 07:55 |
| *** ramishra_ is now known as ramishra | 07:56 | |
| *** velizarx has joined #openstack-lbaas | 08:07 | |
| *** rpittau has joined #openstack-lbaas | 08:13 | |
| *** velizarx has quit IRC | 08:38 | |
| *** yboaron_ has joined #openstack-lbaas | 08:44 | |
| *** velizarx has joined #openstack-lbaas | 09:04 | |
| *** Emine has joined #openstack-lbaas | 09:15 | |
| *** ramishra has quit IRC | 09:22 | |
| *** Emine has quit IRC | 09:26 | |
| *** Emine has joined #openstack-lbaas | 09:26 | |
| *** yboaron_ has quit IRC | 09:26 | |
| *** yboaron_ has joined #openstack-lbaas | 09:27 | |
| *** ramishra has joined #openstack-lbaas | 09:29 | |
| *** Dinesh_Bhor has quit IRC | 10:32 | |
| *** salmankhan has joined #openstack-lbaas | 10:41 | |
| *** sapd1 has quit IRC | 10:45 | |
| *** salmankhan1 has joined #openstack-lbaas | 11:08 | |
| *** Emine has quit IRC | 11:09 | |
| *** salmankhan has quit IRC | 11:09 | |
| *** salmankhan1 is now known as salmankhan | 11:09 | |
| *** Emine has joined #openstack-lbaas | 11:10 | |
| *** velizarx has quit IRC | 11:34 | |
| devfaz | Hi there, is someone already working on "provider driver"-support? | 11:36 |
| *** velizarx has joined #openstack-lbaas | 12:26 | |
| cgoncalves | devfaz, provider support is implemented in octavia from rocky | 12:26 |
| *** emine__ has joined #openstack-lbaas | 12:28 | |
| *** velizarx has quit IRC | 12:30 | |
| *** Emine has quit IRC | 12:30 | |
| *** coreycb has quit IRC | 12:36 | |
| *** coreycb has joined #openstack-lbaas | 12:37 | |
| *** velizarx has joined #openstack-lbaas | 12:38 | |
| devfaz | cgoncalves: thanks a lot! Sorry for not reading carefully the release note :( | 12:41 |
| *** sapd1 has joined #openstack-lbaas | 12:44 | |
| cgoncalves | no problem! | 12:47 |
| *** emine__ has quit IRC | 12:50 | |
| *** mkuf_ has joined #openstack-lbaas | 12:55 | |
| *** mkuf has quit IRC | 12:58 | |
| *** yamamoto has quit IRC | 13:03 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 13:04 | |
| *** yamamoto has joined #openstack-lbaas | 13:08 | |
| *** trown|outtypewww is now known as trown | 13:11 | |
| *** yamamoto has quit IRC | 13:15 | |
| *** velizarx has quit IRC | 13:32 | |
| *** mkuf_ has quit IRC | 13:37 | |
| *** velizarx has joined #openstack-lbaas | 13:37 | |
| *** openstackgerrit has quit IRC | 13:37 | |
| *** yamamoto has joined #openstack-lbaas | 13:46 | |
| *** yamamoto has quit IRC | 13:56 | |
| *** mkuf_ has joined #openstack-lbaas | 14:01 | |
| *** yamamoto has joined #openstack-lbaas | 14:06 | |
| *** yamamoto has quit IRC | 14:06 | |
| *** sapd1 has quit IRC | 14:15 | |
| *** Dinesh_Bhor has quit IRC | 14:18 | |
| *** fnaval has joined #openstack-lbaas | 14:20 | |
| *** AlexStaf has quit IRC | 14:57 | |
| *** sapd1 has joined #openstack-lbaas | 15:02 | |
| *** ianychoi has quit IRC | 15:07 | |
| *** ianychoi has joined #openstack-lbaas | 15:07 | |
| *** yamamoto has joined #openstack-lbaas | 15:16 | |
| *** yamamoto has quit IRC | 15:25 | |
| *** sapd1 has quit IRC | 15:32 | |
| *** gcheresh has quit IRC | 15:50 | |
| colin- | so any tips on threads to chase for this behavior we were discussing yesterday? if not that's fine just feeling a bit aimless at the moment having not found those logs we discussed in my output | 15:50 |
| johnsom | Well, this is a tricky one. If I had the issue, I would probably try to identify on of the "busy" threads/processes and attempt to attach a python debugger to it and see if I can identify what it is doing. | 15:53 |
| johnsom | If that gives me nothing, I might hack on oslo messaging to have it spit out more information about it's connection management to make sure it's really closing things out properly. | 15:54 |
| johnsom | If it can be reproduced in a lab by repeated actions/high change volume, using something like rally, I probably would go into the code and disable the messaging all together and run again. This would help point the finger at sqlalchemy or oslo messaging as the problem area. | 15:55 |
| colin- | good advice, thanks. | 15:56 |
| colin- | can't really operate it with confidence if we don't pin this down i don't feel like | 15:56 |
| colin- | not being able to account for its behavior predictably i mean | 15:57 |
| johnsom | I dug into the oslo messaging code a bit yesterday, it's really hiding the connection management from the user. I would have to spend a bunch of time running it down there further. | 15:57 |
| johnsom | Yep, I am a strong believer in if it's a minor problem today, it will cause more headaches tomorrow. | 15:58 |
| *** yboaron_ has quit IRC | 16:04 | |
| cgoncalves | tomorrow is fine :P | 16:06 |
| colin- | nice blog xgerman had a definite "get off my lawn!" vibe to it :) | 16:07 |
| xgerman | :-) | 16:08 |
| colin- | anwyay your point was well made, we are definitely shifting to a disposable style of infrastructure | 16:08 |
| xgerman | yeah, it’s not really disposable; it’s just that we keep pushing stuff up the stack… | 16:12 |
| *** ccamposr has quit IRC | 16:42 | |
| *** ccamposr has joined #openstack-lbaas | 16:44 | |
| *** ccamposr__ has joined #openstack-lbaas | 16:45 | |
| *** ccamposr has quit IRC | 16:48 | |
| *** ccamposr__ has quit IRC | 16:53 | |
| *** ramishra has quit IRC | 16:57 | |
| *** sapd1 has joined #openstack-lbaas | 17:04 | |
| *** rpittau has quit IRC | 17:08 | |
| *** velizarx has quit IRC | 17:11 | |
| *** sapd1 has quit IRC | 17:14 | |
| *** yamamoto has joined #openstack-lbaas | 17:44 | |
| *** yamamoto has quit IRC | 17:48 | |
| *** salmankhan has quit IRC | 18:03 | |
| *** yamamoto has joined #openstack-lbaas | 18:11 | |
| colin- | heh, found ~6,000,000 amqp connections being held open by the api processes in lsof output | 18:23 |
| colin- | (the actual amqp server obviously does not recognize nearly that many active conns) | 18:23 |
| colin- | for comparison, the other control plane host whose services were restarted ~20h ago has ~650,000 | 18:25 |
| colin- | will continue to explore but wanted to share | 18:25 |
| eandersson | johnsom, rm_work ^ pretty sure that is why the load is crazy high | 18:28 |
| johnsom | Yeah, that was my suspicion, something fishy with oslo messaging | 18:29 |
| johnsom | Thank you for looking into this! | 18:30 |
| *** jlaffaye has quit IRC | 18:37 | |
| *** trown is now known as trown|lunch | 18:41 | |
| *** jlaffaye has joined #openstack-lbaas | 18:51 | |
| eandersson | johnsom, the pattern you use for rabbitmq is very different from everyone elses | 19:00 |
| eandersson | everyone else uses global to define it, while you define it in BaseProdfucer | 19:02 |
| *** mugsie has quit IRC | 19:04 | |
| *** tomtom001 has quit IRC | 19:04 | |
| *** mugsie has joined #openstack-lbaas | 19:05 | |
| eandersson | Pretty sure that whole thing is flawed | 19:07 |
| eandersson | https://github.com/openstack/nova/blob/05d3c01a685de45dd8f98a46bc5c52f6bd08e7cd/nova/rpc.py | 19:08 |
| johnsom | We aren't really using that producer code anymore, it's just for v1, but the v2 code is similar | 19:10 |
| eandersson | ah I see | 19:14 |
| eandersson | Pretty sure it creats multiple instances of transport, when there should only be one per worker | 19:15 |
| eandersson | AmphoraProviderDriver ? | 19:18 |
| eandersson | I'll throw up a WIP patch on what I think is the problem | 19:23 |
| *** abaindur has joined #openstack-lbaas | 19:24 | |
| *** openstackgerrit has joined #openstack-lbaas | 19:25 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/octavia master: [WIP] Re-use oslo messaging transport https://review.openstack.org/636428 | 19:25 |
| eandersson | The above is super wip ^ | 19:26 |
| johnsom | So each request comes in it's own thread, so each establishes a connection. Oslo has problems if the connection is open before it's spun out (at least per their docs). So each request should setup it's transport then tear it down after the request is handled. | 19:27 |
| eandersson | It's just odd as no one else has this problem and we are running lots of Rocky | 19:34 |
| eandersson | So while it can be a oslo bug of course, I suspect it's maybe triggered by your unorthodox pattern | 19:35 |
| eandersson | but we just started doing research on this, so I could be completely off | 19:35 |
| *** trown|lunch is now known as trown | 19:44 | |
| *** kklimonda_ has joined #openstack-lbaas | 20:28 | |
| openstackgerrit | Merged openstack/octavia master: Add amphora agent configuration update admin API https://review.openstack.org/632842 | 20:29 |
| johnsom | eandersson I think that nova code is the "old" way. Those aren't even documented in oslo messaging anymore. | 20:39 |
| johnsom | Plus they are using notification and not the RPC messaging | 20:40 |
| johnsom | But it's hard to say. the oslo messaging docs are rubbish as they say | 20:41 |
| eandersson | Yep | 20:59 |
| eandersson | I agree on that :D | 20:59 |
| johnsom | Yeah, if that works, making the transport global, that would be awesome. | 21:01 |
| johnsom | I wasn't involved the messaging when it was added to octavia. I think there was some concern about this warning: https://docs.openstack.org/oslo.messaging/latest/reference/transport.html#forking-processes-and-oslo-messaging-transport-objects | 21:02 |
| johnsom | But I have no idea. If it works, great. | 21:02 |
| *** mugsie has quit IRC | 21:06 | |
| *** mugsie has joined #openstack-lbaas | 21:06 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Skip amphora scenario test if provider is not set https://review.openstack.org/636436 | 21:10 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/octavia master: [WIP] Re-use oslo messaging transport https://review.openstack.org/636428 | 21:17 |
| *** trown is now known as trown|outtypewww | 21:36 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/octavia master: [WIP] Re-use oslo messaging transport https://review.openstack.org/636428 | 21:46 |
| eandersson | johnsom, still not sure if this ^ is actually the fix - gonna test in a lab soonTM | 21:52 |
| johnsom | Cool, thanks. It seemed like you were working on it, so put one review comment in. | 21:52 |
| eandersson | Yep appriciate it! | 21:53 |
| eandersson | Less work for me to do later | 21:53 |
| rm_work | yeah eandersson / johnsom , that is basically exactly what i was saying too -- i mentioned months ago that i was seeing WAAAAY too many held rmq connections, <_< | 21:59 |
| rm_work | but i couldn't figure out why it wasn't closing them, and as you said, the docs are confusing rubbish | 21:59 |
| rm_work | i am 99% sure our pattern is broken SOMEHOW | 22:00 |
| rm_work | i wonder if we could convince someone who is actually working on oslo.messaging to take a look | 22:00 |
| colin- | would sure welcome some eyes that are more used to oslo things, have been leaning heavily on eandersson for help on this :) | 22:00 |
| colin- | (ty!) | 22:01 |
| johnsom | It looked like the proposed change was working (I.e. didn't break us) so this is likely a very good path. | 22:01 |
| kklimonda_ | hmm, when deploying octavia, how should health monitor be setup to receive health checks on port 5555 from amphorae? If it's configured on the internal API interface, amphorae have no routing.. should it be listening on the interface used by worker to communicate with amphora? | 22:01 |
| johnsom | Well, as your oslo liaison, I can raise it at the next oslo meeting (Monday). BTW, that position is open if someone is interested. | 22:01 |
| johnsom | kklimonda_ Hi, yes, the HM should l have a path, routed or not, from the amphroa so they can send the health heartbeats. Putting them on the same interface as your worker is one option. | 22:03 |
| kklimonda_ | johnsom: is there a more... standard way, or is it simply deployment dependent and my solution is ok as long as it works? | 22:04 |
| rm_work | kinda deployment dependent I think | 22:04 |
| rm_work | I put them on the VIP network :P | 22:05 |
| rm_work | (i put everything on the VIP network) :P | 22:05 |
| colin- | i have a separate network for it | 22:05 |
| johnsom | It's up to the deployment. OpenStack Ansible uses provider networks, OSP13 uses bridges, etc. | 22:05 |
| rm_work | and the VIP network in my case was a provider network | 22:05 |
| colin- | i even considered a non-routed network where the controllers have an inetrface on a common VLAN with the amps | 22:06 |
| johnsom | That is the devstack/OSP way | 22:06 |
| johnsom | finally off of video conferences and can get back to work.... Tuesdays... | 22:08 |
| rm_work | same lol | 22:24 |
| johnsom | rm_work quick question. I am thinking we should not allow a TLS client cert CA to be loaded if there is no TLS cert loaded on a listener. Do you agree? | 22:45 |
| rm_work | uhh | 22:46 |
| rm_work | how would that happen | 22:46 |
| rm_work | and/or, wait, what | 22:46 |
| johnsom | Somebody only specifying a client CA ref when creating or updating a listener | 22:46 |
| johnsom | TLS client authentication | 22:47 |
| rm_work | for the backend members? | 22:47 |
| rm_work | oh | 22:47 |
| johnsom | No, the VIP. Allowing the VIP to require/request a client cert | 22:47 |
| rm_work | ah yeah | 22:47 |
| rm_work | i get it | 22:47 |
| johnsom | I'm reviewing the TLS patch chain | 22:47 |
| rm_work | and if the listener isn't TLS type | 22:48 |
| rm_work | ... | 22:48 |
| rm_work | <_< | 22:48 |
| rm_work | yeah it just does nothing | 22:48 |
| rm_work | i'm not sure if it's ... harmful, exactly | 22:48 |
| rm_work | but it doesn't make any sense | 22:48 |
| johnsom | Yeah, we would have to filter it at the haproxy jinja2 if we don't just outright block it at API validation. | 22:48 |
| rm_work | API IMO | 22:49 |
| johnsom | I just wanted to make sure I wasn't forgetting some reason we would ever allow a client CA but not have a TLS cert | 22:49 |
| johnsom | +1 | 22:49 |
| rm_work | it's just useless | 22:49 |
| rm_work | but yeah, if the type of the listener isn't TLS, then 400 | 22:49 |
| *** fnaval has quit IRC | 23:15 | |
| eandersson | Deployed my patch to the lab, gonna see tomorrow if it worked or not. | 23:25 |
| *** fnaval has joined #openstack-lbaas | 23:31 | |
| rm_work | kk, crossing my fingers | 23:55 |
| rm_work | i will try to review today or tomorrow | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!