| *** yamamoto has quit IRC | 00:41 | |
| *** yamamoto has joined #openstack-lbaas | 00:44 | |
| *** yamamoto has quit IRC | 00:48 | |
| *** takamatsu_ has joined #openstack-lbaas | 01:26 | |
| *** takamatsu has quit IRC | 01:26 | |
| *** yamamoto has joined #openstack-lbaas | 01:41 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: L7rule support client certificate cases https://review.openstack.org/612271 | 01:46 |
|---|---|---|
| openstackgerrit | caoyuan proposed openstack/octavia-tempest-plugin master: Update json module to jsonutils https://review.openstack.org/638332 | 02:04 |
| *** Dinesh_Bhor has joined #openstack-lbaas | 02:04 | |
| eandersson | johnsom, that gate error is an odd one for stable/rocky | 02:08 |
| eandersson | http://logs.openstack.org/54/638454/1/check/octavia-grenade/54ceb48/logs/screen-o-cw.txt.gz | 02:08 |
| eandersson | pretty sure the rocky gate is just broken | 02:10 |
| rm_work | ah i remember there were some issues we were seeing in the grenade test, might be the same | 02:10 |
| rm_work | not sure if cgoncalves actually fixed that yet (or was working on it) | 02:11 |
| eandersson | Out of interest where is that Internal Error coming from? | 02:11 |
| eandersson | > self.client.plug_network(amphora, port_info) | 02:11 |
| rm_work | it's the 500 from the api API | 02:11 |
| eandersson | Reading the code I couldn't understand which api it is hitting | 02:11 |
| rm_work | *amp API | 02:11 |
| cgoncalves | rm_work, I failed to reproduce it locally. ran grenade and all went fine :S | 02:11 |
| cgoncalves | eandersson, that's the million dollar question :) | 02:11 |
| rm_work | the amp API is exploding, usually it's because something in the networking is broken | 02:12 |
| eandersson | Where can I see the amp API logs? | 02:12 |
| rm_work | you using the same versions of everything locally? | 02:12 |
| cgoncalves | rm_work, I guess... | 02:12 |
| rm_work | eandersson: i don't think we ever successfully got those shipped off and stored in test results :/ | 02:12 |
| eandersson | awh :'( | 02:12 |
| cgoncalves | rm_work, ubuntu ctrl and amp | 02:12 |
| rm_work | cgoncalves: i mean, is your build getting the same ubuntu image exactly | 02:12 |
| rm_work | they could have broken stuff in their networking | 02:13 |
| rm_work | basically when i see bit-rot in stable gates, i see "stuff changing that's outside our version control" | 02:13 |
| rm_work | which means ubuntu or a few random other python libs | 02:13 |
| eandersson | I think both Trove and Ironic allows you to access logs from the api right? Maybe that would be nice to have. | 02:13 |
| rm_work | yes | 02:14 |
| rm_work | though if the API is broken... >_> | 02:14 |
| eandersson | haha yea | 02:14 |
| eandersson | I think Ironic pushes the logs | 02:14 |
| rm_work | honestly that's not the worst idea, we have been looking at shipping via SSH but | 02:14 |
| rm_work | it might be ok to just ... uhh | 02:14 |
| cgoncalves | rm_work, yes. I mean, why wouldn't it? I didn't change the default amp settings so it picks ubuntu xenial or whatever | 02:14 |
| rm_work | after we get a 500, immediately hit the "log" API and have it give us back a snapshot of the last 100 lines of the agent log or something | 02:15 |
| rm_work | rofl | 02:15 |
| cgoncalves | also ran xenial on controller | 02:15 |
| rm_work | cgoncalves: i mean, you JUST built that image? not sure if you are an "always clean env" kind of person or not LP | 02:15 |
| rm_work | :P | 02:15 |
| rm_work | i never use the same devstack twice, but a lot of people on my old teams thought i was weird | 02:16 |
| cgoncalves | rm_work, always clean | 02:17 |
| cgoncalves | grenade has a clean up script that deletes everything | 02:17 |
| rm_work | ;) | 02:17 |
| johnsom | So, I think we need to enable the dib log output for that job. I think there is an image issue. We need the SHA for the amphora-agent version inside that image | 02:17 |
| cgoncalves | I even pip uninstall all | 02:17 |
| johnsom | We are close to having log shipping from the amp, it just needs some more thought/work. We talked about it two or so meetings ago. | 02:19 |
| johnsom | But, first and easy things, get the DIB log captured. | 02:21 |
| eandersson | btw another topic - would it be impossible to move some of the biz logic from the health driver away from UpdateHealthDb? Ideally like UpdateStatsDb | 02:22 |
| eandersson | That call is so expensive on large load balancers with lots of members | 02:23 |
| johnsom | I have to run tonight, but we can chat about that tomorrow | 02:24 |
| eandersson | Sounds good | 02:24 |
| *** fnaval has quit IRC | 02:29 | |
| rm_work | eandersson: the main problem is that the stats data *comes from* the health call, so we have to do SOMETHING with it there | 02:48 |
| rm_work | it's possible we could like... dump one giant string somewhere and have another process come along and actually process (HK?) | 02:48 |
| rm_work | but that seems a bit messy | 02:48 |
| eandersson | For sure | 02:48 |
| eandersson | But at the moment getting a lb with 400+ members takes like 40s | 02:49 |
| rm_work | i thought we passed that off to another set of threads at least? | 02:49 |
| rm_work | augh what LB is that | 02:49 |
| eandersson | Worst case scenario of course | 02:49 |
| rm_work | i feel like once you have 400 members you need more than one LB and like, RR-DNS or something | 02:49 |
| eandersson | but even with lets say 50 members the call takes ~10s | 02:49 |
| rm_work | yeah that's not great | 02:50 |
| rm_work | the time is in the DB fetch? | 02:50 |
| eandersson | Yea | 02:50 |
| eandersson | Building that db object is super expensive | 02:50 |
| rm_work | I still feel like it SHOULDN'T take that long | 02:50 |
| rm_work | eugh | 02:50 |
| rm_work | we could avoid objects if we hate ourselves | 02:50 |
| rm_work | we've done it in one other place I thought | 02:51 |
| rm_work | actually, possibly in the health stuff? | 02:51 |
| rm_work | I remember johnsom had some raw querying going on | 02:51 |
| eandersson | Interesting - I might have missed something | 02:51 |
| rm_work | because it just SHOULD NOT take that long to deal with 40 or even 1000 members | 02:51 |
| rm_work | DBs are super quick | 02:51 |
| rm_work | that's the point | 02:51 |
| rm_work | (well, part of the point) | 02:52 |
| eandersson | https://github.com/openstack/octavia/blob/22df2e0f484b3560cbb47d33b7f2f62cb57519fd/octavia/db/repositories.py#L1252 | 02:52 |
| rm_work | whenever i see DB loading taking that long, i remember running queries on a 1billion+ row table that returned 100s of rows of results and that taking subsecond times at my first job | 02:52 |
| rm_work | but that was MSSQL so | 02:52 |
| eandersson | I based my stats on just in general getting a lb, not how long it takes in the health manager call | 02:53 |
| eandersson | So it might not be as bad as 10s for 50 memmbers | 02:53 |
| eandersson | I'll probably need to profile that whole function | 02:53 |
| rm_work | ah yeah there it is, you found it | 02:53 |
| eandersson | https://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/health_drivers/update_db.py#L128 | 02:54 |
| eandersson | It's... so massive | 02:54 |
| rm_work | no objects for us there :P | 02:54 |
| eandersson | Yea might have been a bad assumption by me | 02:54 |
| johnsom | Can’t talk tonight, but would be interested in exactly your use case. That query is super optimized and fast. Plus I can simulate configs for hm testing. | 02:56 |
| johnsom | Already getting very bad looks from the spouse. | 02:56 |
| rm_work | lolol | 02:56 |
| rm_work | put down the phone | 02:56 |
| *** yamamoto has quit IRC | 02:57 | |
| eandersson | We are just worried about the usage we are seeing at the moment, I think colin- mentioned this already | 02:58 |
| eandersson | and we are not really using octavia much yet | 02:59 |
| eandersson | It's not the end of the world, but just something we are worried about. | 03:00 |
| *** yamamoto has joined #openstack-lbaas | 03:02 | |
| *** psachin has joined #openstack-lbaas | 03:04 | |
| rm_work | yeah, i agree with you about the HM usage, any way we can possibly get that to be more efficient, we should pursue | 03:06 |
| rm_work | having it run too slowly can have catastrophic results (i know because i have dealt with related issues, in real production environments, more than once) | 03:06 |
| *** abaindur has quit IRC | 03:07 | |
| *** abaindur has joined #openstack-lbaas | 03:08 | |
| *** abaindur has quit IRC | 03:08 | |
| *** abaindur has joined #openstack-lbaas | 03:09 | |
| *** abaindur has quit IRC | 03:10 | |
| *** Dinesh_Bhor has quit IRC | 03:55 | |
| *** yamamoto has quit IRC | 03:56 | |
| *** yamamoto has joined #openstack-lbaas | 03:59 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 04:00 | |
| *** yamamoto has quit IRC | 04:05 | |
| cgoncalves | LB with 400+ members but "not really using octavia much yet". that's quite an underestimation | 04:23 |
| *** Dinesh_Bhor has quit IRC | 04:47 | |
| *** ramishra has joined #openstack-lbaas | 04:48 | |
| *** abaindur has joined #openstack-lbaas | 04:49 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 04:54 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: WIP: Add RHEL 8 amphora support https://review.openstack.org/638581 | 04:56 |
| cgoncalves | RHEL8-based LB is ACTIVE :) | 04:56 |
| *** yamamoto has joined #openstack-lbaas | 05:12 | |
| *** pck has joined #openstack-lbaas | 05:13 | |
| *** yamamoto has quit IRC | 05:16 | |
| *** abaindur has quit IRC | 05:26 | |
| *** yamamoto has joined #openstack-lbaas | 05:37 | |
| *** yamamoto has quit IRC | 06:50 | |
| *** yamamoto has joined #openstack-lbaas | 06:50 | |
| *** Dinesh_Bhor has quit IRC | 07:13 | |
| *** ivve has joined #openstack-lbaas | 07:14 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 07:15 | |
| openstackgerrit | Jacky Hu proposed openstack/octavia-dashboard master: WIP: Add load balancer flavor support https://review.openstack.org/638365 | 07:35 |
| *** ccamposr has joined #openstack-lbaas | 07:44 | |
| *** yamamoto has quit IRC | 07:56 | |
| *** yamamoto has joined #openstack-lbaas | 07:58 | |
| *** yamamoto has quit IRC | 08:02 | |
| *** yamamoto has joined #openstack-lbaas | 08:09 | |
| *** celebdor has joined #openstack-lbaas | 08:11 | |
| *** yamamoto has quit IRC | 08:13 | |
| *** celebdor has quit IRC | 08:19 | |
| *** yamamoto has joined #openstack-lbaas | 08:22 | |
| *** abaindur has joined #openstack-lbaas | 08:27 | |
| dayou | Cool | 08:34 |
| *** yamamoto has quit IRC | 09:28 | |
| *** ivve has quit IRC | 09:34 | |
| *** celebdor has joined #openstack-lbaas | 09:43 | |
| *** abaindur has quit IRC | 09:46 | |
| *** takamatsu_ has quit IRC | 10:04 | |
| *** takamatsu has joined #openstack-lbaas | 10:05 | |
| *** yamamoto has joined #openstack-lbaas | 10:08 | |
| *** yamamoto has quit IRC | 10:13 | |
| *** Dinesh_Bhor has quit IRC | 10:23 | |
| *** takamatsu_ has joined #openstack-lbaas | 10:23 | |
| *** takamatsu has quit IRC | 10:24 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 10:25 | |
| *** rcernin has quit IRC | 10:31 | |
| *** takamatsu_ has quit IRC | 10:48 | |
| *** takamatsu has joined #openstack-lbaas | 10:49 | |
| *** takamatsu has quit IRC | 10:54 | |
| *** takamatsu has joined #openstack-lbaas | 10:58 | |
| *** yamamoto has joined #openstack-lbaas | 11:30 | |
| *** yamamoto has quit IRC | 11:34 | |
| *** Dinesh_Bhor has quit IRC | 11:38 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 11:55 | |
| *** Dinesh_Bhor has quit IRC | 11:57 | |
| *** yamamoto has joined #openstack-lbaas | 12:16 | |
| *** ivve has joined #openstack-lbaas | 12:35 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Add RHEL 8 amphora support https://review.openstack.org/638581 | 12:43 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: octavia-v2-dsvm-scenario-rhel-8 job https://review.openstack.org/638656 | 12:48 |
| *** trown|outtypewww is now known as trown | 13:03 | |
| *** psachin has quit IRC | 13:35 | |
| *** yamamoto has quit IRC | 13:43 | |
| *** yamamoto has joined #openstack-lbaas | 13:43 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: octavia-v2-dsvm-scenario-rhel-8 job https://review.openstack.org/638656 | 13:47 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: octavia-v2-dsvm-scenario-rhel-8 job https://review.openstack.org/638656 | 14:28 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Add Python 3.7 support https://review.openstack.org/635236 | 15:09 |
| *** yamamoto has quit IRC | 15:51 | |
| *** yamamoto has joined #openstack-lbaas | 16:28 | |
| *** yamamoto has quit IRC | 16:32 | |
| *** gcheresh has joined #openstack-lbaas | 17:17 | |
| *** trown is now known as trown|lunch | 17:18 | |
| *** celebdor has quit IRC | 17:18 | |
| *** yamamoto has joined #openstack-lbaas | 17:24 | |
| *** yamamoto has quit IRC | 17:29 | |
| *** cranges has joined #openstack-lbaas | 17:29 | |
| *** takamatsu_ has joined #openstack-lbaas | 17:48 | |
| *** takamatsu has quit IRC | 17:48 | |
| *** ccamposr has quit IRC | 17:51 | |
| *** cranges has quit IRC | 18:00 | |
| *** takamatsu_ has quit IRC | 18:03 | |
| *** takamatsu_ has joined #openstack-lbaas | 18:06 | |
| xgerman | well, we can get amp logs if we merge my log patch | 18:12 |
| xgerman | (scrollback reading) | 18:12 |
| *** gcheresh has quit IRC | 18:12 | |
| johnsom | First step is to turn on the dib log collection there.... | 18:14 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: L7rule support client certificate cases https://review.openstack.org/612271 | 18:25 |
| xgerman | yep, jumping ahead | 18:37 |
| *** ramishra has quit IRC | 18:40 | |
| *** trown|lunch is now known as trown | 18:40 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: L7rule support client certificate cases https://review.openstack.org/612271 | 18:42 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Add 'client_ca_tls_container_ref' in Listener on client side https://review.openstack.org/616158 | 18:47 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Add 'client_authentication' in Listener on client https://review.openstack.org/616879 | 18:59 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Add client_crl_container_ref for Listener API in CLI https://review.openstack.org/617619 | 19:06 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Add 4 l7rule types into Octavia CLI https://review.openstack.org/618716 | 19:09 |
| *** yamamoto has joined #openstack-lbaas | 19:12 | |
| *** yamamoto has quit IRC | 19:16 | |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Add client_crl_container_ref for Listener API in CLI https://review.openstack.org/617619 | 19:22 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Add 4 l7rule types into Octavia CLI https://review.openstack.org/618716 | 19:23 |
| *** yamamoto has joined #openstack-lbaas | 19:52 | |
| *** yamamoto has quit IRC | 19:57 | |
| johnsom | rm_work Are you around, I have a certs question | 20:06 |
| eandersson | xgerman, what patch is that? | 20:14 |
| johnsom | eandersson It's this one: https://review.openstack.org/624835 But we are talking about doing it a bit differently | 20:15 |
| johnsom | eandersson Are you going to be around in ~30 minutes? I need to go make lunch, but would like to talk about HM and your setup. | 20:16 |
| eandersson | Getting lunch but should be back around then | 20:18 |
| rm_work | Yeah johnsom | 20:19 |
| johnsom | rm_work Doh, bad timing on my part sorry about that. So the question is about the CA cert for client authentication. | 20:44 |
| johnsom | Should we make people stuff that in a pkcs12 or should we store it separate? | 20:44 |
| johnsom | Right now, the cert parser stuff pukes with a pkcs12 with no main cert in it. | 20:45 |
| *** hyang has joined #openstack-lbaas | 20:56 | |
| *** hyang has quit IRC | 20:59 | |
| *** yamamoto has joined #openstack-lbaas | 21:04 | |
| *** notcolin has joined #openstack-lbaas | 21:06 | |
| *** notcolin has quit IRC | 21:08 | |
| *** yamamoto has quit IRC | 21:09 | |
| *** notcolin has joined #openstack-lbaas | 21:09 | |
| *** notcolin has quit IRC | 21:16 | |
| *** henriqueof has quit IRC | 21:20 | |
| johnsom | eandersson Available to chat about HM? | 21:37 |
| eandersson | Sure | 21:37 |
| eandersson | colin-, is here as well | 21:38 |
| johnsom | Excellent. So if I remember right, you have ~650 amphora some or all have 400 members? | 21:38 |
| eandersson | So 400 members is not octavia at the moment | 21:38 |
| johnsom | And you are running one instance of the HM process | 21:38 |
| *** yamamoto has joined #openstack-lbaas | 21:39 | |
| eandersson | I think 8 | 21:39 |
| johnsom | Oh, ok, 8 instances of the HM | 21:39 |
| johnsom | I'm just trying to get a feel for the scale. I can then use the simulator to reproduce your performance concern. | 21:40 |
| johnsom | An you said the status update section of the HM processing is taking seconds? | 21:40 |
| eandersson | Sorry, that was actually assumptions, not facts. | 21:41 |
| eandersson | My concern is the load it's generating | 21:42 |
| eandersson | So with 8 workers they are constantly at 40% cpu usage (each) | 21:42 |
| eandersson | And this is without the larger use cases (e.g. 100+ members) | 21:42 |
| eandersson | My concern is really that this increase almost looks exponential | 21:43 |
| *** yamamoto has quit IRC | 21:43 | |
| eandersson | The good thing here is that this seems to be horizontally scalable? so we can just keep adding more HMs | 21:44 |
| johnsom | What do you have set for these settings? [health_manager] health_update_threads and stats_update_threads | 21:44 |
| eandersson | 8 | 21:45 |
| johnsom | So you have 8 instances, each capped at 8 processes? | 21:45 |
| eandersson | When you say instances, do you mean amps? | 21:45 |
| johnsom | No, systemd started instances of HM | 21:46 |
| eandersson | or talking about "control nodes" | 21:46 |
| eandersson | 2x8 | 21:46 |
| eandersson | So 16 | 21:47 |
| *** ivve has quit IRC | 21:47 | |
| eandersson | Are stats and health updates sent at the same frequency? | 21:47 |
| johnsom | It is all in one UDP packet | 21:48 |
| eandersson | ah | 21:48 |
| johnsom | So you have 2 systemd started HMs, each configured to have up to 8 sub processes. | 21:48 |
| eandersson | Yea so I see that hm has 2x8 per control node | 21:49 |
| eandersson | and in this case we have two control nodes | 21:49 |
| eandersson | So just adding some information we were trying to figure out why octavia was capping out 20+ cores | 21:50 |
| eandersson | We assumed it was the HM's, but in reality it was the oslo connection leak causing most of this load | 21:51 |
| johnsom | Ok, And you don't have sync_provisioning_status or event_streamer_driver enabled right? | 21:51 |
| eandersson | Yea - that is disabled | 21:51 |
| johnsom | What is your heartbeat_interval and health_check_interval setting? | 21:51 |
| eandersson | It's set to 10, but... defaults to 3 right? | 21:51 |
| eandersson | and that hasn't replicated yet | 21:52 |
| eandersson | Since all the amps need to be re-created right? | 21:52 |
| johnsom | Well, these two settings have very different meanings. | 21:53 |
| johnsom | health check interval is how often the HM hits the database looking for missing amps. This can change at any time as it is controller side. | 21:53 |
| eandersson | ah sorry, heartbeat_interval is probably set to default | 21:54 |
| johnsom | That has a default of 3 seconds | 21:54 |
| johnsom | heartbeat interval is how often an amp sends that UDP packet with the stats/status in it. This defaults to once every 10 seconds. | 21:55 |
| rm_work | Probably a lone cert is fine, it doesn't really meet the qualifications for a pkcs12 | 21:57 |
| johnsom | That one is stored in the amphroa and would require a rebuild. However Stein will have a new API that lets you update it without rebuild | 21:57 |
| eandersson | So johnsom increase the sleep etc will obviously lower cpu time used by the HM | 21:57 |
| eandersson | That is awesome | 21:57 |
| rm_work | Yeah but increase failure detection time | 21:58 |
| eandersson | My only concern there is that it would make health checks less responsive right? | 21:58 |
| rm_work | It's a trade-off | 21:58 |
| eandersson | Yea | 21:58 |
| johnsom | Yeah, I'm just trying to dive into the situation. I wouldn't really expect it to be a lot of load | 21:58 |
| rm_work | Ah are you using py2 or py3 | 21:58 |
| eandersson | tbh what ever data we have right now is invalid until we have pushed the oslo fix out | 21:58 |
| eandersson | 2 | 21:58 |
| rm_work | I noticed the sleeps in that code in native python are shitty, it's fixed in py3 to be less shit | 21:59 |
| rm_work | So that might also help | 21:59 |
| eandersson | because at first I made the assumption about SQL based on the load on the (only reason I could see something generating that much load) | 21:59 |
| johnsom | Ok, so the 40% was for all of the controller processes and not just the HM? | 21:59 |
| rm_work | But that's theoretical, I'm just postulating | 21:59 |
| eandersson | cpu is 40% per process, but load on box did not add up | 21:59 |
| eandersson | (basically load was 20+, but cpu usage per process was lower and did not add up) | 22:00 |
| rm_work | (thread sleeps) | 22:00 |
| johnsom | So 40% of one core? | 22:00 |
| eandersson | Yea | 22:00 |
| eandersson | If you added that up load should have been at 4 | 22:00 |
| eandersson | but having 6-20 million connections established in the background threw those numbers off | 22:01 |
| johnsom | Yeah, that is only on the API process though | 22:01 |
| johnsom | There is no rabbit in HM | 22:01 |
| eandersson | api process usage per core was like 20% :p | 22:01 |
| eandersson | so nothing added up | 22:01 |
| johnsom | Ok, so maybe we need to come back to this discussion once you have deployed the rabbit fix. | 22:02 |
| eandersson | Yea I think so | 22:02 |
| eandersson | I mean there are for sure improvements that can be done like you said | 22:03 |
| eandersson | but I wouldn't mind making sure there are no outside factors first | 22:03 |
| johnsom | Ok. I spent some time last year optimizing the HM status update process. It should be pretty lean assuming you are not failing over a bunch of amps all the time. | 22:03 |
| johnsom | Under load, if I remember right, it was about .006 seconds per UDP packet. I think that was with a few thousand amps | 22:04 |
| colin- | nice that's a good metric for us to judge against | 22:07 |
| johnsom | rm_work On the cert thing, so no pkcs12 wrapper for the ca cert. That means I need to extend the cert manager again so we have a basic secret api. | 22:08 |
| johnsom | I did this for the CRL: https://review.openstack.org/#/c/612269/13/octavia/certificates/manager/barbican.py | 22:09 |
| johnsom | I probably should just change that to get_secret | 22:09 |
| rm_work | Hmm | 22:17 |
| johnsom | I mean if I had time and could get it reviewed I would probably refactor the whole manager to be more generic and move the PKCS12 stuff outside the manager. | 22:18 |
| *** celebdor has joined #openstack-lbaas | 22:54 | |
| *** yamamoto has joined #openstack-lbaas | 22:56 | |
| *** yamamoto has quit IRC | 23:00 | |
| eandersson | The code path is pretty fast johnsom, at least without actual db codes | 23:13 |
| johnsom | Yeah, typically it's the DB round trips that are the slowest part. That is why I hand optimized that DB query so we limit the number of DB calls to only those necessary. | 23:14 |
| rm_work | yeah | 23:15 |
| rm_work | the DB is super important there | 23:16 |
| rm_work | which is why we originally intended to put that stuff in redis or similar | 23:16 |
| rm_work | I wonder if next cycle would be a good time to revisit that | 23:16 |
| rm_work | we may finally be at a point where supporting that would be good | 23:16 |
| rm_work | and maybe we could get cycles from eandersson and crew to help? :P | 23:16 |
| rm_work | discussion at denver? | 23:16 |
| eandersson | I think you have to convince Jude :p | 23:17 |
| johnsom | I'm still not sure we actually *need* that. I didn't have enough tester horsepower to really see where the tipping point is. | 23:17 |
| rm_work | yeah but if we can tell the DB is the bottleneck | 23:17 |
| rm_work | and we want MORE POWER (hur hur hur) that is where we'd make the improvement | 23:18 |
| johnsom | But if we don't hit that until 100,000 LBs???? | 23:18 |
| johnsom | Or 1,000,000.... I don't know. I needed more tester CPU cores than I have. | 23:19 |
| eandersson | https://www.openstack.org/summit/denver-2019/summit-schedule/events/23379/how-blizzard-entertainment-uses-autoscaling-with-overwatch | 23:19 |
| johnsom | I bet you are really interested in the active/active work.... lol | 23:20 |
| rm_work | it is cool I can say the project I work on powers Overwatch :P | 23:24 |
| rm_work | I think I've mentioned that a few times when friends ask what I do and then are like "lolwat k" | 23:25 |
| rm_work | makes things more relatable :P | 23:25 |
| rm_work | too bad i'm horrible at overwatch >_< | 23:25 |
| johnsom | I haven't played it | 23:25 |
| rm_work | it's too fast for me, I prefer shooter games where people move at like... normal human-ish speeds | 23:26 |
| rm_work | CS:GO, PUBG/RoE, etc | 23:26 |
| rm_work | Fortnite, Apex Legends, Overwatch are all way too fast | 23:26 |
| rm_work | I can't track targets <_< | 23:26 |
| johnsom | Yeah, last game I played was sniper elite 4 | 23:26 |
| eandersson | I like BR... but Fortnite is wayyy to fast | 23:29 |
| eandersson | I can play Apex, but seriously looking at kids playing Fornite is just nuts | 23:29 |
| eandersson | Makes me feel old | 23:30 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Fix LB failover when in ERROR https://review.openstack.org/638790 | 23:41 |
| johnsom | cgoncalves I think there is a story for that | 23:42 |
| cgoncalves | johnsom, cool! | 23:42 |
| cgoncalves | I was about to ask you folks if you were okay with this change | 23:42 |
| cgoncalves | I think this was discussed before. sadly I don't remember what the conclusion was | 23:43 |
| cgoncalves | can someone teach me how to search on storyboard? :S | 23:45 |
| johnsom | Yeah, I have been looking for the story too. Storyboard has gotten even slower.... | 23:45 |
| cgoncalves | guessing you're not referring to https://storyboard.openstack.org/#!/story/2002634 | 23:47 |
| johnsom | No | 23:48 |
| johnsom | Well, I can't find it, so maybe it wasn't a story | 23:49 |
| cgoncalves | are you generally okay with allowing LBs in ERROR to be failed over? | 23:51 |
| cgoncalves | if so, I'll open a story | 23:51 |
| cgoncalves | I have received numerous requests on how to recover LBs in ERROR | 23:52 |
| johnsom | Yes, the original intent of the failover API was to recover ERROR LBs. That is a bug. | 23:53 |
| johnsom | Though I just commented on the patch | 23:53 |
| johnsom | Ugh, I have to stop looking at bugs or we are not going to get a bunch of features in stein.... | 23:53 |
| cgoncalves | ok, great | 23:56 |
| johnsom | Oh, hmm, maybe that isn't doing what I thought at first glance | 23:56 |
| johnsom | my comment might be bogus | 23:56 |
| cgoncalves | I think so | 23:56 |
| johnsom | Yeah, my bad. | 23:59 |
| johnsom | Ok, back to working on getting these features in. Bugs are in two weeks. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!