| *** catintheroof has quit IRC | 00:06 | |
| *** rcernin has quit IRC | 00:29 | |
| *** dayou_ has joined #openstack-lbaas | 00:44 | |
| *** yamamoto has joined #openstack-lbaas | 01:36 | |
| *** yamamoto has quit IRC | 01:44 | |
| *** rcernin has joined #openstack-lbaas | 02:20 | |
| *** yamamoto has joined #openstack-lbaas | 02:36 | |
| *** dayou_ has quit IRC | 02:54 | |
| *** armax has quit IRC | 03:04 | |
| *** sanfern has joined #openstack-lbaas | 03:29 | |
| *** dayou_ has joined #openstack-lbaas | 03:36 | |
| *** dayou has quit IRC | 03:38 | |
| *** links has joined #openstack-lbaas | 04:02 | |
| *** SumitNaiksatam has joined #openstack-lbaas | 04:22 | |
| *** aojea has joined #openstack-lbaas | 04:46 | |
| *** aojea has quit IRC | 04:50 | |
| *** gcheresh_ has joined #openstack-lbaas | 05:09 | |
| *** ianychoi has quit IRC | 05:11 | |
| *** ianychoi has joined #openstack-lbaas | 05:12 | |
| *** rcernin has quit IRC | 05:15 | |
| *** armax has joined #openstack-lbaas | 05:23 | |
| *** pcaruana has joined #openstack-lbaas | 05:24 | |
| *** gcheresh_ has quit IRC | 05:26 | |
| *** aojea has joined #openstack-lbaas | 05:26 | |
| *** atoth has quit IRC | 05:26 | |
| *** atoth has joined #openstack-lbaas | 05:26 | |
| *** gcheresh_ has joined #openstack-lbaas | 05:28 | |
| *** pcaruana has quit IRC | 05:29 | |
| *** gcheresh_ has quit IRC | 05:30 | |
| *** dayou_ has quit IRC | 05:44 | |
| *** rcernin has joined #openstack-lbaas | 05:52 | |
| *** yamamoto_ has joined #openstack-lbaas | 05:53 | |
| *** yamamoto has quit IRC | 05:57 | |
| *** aojea has quit IRC | 06:00 | |
| *** aojea has joined #openstack-lbaas | 06:10 | |
| openstackgerrit | Santhosh Fernandes proposed openstack/octavia master: [WIP] Adding exabgp-speaker element to amphora image https://review.openstack.org/490164 | 06:28 |
|---|---|---|
| *** eezhova has joined #openstack-lbaas | 06:36 | |
| *** pcaruana has joined #openstack-lbaas | 07:04 | |
| *** dulek has joined #openstack-lbaas | 07:16 | |
| *** eezhova has quit IRC | 07:18 | |
| *** aojea has quit IRC | 07:30 | |
| *** eezhova has joined #openstack-lbaas | 07:46 | |
| *** sanfern has quit IRC | 07:53 | |
| *** armax has quit IRC | 07:57 | |
| *** sanfern has joined #openstack-lbaas | 08:04 | |
| *** yamamoto_ has quit IRC | 09:01 | |
| *** sanfern has quit IRC | 09:14 | |
| *** aojea has joined #openstack-lbaas | 09:15 | |
| *** grim-lock has joined #openstack-lbaas | 09:26 | |
| *** yamamoto has joined #openstack-lbaas | 09:29 | |
| *** yamamoto has quit IRC | 09:32 | |
| *** sanfern has joined #openstack-lbaas | 09:36 | |
| *** gcheresh_ has joined #openstack-lbaas | 09:38 | |
| *** yamamoto has joined #openstack-lbaas | 09:40 | |
| *** salmankhan has joined #openstack-lbaas | 09:41 | |
| *** grim-lock has quit IRC | 09:42 | |
| *** aojea has quit IRC | 09:49 | |
| *** gcheresh_ has quit IRC | 09:54 | |
| *** yamamoto has quit IRC | 09:54 | |
| *** yamamoto has joined #openstack-lbaas | 10:01 | |
| *** numans has quit IRC | 10:08 | |
| *** aojea has joined #openstack-lbaas | 10:09 | |
| *** numans has joined #openstack-lbaas | 10:10 | |
| *** aojea has quit IRC | 10:12 | |
| *** aojea has joined #openstack-lbaas | 10:12 | |
| *** aojea_ has joined #openstack-lbaas | 10:18 | |
| *** aojea has quit IRC | 10:19 | |
| *** Alex_Staf has joined #openstack-lbaas | 10:28 | |
| *** aojea_ has quit IRC | 10:35 | |
| *** salmankhan has quit IRC | 10:39 | |
| *** salmankhan has joined #openstack-lbaas | 10:40 | |
| *** aojea has joined #openstack-lbaas | 10:50 | |
| *** sanfern has quit IRC | 10:56 | |
| *** gcheresh_ has joined #openstack-lbaas | 11:07 | |
| *** sapd_ has quit IRC | 11:17 | |
| *** sapd_ has joined #openstack-lbaas | 11:17 | |
| *** sapd_ has quit IRC | 11:17 | |
| *** sapd_ has joined #openstack-lbaas | 11:18 | |
| *** salmankhan has quit IRC | 11:21 | |
| *** sapd_ has quit IRC | 11:23 | |
| *** sapd_ has joined #openstack-lbaas | 11:24 | |
| *** aojea has quit IRC | 11:27 | |
| *** salmankhan has joined #openstack-lbaas | 11:33 | |
| *** armax has joined #openstack-lbaas | 11:53 | |
| *** sapd__ has joined #openstack-lbaas | 12:03 | |
| *** sapd__ has quit IRC | 12:03 | |
| *** sapd_ has quit IRC | 12:03 | |
| *** gcheresh_ has quit IRC | 12:03 | |
| *** sapd__ has joined #openstack-lbaas | 12:04 | |
| *** salmankhan has quit IRC | 12:09 | |
| *** salmankhan has joined #openstack-lbaas | 12:09 | |
| *** sanfern has joined #openstack-lbaas | 12:19 | |
| *** aojea has joined #openstack-lbaas | 12:20 | |
| *** tonygunk has joined #openstack-lbaas | 12:41 | |
| *** armax has quit IRC | 12:54 | |
| *** catintheroof has joined #openstack-lbaas | 13:03 | |
| *** leitan has joined #openstack-lbaas | 13:06 | |
| *** gcheresh_ has joined #openstack-lbaas | 13:18 | |
| *** slaweq has quit IRC | 13:30 | |
| *** gcheresh_ has quit IRC | 13:37 | |
| *** gcheresh_ has joined #openstack-lbaas | 13:38 | |
| *** links has quit IRC | 13:39 | |
| *** gcheresh_ has quit IRC | 13:57 | |
| openstackgerrit | Lingxian Kong proposed openstack/octavia-tempest-plugin master: [WIP] Add basic loadbalancer functionality test https://review.openstack.org/508516 | 14:04 |
| *** aojea has quit IRC | 14:16 | |
| *** tonygunk has quit IRC | 14:21 | |
| *** tonygunk has joined #openstack-lbaas | 14:21 | |
| *** aojea has joined #openstack-lbaas | 14:24 | |
| *** aojea has quit IRC | 14:42 | |
| *** apuimedo has quit IRC | 14:43 | |
| *** aojea has joined #openstack-lbaas | 14:46 | |
| johnsom | Ouch, looks like we have some Zuulv3 issues: https://review.openstack.org/#/c/490164/ | 14:47 |
| *** tonygunk has quit IRC | 14:56 | |
| *** tonygunk has joined #openstack-lbaas | 14:57 | |
| *** eezhova has quit IRC | 15:01 | |
| *** rcernin has quit IRC | 15:01 | |
| *** tonygunk has quit IRC | 15:03 | |
| dmellado | johnsom, I guess like everyone today | 15:05 |
| dmellado | I've been hitting the 'out of stream' issue endlessly :\ | 15:05 |
| *** aojea has quit IRC | 15:05 | |
| *** apuimedo has joined #openstack-lbaas | 15:10 | |
| johnsom | Yep | 15:13 |
| *** aojea has joined #openstack-lbaas | 15:15 | |
| *** yamamoto has quit IRC | 15:18 | |
| *** aojea has quit IRC | 15:25 | |
| *** chlong has quit IRC | 15:35 | |
| *** eezhova has joined #openstack-lbaas | 15:38 | |
| *** tonygunk has joined #openstack-lbaas | 15:46 | |
| *** yamamoto has joined #openstack-lbaas | 16:18 | |
| *** yamamoto has quit IRC | 16:24 | |
| *** pcaruana has quit IRC | 16:26 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-dashboard master: DO-NOT-MERGE: Gate test patch https://review.openstack.org/508560 | 16:32 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: DO-NOT-MERGE: Gate test patch https://review.openstack.org/508561 | 16:33 |
| *** eezhova has quit IRC | 16:34 | |
| *** sshank has joined #openstack-lbaas | 16:48 | |
| *** rtjure has quit IRC | 16:49 | |
| *** salmankhan has quit IRC | 16:59 | |
| *** salmankhan has joined #openstack-lbaas | 17:00 | |
| *** bbzhao has quit IRC | 17:00 | |
| *** bbzhao has joined #openstack-lbaas | 17:00 | |
| *** tongl has joined #openstack-lbaas | 17:02 | |
| *** salmankhan has quit IRC | 17:11 | |
| *** salmankhan has joined #openstack-lbaas | 17:12 | |
| *** SumitNaiksatam has quit IRC | 17:15 | |
| *** salmankhan has quit IRC | 17:16 | |
| tongl | When we do redirect_to_url in octavia, what's the response code for redirect? Is it 301 by default? | 17:17 |
| johnsom | Just a sec, I have to look | 17:18 |
| johnsom | tongl 302 | 17:19 |
| tongl | i see, we just use 302 Found for temporary redirect. | 17:20 |
| *** mugsie has joined #openstack-lbaas | 17:20 | |
| *** yamamoto has joined #openstack-lbaas | 17:20 | |
| tongl | maybe we can add a redirect_code to allow user to set response code if needed. | 17:20 |
| tongl | The redirect may be permanent | 17:21 |
| johnsom | Yeah, that would be a feature enhancement to the API. | 17:23 |
| johnsom | We technically support 301, 302, 303, 307 and 308 but currently it only offers 302 | 17:24 |
| tongl | Let me draft a spec for that | 17:25 |
| *** yamamoto has quit IRC | 17:25 | |
| tongl | Can you paste a link to where 302 is defined? | 17:26 |
| johnsom | It's missing from our API-ref, but here is the root docs: http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4.2-redirect%20location | 17:28 |
| tongl | @johnsom: what's the response code for REJECT action? | 17:28 |
| johnsom | https://www.irccloud.com/pastebin/QtIVqZew/ | 17:28 |
| tongl | Do we just use 501 for reject? | 17:29 |
| johnsom | 403 | 17:30 |
| tongl | thx | 17:34 |
| johnsom | I will put up an api-ref patch that specifies these | 17:35 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: L7 policy API-REF update for result codes https://review.openstack.org/508575 | 17:38 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: L7 policy API-REF update for result codes https://review.openstack.org/508575 | 17:40 |
| johnsom | Didn't like the wording, hopefully that is better. | 17:40 |
| tongl | nice | 17:41 |
| *** SumitNaiksatam has joined #openstack-lbaas | 17:43 | |
| *** rtjure has joined #openstack-lbaas | 17:53 | |
| *** sshank has quit IRC | 18:05 | |
| *** sshank has joined #openstack-lbaas | 18:06 | |
| rm_work | so basically, we're not merging anything in the near future | 18:07 |
| *** yamamoto has joined #openstack-lbaas | 18:22 | |
| johnsom | Yep | 18:23 |
| johnsom | Things are getting better though | 18:23 |
| johnsom | This one has me scratching my head however: http://logs.openstack.org/75/508575/2/check/openstack-tox-py27/6208b6f/job-output.txt.gz#_2017-09-29_18_01_36_558322 | 18:25 |
| *** yamamoto has quit IRC | 18:26 | |
| rm_work | lol of course | 18:33 |
| rm_work | because carlos | 18:33 |
| johnsom | Do you see the issue? | 18:33 |
| rm_work | or actually that might have been barclac | 18:33 |
| rm_work | one sec | 18:34 |
| rm_work | i just know that bit is really fragile because they're doing a bunch of low-level stuff | 18:34 |
| johnsom | Oh, I think I see it | 18:35 |
| johnsom | hmm | 18:35 |
| *** dayou_ has joined #openstack-lbaas | 18:35 | |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/health_daemon/status_message.py#L72 | 18:36 |
| johnsom | digest is not ascii | 18:36 |
| rm_work | so in py2 hmac doesn't have the compare method native, so they do their thing | 18:36 |
| johnsom | But why was this always working until now? | 18:37 |
| *** tonygunk has quit IRC | 18:38 | |
| *** sshank has quit IRC | 18:41 | |
| *** sshank has joined #openstack-lbaas | 18:41 | |
| rm_work | probably they switched python versions slightly or something when they made the new backend nodes | 18:41 |
| rm_work | i'm trying to see if there's a version where it'll fail | 18:41 |
| johnsom | Thanks | 18:42 |
| *** tongl has quit IRC | 18:42 | |
| rm_work | i have some questions about performance testing today | 18:47 |
| rm_work | i have no idea what the numbers coming back from apachebench actually MEAN or how to tell whether i've got the right bottleneck (the LB) or not | 18:48 |
| rm_work | i set up a very simple test scenario right now... i have two backends running the c10k Golang server, and a LB, and two load generator VMs with apachebench | 18:49 |
| *** sshank has quit IRC | 18:50 | |
| johnsom | Yeah, so there is an art to this... | 18:50 |
| johnsom | I will also mention that apache bench is not the best benchmark suite out there... | 18:51 |
| rm_work | k | 18:51 |
| rm_work | it seemed like the easiest (and was recommended in the c10k code) | 18:51 |
| rm_work | but I am open to whatever | 18:51 |
| rm_work | I used Tsung in the past | 18:51 |
| rm_work | was thinking about poking at that again | 18:52 |
| johnsom | Yeah, I wrote up an etherpad for the bluebox folks poking at this, not sure if I can find it | 18:52 |
| johnsom | Tsung is a nice multi-client suite. For single I like weighttp: http://redmine.lighttpd.net/projects/weighttp/wiki | 18:53 |
| rm_work | will that generate enough load? | 18:54 |
| johnsom | This describes the ab output: https://httpd.apache.org/docs/2.4/programs/ab.html#output | 18:54 |
| johnsom | weighttp does, but single client. It will beat the tar out of web servers | 18:55 |
| *** Pernille has joined #openstack-lbaas | 18:56 | |
| johnsom | This page has some useful reading: http://gwan.com/en_apachebench_httperf.html | 18:58 |
| johnsom | gwan is a great static content web server for benchmarking too. compiled, serve from memory, etc. | 18:59 |
| johnsom | I like to compare with a LB bypass route too | 19:01 |
| rm_work | yeah just direct to the member? | 19:02 |
| johnsom | Yeah, basline comparison | 19:02 |
| rm_work | hmm weighttp is getting a lot of "connection reset by peer", wonder if that means the LB is maxxed | 19:02 |
| johnsom | Or check your timed_waits and open file descriptors... | 19:03 |
| johnsom | max conn settings for the LB | 19:03 |
| johnsom | etc. | 19:03 |
| rm_work | i did ulimit -n 100000 | 19:03 |
| johnsom | Lots-o-knobs to turn | 19:03 |
| rm_work | ah right max connections options on the listener.... | 19:03 |
| johnsom | Even kernel conntrack might need tuning on client/web server side | 19:04 |
| johnsom | We do some of that for the amps in our elements | 19:04 |
| *** tongl has joined #openstack-lbaas | 19:05 | |
| johnsom | Actually, I think we disable conntrack in the amp since it can dog the flows | 19:05 |
| *** sshank has joined #openstack-lbaas | 19:05 | |
| *** sshank has quit IRC | 19:09 | |
| rm_work | yeah i am wondering if i am hitting bandwidth issues or what | 19:14 |
| rm_work | even the baseline without the LB involved is a little weird | 19:14 |
| rm_work | i'm doing like -n 10000 -c 10000 | 19:14 |
| rm_work | no idea if that's good or not | 19:14 |
| rm_work | lol taking -c down to 1000 makes it ... instant | 19:15 |
| rm_work | lol | 19:15 |
| rm_work | so obviously -c 10000 is not good | 19:15 |
| *** catintheroof has quit IRC | 19:20 | |
| *** catintheroof has joined #openstack-lbaas | 19:20 | |
| rm_work | hmmm 2.7.14 doesn't make that failure happen | 19:21 |
| rm_work | need to look at what's on the box i guess | 19:21 |
| xgerman_ | Yep, Jason reported 10-16K maybe you can ask how they tester | 19:22 |
| *** yamamoto has joined #openstack-lbaas | 19:23 | |
| xgerman_ | Met, with blogan they are trying to get rid of A10, too | 19:23 |
| xgerman_ | Lol - since dougwig is driving that ;-) | 19:24 |
| *** catintheroof has quit IRC | 19:24 | |
| rm_work | lol | 19:25 |
| *** yamamoto has quit IRC | 19:28 | |
| rm_work | so basically what i can see is, dougwig leaves a10, everyone drops a10 | 19:28 |
| rm_work | :P | 19:28 |
| rm_work | i'm getting a lot of connection errors with the LB in the loop... but the amp seems to be underutilized still, haproxy isn't going above like 20% CPU and 10% RAM | 19:59 |
| *** sshank has joined #openstack-lbaas | 20:03 | |
| *** tonygunk has joined #openstack-lbaas | 20:08 | |
| *** aojea has joined #openstack-lbaas | 20:09 | |
| *** aojea has quit IRC | 20:13 | |
| xgerman_ | mmh, you fixed ulimit might be at the hypervisor or is the backend without the LB fairing better (aka doing twice the connections) — after all you are going in and out | 20:21 |
| xgerman_ | in the amp | 20:21 |
| rm_work | yeah | 20:22 |
| rm_work | direct to LB doesn't get the connection errors | 20:22 |
| rm_work | eerrrr | 20:23 |
| rm_work | sorry, direct to member node | 20:23 |
| rm_work | to LB is where the connect errors appear | 20:23 |
| rm_work | weighttp -n 100000 -c 6000 -t 12 -k http://$LB_IP/slow | 20:24 |
| rm_work | finished in 163 sec, 265 millisec and 920 microsec, 612 req/s, 84 kbyte/s | 20:24 |
| rm_work | requests: 100000 total, 100000 started, 100000 done, 99322 succeeded, 678 failed, 0 errored | 20:24 |
| *** yamamoto has joined #openstack-lbaas | 20:24 | |
| rm_work | weighttp -n 100000 -c 6000 -t 12 -k http://$MEMBER_IP/slow | 20:24 |
| rm_work | finished in 52 sec, 310 millisec and 604 microsec, 1911 req/s, 265 kbyte/s | 20:24 |
| rm_work | requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored | 20:24 |
| rm_work | ^^ so that's not ideal | 20:24 |
| rm_work | i don't know if my c/t numbers are way off though | 20:25 |
| dayou_ | johnsom: Could you help to do a pip list | grep futu on your devstack machine that has problem restarting health manager with new patch? | 20:25 |
| dayou_ | See it the futures and futurist get updated recently, wondering that was breaking | 20:26 |
| *** tonygunk has quit IRC | 20:27 | |
| dayou_ | It's change in futurist and fucsigs that I am suspecting | 20:28 |
| *** yamamoto has quit IRC | 20:29 | |
| *** tongl has quit IRC | 20:31 | |
| *** eezhova has joined #openstack-lbaas | 20:33 | |
| *** Pernille has quit IRC | 20:34 | |
| johnsom | 612 req/s ???? Something is seriously wrong with that setup..... | 20:37 |
| rm_work | :/ | 20:38 |
| johnsom | dayou_ Just a minute, will look | 20:38 |
| johnsom | future (0.16.0) | 20:39 |
| johnsom | futures (3.1.1) | 20:39 |
| johnsom | futurist (1.4.0) | 20:39 |
| johnsom | rm_work I'm going to setup a devstack and give it a go too | 20:39 |
| rm_work | ^^ dayou_ | 20:41 |
| rm_work | if you want to look at performance data for a minute, i'm totally down to screenshare and walk through some tests / this testbed | 20:42 |
| rm_work | but if you want to get your own actual work done, i'm continuing on my own anyway | 20:42 |
| *** KeithMnemonic has quit IRC | 20:45 | |
| *** eezhova has quit IRC | 20:47 | |
| johnsom | Well, I will load up a clean stack and kick the tires as I'm curious now | 20:47 |
| johnsom | Otherwise I'm mostly looking at zuulv3 stuff, but more dust needs to clear there IMO | 20:47 |
| rm_work | johnsom: load a clean stack for dayou_'s thing or for perf? | 20:48 |
| rm_work | I assume perf is my env | 20:48 |
| johnsom | perf | 20:48 |
| johnsom | The package answer was easy as I still had that stack up and running | 20:49 |
| dayou_ | Well, hopefully you can help testing the health manager thing after perf | 20:49 |
| rm_work | i mean, i can just show you my env and run whatever you want, too | 20:50 |
| johnsom | dayou_ Sure, what is it you would like me to test? | 20:50 |
| dayou_ | Just load the health manager patch and try restart and see whether it works | 20:51 |
| dayou_ | I see they upgraded futurist requirements from >= 0.11.0 != 0.15.0 to >=1.2.0 on Sep 7th | 20:53 |
| dayou_ | I did recreate the stack on Sept 12th | 20:54 |
| dayou_ | So that's I am wondering it might be broken before | 20:54 |
| *** armax has joined #openstack-lbaas | 21:00 | |
| *** tongl has joined #openstack-lbaas | 21:07 | |
| johnsom | https://www.irccloud.com/pastebin/AEQGde4S/ | 21:16 |
| johnsom | rm_work ^^^^ On my lowly vmware workstation desktop devstack | 21:16 |
| rm_work | hmmm | 21:16 |
| rm_work | what command did you run | 21:16 |
| johnsom | ./weighttp -n 100000 -c 6000 -t 12 -k http://10.0.0.12 | 21:16 |
| rm_work | hmmm | 21:16 |
| johnsom | Copied yours, though I think there is room for improvement there.... | 21:16 |
| rm_work | so that's with a LB or direct to member | 21:17 |
| johnsom | That is the LB VIP | 21:17 |
| rm_work | hmmmmm | 21:18 |
| rm_work | oh hold on | 21:18 |
| rm_work | you're not using /slow | 21:18 |
| *** aojea has joined #openstack-lbaas | 21:18 | |
| rm_work | direct to member without /slow is: | 21:18 |
| rm_work | finished in 3 sec, 794 millisec and 283 microsec, 26355 req/s, 3654 kbyte/s | 21:18 |
| rm_work | requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored | 21:18 |
| johnsom | Ok, that is more like it | 21:18 |
| rm_work | LB without /slow is: | 21:19 |
| rm_work | finished in 15 sec, 152 millisec and 978 microsec, 6599 req/s, 915 kbyte/s | 21:19 |
| rm_work | requests: 100000 total, 100000 started, 100000 done, 99986 succeeded, 14 failed, 0 errored | 21:19 |
| rm_work | so still significantly worse | 21:19 |
| rm_work | is that ... expected? | 21:19 |
| johnsom | Not that low | 21:20 |
| rm_work | :/ | 21:20 |
| rm_work | haproxy is at ~80% CPU there | 21:20 |
| rm_work | still 7% memory | 21:20 |
| rm_work | lol and systemd-journal is at 50% | 21:21 |
| rm_work | finished in 72 sec, 304 millisec and 733 microsec, 11064 req/s, 1533 kbyte/s | 21:22 |
| rm_work | requests: 800000 total, 800000 started, 800000 done, 799656 succeeded, 344 failed, 0 errored | 21:22 |
| rm_work | increased request count | 21:22 |
| johnsom | So slow delays each of the go threads three seconds... | 21:22 |
| rm_work | yes | 21:22 |
| rm_work | to simulate "work" happening | 21:22 |
| johnsom | Well, that will tank the req/s | 21:23 |
| rm_work | ah yeah i guess so | 21:23 |
| rm_work | we're looking for max RPS | 21:23 |
| rm_work | so ok, using not /slow | 21:23 |
| rm_work | but why am i getting failures T_T | 21:23 |
| johnsom | I only get them with /slow | 21:23 |
| rm_work | hmm | 21:24 |
| johnsom | Likely because weighttp gives up waiting. | 21:24 |
| rm_work | well, i'm getting them here too | 21:24 |
| johnsom | https://www.irccloud.com/pastebin/Hl4OhTSc/ | 21:24 |
| rm_work | lots of "Connection reset by peer" | 21:24 |
| *** ltomasbo has quit IRC | 21:24 | |
| rm_work | if i take it down to 5000 concurrent, i don't see them I think | 21:25 |
| rm_work | ah no, still some, but less | 21:25 |
| rm_work | lol and it finishes quicker | 21:25 |
| *** yamamoto has joined #openstack-lbaas | 21:25 | |
| rm_work | down to 4000 and again, finishes faster and less errors | 21:26 |
| rm_work | it always gets to 99% pretty quick and then sits... | 21:26 |
| rm_work | like the last few requests aren't finishing? | 21:26 |
| rm_work | at -c 2000 : | 21:26 |
| rm_work | finished in 21 sec, 724 millisec and 755 microsec, 9206 req/s, 1276 kbyte/s | 21:26 |
| rm_work | requests: 200000 total, 200000 started, 200000 done, 200000 succeeded, 0 failed, 0 errored | 21:26 |
| rm_work | err sorry, 4000 | 21:26 |
| rm_work | going below that seems to still be faster lol | 21:27 |
| rm_work | at 2000 now | 21:27 |
| rm_work | finished in 19 sec, 180 millisec and 722 microsec, 10427 req/s, 1445 kbyte/s | 21:27 |
| rm_work | requests: 200000 total, 200000 started, 200000 done, 200000 succeeded, 0 failed, 0 errored | 21:27 |
| rm_work | is that an issue on this side (client)? | 21:27 |
| rm_work | like, this machine sucks at generating so many requests? | 21:27 |
| rm_work | too many threads? | 21:27 |
| *** ltomasbo has joined #openstack-lbaas | 21:28 | |
| rm_work | seems like I am topping out around 10k req/s | 21:29 |
| johnsom | Yeah, you are asking for a lot of threads for cores you probably don't have | 21:29 |
| rm_work | running like this: weighttp -n 200000 -c 2000 -t 12 -k http://10.32.156.192 | 21:29 |
| rm_work | if i run on one box i get 10k | 21:29 |
| rm_work | if i run it simultaneously on two boxes, it takes twice as long on both and gets about 5k/each | 21:30 |
| rm_work | down to 8 and then 4 threads and results are nearly identical on one box (around 10k rps) | 21:31 |
| *** yamamoto has quit IRC | 21:31 | |
| rm_work | yeah tweaking the -c and -t params around seems to pretty much always end with 10k rps | 21:31 |
| rm_work | and HAProxy hovers around 80% +/- 10% CPU | 21:32 |
| rm_work | we looked at enabling haproxy on multi-core and decided it wasn't worth it? or caused problems? or we would just leave it up to the deployer? | 21:34 |
| johnsom | Well, long, long ago, the multi-proc model in haproxy had issues with peer sync and seamless restarts. We never really went back and checked what all was fixed. | 21:37 |
| johnsom | Frankly, we have done zero tuning | 21:37 |
| johnsom | It's all been about "make it work" | 21:37 |
| rm_work | k | 21:38 |
| rm_work | anyway, unimportant | 21:38 |
| rm_work | i am thinking 10k is about the threshold | 21:38 |
| rm_work | for what this env can handle | 21:38 |
| rm_work | that seems... a little low? but maybe that's what we get with this tuning level in our cloud | 21:38 |
| johnsom | It's odd that I'm seeing about the same. You aren't running devstack right? | 21:38 |
| rm_work | no this is production | 21:38 |
| rm_work | real prod cloud here | 21:39 |
| johnsom | Yeah, so hmmm, seems like we are hitting something artificial | 21:39 |
| johnsom | Because, I mean, I'm running in vmware workstation on top of windows.... | 21:39 |
| rm_work | reading through this guide and the previous two parts: https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to-achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27 | 21:40 |
| rm_work | https://medium.freecodecamp.org/load-testing-haproxy-part-1-f7d64500b75d | 21:40 |
| rm_work | and https://medium.freecodecamp.org/load-testing-haproxy-part-2-4c8677780df6 | 21:40 |
| rm_work | sockets? | 21:40 |
| rm_work | (part 2) | 21:40 |
| johnsom | I'm wondering about the cirros | 21:41 |
| rm_work | well i am not using cirros at all | 21:41 |
| johnsom | It looks like conntrack on my devstack host is playing a factor | 21:48 |
| rm_work | hmm | 21:48 |
| rm_work | ah but given that i really am seeing about 10k limit on the LB side, not client | 21:49 |
| rm_work | eugh | 22:04 |
| rm_work | yeah it's not the load-generating side that's bottlenecking now | 22:04 |
| rm_work | that's clear at least | 22:04 |
| johnsom | I think it might be in my case. The amp isn't really sweating | 22:05 |
| johnsom | after I turn off logging for example | 22:05 |
| rm_work | ah i wonder how much that affects things | 22:05 |
| rm_work | i did note that the logger was taking a lot of CPU | 22:05 |
| rm_work | 10k just seems like a low-ish number | 22:06 |
| rm_work | especially given that i can get better performance with direct-to-member | 22:06 |
| rm_work | i mean, is direct-to-member ALWAYS going to give better performance? | 22:06 |
| rm_work | (in the case that what the member is doing is just statically returning one byte) | 22:07 |
| johnsom | Yes, direct to member will always be higher | 22:07 |
| *** leitan has quit IRC | 22:08 | |
| rm_work | k | 22:09 |
| rm_work | i can get about 55k RPS direct-to-member, it seems | 22:09 |
| rm_work | LB seems to really top out at 10k | 22:09 |
| rm_work | LOL well | 22:13 |
| rm_work | if i remove all but one member | 22:13 |
| rm_work | the LB suddenly jumps to 25k RPS | 22:13 |
| rm_work | wtf | 22:13 |
| rm_work | did i have slow members throwing it off? | 22:14 |
| rm_work | actually half the RPS makes sense, right? because it has to do user-->haproxy<--member | 22:14 |
| rm_work | so it has to do twice the jumps | 22:14 |
| rm_work | adding a second member halves the RPS | 22:23 |
| rm_work | johnsom: ^^ | 22:23 |
| johnsom | Yeah, not sure on that | 22:23 |
| johnsom | Seems odd | 22:23 |
| rm_work | yeah both members test at 55kRPS | 22:24 |
| rm_work | put one behind the LB, halve that to ~26kRPS | 22:25 |
| rm_work | put a second, halve THAT to ~12kRPS | 22:25 |
| rm_work | i think it's the act of having to select a member at all? | 22:26 |
| rm_work | oh, maybe the algorithm?!? | 22:26 |
| rm_work | eh, this is ROUND_ROBIN, so nm, shouldn't be much calculation | 22:26 |
| rm_work | but i guess maybe there's SOME check that has to take place to decide where to send traffic? | 22:26 |
| *** armax has quit IRC | 22:26 | |
| *** yamamoto has joined #openstack-lbaas | 22:27 | |
| rm_work | on the amp, I see this during load: http://paste.openstack.org/show/622353/ | 22:29 |
| rm_work | i guess "max concurrent connections" is something I wanted to test as well, which is maybe more meaningful that RPS | 22:31 |
| rm_work | *than | 22:31 |
| rm_work | but i'm not sure how to test that, besides noting how many I try to make before it starts giving connection errors? | 22:31 |
| rm_work | which i assume is the thing | 22:31 |
| johnsom | Anyone know the new cirros password? | 22:31 |
| rm_work | err did it change? | 22:31 |
| rm_work | it isn't cubswin;) anymore? | 22:31 |
| johnsom | trying to su in an image and getting no where | 22:32 |
| rm_work | errr | 22:32 |
| rm_work | cubswin:) | 22:32 |
| *** aojea has quit IRC | 22:32 | |
| rm_work | i guess that's the cirros user password? | 22:32 |
| rm_work | can you not passwordless sudo? | 22:32 |
| *** yamamoto has quit IRC | 22:32 | |
| johnsom | Yeah, I'm trying to bump the ulimit with no luck | 22:33 |
| johnsom | ah, yeah, nevermind. Just no bash alias there | 22:33 |
| rm_work | OH yeah that | 22:34 |
| rm_work | you have to sudo su - | 22:34 |
| rm_work | and then do ulimit | 22:34 |
| rm_work | or did you find another way? | 22:34 |
| johnsom | built myself a staticly linked weighttp | 22:34 |
| rm_work | heh | 22:34 |
| rm_work | my problem is definitely not client side | 22:35 |
| rm_work | it's definitely the LB limiting me to ~11kRPS | 22:35 |
| rm_work | i've been running weighttp from /dev/shm anyway | 22:36 |
| rm_work | i had to build my own because of course there's no CentOS package <_< | 22:36 |
| johnsom | https://www.irccloud.com/pastebin/wquPhAUr/ | 22:36 |
| johnsom | straight to the member web server | 22:37 |
| johnsom | so, something is fishy with cirros | 22:37 |
| rm_work | hmm | 22:38 |
| rm_work | i mean | 22:38 |
| rm_work | load generation on the same box you're serving from... | 22:38 |
| johnsom | No, I booted another | 22:38 |
| rm_work | yeah it's still on your devstack HV | 22:39 |
| rm_work | which is a piece of your windows machine <_< | 22:39 |
| rm_work | box == your laptop | 22:39 |
| *** aojea has joined #openstack-lbaas | 22:40 | |
| *** aojea has quit IRC | 22:45 | |
| *** aojea has joined #openstack-lbaas | 22:57 | |
| johnsom | Yeah, so hacked haproxy to just 503 (no members), still right around 10k | 23:00 |
| johnsom | Something outside is limiting it | 23:00 |
| rm_work | going to try with Vegeta! | 23:00 |
| rm_work | it has the appropriate power level for what i'm going for | 23:00 |
| johnsom | I'm thinking it's the security groups conntrack | 23:01 |
| rm_work | hmm | 23:01 |
| rm_work | well i doubt that's the issue in my env? | 23:01 |
| johnsom | Did you take the security group code out of octavia ? | 23:01 |
| johnsom | I mean you are still running neutron right? | 23:01 |
| *** aojea has quit IRC | 23:02 | |
| *** sshank has quit IRC | 23:02 | |
| rm_work | no? | 23:02 |
| rm_work | but like | 23:02 |
| rm_work | direct to member -> member still has a security group | 23:02 |
| rm_work | and i get 55k to members | 23:03 |
| rm_work | they're all just VMs | 23:03 |
| rm_work | also, i'm not limited to 10k | 23:03 |
| rm_work | my LB will do 25k with one member | 23:03 |
| rm_work | Vegeta seems to be a lot better tool AFAICT | 23:08 |
| rm_work | hmm though i can't get the same request rate direct-to-member with it :/ | 23:17 |
| rm_work | Vegeta is showing me the same thing though -- about 11k is the best I can do | 23:19 |
| rm_work | before everything goes to shit | 23:19 |
| johnsom | Yeah, my host running qemu is maxing out conntrackd | 23:19 |
| rm_work | hmm | 23:20 |
| johnsom | -rw------- 1 root root 599739930 Sep 29 16:22 conntrackd-stats.log | 23:22 |
| johnsom | Well, that is probably part of it... | 23:23 |
| johnsom | Sigh | 23:23 |
| johnsom | appears to be logging for each connection | 23:23 |
| *** tongl has quit IRC | 23:25 | |
| rm_work | hmm | 23:28 |
| *** yamamoto has joined #openstack-lbaas | 23:29 | |
| rm_work | oh | 23:32 |
| rm_work | how do we disable conntrack on the amps exactly? | 23:32 |
| rm_work | isn't it slightly different on like, centos7 vs ubuntu? | 23:32 |
| rm_work | on my amp, if I do "lsmod" i definitely see nf_conntrack loaded | 23:33 |
| rm_work | johnsom: ^^ | 23:34 |
| johnsom | Yeah, I don't | 23:34 |
| johnsom | I don't think it's in the ubuntu cloud image | 23:34 |
| *** yamamoto has quit IRC | 23:35 | |
| rm_work | hmmmmmm crap | 23:35 |
| rm_work | i may need to disable it | 23:35 |
| johnsom | Yeah, try disabling it in one of your amps and see what you get | 23:35 |
| johnsom | I think you can just rmmod it and it's friends | 23:36 |
| rm_work | ip_vs ? | 23:36 |
| rm_work | let's see | 23:36 |
| johnsom | No, I don't htink so | 23:36 |
| rm_work | i mean, YES, ip_vs is one | 23:36 |
| johnsom | Just do the conntrack one and it should whine if there are friends | 23:37 |
| rm_work | yes, ip_vs | 23:37 |
| johnsom | Hmm | 23:37 |
| johnsom | Ugh, these ubuntu cloud images... joydev module loaded, pata, iscsi, parport, sigh | 23:38 |
| rm_work | time to add to our elements to do some blacklisting? :P | 23:38 |
| johnsom | Well, it wastes so much disk space too. Which someone would pay me to just make a small OS image build system. I have yet to see a distro that isn't bloated | 23:39 |
| rm_work | removing the module appears to make zero difference :/ | 23:42 |
| johnsom | I think to get better numbers I would need to setup multiple VMs and pop the traffic out a provider network. Get client and servers out of the hands of neutron/qemu/etc | 23:44 |
| rm_work | ahhh ummmmmm | 23:56 |
| rm_work | i might have found a problem | 23:56 |
| rm_work | so haproxy seems to have a maxconn default ?? | 23:57 |
| johnsom | Yeah, like 2000 | 23:57 |
| rm_work | i added "maxconn 200000" to the config in global and defaults | 23:57 |
| rm_work | reloaded it | 23:57 |
| rm_work | and now it's getting a bit higher RPS | 23:57 |
| rm_work | but also ulimit errors | 23:57 |
| rm_work | i thought it was supposed to auto-adjust the process ulimit :/ | 23:57 |
| johnsom | I told you to increase your listener max conn | 23:57 |
| rm_work | not sure how to fix it for haproxy | 23:57 |
| rm_work | the default listener max conn is *unlimited* according to our docs | 23:58 |
| johnsom | Yeah, I think I opened a bug against that | 23:58 |
| rm_work | so what am i supposed to increase it to <_< | 23:58 |
| johnsom | A large number | 23:58 |
| johnsom | grin | 23:58 |
| rm_work | k | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!