| *** itsuugo has joined #openstack-lbaas | 00:00 | |
| *** diogogmt has quit IRC | 00:09 | |
| openstackgerrit | Dustin Lundquist proposed openstack/neutron-lbaas: haproxy ns driver: set global haproxy maxconn https://review.openstack.org/369088 | 00:14 |
|---|---|---|
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Set haproxy global connection limit https://review.openstack.org/369089 | 00:18 |
| *** diogogmt has joined #openstack-lbaas | 00:24 | |
| *** itsuugo has quit IRC | 00:24 | |
| *** itsuugo has joined #openstack-lbaas | 00:26 | |
| *** diogogmt has quit IRC | 00:27 | |
| *** ducttape_ has joined #openstack-lbaas | 00:35 | |
| *** fnaval has quit IRC | 00:36 | |
| *** fnaval has joined #openstack-lbaas | 00:37 | |
| *** itsuugo has quit IRC | 00:40 | |
| *** itsuugo has joined #openstack-lbaas | 00:42 | |
| *** itsuugo has quit IRC | 00:48 | |
| *** itsuugo has joined #openstack-lbaas | 00:49 | |
| *** itsuugo has quit IRC | 00:59 | |
| *** itsuugo has joined #openstack-lbaas | 01:00 | |
| *** gongysh has joined #openstack-lbaas | 01:06 | |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Set fullconn parameter in haproxy backends https://review.openstack.org/369102 | 01:07 |
| *** bana_k has quit IRC | 01:11 | |
| *** bana_k has joined #openstack-lbaas | 01:12 | |
| *** bana_k has quit IRC | 01:13 | |
| *** kevo has quit IRC | 01:13 | |
| *** chlong_ has joined #openstack-lbaas | 01:14 | |
| *** carrbs has quit IRC | 01:15 | |
| *** itsuugo has quit IRC | 01:25 | |
| *** itsuugo has joined #openstack-lbaas | 01:25 | |
| *** diogogmt has joined #openstack-lbaas | 01:28 | |
| *** itsuugo has quit IRC | 01:34 | |
| *** itsuugo has joined #openstack-lbaas | 01:36 | |
| *** itsuugo has quit IRC | 01:41 | |
| *** itsuugo has joined #openstack-lbaas | 01:42 | |
| *** chlong_ has quit IRC | 01:49 | |
| *** ducttape_ has quit IRC | 01:49 | |
| *** ducttape_ has joined #openstack-lbaas | 01:50 | |
| *** itsuugo has quit IRC | 01:50 | |
| *** itsuugo has joined #openstack-lbaas | 01:51 | |
| *** ducttape_ has quit IRC | 01:54 | |
| *** itsuugo has quit IRC | 01:58 | |
| *** itsuugo has joined #openstack-lbaas | 02:00 | |
| *** chlong_ has joined #openstack-lbaas | 02:06 | |
| *** itsuugo has quit IRC | 02:09 | |
| *** itsuugo has joined #openstack-lbaas | 02:10 | |
| *** itsuugo has quit IRC | 02:15 | |
| *** itsuugo has joined #openstack-lbaas | 02:17 | |
| *** ducttape_ has joined #openstack-lbaas | 02:20 | |
| *** itsuugo has quit IRC | 02:21 | |
| *** itsuugo has joined #openstack-lbaas | 02:22 | |
| *** itsuugo has quit IRC | 02:27 | |
| *** itsuugo has joined #openstack-lbaas | 02:28 | |
| *** itsuugo has quit IRC | 02:34 | |
| *** itsuugo has joined #openstack-lbaas | 02:37 | |
| *** itsuugo has quit IRC | 02:50 | |
| *** itsuugo has joined #openstack-lbaas | 02:51 | |
| *** itsuugo has quit IRC | 03:20 | |
| *** itsuugo has joined #openstack-lbaas | 03:22 | |
| *** ducttape_ has quit IRC | 03:25 | |
| *** itsuugo has quit IRC | 03:29 | |
| *** itsuugo has joined #openstack-lbaas | 03:31 | |
| *** ducttape_ has joined #openstack-lbaas | 03:32 | |
| *** armax has quit IRC | 03:35 | |
| *** bana_k has joined #openstack-lbaas | 03:36 | |
| *** sticker has joined #openstack-lbaas | 03:39 | |
| *** Alex_Stef has joined #openstack-lbaas | 03:39 | |
| *** ducttape_ has quit IRC | 03:53 | |
| *** ducttape_ has joined #openstack-lbaas | 03:56 | |
| *** itsuugo has quit IRC | 04:01 | |
| *** ducttape_ has quit IRC | 04:01 | |
| *** bana_k has quit IRC | 04:01 | |
| *** itsuugo has joined #openstack-lbaas | 04:02 | |
| *** itsuugo has quit IRC | 04:11 | |
| *** itsuugo has joined #openstack-lbaas | 04:12 | |
| *** itsuugo has quit IRC | 04:16 | |
| *** itsuugo has joined #openstack-lbaas | 04:17 | |
| *** itsuugo has quit IRC | 04:22 | |
| *** itsuugo has joined #openstack-lbaas | 04:24 | |
| *** itsuugo has quit IRC | 04:31 | |
| *** itsuugo has joined #openstack-lbaas | 04:32 | |
| *** itsuugo has quit IRC | 04:36 | |
| *** itsuugo has joined #openstack-lbaas | 04:37 | |
| *** itsuugo has quit IRC | 04:42 | |
| *** itsuugo has joined #openstack-lbaas | 04:43 | |
| *** itsuugo has quit IRC | 04:48 | |
| *** itsuugo has joined #openstack-lbaas | 04:49 | |
| *** itsuugo has quit IRC | 04:53 | |
| *** itsuugo has joined #openstack-lbaas | 04:55 | |
| *** ducttape_ has joined #openstack-lbaas | 04:57 | |
| *** ducttape_ has quit IRC | 05:03 | |
| *** itsuugo has quit IRC | 05:06 | |
| *** itsuugo has joined #openstack-lbaas | 05:06 | |
| *** cody-somerville has joined #openstack-lbaas | 05:11 | |
| *** itsuugo has quit IRC | 05:11 | |
| *** itsuugo has joined #openstack-lbaas | 05:12 | |
| *** amitkqed has quit IRC | 05:16 | |
| *** amitkqed has joined #openstack-lbaas | 05:16 | |
| *** itsuugo has quit IRC | 05:17 | |
| *** itsuugo has joined #openstack-lbaas | 05:17 | |
| *** itsuugo has quit IRC | 05:24 | |
| *** csomerville has joined #openstack-lbaas | 05:25 | |
| *** itsuugo has joined #openstack-lbaas | 05:25 | |
| *** cody-somerville has quit IRC | 05:28 | |
| *** itsuugo has quit IRC | 05:30 | |
| *** itsuugo has joined #openstack-lbaas | 05:31 | |
| *** itsuugo has quit IRC | 05:36 | |
| *** itsuugo has joined #openstack-lbaas | 05:36 | |
| *** itsuugo has quit IRC | 05:44 | |
| *** Alex_Stef has quit IRC | 05:44 | |
| *** itsuugo has joined #openstack-lbaas | 05:46 | |
| *** eezhova has joined #openstack-lbaas | 05:49 | |
| *** itsuugo has quit IRC | 05:54 | |
| *** itsuugo has joined #openstack-lbaas | 05:54 | |
| *** ducttape_ has joined #openstack-lbaas | 05:59 | |
| *** itsuugo has quit IRC | 05:59 | |
| *** itsuugo has joined #openstack-lbaas | 06:00 | |
| *** ducttape_ has quit IRC | 06:03 | |
| *** kevo has joined #openstack-lbaas | 06:03 | |
| *** eezhova has quit IRC | 06:08 | |
| *** itsuugo has quit IRC | 06:12 | |
| *** itsuugo has joined #openstack-lbaas | 06:13 | |
| *** itsuugo has quit IRC | 06:18 | |
| *** rcernin has joined #openstack-lbaas | 06:19 | |
| *** itsuugo has joined #openstack-lbaas | 06:19 | |
| *** itsuugo has quit IRC | 06:24 | |
| *** itsuugo has joined #openstack-lbaas | 06:25 | |
| *** kobis has joined #openstack-lbaas | 06:27 | |
| *** itsuugo has quit IRC | 06:36 | |
| *** itsuugo has joined #openstack-lbaas | 06:37 | |
| *** fnaval has quit IRC | 06:47 | |
| *** itsuugo has quit IRC | 06:49 | |
| *** itsuugo has joined #openstack-lbaas | 06:50 | |
| *** fnaval has joined #openstack-lbaas | 06:50 | |
| *** fnaval has quit IRC | 06:55 | |
| *** woodster_ has quit IRC | 06:59 | |
| *** ducttape_ has joined #openstack-lbaas | 07:00 | |
| *** pcaruana has joined #openstack-lbaas | 07:02 | |
| *** ducttape_ has quit IRC | 07:04 | |
| *** nmagnezi has joined #openstack-lbaas | 07:05 | |
| *** fnaval has joined #openstack-lbaas | 07:06 | |
| *** tesseract- has joined #openstack-lbaas | 07:08 | |
| *** fnaval has quit IRC | 07:10 | |
| *** fnaval has joined #openstack-lbaas | 07:24 | |
| *** fnaval has quit IRC | 07:32 | |
| *** fnaval has joined #openstack-lbaas | 07:33 | |
| *** itsuugo has quit IRC | 07:37 | |
| *** fnaval has quit IRC | 07:38 | |
| *** itsuugo has joined #openstack-lbaas | 07:39 | |
| *** Alex_Stef has joined #openstack-lbaas | 07:40 | |
| *** fnaval has joined #openstack-lbaas | 07:45 | |
| *** itsuugo has quit IRC | 07:45 | |
| *** itsuugo has joined #openstack-lbaas | 07:47 | |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** openstackgerrit has joined #openstack-lbaas | 07:49 | |
| openstackgerrit | Nir Magnezi proposed openstack/neutron-lbaas: Adds neutron_lbaas.conf and services_lbaas.conf to q-svc command line https://review.openstack.org/366690 | 07:52 |
| *** kevo has quit IRC | 07:55 | |
| *** itsuugo has quit IRC | 07:56 | |
| *** csomerville has quit IRC | 07:57 | |
| *** itsuugo has joined #openstack-lbaas | 07:57 | |
| *** fnaval has quit IRC | 07:58 | |
| *** csomerville has joined #openstack-lbaas | 07:58 | |
| *** fnaval has joined #openstack-lbaas | 08:01 | |
| *** ducttape_ has joined #openstack-lbaas | 08:01 | |
| *** itsuugo has quit IRC | 08:02 | |
| *** itsuugo has joined #openstack-lbaas | 08:03 | |
| *** ducttape_ has quit IRC | 08:06 | |
| *** csomerville has quit IRC | 08:08 | |
| *** cody-somerville has joined #openstack-lbaas | 08:09 | |
| *** itsuugo has quit IRC | 08:10 | |
| *** itsuugo has joined #openstack-lbaas | 08:12 | |
| *** lmiccini_ has joined #openstack-lbaas | 08:19 | |
| *** cgross has quit IRC | 08:19 | |
| *** lmiccini has quit IRC | 08:20 | |
| *** cgross has joined #openstack-lbaas | 08:21 | |
| *** cody-somerville has quit IRC | 08:22 | |
| *** lmiccini_ is now known as lmiccini | 08:25 | |
| *** eezhova has joined #openstack-lbaas | 08:26 | |
| *** amotoki has joined #openstack-lbaas | 08:32 | |
| *** zigo_ is now known as zigo | 08:34 | |
| *** yamamoto has joined #openstack-lbaas | 08:39 | |
| *** amotoki has quit IRC | 08:44 | |
| *** itsuugo has quit IRC | 08:53 | |
| *** itsuugo has joined #openstack-lbaas | 08:54 | |
| *** itsuugo has quit IRC | 08:59 | |
| *** itsuugo has joined #openstack-lbaas | 09:00 | |
| *** amotoki has joined #openstack-lbaas | 09:05 | |
| *** gongysh has quit IRC | 09:22 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia: Amphora agent refactor to classes https://review.openstack.org/346728 | 09:35 |
| *** amotoki has quit IRC | 09:37 | |
| *** itsuugo has quit IRC | 09:45 | |
| *** amotoki has joined #openstack-lbaas | 09:46 | |
| *** itsuugo has joined #openstack-lbaas | 09:47 | |
| *** amotoki has quit IRC | 09:56 | |
| *** ducttape_ has joined #openstack-lbaas | 10:02 | |
| *** itsuugo has quit IRC | 10:06 | |
| *** amotoki has joined #openstack-lbaas | 10:07 | |
| *** ducttape_ has quit IRC | 10:07 | |
| *** itsuugo has joined #openstack-lbaas | 10:08 | |
| *** permalac has joined #openstack-lbaas | 10:17 | |
| *** lmiccini has quit IRC | 10:25 | |
| *** cgross has quit IRC | 10:25 | |
| *** itsuugo has quit IRC | 10:43 | |
| *** itsuugo has joined #openstack-lbaas | 10:44 | |
| *** permalac has quit IRC | 10:51 | |
| *** itsuugo has quit IRC | 10:58 | |
| *** itsuugo has joined #openstack-lbaas | 10:59 | |
| *** ducttape_ has joined #openstack-lbaas | 11:03 | |
| *** ducttape_ has quit IRC | 11:07 | |
| *** itsuugo has quit IRC | 11:09 | |
| *** itsuugo has joined #openstack-lbaas | 11:09 | |
| openstackgerrit | Nir Magnezi proposed openstack/neutron-lbaas: Adds neutron_lbaas.conf and services_lbaas.conf to q-svc command line https://review.openstack.org/366690 | 11:17 |
| *** itsuugo has quit IRC | 11:22 | |
| *** itsuugo has joined #openstack-lbaas | 11:23 | |
| *** yamamoto has quit IRC | 11:35 | |
| *** itsuugo has quit IRC | 11:48 | |
| *** amotoki has quit IRC | 11:49 | |
| *** itsuugo has joined #openstack-lbaas | 11:49 | |
| *** itsuugo has quit IRC | 11:59 | |
| *** itsuugo has joined #openstack-lbaas | 12:00 | |
| *** ducttape_ has joined #openstack-lbaas | 12:04 | |
| *** ducttape_ has quit IRC | 12:09 | |
| *** ducttape_ has joined #openstack-lbaas | 12:13 | |
| *** ducttape_ has quit IRC | 12:14 | |
| *** yamamoto has joined #openstack-lbaas | 12:17 | |
| *** ducttape_ has joined #openstack-lbaas | 12:17 | |
| *** permalac has joined #openstack-lbaas | 12:33 | |
| *** ducttape_ has quit IRC | 12:38 | |
| *** itsuugo has quit IRC | 12:42 | |
| *** yamamoto has quit IRC | 12:42 | |
| *** yamamoto has joined #openstack-lbaas | 12:43 | |
| *** itsuugo has joined #openstack-lbaas | 12:43 | |
| *** yamamoto has quit IRC | 12:47 | |
| *** portdirect has joined #openstack-lbaas | 12:49 | |
| *** ducttape_ has joined #openstack-lbaas | 12:59 | |
| *** yamamoto has joined #openstack-lbaas | 13:09 | |
| *** ducttape_ has quit IRC | 13:09 | |
| *** nmagnezi_ has joined #openstack-lbaas | 13:12 | |
| *** nmagnezi has quit IRC | 13:13 | |
| *** nmagnezi_ is now known as nmagnezi | 13:14 | |
| *** woodster_ has joined #openstack-lbaas | 13:18 | |
| *** matt-borland has joined #openstack-lbaas | 13:26 | |
| *** amotoki has joined #openstack-lbaas | 13:26 | |
| *** amotoki has quit IRC | 13:28 | |
| *** brad_behle has joined #openstack-lbaas | 13:47 | |
| *** amotoki has joined #openstack-lbaas | 13:59 | |
| *** yamamoto has quit IRC | 13:59 | |
| *** diogogmt has quit IRC | 14:00 | |
| *** ducttape_ has joined #openstack-lbaas | 14:04 | |
| *** nmagnezi has quit IRC | 14:05 | |
| *** ducttape_ has quit IRC | 14:08 | |
| *** carrbs has joined #openstack-lbaas | 14:12 | |
| *** yamamoto has joined #openstack-lbaas | 14:25 | |
| *** pglass has joined #openstack-lbaas | 14:33 | |
| *** ducttape_ has joined #openstack-lbaas | 14:43 | |
| *** amotoki has quit IRC | 14:44 | |
| *** amotoki has joined #openstack-lbaas | 14:44 | |
| *** amotoki has quit IRC | 14:52 | |
| *** itsuugo has quit IRC | 14:53 | |
| *** itsuugo has joined #openstack-lbaas | 14:54 | |
| *** yamamoto has quit IRC | 15:04 | |
| *** armax has joined #openstack-lbaas | 15:09 | |
| *** reedip has quit IRC | 15:12 | |
| *** ducttape_ has quit IRC | 15:13 | |
| *** reedip has joined #openstack-lbaas | 15:13 | |
| *** ducttape_ has joined #openstack-lbaas | 15:13 | |
| *** yamamoto has joined #openstack-lbaas | 15:20 | |
| *** itsuugo has quit IRC | 15:23 | |
| *** itsuugo has joined #openstack-lbaas | 15:24 | |
| *** diogogmt has joined #openstack-lbaas | 15:25 | |
| *** cody-somerville has joined #openstack-lbaas | 15:25 | |
| *** cody-somerville has quit IRC | 15:25 | |
| *** cody-somerville has joined #openstack-lbaas | 15:25 | |
| *** mixos has joined #openstack-lbaas | 15:26 | |
| *** ducttape_ has quit IRC | 15:40 | |
| *** ducttape_ has joined #openstack-lbaas | 15:41 | |
| sbalukoff | diltram: Please have a look at my response to your comments on the documentation patch. I know I'm coming across as pretty harsh there, but I'm honestly getting sick of quibbling over this, and having half-helpful feedback. (Saying, "This is wrong" is only half as helpful as "this is wrong, and here's how you correct it...") | 15:50 |
| *** cody-somerville has quit IRC | 15:52 | |
| *** cody-somerville has joined #openstack-lbaas | 16:09 | |
| *** permalac has quit IRC | 16:20 | |
| *** cody-somerville has quit IRC | 16:23 | |
| *** Alex_Stef has quit IRC | 16:25 | |
| *** kobis has quit IRC | 16:33 | |
| *** cody-somerville has joined #openstack-lbaas | 16:34 | |
| *** cody-somerville has quit IRC | 16:34 | |
| *** cody-somerville has joined #openstack-lbaas | 16:34 | |
| *** BjoernT has joined #openstack-lbaas | 16:38 | |
| sbalukoff | Gerrit's being it's awesome "responsive" self today. Is there a deadline today that I don't know about? | 16:42 |
| sbalukoff | er... its. | 16:42 |
| *** yamamoto has quit IRC | 16:45 | |
| *** tesseract- has quit IRC | 16:46 | |
| *** kevo has joined #openstack-lbaas | 16:48 | |
| *** pcaruana has quit IRC | 16:59 | |
| *** rcernin has quit IRC | 17:03 | |
| johnsom | RC1 is today | 17:05 |
| *** cody-somerville has quit IRC | 17:05 | |
| *** eezhova has quit IRC | 17:07 | |
| *** cody-somerville has joined #openstack-lbaas | 17:08 | |
| *** cody-somerville has quit IRC | 17:08 | |
| *** cody-somerville has joined #openstack-lbaas | 17:08 | |
| sbalukoff | Aah! Ok. | 17:10 |
| *** itsuugo has quit IRC | 17:14 | |
| *** fishbone has joined #openstack-lbaas | 17:15 | |
| fishbone | is it possible to ask questions about lbaasv2 with haproxy in this chat? | 17:16 |
| *** itsuugo has joined #openstack-lbaas | 17:17 | |
| *** ducttape_ has quit IRC | 17:18 | |
| *** bana_k has joined #openstack-lbaas | 17:20 | |
| sbalukoff | fishbone: You mean with the namespace driver? This would be the place to ask questions, but to be honest, most people here are working on the reference driver (Octavia). | 17:25 |
| johnsom | fishbone This is the right place | 17:25 |
| *** cody-somerville has quit IRC | 17:27 | |
| *** kevo has quit IRC | 17:29 | |
| dougwig | you scared off fishbone | 17:36 |
| sbalukoff | Well... I am an asshole. | 17:36 |
| *** brad_behle has quit IRC | 17:38 | |
| fishbone | haha sorry, I am at work | 17:38 |
| fishbone | I have an error in the lbaasv2 dashboard panel when I try to create a load balancer | 17:38 |
| fishbone | when I get to the final step and submit it says "error: source_type" | 17:39 |
| fishbone | I am able to create and list all load balancer items in the command line | 17:39 |
| fishbone | but if i create anything the panel changes error to "cannot retrieve pool list" | 17:40 |
| fishbone | I am using lbaasv2 with haproxy, and have the lbaasv2 panel installed in horizon | 17:40 |
| zetab | someone could code nginx for lbaasv2 ;) | 17:41 |
| dougwig | fishbone: are you using the lbaasv1 or v2 horizon dashboard? | 17:42 |
| *** cody-somerville has joined #openstack-lbaas | 17:43 | |
| *** yamamoto has joined #openstack-lbaas | 17:46 | |
| fishbone | I am using lbaasv2 | 17:49 |
| sbalukoff | johnsom: Could I ask your opinion on something: Should we just disable ulimits for filehandles and whatnot on amphorae, or set them to some arbitrarily high value that we hope won't cause the amphorae to crash if a tenant sets a listener connection limit to some ridiculously high value? On that note as well, should we set a maximum value to which the tenant can set the connection limit? | 17:49 |
| fishbone | I don't know what the version of horizon is but it came from the mitaka repository package | 17:49 |
| dougwig | sbalukoff: shouldn't we set them to some multiple of the requested limits? | 17:49 |
| dougwig | fishbone: any errors in the neutron server log, or is the dashboard never even making the lbaas api request? | 17:50 |
| sbalukoff | dougwig: We could potentially adjust this on the fly, but if we're going to have a maximum possible limit that we allow, there's no reason we can't just do this on the amphora image at boot (ie. with a disimagebuilder element) | 17:51 |
| *** yamamoto has quit IRC | 17:51 | |
| dougwig | sbalukoff: true. i was thinking of catching leak conditions quicker, but don't have a strong preference. | 17:51 |
| sbalukoff | I think haproxy is "being good" with how it pre-allocates filehandles and whatnot. | 17:52 |
| *** ducttape_ has joined #openstack-lbaas | 17:52 | |
| sbalukoff | So if there's a leak, I wouldn't expect to see it in haproxy's file handles. | 17:52 |
| johnsom | sbalukoff A couple things to note: | 17:53 |
| johnsom | https://github.com/openstack/octavia/blob/master/elements/haproxy-octavia-ubuntu/post-install.d/20-haproxy-tune-kernel#L19 | 17:53 |
| *** amotoki has joined #openstack-lbaas | 17:53 | |
| sbalukoff | Oh! So we're already trying to tune it. So... yeah, I can just fix that bug. | 17:54 |
| johnsom | Aside from that, doesn't the ulimit get set by haproxy based on maxconn? | 17:55 |
| sbalukoff | It's supposed to. But when I'm testing the maxconn patch I wrote, I'm seeing this in the haproxy logs: [WARNING] 256/165039 (20882) : [/usr/sbin/haproxy.main()] FD limit (1024) too low for maxconn=1000000/maxsock=2000030. Please raise 'ulimit-n' to 2000030 or more to avoid any trouble. | 17:56 |
| *** cody-somerville has quit IRC | 17:57 | |
| johnsom | So, this isn't set/working? https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#maxconn | 17:57 |
| *** amotoki has quit IRC | 17:58 | |
| johnsom | Or is is just a bogus warning | 17:58 |
| openstackgerrit | Cedric Shock proposed openstack/neutron-lbaas: Allow creating loadbalancer with network_id https://review.openstack.org/363302 | 17:59 |
| johnsom | I guess we could set this too: https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#ulimit-n | 17:59 |
| sbalukoff | I'm still looking into it. | 17:59 |
| sbalukoff | In any case, our past intent I think is clearer to me now. | 18:00 |
| johnsom | Ok, cool. I have been looking at these patches today. I hope my afternoon stays free and I can crank through a bunch of reviews today. | 18:01 |
| sbalukoff | Nice! | 18:01 |
| openstackgerrit | Phillip Toohill proposed openstack/octavia: WIP: Octavia: Basic LoadBalancer Scenario Test https://review.openstack.org/172199 | 18:01 |
| sbalukoff | I think Dustin's patches should be good to go-- they wouldn't have the same ulimit concern anyway. | 18:01 |
| johnsom | In theory they would, wouldn't they? | 18:02 |
| sbalukoff | Well, the namespace driver is less concerned with the underlying host's tuning. | 18:02 |
| sbalukoff | On Octavia, we have to be concerned with this, since the amphora is supposed to act like a virtual appliance. | 18:02 |
| *** ducttape_ has quit IRC | 18:03 | |
| *** cody-somerville has joined #openstack-lbaas | 18:03 | |
| *** cody-somerville has quit IRC | 18:03 | |
| *** cody-somerville has joined #openstack-lbaas | 18:03 | |
| johnsom | Yes, true, but we should set the haproxy settings up similar I expect. | 18:03 |
| sbalukoff | In other words, with the namespace driver, if you get a log warning like that, it's up to the operator to re-tune the hosts running the lbassv2 namespace agent. From within lbaasv2, I think it's generally a bad idea to try to tune the host since were really have no idea how it's deployed. | 18:04 |
| sbalukoff | Oh yes! His patches do that. | 18:04 |
| johnsom | Yeah, I agree we should not mess with the host | 18:04 |
| *** portdirect has quit IRC | 18:04 | |
| sbalukoff | But Octavia "owns" the whole amphora. So I'll need to fix the amphora VM tuning (if necessary) in my patch for the patch to be meaningful. | 18:05 |
| johnsom | Yep | 18:05 |
| sbalukoff | Ok. Gonna go heads down for while and poke at this problem. | 18:06 |
| dlundquist | This article has a number of tuning suggestions for proxying a large number of connections https://mrotaru.wordpress.com/2015/05/20/how-migratorydata-solved-the-c10m-problem-10-million-concurrent-connections-on-a-single-commodity-server/. I wanted to take care of the 2k haproxy default limit first, then look at further optmizations. | 18:12 |
| *** rcernin has joined #openstack-lbaas | 18:12 | |
| *** ducttape_ has joined #openstack-lbaas | 18:13 | |
| dlundquist | sbalukoff: ^^^ | 18:13 |
| sbalukoff | dlundquist: Aah! Yes, thanks for that! | 18:15 |
| dlundquist | Raising the ulimit value to unlimited for the amphora does seem like a good optimization, but I think there is a fair amount of work down this path, but a 1m concurrent connection Octavia demo would be pretty cool. | 18:16 |
| sbalukoff | Well... it may make sense to just make a separate tuning patch based on what I'm seeing. I mean, the one there addresses the haproxy config problem, but tuning is probably a seperate concern. | 18:17 |
| sbalukoff | So yeah... let me get rid of the -workflow on that. | 18:17 |
| dlundquist | Also, network tuning sysctl's still apply to namespace driver. | 18:18 |
| sbalukoff | dlundquist: I would consider that a deployment problem. | 18:19 |
| dlundquist | actualy not, since each new namespace has default network stack tuning | 18:19 |
| dlundquist | the deployer can not configure those | 18:19 |
| sbalukoff | Oh? | 18:19 |
| dlundquist | Since they are create on demand, we could ship a lbaas-ns-sysctl file and let them customize as needed | 18:20 |
| sbalukoff | Er... where does it get the 'default network stack tuning'? | 18:20 |
| dlundquist | ferry docking now, back in a bit | 18:20 |
| dlundquist | complied into linux | 18:20 |
| sbalukoff | Aah, crapsticks. | 18:20 |
| sbalukoff | Well, that probably explains why the haproxy running in a namespace on the amphora is seeing a 1024 file limit. :P | 18:21 |
| *** pcaruana has joined #openstack-lbaas | 18:36 | |
| fishbone | dougwig: sorry for the late response, no there is nothing in the lbaasv2 logs and the neutron-server log has no errors or warnings | 18:36 |
| openstackgerrit | Trevor Vardeman proposed openstack/octavia: Adding a unit test for graph create failure case https://review.openstack.org/369627 | 18:37 |
| fishbone | dougwig: how would I know if the api requests are being made? That sounds like it may be the issue | 18:37 |
| johnsom | fishbone You can check the q-svc.log The API calls should be logged in there for LBaaSv2 | 18:40 |
| fishbone | johnsom: thanks I will check and get back | 18:41 |
| johnsom | dlundquist sbalukoff I see the kernel tuning settings we have setup for the amp visible when I sysctl -a from inside the namespace. Are you sure they aren't using the /etc/sysctl.d settings? | 18:45 |
| dlundquist | johnsom: some of them apply, only network ones do not | 18:46 |
| dlundquist | Example woudl net.ipv4.conf.all.forwarding | 18:47 |
| johnsom | Yep, ok, I see it now. I was too lucky on what I queried the first time | 18:48 |
| dlundquist | Ulimit should not be different in a network namespace since it is a filesystem configuration option | 18:48 |
| johnsom | Interesting, I wonder how we aren't having issues with the nonlocal_bind not being set | 18:48 |
| dlundquist | maybe it's more nuanced... It's easy enough to test | 18:51 |
| johnsom | Interesting to note as well, sysctl -- system seems to have applied some of the settings, but not all. port_range came in, but rmem did not | 18:53 |
| fishbone | johnsom I don't have a file with that name on my controller node | 18:55 |
| TrevorV | diltram, you around? | 19:03 |
| *** johnsom has quit IRC | 19:16 | |
| *** amitkqed has quit IRC | 19:16 | |
| *** markd_ has quit IRC | 19:16 | |
| *** Dave has quit IRC | 19:16 | |
| *** bedis has quit IRC | 19:16 | |
| *** davidlenwell has quit IRC | 19:17 | |
| *** rm_work has quit IRC | 19:17 | |
| *** bcafarel has quit IRC | 19:17 | |
| *** mhayden has quit IRC | 19:17 | |
| *** Kdecherf has quit IRC | 19:17 | |
| *** logan- has quit IRC | 19:17 | |
| *** jorgem has quit IRC | 19:17 | |
| *** zigo has quit IRC | 19:17 | |
| *** yuanying has quit IRC | 19:17 | |
| *** ctracey has quit IRC | 19:17 | |
| *** HenryG has quit IRC | 19:17 | |
| *** nagyz has quit IRC | 19:17 | |
| *** dougwig has quit IRC | 19:17 | |
| *** fnaval has quit IRC | 19:17 | |
| *** kbyrne has quit IRC | 19:17 | |
| *** darrenc has quit IRC | 19:17 | |
| *** jidar has quit IRC | 19:17 | |
| *** kong has quit IRC | 19:17 | |
| *** dlundquist has quit IRC | 19:17 | |
| *** fishbone has quit IRC | 19:17 | |
| *** sbalukoff has quit IRC | 19:17 | |
| *** barclaac has quit IRC | 19:17 | |
| *** blogan has quit IRC | 19:17 | |
| *** johnsom_ has quit IRC | 19:17 | |
| *** mjblack has quit IRC | 19:17 | |
| *** ptoohill has quit IRC | 19:17 | |
| *** xgerman has quit IRC | 19:17 | |
| *** bana_k has quit IRC | 19:17 | |
| *** reedip has quit IRC | 19:17 | |
| *** carrbs has quit IRC | 19:17 | |
| *** harlowja has quit IRC | 19:17 | |
| *** ipsecguy has quit IRC | 19:17 | |
| *** tries_ has quit IRC | 19:17 | |
| *** rcernin has quit IRC | 19:17 | |
| *** woodster_ has quit IRC | 19:17 | |
| *** AlexeyAbashkin has quit IRC | 19:17 | |
| *** amit213 has quit IRC | 19:17 | |
| *** Kiall_ has quit IRC | 19:17 | |
| *** amitry has quit IRC | 19:17 | |
| *** _laco has quit IRC | 19:17 | |
| *** malos has quit IRC | 19:17 | |
| *** cody-somerville has quit IRC | 19:17 | |
| *** matt-borland has quit IRC | 19:17 | |
| *** njohnston has quit IRC | 19:18 | |
| *** ajo has quit IRC | 19:18 | |
| *** zetab has quit IRC | 19:18 | |
| *** BjoernT has quit IRC | 19:18 | |
| *** diogogmt has quit IRC | 19:18 | |
| *** chlong_ has quit IRC | 19:18 | |
| *** eandersson has quit IRC | 19:18 | |
| *** diltram has quit IRC | 19:18 | |
| *** frickler has quit IRC | 19:18 | |
| *** crc32|znc has quit IRC | 19:18 | |
| *** afranc has quit IRC | 19:18 | |
| *** itsuugo has quit IRC | 19:18 | |
| *** pck has quit IRC | 19:18 | |
| *** bradjones has quit IRC | 19:18 | |
| *** kevinbenton has quit IRC | 19:18 | |
| *** TrevorV has quit IRC | 19:18 | |
| *** sirushti has quit IRC | 19:18 | |
| *** zw has quit IRC | 19:18 | |
| *** pcaruana has quit IRC | 19:18 | |
| *** mixos has quit IRC | 19:18 | |
| *** pglass has quit IRC | 19:18 | |
| *** sbelous_ has quit IRC | 19:18 | |
| *** vmtyler has quit IRC | 19:18 | |
| *** mestery has quit IRC | 19:18 | |
| *** basilAB has quit IRC | 19:18 | |
| *** greghaynes has quit IRC | 19:18 | |
| *** openstackgerrit has quit IRC | 19:18 | |
| *** sticker has quit IRC | 19:18 | |
| *** dasm has quit IRC | 19:18 | |
| *** zw has joined #openstack-lbaas | 19:23 | |
| *** basilAB has joined #openstack-lbaas | 19:23 | |
| *** johnsom has joined #openstack-lbaas | 19:23 | |
| *** crc32|znc has joined #openstack-lbaas | 19:23 | |
| *** markd_ has joined #openstack-lbaas | 19:23 | |
| *** pglass has joined #openstack-lbaas | 19:23 | |
| *** chlong_ has joined #openstack-lbaas | 19:23 | |
| *** sirushti has joined #openstack-lbaas | 19:23 | |
| *** pcaruana has joined #openstack-lbaas | 19:23 | |
| *** xgerman has joined #openstack-lbaas | 19:23 | |
| *** ptoohill has joined #openstack-lbaas | 19:23 | |
| *** mjblack has joined #openstack-lbaas | 19:23 | |
| *** johnsom_ has joined #openstack-lbaas | 19:23 | |
| *** blogan has joined #openstack-lbaas | 19:23 | |
| *** barclaac has joined #openstack-lbaas | 19:23 | |
| *** sbalukoff has joined #openstack-lbaas | 19:23 | |
| *** pck has joined #openstack-lbaas | 19:23 | |
| *** malos has joined #openstack-lbaas | 19:23 | |
| *** _laco has joined #openstack-lbaas | 19:23 | |
| *** Kiall_ has joined #openstack-lbaas | 19:23 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 19:23 | |
| *** rcernin has joined #openstack-lbaas | 19:23 | |
| *** itsuugo has joined #openstack-lbaas | 19:23 | |
| *** zigo has joined #openstack-lbaas | 19:23 | |
| *** yuanying has joined #openstack-lbaas | 19:23 | |
| *** HenryG has joined #openstack-lbaas | 19:23 | |
| *** nagyz has joined #openstack-lbaas | 19:23 | |
| *** mestery has joined #openstack-lbaas | 19:23 | |
| *** amitkqed has joined #openstack-lbaas | 19:23 | |
| *** cody-somerville has joined #openstack-lbaas | 19:23 | |
| *** matt-borland has joined #openstack-lbaas | 19:23 | |
| *** njohnston has joined #openstack-lbaas | 19:23 | |
| *** zetab has joined #openstack-lbaas | 19:23 | |
| *** ajo has joined #openstack-lbaas | 19:23 | |
| *** bedis_ has joined #openstack-lbaas | 19:23 | |
| *** davidlenwell has joined #openstack-lbaas | 19:23 | |
| *** rm_work has joined #openstack-lbaas | 19:23 | |
| *** bcafarel has joined #openstack-lbaas | 19:23 | |
| *** mhayden has joined #openstack-lbaas | 19:23 | |
| *** Kdecherf has joined #openstack-lbaas | 19:23 | |
| *** logan- has joined #openstack-lbaas | 19:23 | |
| *** jorgem has joined #openstack-lbaas | 19:23 | |
| *** eandersson has joined #openstack-lbaas | 19:23 | |
| *** openstackgerrit has joined #openstack-lbaas | 19:23 | |
| *** sticker has joined #openstack-lbaas | 19:23 | |
| *** dasm has joined #openstack-lbaas | 19:23 | |
| *** diltram has joined #openstack-lbaas | 19:23 | |
| *** frickler has joined #openstack-lbaas | 19:23 | |
| *** kevinbenton has joined #openstack-lbaas | 19:23 | |
| *** bradjones has joined #openstack-lbaas | 19:23 | |
| *** afranc has joined #openstack-lbaas | 19:23 | |
| *** fishbone has joined #openstack-lbaas | 19:23 | |
| *** greghaynes has joined #openstack-lbaas | 19:23 | |
| *** bana_k has joined #openstack-lbaas | 19:23 | |
| *** reedip has joined #openstack-lbaas | 19:23 | |
| *** carrbs has joined #openstack-lbaas | 19:23 | |
| *** harlowja has joined #openstack-lbaas | 19:23 | |
| *** tries_ has joined #openstack-lbaas | 19:23 | |
| *** ipsecguy has joined #openstack-lbaas | 19:23 | |
| *** fnaval has joined #openstack-lbaas | 19:23 | |
| *** kbyrne has joined #openstack-lbaas | 19:23 | |
| *** darrenc has joined #openstack-lbaas | 19:23 | |
| *** jidar has joined #openstack-lbaas | 19:23 | |
| *** fishbone has quit IRC | 19:24 | |
| *** fishbone has joined #openstack-lbaas | 19:24 | |
| *** bradjones has quit IRC | 19:24 | |
| *** bradjones has joined #openstack-lbaas | 19:24 | |
| openstackgerrit | Dustin Lundquist proposed openstack/neutron-lbaas: haproxy ns driver: set global haproxy maxconn https://review.openstack.org/369088 | 19:25 |
| *** TrevorV has joined #openstack-lbaas | 19:26 | |
| *** Dave has joined #openstack-lbaas | 19:30 | |
| *** csomerville has joined #openstack-lbaas | 19:43 | |
| *** sbelous_ has joined #openstack-lbaas | 19:43 | |
| fishbone | johnsom also when I do an ip netns command the loadbalancers created in command line do not show | 19:44 |
| *** cody-somerville has quit IRC | 19:47 | |
| *** woodster_ has joined #openstack-lbaas | 19:48 | |
| *** dlundquist has joined #openstack-lbaas | 19:48 | |
| *** johnsom has quit IRC | 19:54 | |
| openstackgerrit | Dustin Lundquist proposed openstack/neutron-lbaas: haproxy ns driver: set global haproxy maxconn https://review.openstack.org/369088 | 19:55 |
| *** amit213 has joined #openstack-lbaas | 19:59 | |
| *** kev0 has joined #openstack-lbaas | 20:00 | |
| *** amitry has joined #openstack-lbaas | 20:00 | |
| *** eezhova has joined #openstack-lbaas | 20:00 | |
| *** kong has joined #openstack-lbaas | 20:01 | |
| *** ctracey has joined #openstack-lbaas | 20:03 | |
| *** dougwig has joined #openstack-lbaas | 20:03 | |
| *** dougw2 has joined #openstack-lbaas | 20:06 | |
| *** vmtyler has joined #openstack-lbaas | 20:08 | |
| *** pcaruana has quit IRC | 20:11 | |
| *** johnsom has joined #openstack-lbaas | 20:13 | |
| fishbone | when I create a load balancer in the lbaasv2 logs it says stats socket not found, I think the issue is with the haproxy namespace driver | 20:20 |
| *** dougw21 has joined #openstack-lbaas | 20:25 | |
| *** dougw2 has quit IRC | 20:27 | |
| *** dougw21 has quit IRC | 20:32 | |
| *** harlowja has quit IRC | 21:04 | |
| *** fishbone has quit IRC | 21:07 | |
| *** markd_ has quit IRC | 21:12 | |
| *** amitry has quit IRC | 21:27 | |
| *** amit213 has quit IRC | 21:27 | |
| *** greghaynes has quit IRC | 21:27 | |
| *** bana_k has quit IRC | 21:27 | |
| *** reedip has quit IRC | 21:27 | |
| *** carrbs has quit IRC | 21:27 | |
| *** ipsecguy has quit IRC | 21:27 | |
| *** tries_ has quit IRC | 21:28 | |
| *** ipsecguy has joined #openstack-lbaas | 21:28 | |
| *** tries_ has joined #openstack-lbaas | 21:28 | |
| *** tries_ has quit IRC | 21:28 | |
| *** tries_ has joined #openstack-lbaas | 21:28 | |
| *** reedip has joined #openstack-lbaas | 21:28 | |
| *** bana_k has joined #openstack-lbaas | 21:28 | |
| *** greghaynes has joined #openstack-lbaas | 21:28 | |
| *** carrbs has joined #openstack-lbaas | 21:31 | |
| *** matt-borland has quit IRC | 21:33 | |
| *** amit213 has joined #openstack-lbaas | 21:34 | |
| *** amitry has joined #openstack-lbaas | 21:35 | |
| *** markd_ has joined #openstack-lbaas | 21:43 | |
| sbalukoff | That's bizarre... so dlundquist's patch to restore those unit tests is failing in the gate now, and the specific tests that are failing are not related to the tests he restored. | 21:49 |
| sbalukoff | But it's failing in the merge gate where it passed the tests previously. | 21:49 |
| sbalukoff | (The fix looks like it should be simple-- the six failing tests are all API tests that are now including a project_id whereas they didn't before.) | 21:50 |
| sbalukoff | But it's a little disturbing to me to see the merge gate spontaneously fail like this. | 21:51 |
| sbalukoff | Aah-- Ok, looks like something merged upstream (again) last night which is causing the gate to fail (again) (both test and merge). Dang, that's annoying. :P | 21:52 |
| dlundquist | Yeah, my dependent patch just started failing (passed locally). Removed local .tox environment and it fail locally now | 21:52 |
| sbalukoff | Ok. | 21:53 |
| sbalukoff | So yeah, it's a Neutron thing. | 21:53 |
| sbalukoff | Do you want to make a patch to fix this, or shall I? | 21:53 |
| dlundquist | I don't see the cause, but would like to learn what it is from your patch. | 21:54 |
| sbalukoff | Ok. | 21:54 |
| sbalukoff | Also, in my poking around on the amphora with ulimits, I see that our setting for file-max actually was working. I was just trying to raise the connection limit too high for it-- the connection limit can be at most approx. half of the file-max limit. Which means we're currently at 500k connections as an effective limit. | 21:58 |
| sbalukoff | And that seems... adequate. :) | 21:58 |
| sbalukoff | But the error that gets reported is reporting the wrong number. Someone should open a bug report with haproxy. ;) | 21:58 |
| sbalukoff | In any case, I'm going to open an RFE bug to do C10k stuff with Octavia. But it'll probably be a pretty low priority for the time being (as we're on countdown-to-release mode right now). | 21:59 |
| dlundquist | I agree, C10k (actually much higher 15+ years later) sounds like RFE work for Ocata | 22:01 |
| *** harlowja has joined #openstack-lbaas | 22:03 | |
| rm_work | lol C10M | 22:05 |
| rm_work | according to wikipedia | 22:05 |
| * rm_work is reading up :P | 22:05 | |
| dasm | dlundquist: sbalukoff if you're talking about http://logs.openstack.org/82/369082/2/gate/gate-neutron-lbaas-python35/4bee452/testr_results.html.gz it looks like affected by https://review.openstack.org/#/c/357977/ | 22:11 |
| *** pglass has quit IRC | 22:11 | |
| sbalukoff | dasm: Yeah, that's what I'm seeing. I'm guessing that patch is not going to get rolled back, eh? | 22:12 |
| sbalukoff | (I'm working on a fix for our tests to expect the project_id) | 22:13 |
| openstackgerrit | Trevor Vardeman proposed openstack/octavia: Add quota support to Octavia https://review.openstack.org/360794 | 22:13 |
| dasm | sbalukoff: not likely. HenryG came up with something like this for vpnaas: https://review.openstack.org/#/c/369717/ | 22:14 |
| dasm | sbalukoff: it's one-way ticket. | 22:14 |
| *** ducttape_ has quit IRC | 22:15 | |
| sbalukoff | dasm: Eh... It's just six failing tests with us; And since that project_id is going to be there in the API request going forward, I don't mind just updating these tests to look specifically for it. | 22:16 |
| dasm | sbalukoff: ack. | 22:18 |
| sbalukoff | Awesome! Launchpad won't let me report this as a bug. Timeout errors. Yay for RC1 day! | 22:35 |
| *** itsuugo has quit IRC | 22:42 | |
| *** itsuugo has joined #openstack-lbaas | 22:45 | |
| openstackgerrit | Stephen Balukoff proposed openstack/neutron-lbaas: Expect project_id in API calls https://review.openstack.org/369743 | 22:47 |
| *** reedip has quit IRC | 22:55 | |
| openstackgerrit | Stephen Balukoff proposed openstack/neutron-lbaas: Expect project_id in API calls https://review.openstack.org/369743 | 23:08 |
| *** reedip has joined #openstack-lbaas | 23:09 | |
| *** ducttape_ has joined #openstack-lbaas | 23:16 | |
| *** itsuugo has quit IRC | 23:20 | |
| sbalukoff | rm_work, dougwig, and/or johnsom: I think this neutron-lbaas gate fix should be ready: https://review.openstack.org/#/c/369743/ | 23:22 |
| sbalukoff | It's not through the check gate yet, but the tests that were failing are now passing. | 23:22 |
| *** itsuugo has joined #openstack-lbaas | 23:22 | |
| johnsom | Ok, will have a look | 23:22 |
| *** rcernin has quit IRC | 23:23 | |
| dougwig | sbalukoff: should we be passing *both* ? and shouldn't the old way work long-term, so why was it breaking? | 23:28 |
| sbalukoff | dougwig: It doesn't break things to pass both. And no matter which one we pass, both will be returned (which is what broke us). | 23:29 |
| dougwig | sbalukoff: oh, i see. | 23:29 |
| johnsom | Yeah, seems like a big change to land this late in the cycle | 23:30 |
| *** gongysh has joined #openstack-lbaas | 23:31 | |
| sbalukoff | Heh! Indeed. | 23:31 |
| sbalukoff | Having said that, multiple gate breaks due to related projects is sort of par for the course the week people are cutting releases. | 23:33 |
| sbalukoff | Also, if y'all agree with Henry's comment on that, I can cut a new patch set which solves things similarly to what he suggested. (I disagree with him, but, I'm happy to be overridden if y'all think differently.) | 23:34 |
| sbalukoff | I'll be working for at least another hour today. :) | 23:34 |
| *** eezhova has quit IRC | 23:36 | |
| johnsom | I'm on the fence. I'm not sure I care too much that they are giving use extra data, but on the other hand notice that "project_id" is coming is useful. | 23:36 |
| johnsom | More to the point, this is done, so I really don't have a problem kicking the can | 23:37 |
| sbalukoff | Heh! | 23:37 |
| dougwig | sbalukoff: i agree with henry, but i'm fine with a follow-up, now or later. | 23:45 |
| dougwig | i'm not going to hold the gate hostage for my opinion. | 23:46 |
| sbalukoff | Ok. | 23:46 |
| sbalukoff | Sounds like we need to start rooting tenant_id out of the code tree anyway. | 23:46 |
| *** itsuugo has quit IRC | 23:46 | |
| sbalukoff | But again, yes, that can happen after the gate is fixed. | 23:46 |
| johnsom | I thought we left it for backward compat. | 23:47 |
| *** itsuugo has joined #openstack-lbaas | 23:47 | |
| dougwig | we have to accept it and return it long-term, we don't have to use it. | 23:47 |
| sbalukoff | Yeah, it'll be hard to get rid of it where we touch 3rd parties (or vendor drivers) | 23:47 |
| dougwig | my gate will be a test that tenant_id still works, since i hate the term 'project'. | 23:48 |
| sbalukoff | Haha! | 23:48 |
| johnsom | Let's be honest and call it bozo_id.... | 23:49 |
| sbalukoff | 'project' is a sort of useless word in the IT industry. It's got as many meanings as "thing" or "entity" | 23:49 |
| sbalukoff | Passive-aggressive variable naming. How very Pacific Northwest of you. ;) | 23:50 |
| johnsom | Ha, I just googled and bozo has a harsher connotation than it did when I was a kid... | 23:51 |
| dougwig | sbalukoff: one of the benefits of evil vendor code is that i can be passive aggressive. | 23:52 |
| johnsom | http://www.waymarking.com/waymarks/WMGJ8M_Bozo_the_Clowns_Childhood_Home_Jacksonville_OR | 23:53 |
| sbalukoff | Oh nice! | 23:54 |
| sbalukoff | The local Seattle clown celebrity is Patches. | 23:54 |
| johnsom | That town is next door to where I grew up. The museum had an exhibit about Bozo the clown.... | 23:54 |
| johnsom | Anyway, probably why there is a regional difference in connotation. | 23:55 |
| *** ducttape_ has quit IRC | 23:55 | |
| sbalukoff | Yep. | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!