| *** oomichi has joined #openstack-lbaas | 00:02 | |
| *** yamamoto has joined #openstack-lbaas | 00:03 | |
| *** cody-somerville has joined #openstack-lbaas | 00:06 | |
| *** yamamoto has quit IRC | 00:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-octaviaclient master: Updated from global requirements https://review.openstack.org/485885 | 00:21 |
|---|---|---|
| *** diltram has quit IRC | 00:24 | |
| *** diltram has joined #openstack-lbaas | 00:31 | |
| *** ssmith has quit IRC | 00:46 | |
| rm_work | johnsom: ah, FYI harlowja pointed out that threads in gunicorn/uwsgi suck a lot | 00:51 |
| rm_work | johnsom: seems like instead of combo of threads/workers, it's almost unilaterally better to just use as however many workers * threads, as JUST WORKERS <_< | 00:51 |
| johnsom | Threads in python just suck... | 00:52 |
| rm_work | so switching from -w4 -t2 to -w8 and things seem... more stable | 00:52 |
| johnsom | Sorry, "threads" | 00:53 |
| johnsom | Air quotes | 00:54 |
| rm_work | heh | 00:57 |
| *** JudeC has quit IRC | 01:06 | |
| *** tongl has quit IRC | 01:07 | |
| *** csomerville has joined #openstack-lbaas | 01:11 | |
| *** cody-somerville has quit IRC | 01:14 | |
| *** armax has joined #openstack-lbaas | 01:26 | |
| *** sanfern has quit IRC | 01:29 | |
| *** sanfern has joined #openstack-lbaas | 01:32 | |
| *** armax has quit IRC | 01:40 | |
| *** sanfern has quit IRC | 02:18 | |
| *** yamamoto has joined #openstack-lbaas | 02:36 | |
| *** kobis has joined #openstack-lbaas | 02:51 | |
| *** kobis has quit IRC | 02:57 | |
| *** yamamoto has quit IRC | 03:08 | |
| *** armax has joined #openstack-lbaas | 03:18 | |
| *** sanfern has joined #openstack-lbaas | 03:49 | |
| *** yamamoto has joined #openstack-lbaas | 04:39 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-octaviaclient master: Updated from global requirements https://review.openstack.org/485885 | 04:46 |
| *** yamamoto has quit IRC | 04:48 | |
| *** yamamoto has joined #openstack-lbaas | 04:51 | |
| *** yamamoto has quit IRC | 05:15 | |
| *** gcheresh has joined #openstack-lbaas | 05:16 | |
| *** armax has quit IRC | 05:22 | |
| *** rcernin has quit IRC | 05:22 | |
| *** gcheresh has quit IRC | 05:29 | |
| *** rcernin has joined #openstack-lbaas | 05:36 | |
| *** gcheresh has joined #openstack-lbaas | 05:52 | |
| *** gcheresh has quit IRC | 06:18 | |
| *** links has joined #openstack-lbaas | 06:37 | |
| *** csomerville has quit IRC | 06:50 | |
| *** rcernin has quit IRC | 06:58 | |
| *** rcernin has joined #openstack-lbaas | 06:58 | |
| *** ndahiwade has quit IRC | 07:08 | |
| *** aojea has joined #openstack-lbaas | 07:16 | |
| *** tesseract has joined #openstack-lbaas | 07:16 | |
| *** yamamoto has joined #openstack-lbaas | 07:20 | |
| *** JudeC has joined #openstack-lbaas | 07:21 | |
| *** yamamoto has quit IRC | 07:42 | |
| *** gcheresh has joined #openstack-lbaas | 07:58 | |
| *** sanfern has quit IRC | 08:30 | |
| *** krypto has joined #openstack-lbaas | 09:05 | |
| *** krypto has quit IRC | 09:10 | |
| *** krypto has joined #openstack-lbaas | 09:10 | |
| *** gcheresh has quit IRC | 09:27 | |
| *** gcheresh has joined #openstack-lbaas | 10:20 | |
| *** yamamoto_ has joined #openstack-lbaas | 10:48 | |
| *** rtjure has quit IRC | 11:16 | |
| *** diltram has quit IRC | 11:21 | |
| *** diltram has joined #openstack-lbaas | 11:22 | |
| *** aojea has quit IRC | 11:22 | |
| *** aojea has joined #openstack-lbaas | 11:30 | |
| *** yamamoto_ has quit IRC | 11:32 | |
| *** aojea has quit IRC | 11:34 | |
| *** aojea has joined #openstack-lbaas | 11:39 | |
| *** aojea has quit IRC | 11:43 | |
| *** aojea has joined #openstack-lbaas | 11:48 | |
| *** diltram has quit IRC | 11:51 | |
| *** yamamoto has joined #openstack-lbaas | 11:52 | |
| *** aojea has quit IRC | 11:53 | |
| *** diltram has joined #openstack-lbaas | 11:56 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** aojea has joined #openstack-lbaas | 11:57 | |
| *** aojea has quit IRC | 12:01 | |
| *** aojea has joined #openstack-lbaas | 12:06 | |
| *** aojea has quit IRC | 12:09 | |
| *** aojea has joined #openstack-lbaas | 12:09 | |
| *** diltram has quit IRC | 12:26 | |
| *** atoth is now known as atoth_guinea_pig | 12:37 | |
| *** atoth_guinea_pig is now known as atoth | 12:37 | |
| *** yamamoto has joined #openstack-lbaas | 12:42 | |
| *** leitan has joined #openstack-lbaas | 12:48 | |
| *** diltram has joined #openstack-lbaas | 12:50 | |
| *** yamamoto has quit IRC | 12:52 | |
| *** yamamoto has joined #openstack-lbaas | 12:52 | |
| *** catintheroof has joined #openstack-lbaas | 13:25 | |
| *** catinthe_ has joined #openstack-lbaas | 13:26 | |
| *** yamamoto_ has joined #openstack-lbaas | 13:28 | |
| *** yamamoto_ has quit IRC | 13:28 | |
| *** yamamoto has quit IRC | 13:28 | |
| *** catintheroof has quit IRC | 13:30 | |
| *** links has quit IRC | 13:43 | |
| *** yamamoto has joined #openstack-lbaas | 13:47 | |
| *** ssmith has joined #openstack-lbaas | 13:50 | |
| *** cpusmith has joined #openstack-lbaas | 13:51 | |
| *** ssmith has quit IRC | 13:55 | |
| *** cpusmith_ has joined #openstack-lbaas | 13:58 | |
| *** ssmith has joined #openstack-lbaas | 14:00 | |
| *** cpusmith_ has quit IRC | 14:00 | |
| *** cpusmith has quit IRC | 14:02 | |
| *** pksingh has joined #openstack-lbaas | 14:09 | |
| pksingh | johnsom: hello | 14:10 |
| pksingh | johnsom: can you please have a look on https://review.openstack.org/#/c/484325/, when you get some time | 14:11 |
| *** armax has joined #openstack-lbaas | 14:16 | |
| *** cpusmith has joined #openstack-lbaas | 14:23 | |
| *** ssmith has quit IRC | 14:27 | |
| *** diltram has quit IRC | 14:37 | |
| *** diltram has joined #openstack-lbaas | 14:41 | |
| *** atoth has quit IRC | 14:56 | |
| *** kobis has joined #openstack-lbaas | 15:04 | |
| johnsom | o/ | 15:13 |
| johnsom | Yeah, I looked at it again yesterday, I will try to comment today | 15:13 |
| *** rcernin has quit IRC | 15:14 | |
| *** rtjure has joined #openstack-lbaas | 15:16 | |
| *** kobis has quit IRC | 15:17 | |
| openstackgerrit | Merged openstack/python-octaviaclient master: Updated from global requirements https://review.openstack.org/485885 | 15:21 |
| *** gcheresh has quit IRC | 15:25 | |
| xgerman_ | o/ | 15:26 |
| *** dayou has quit IRC | 15:26 | |
| *** aojea has quit IRC | 15:30 | |
| *** aojea has joined #openstack-lbaas | 15:30 | |
| *** dayou has joined #openstack-lbaas | 15:32 | |
| *** aojea has quit IRC | 15:35 | |
| *** yamamoto has quit IRC | 15:44 | |
| *** yamamoto has joined #openstack-lbaas | 15:44 | |
| *** yamamoto has quit IRC | 15:49 | |
| *** pksingh has quit IRC | 15:56 | |
| *** csomerville has joined #openstack-lbaas | 16:05 | |
| *** pksingh has joined #openstack-lbaas | 16:20 | |
| *** rtjure has quit IRC | 16:23 | |
| *** pksingh has quit IRC | 16:28 | |
| *** greghaynes has quit IRC | 16:40 | |
| *** pksingh has joined #openstack-lbaas | 16:40 | |
| *** yamamoto has joined #openstack-lbaas | 16:50 | |
| *** bcafarel has quit IRC | 16:54 | |
| *** krypto has quit IRC | 16:56 | |
| *** krypto has joined #openstack-lbaas | 16:57 | |
| *** krypto has quit IRC | 16:57 | |
| *** krypto has joined #openstack-lbaas | 16:57 | |
| -openstackstatus- NOTICE: Restarting Gerrit for our weekly memory leak cleanup. | 17:05 | |
| *** yamamoto has quit IRC | 17:05 | |
| *** bcafarel has joined #openstack-lbaas | 17:08 | |
| *** atoth has joined #openstack-lbaas | 17:11 | |
| *** tongl has joined #openstack-lbaas | 17:15 | |
| *** greghaynes has joined #openstack-lbaas | 17:16 | |
| *** krypto has quit IRC | 17:17 | |
| *** sshank has joined #openstack-lbaas | 17:27 | |
| rm_work | johnsom: i guess probably we won't get to the members thing | 17:38 |
| rm_work | do we have any other *features* waiting that'd be blocked after next week tho? | 17:39 |
| *** cpusmith has quit IRC | 17:39 | |
| *** cpusmith_ has joined #openstack-lbaas | 17:40 | |
| johnsom | We can probably get members in | 17:50 |
| johnsom | I was going to try to do a scrub on the priority list today. | 17:51 |
| johnsom | Juggling too much at the moment | 17:51 |
| rm_work | yeah you and me both | 17:58 |
| rm_work | and this performance thing is killing me | 17:58 |
| rm_work | ruled out the DB | 17:59 |
| rm_work | (i'm just hitting / now, to get the static APIversion response) | 17:59 |
| johnsom | Oh, really, now that is super interesting | 17:59 |
| johnsom | That is like a static page | 17:59 |
| rm_work | it seems to be "first response is 10s", then every 5 minutes one response is 10s | 18:00 |
| rm_work | and then after some hours, it becomes 2 responses are slow, then 3 | 18:00 |
| rm_work | every 5 min | 18:00 |
| rm_work | http://i.imgur.com/QfCv4jI.png | 18:00 |
| rm_work | it MUST be uwsgi... | 18:01 |
| rm_work | (i'm back on uwsgi) | 18:01 |
| rm_work | also it has interesting stratum | 18:01 |
| rm_work | you can see it better in this one: | 18:01 |
| rm_work | http://i.imgur.com/WHzE0if.png | 18:02 |
| rm_work | most requests are like .5s | 18:02 |
| rm_work | but then at 1.5s and 2.5s there's other groupings | 18:02 |
| rm_work | with outliers around 3.5s | 18:02 |
| rm_work | it's not just a random cloud between 0.5 and 3s | 18:03 |
| johnsom | FYI, I am also looking at our image. It looks like Ubuntu has been loading up the cloud image with more stuff. The compressed image has almost doubled in size over the last year and half. Also the un-compressed amp only has like 150MB free. | 18:04 |
| johnsom | I did a quick build with ubuntu-minimal element, this seems to be better (builds with debbootstrap as opposed to just grabbing the cloud image from ubuntu), but I need to do some testing and figure out some things I had to work around. | 18:05 |
| rm_work | yeah | 18:06 |
| rm_work | also i think the Alpine story looks a lot better now | 18:06 |
| rm_work | haproxy is packaged | 18:06 |
| rm_work | I was going tor evisit | 18:06 |
| rm_work | *to revisit | 18:06 |
| rm_work | also I know someone here who has *done* haproxy+pythonagent on Alpine with docker | 18:07 |
| rm_work | so I am hoping if I show him some DIB stuff I might be able to steal some time to do pair-work to knock that out | 18:07 |
| *** atoth has quit IRC | 18:07 | |
| rm_work | just don't know when that could happen | 18:07 |
| rm_work | (last time I tried, haproxy was going to be a custom compile, and python was acting wonky, so i just gave up) | 18:07 |
| rm_work | though I don't know about the netns story, but I assume it's good since Alpine is a security distro? | 18:08 |
| rm_work | yeah the centos image makes me hurt inside, it's so big | 18:09 |
| rm_work | also, the images kolla builds for octavia-api and such are ginormous too | 18:09 |
| johnsom | Well, netns is kernel level, so would expect some support | 18:09 |
| rm_work | 830mb for a *docker image* | 18:09 |
| johnsom | WHAT? | 18:09 |
| johnsom | hahaha | 18:09 |
| rm_work | yes | 18:09 |
| johnsom | Wow | 18:09 |
| rm_work | it's *stupid* | 18:09 |
| rm_work | not sure what to do about it | 18:09 |
| rm_work | but it's way down on my todo list | 18:09 |
| rm_work | they have like... 20 layers too | 18:09 |
| rm_work | and they install our test-requirements and also leave all the build-deps for stuff on there, like gcc | 18:10 |
| johnsom | Well, I think minimal would be a good short term fix at least | 18:10 |
| johnsom | Yeah, we do in our amps too. If I really wanted to go deep I could create an element that strips out stuff at the end. Not sure I'm there at the moment | 18:11 |
| rm_work | yeah | 18:12 |
| rm_work | well in docker images it's hard because "stripping it out" is a layer... so it still has to download it all, to get to the state where it's not there | 18:13 |
| rm_work | <_< | 18:13 |
| rm_work | the k8s team here is like "you guys are dumb" | 18:13 |
| johnsom | I also have two modules left and I'm done with the python-openstacksdk | 18:13 |
| johnsom | Though I'm not getting reviews so that is worrying me | 18:13 |
| rm_work | T_T | 18:13 |
| rm_work | I can look maybe... | 18:13 |
| rm_work | maybe if they get +1 | 18:13 |
| rm_work | it'd help | 18:13 |
| johnsom | https://review.openstack.org/#/q/project:openstack/python-openstacksdk+status:open | 18:14 |
| johnsom | I have l7rule and quota to do | 18:15 |
| *** gcheresh has joined #openstack-lbaas | 18:16 | |
| *** pksingh has quit IRC | 18:17 | |
| *** ajo has quit IRC | 18:18 | |
| *** cpusmith has joined #openstack-lbaas | 18:22 | |
| johnsom | Yeah, ok, minimal is going to be some work. It looks like it doesn't have cloud-init and maybe not even dhcp | 18:23 |
| *** cpusmith_ has quit IRC | 18:25 | |
| -openstackstatus- NOTICE: docs.o.o is currently broken, we're investigating | 18:39 | |
| *** sshank has quit IRC | 18:39 | |
| rm_work | yeah i remember last time discovering that it had... issues | 18:42 |
| rm_work | I mean this is just... WHAT | 19:07 |
| rm_work | http://paste.openstack.org/show/616199/ | 19:07 |
| rm_work | ahhhh i wonder | 19:11 |
| rm_work | http://paste.openstack.org/show/616200/ | 19:11 |
| rm_work | err sorry fixed linebreaks | 19:11 |
| rm_work | http://paste.openstack.org/show/616201/ | 19:11 |
| rm_work | so at 4:30:14 is the last request issued | 19:11 |
| rm_work | and ALL stop on that worker until about 4:30:40 where they resume | 19:12 |
| rm_work | and we see a few of the ones from 4:30:14 interspersed at that point | 19:12 |
| rm_work | I have two nodes up... i wonder if the nginx in front of them (LEAST_CONNECTIONS) is actually smartly noticing the slow response and directing queries to the other one | 19:13 |
| rm_work | which is not being slow | 19:13 |
| rm_work | nope, BOTH locked up exactly the same time | 19:15 |
| rm_work | wut | 19:15 |
| rm_work | that makes very little sense | 19:15 |
| xgerman_ | rm_work WTF? https://bugs.launchpad.net/octavia/+bug/1705764 | 19:21 |
| openstack | Launchpad bug 1705764 in octavia "Deleting a listener during failover causes it to go into ERROR (edge case)" [Undecided,New] | 19:21 |
| rm_work | uhhh | 19:21 |
| xgerman_ | yeah, I have been looking at Octavia failures the last couple of days. Getting to head against wall state | 19:21 |
| xgerman_ | How, can people hit every edge case? | 19:22 |
| xgerman_ | in that case we hit the 404 just before the listener was built on the amp — what are the changes? | 19:22 |
| xgerman_ | chances | 19:22 |
| rm_work | [Adam Harwell] [10:44] | 19:32 |
| rm_work | yeah I’m scheduled to bang my head against a wall for a couple more hours, then in the afternoon I have two hours of crying softly to myself in a corner, and then i’m planning to take an early friday | 19:32 |
| rm_work | ^^ me this morning in Slack | 19:32 |
| *** dougwig has joined #openstack-lbaas | 19:41 | |
| *** sshank has joined #openstack-lbaas | 19:53 | |
| *** sshank has quit IRC | 19:57 | |
| *** catinthe_ has quit IRC | 19:58 | |
| *** aojea has joined #openstack-lbaas | 20:01 | |
| *** sshank has joined #openstack-lbaas | 20:06 | |
| *** cpusmith has quit IRC | 20:10 | |
| *** sshank has quit IRC | 20:31 | |
| *** sshank has joined #openstack-lbaas | 20:31 | |
| *** sshank has quit IRC | 20:31 | |
| *** sshank has joined #openstack-lbaas | 20:31 | |
| openstackgerrit | Erwan Gallen proposed openstack/octavia-dashboard master: Replace patterns syntax by url list https://review.openstack.org/486220 | 20:34 |
| *** gcheresh has quit IRC | 21:13 | |
| *** ssmith has joined #openstack-lbaas | 21:18 | |
| johnsom | rm_work Are you crying in the corner yet? (i.e. how is it going?) | 21:26 |
| *** mestery has quit IRC | 21:40 | |
| *** mestery_ has joined #openstack-lbaas | 21:40 | |
| *** mestery_ is now known as mestery | 21:42 | |
| rm_work | eh | 22:03 |
| rm_work | maybe new hints | 22:03 |
| rm_work | dunno | 22:03 |
| rm_work | still poking | 22:03 |
| johnsom | Do tell | 22:03 |
| rm_work | i heinously murdered the workers via W on the masterfifo for one uwsgi | 22:03 |
| rm_work | and it stopped doing the 5-min thing | 22:03 |
| rm_work | so i did it to the other one | 22:04 |
| rm_work | seeing if they disappear altogether | 22:04 |
| rm_work | they did for 20min... | 22:04 |
| *** leitan has quit IRC | 22:04 | |
| *** tesseract has quit IRC | 22:25 | |
| *** sshank has quit IRC | 22:27 | |
| *** sshank has joined #openstack-lbaas | 22:33 | |
| xgerman_ | yep, taking the early Friday as well. | 22:35 |
| johnsom | Enjoy! | 22:35 |
| johnsom | Wait, it is already after 5 for you... | 22:35 |
| johnsom | I am working through some reviews | 22:36 |
| *** sshank has quit IRC | 22:38 | |
| *** tongl has quit IRC | 22:39 | |
| *** aojea has quit IRC | 22:51 | |
| *** aojea has joined #openstack-lbaas | 22:52 | |
| *** aojea has quit IRC | 22:56 | |
| *** sshank has joined #openstack-lbaas | 22:57 | |
| *** sshank has quit IRC | 23:08 | |
| *** sshank has joined #openstack-lbaas | 23:08 | |
| *** sshank has quit IRC | 23:20 | |
| *** dougwig has quit IRC | 23:22 | |
| *** csomerville has quit IRC | 23:39 | |
| rm_work | whelp now the results are just utterly perplexing again | 23:51 |
| rm_work | FML | 23:51 |
| rm_work | I'm done | 23:51 |
| rm_work | (for the day) | 23:51 |
| *** catintheroof has joined #openstack-lbaas | 23:58 | |
| *** JudeC has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
