| openstackgerrit | Michael Johnson proposed openstack/octavia master: Adds flavor support to the amphora driver https://review.openstack.org/621323 | 00:37 |
|---|---|---|
| *** yamamoto has joined #openstack-lbaas | 00:45 | |
| *** Swami has quit IRC | 00:52 | |
| lxkong | johnsom: hi, we noticed that there are a lot of threads for octavia-worker service and too much memory it is eating, https://paste.pics/d2e89878ed8a0ea36de1acc2b6614a35 | 00:52 |
| lxkong | wouldn't you happen to know the situation before, would you? | 00:52 |
| lxkong | xgerman, rm_work ^^ | 00:53 |
| johnsom | lxkong The worker, no, can't say I have seen that. Unless you have a ton of user requests coming in or the threads are backed up waiting for nova to build instances. What do the logs look like, normal? | 00:54 |
| lxkong | normal, we didn't enable the debug, so nothing happens in the log in the last 2 hours | 00:55 |
| lxkong | seems like all the threads are just listening or waiting for something | 00:55 |
| lxkong | we are using Queens | 00:55 |
| johnsom | The only thing the worker listens on is the rabbit queue | 00:56 |
| johnsom | If it was processing flows and waiting on nova, you would see info or warnings in the logs. | 00:56 |
| lxkong | i can confirm that no lb creation requests are coming | 00:57 |
| lxkong | weird | 00:57 |
| johnsom | But even then, the threads should not be listening/waiting, they are spun out as stuff comes in on rabbit | 00:57 |
| johnsom | Would be interested in a trace if you attach a debugger to one of those threads | 01:03 |
| johnsom | My worker is under two megabytes with all three processes combined | 01:08 |
| johnsom | You could try pmap -x as well and see if something jumps out there as to what is using that memory | 01:09 |
| lxkong | johnsom: hmm...i've no idea about what the pmap -x PID command output means | 01:15 |
| johnsom | It show the memory usage of a process, broken down by library if it can determine what it is | 01:16 |
| lxkong | there are several lines like this `00007f774818d000 63948 0 0 ----- [ anon ]` | 01:16 |
| johnsom | One of my two octavia-worker consumer processes look like this: | 01:17 |
| johnsom | https://www.irccloud.com/pastebin/ewuE95e6/ | 01:17 |
| *** yamamoto has quit IRC | 01:18 | |
| *** yamamoto has joined #openstack-lbaas | 01:18 | |
| lxkong | ours | 01:18 |
| lxkong | http://termbin.com/q9xw | 01:18 |
| lxkong | didn't see anyting special | 01:18 |
| lxkong | except for the total memory | 01:19 |
| johnsom | Yeah, that is big and you have a lot more threads than I do | 01:20 |
| lxkong | 65 threads | 01:20 |
| johnsom | I have like 9 | 01:22 |
| johnsom | per process | 01:22 |
| lxkong | will the 'Socket closed' error by rabbitmq lead to create new worker thread? | 01:23 |
| lxkong | we have some 'Socket closed' error in the log caused by haproxy timeout | 01:23 |
| johnsom | maybe. The oslo messaging stuff does seem a bit fragile | 01:24 |
| lxkong | http://paste.openstack.org/show/736738/ | 01:24 |
| *** sapd1 has quit IRC | 01:40 | |
| *** sapd1 has joined #openstack-lbaas | 01:40 | |
| *** abaindur has quit IRC | 02:18 | |
| *** hongbin has joined #openstack-lbaas | 02:35 | |
| *** yamamoto has quit IRC | 02:45 | |
| *** yamamoto has joined #openstack-lbaas | 03:24 | |
| *** yamamoto has quit IRC | 03:34 | |
| *** ramishra has quit IRC | 03:36 | |
| *** yamamoto has joined #openstack-lbaas | 04:29 | |
| *** hongbin has quit IRC | 04:33 | |
| *** ramishra has joined #openstack-lbaas | 04:35 | |
| *** yboaron has joined #openstack-lbaas | 05:02 | |
| *** yboaron has quit IRC | 07:26 | |
| *** yboaron has joined #openstack-lbaas | 07:26 | |
| *** ataraday has quit IRC | 07:49 | |
| *** aojea has joined #openstack-lbaas | 07:52 | |
| *** rcernin has quit IRC | 07:56 | |
| *** pcaruana has joined #openstack-lbaas | 07:58 | |
| *** pcaruana is now known as muttley | 07:58 | |
| *** abaindur has joined #openstack-lbaas | 08:00 | |
| *** aojea has quit IRC | 08:00 | |
| *** rcernin has joined #openstack-lbaas | 08:03 | |
| *** abaindur has quit IRC | 08:11 | |
| *** abaindur has joined #openstack-lbaas | 08:12 | |
| *** velizarx has joined #openstack-lbaas | 08:12 | |
| *** abaindur has quit IRC | 08:22 | |
| *** abaindur has joined #openstack-lbaas | 08:23 | |
| *** abaindur has quit IRC | 08:24 | |
| *** velizarx has quit IRC | 08:28 | |
| *** velizarx has joined #openstack-lbaas | 08:33 | |
| *** rcernin has quit IRC | 08:33 | |
| *** abaindur has joined #openstack-lbaas | 09:05 | |
| *** ramishra has quit IRC | 09:19 | |
| *** ramishra has joined #openstack-lbaas | 09:21 | |
| *** AlexStaf has quit IRC | 09:30 | |
| *** abaindur has quit IRC | 09:30 | |
| *** ccamposr has quit IRC | 09:32 | |
| *** salmankhan has joined #openstack-lbaas | 10:13 | |
| *** salmankhan1 has joined #openstack-lbaas | 10:22 | |
| *** salmankhan has quit IRC | 10:23 | |
| *** salmankhan1 is now known as salmankhan | 10:23 | |
| *** yboaron_ has joined #openstack-lbaas | 10:26 | |
| *** yboaron has quit IRC | 10:29 | |
| *** Krast has quit IRC | 10:53 | |
| *** yboaron_ has quit IRC | 11:05 | |
| *** yboaron_ has joined #openstack-lbaas | 11:08 | |
| *** AlexStaf has joined #openstack-lbaas | 11:11 | |
| *** yboaron_ has quit IRC | 11:33 | |
| *** huseyin_ has joined #openstack-lbaas | 11:47 | |
| huseyin_ | Hello everybody, I want to forward terminated SSL traffic to the backend servers in http ssl mode. After adding x-forwarded-for headers on the http frontend, I also want to send HTTPS traffic to the backend servers. Because a wsdl security framework works at the backend, and they specifically look for ssl headers. | 12:11 |
| huseyin_ | When i use http pool, terminated-ssl listener works good | 12:12 |
| huseyin_ | But when i try to add https pool to the tls-terminated listener, it gives an error | 12:13 |
| huseyin_ | Amphora agent returned unexpected result code 400 with response {u'message': u'Invalid request', u'details': u"[ALERT] 339/111101 (2273) : http frontend '5a3c58a4-27d8-462b-84ca-18e57a41383e' (/var/lib/octavia/5a3c58a4-27d8-462b-84ca-18e57a41383e/haproxy.cfg.new:20) tries to use incompatible tcp backend 'a8dee118-e706-4d5b-8ecf-0efe5816bf79' (/var/lib/octavia/5a3c58a4-27d8-462b-84ca-18e57a41383e/haproxy.cfg.new:29) as its default backend (see | 12:13 |
| huseyin_ | 'mode').\n[WARNING] 339/111101 (2273) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.\n[ALERT] 339/111101 (2273) : Fatal errors found in configuration.\n"} | 12:13 |
| huseyin_ | As far as understand, amphora tries to add backend servers with tcp mode | 12:13 |
| huseyin_ | But frontend is http, and they are incompatible | 12:14 |
| huseyin_ | In a manual haproxy configuration, ssl backend definition works with tls-terminated frontend | 12:15 |
| huseyin_ | Is there any way to do this in octavia? | 12:15 |
| huseyin_ | Any idea? | 12:16 |
| *** AlexStaf has quit IRC | 12:20 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** yboaron_ has joined #openstack-lbaas | 12:30 | |
| *** yamamoto has joined #openstack-lbaas | 12:31 | |
| *** AlexStaf has joined #openstack-lbaas | 12:35 | |
| *** pbourke has quit IRC | 12:40 | |
| *** pbourke has joined #openstack-lbaas | 12:40 | |
| *** muttley has quit IRC | 13:08 | |
| *** rcernin has joined #openstack-lbaas | 13:17 | |
| *** muttley has joined #openstack-lbaas | 13:21 | |
| *** muttley has quit IRC | 13:25 | |
| *** muttley has joined #openstack-lbaas | 13:26 | |
| *** rcernin has quit IRC | 13:29 | |
| *** muttley has quit IRC | 13:29 | |
| *** yboaron_ has quit IRC | 13:29 | |
| *** pcaruana has joined #openstack-lbaas | 13:34 | |
| *** pcaruana has quit IRC | 13:39 | |
| cgoncalves | centos7 job green again | 13:41 |
| *** yamamoto has quit IRC | 13:41 | |
| *** pcaruana has joined #openstack-lbaas | 13:43 | |
| *** pcaruana has quit IRC | 13:47 | |
| openstackgerrit | Reedip proposed openstack/octavia-tempest-plugin master: Add configuration to enable/disable L7 tests https://review.openstack.org/621493 | 14:06 |
| *** yamamoto has joined #openstack-lbaas | 14:24 | |
| *** yamamoto has quit IRC | 14:37 | |
| *** Swami has joined #openstack-lbaas | 14:44 | |
| johnsom | huseyin_: Backend re-encryption is planned for Stein as a new feature. Today we can only do HTTPS pass through is the backend server needs an HTTPS connection. | 14:47 |
| huseyin_ | johnsom: Thank you very much for the clarification | 14:49 |
| *** huseyin_ has left #openstack-lbaas | 14:50 | |
| *** Swami has quit IRC | 14:55 | |
| *** Swami has joined #openstack-lbaas | 14:56 | |
| *** ramishra has quit IRC | 15:00 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Enable debug for Octavia services in grenade job https://review.openstack.org/622319 | 15:18 |
| *** AlexStaf has quit IRC | 15:34 | |
| xgerman | cgoncalves: I guess making it non-voting jinxed it | 15:34 |
| cgoncalves | RDO imported golang, that's what fixed it | 15:35 |
| *** AlexStaf has joined #openstack-lbaas | 15:48 | |
| *** roukoswarf has quit IRC | 15:53 | |
| *** ramishra has joined #openstack-lbaas | 15:59 | |
| *** takamatsu has quit IRC | 16:10 | |
| *** velizarx_ has joined #openstack-lbaas | 16:18 | |
| *** velizarx has quit IRC | 16:18 | |
| jiteka | xgerman johnsom : hello, is Hongbin Lu on that IRC channel ? I've got couple of question for him as I'm interested in Zun integration in Octavia | 16:39 |
| jiteka | https://review.openstack.org/#/c/579405/ | 16:39 |
| *** velizarx_ has quit IRC | 16:47 | |
| *** velizarx has joined #openstack-lbaas | 16:48 | |
| *** Swami has quit IRC | 17:14 | |
| *** velizarx has quit IRC | 17:15 | |
| *** salmankhan has quit IRC | 17:36 | |
| *** AlexStaf has quit IRC | 17:39 | |
| *** velizarx has joined #openstack-lbaas | 17:45 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Add v2 two-node scenario test https://review.openstack.org/605163 | 17:50 |
| *** huseyin_ has joined #openstack-lbaas | 18:00 | |
| *** fnaval has joined #openstack-lbaas | 18:00 | |
| *** Swami has joined #openstack-lbaas | 18:05 | |
| *** velizarx has quit IRC | 18:29 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Add v2 two-node scenario test https://review.openstack.org/605163 | 18:31 |
| *** yamamoto has joined #openstack-lbaas | 18:34 | |
| rm_work | jiteka: i believe he's in the zun channel as hongbin | 18:34 |
| *** fnaval has quit IRC | 18:37 | |
| *** yamamoto has quit IRC | 18:38 | |
| *** fnaval has joined #openstack-lbaas | 18:44 | |
| *** ramishra has quit IRC | 18:47 | |
| *** fnaval has quit IRC | 18:52 | |
| jiteka | rm_work: thanks ! I will check there | 18:56 |
| *** salmankhan has joined #openstack-lbaas | 19:13 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Support remote debugging with PyDev https://review.openstack.org/619944 | 19:16 |
| *** salmankhan has quit IRC | 19:18 | |
| openstackgerrit | Merged openstack/octavia master: Fix the grenade gate https://review.openstack.org/623102 | 19:53 |
| rm_work | cgoncalves: you are a victim of my first review :P | 19:55 |
| rm_work | 😂😂😂💕👌😜😇 | 19:55 |
| rm_work | (I just found out my new macbook has emoji on the touchbar lol 😅) | 19:56 |
| nmagnezi | rm_work, don't forget to cancel Siri :) | 19:59 |
| rm_work | yeah i disabled that like instantly | 20:00 |
| rm_work | and removed it from the touchbar | 20:00 |
| nmagnezi | rm_work, It can also be removed from the upper menu bar | 20:00 |
| rm_work | yeah, i did, disabled it completely | 20:00 |
| rm_work | i gave her one chance to be useful, and she failed | 20:01 |
| *** aojea has joined #openstack-lbaas | 20:02 | |
| *** aojea has quit IRC | 20:07 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: Fix IPv6 in Active/Standby topology https://review.openstack.org/589292 | 20:10 |
| *** rcernin has joined #openstack-lbaas | 20:12 | |
| openstackgerrit | German Eichberger proposed openstack/octavia-tempest-plugin master: Adds a mixed IPv4/IPv6 members traffic test https://review.openstack.org/611405 | 20:20 |
| nmagnezi | xgerman, that's cool ^^ | 20:21 |
| nmagnezi | xgerman, oh actually I've seen it before :) | 20:22 |
| xgerman | yeah, just rebasing... | 20:22 |
| johnsom | lol | 20:22 |
| johnsom | There are five patches up for fixing IPv6 and adding tempest tests for it.... | 20:23 |
| nmagnezi | johnsom, was just looking at what IPv6 tests we have to review that patch | 20:23 |
| johnsom | Fix IPv6 in Active/Standby topology - https://review.openstack.org/#/c/589292/ - Backport Candidate +2 | 20:24 |
| johnsom | Bring up secondary IPs on member networks - https://review.openstack.org/611460 - Backport Candidate | 20:24 |
| johnsom | Add traffic tests using an IPv6 VIP - https://review.openstack.org/611980 | 20:24 |
| johnsom | Adds a mixed IPv4/IPv6 members traffic test - https://review.openstack.org/611405 | 20:24 |
| nmagnezi | johnsom, yup, I look into 589292 | 20:24 |
| cgoncalves | rm_work, hahah! I'll fix it, also need to fix a long line | 20:24 |
| johnsom | And this one: https://review.openstack.org/614449 | 20:24 |
| cgoncalves | dinner distracted me | 20:25 |
| johnsom | I probably should have set the topic to IPv6 on all of those or something | 20:25 |
| johnsom | But that is the stack that will fix and add tests for IPv6. | 20:25 |
| johnsom | nmagnezi It also solves the keepalived both interfaces up issue. | 20:25 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Support remote debugging with PyDev https://review.openstack.org/619944 | 20:26 |
| cgoncalves | rm_work, if you have some time, please revisit https://review.openstack.org/#/c/553381/ | 20:28 |
| *** salmankhan has joined #openstack-lbaas | 20:31 | |
| *** fnaval has joined #openstack-lbaas | 20:39 | |
| nmagnezi | johnsom, a question re: https://review.openstack.org/#/c/589292/20/octavia/amphorae/backends/agent/agent_jinja_cfg.py | 20:42 |
| nmagnezi | johnsom, hmm actually give me a sec before I ask | 20:42 |
| nmagnezi | :) | 20:42 |
| nmagnezi | johnsom, okay so IIRC agent_jinja_cfg.py runs on the Octavia controller (worker). Meaning that topology value is being Read from octavia.conf | 20:46 |
| nmagnezi | And the jinja rendering will put that value on amphora-agent.conf in the Amphora VM | 20:47 |
| nmagnezi | But https://review.openstack.org/#/c/589292/20/octavia/amphorae/backends/agent/api_server/osutils.py@129 run on the Amphora itself | 20:47 |
| nmagnezi | So that value is now coming from amphora-agent.conf | 20:47 |
| nmagnezi | So to my question: what happens if someone upgrades the Amphora image but not the controller | 20:47 |
| nmagnezi | Maybe we should default to some empty value? I did not actually check this | 20:48 |
| nmagnezi | Just thinking out loud | 20:48 |
| johnsom | nmagnezi https://github.com/openstack/octavia/blob/master/octavia/common/config.py#L376 | 20:50 |
| nmagnezi | johnsom, yes but will that translate to amphora-agent.conf with an old controller that does not have the addition to https://review.openstack.org/#/c/589292/20/octavia/amphorae/backends/agent/agent_jinja_cfg.py ? | 20:51 |
| nmagnezi | Meaning that controller will still render that file without the topology option | 20:51 |
| johnsom | nmagnezi It's the same common/config.py that is used inside the amphora to read the amphora-agent.conf. That is why the path is "CONF.controller_worker.loadbalancer_topology" even inside the agent | 20:52 |
| johnsom | So this line in osutils.py "topology=CONF.controller_worker.loadbalancer_topology," will fill in the default | 20:52 |
| johnsom | Since it is looking it up via CONF | 20:53 |
| colin- | wow, did not know irssi supported emojis | 20:53 |
| colin- | that was a bit startling | 20:53 |
| nmagnezi | colin-, ¯\_(ツ)_/¯ | 20:54 |
| nmagnezi | johnsom, /me looking again | 20:54 |
| johnsom | So does IRC cloud, but you can turn them off | 20:54 |
| xgerman | why would you? they are fun! | 20:56 |
| xgerman | now we need to upload meme images and my life will be complete | 20:56 |
| xgerman | https://i.kym-cdn.com/photos/images/newsfeed/000/115/654/16behwy.jpg | 20:57 |
| colin- | +2 | 20:57 |
| nmagnezi | johnsom, ah, you're right. In the amp we don't have the octavia.conf file but we will default to the same thing as you said. Great, thanks :) | 20:59 |
| cgoncalves | xgerman, how can I help turning your +1 a +2 on the remote debugging patch? | 21:25 |
| xgerman | I need to have. a minute to test | 21:26 |
| rm_work | working on getting my new devstack workflow ironed out, then i'll be able to really review patches again :) | 21:40 |
| cgoncalves | octavia-v2-dsvm-scenario-two-node job took only 1 hour. fantastic! | 21:42 |
| *** salmankhan has quit IRC | 21:53 | |
| *** abaindur has joined #openstack-lbaas | 22:40 | |
| *** abaindur has quit IRC | 22:41 | |
| *** abaindur has joined #openstack-lbaas | 22:42 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Add v2 two-node scenario test https://review.openstack.org/605163 | 23:20 |
| *** fnaval has quit IRC | 23:31 | |
| rm_work | woo, *almost* got devstack up and going consistently... though no nested-virt :/ | 23:33 |
| *** yamamoto has joined #openstack-lbaas | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
