| *** longkb has joined #openstack-lbaas | 00:30 | |
| *** hongbin has joined #openstack-lbaas | 01:39 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: [UDP] Update amphora agent api ref https://review.openstack.org/588893 | 01:52 |
|---|---|---|
| *** gans has joined #openstack-lbaas | 03:58 | |
| *** gans has quit IRC | 04:05 | |
| *** hongbin has quit IRC | 04:09 | |
| *** nmagnezi_ has joined #openstack-lbaas | 06:03 | |
| *** pcaruana has joined #openstack-lbaas | 06:07 | |
| *** luksky has joined #openstack-lbaas | 06:42 | |
| *** ivve has joined #openstack-lbaas | 06:46 | |
| *** rcernin has quit IRC | 06:52 | |
| *** ramishra has joined #openstack-lbaas | 07:03 | |
| bzhao__ | cgoncalves: Thanks for test. That error maybe extend my knowledge, ;-( . Is that the keepalived kernel error? | 07:26 |
| *** velizarx has joined #openstack-lbaas | 07:36 | |
| *** rpittau has joined #openstack-lbaas | 07:53 | |
| *** yamamoto has quit IRC | 08:09 | |
| cgoncalves | bzhao__, keepalived segmentation fault | 08:10 |
| cgoncalves | bzhao__, before I get lots of those errors (because systemd restarts keepalived), there are errors running the health check. I will try to find what's exactly causing it | 08:11 |
| cgoncalves | ok, found it. the problem is that the virtual server ip address is v4 but members were configured with ipv6 addresses | 08:14 |
| bzhao__ | cgoncalves: Thanks for explain. That's the real ip packet, maybe the keepalived community can search the reason. | 08:14 |
| bzhao__ | ...=.= | 08:14 |
| *** crazik has left #openstack-lbaas | 08:15 | |
| bzhao__ | cgoncalves: does haproxy support transfer from ipv4 to ipv6? | 08:16 |
| cgoncalves | bzhao__, there are reports of this upstream. one dev is saying to create multiple vrrp_instances | 08:18 |
| cgoncalves | https://github.com/acassen/keepalived/issues/497#issuecomment-273122561 | 08:18 |
| cgoncalves | bzhao__, could you check whether a mix of v4 and v6 works for you? also the keepalived version you're running | 08:19 |
| cgoncalves | keepalived v1.3.5 here (centos 7.5) | 08:19 |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: systemd should recover Keepalived https://review.openstack.org/588993 | 08:21 |
| bzhao__ | cgoncalves: I imaging that if we configure v4 VS, but v6 RS, I think it won't work as wish, as currently we support nat mode only, and we configure the iptables based on VIP address version on ubuntu.. | 08:22 |
| bzhao__ | cgoncalves: I will recreate the test env. | 08:22 |
| cgoncalves | ok | 08:25 |
| bbbbzhao_ | https://www.irccloud.com/pastebin/gQUNpDud/ | 08:38 |
| bbbbzhao_ | https://www.irccloud.com/pastebin/mbZP3Fx5/ | 08:39 |
| bbbbzhao_ | Oh, I remove the persistence first... | 08:39 |
| bbbbzhao_ | sorry | 08:39 |
| cgoncalves | http://paste.openstack.org/show/727386/ is what I have | 08:43 |
| cgoncalves | note that I manually changed the RS IP addresses to v4 | 08:43 |
| *** yamamoto has joined #openstack-lbaas | 08:43 | |
| cgoncalves | I can netcat to each RS but not to the virtual server | 08:43 |
| bbbbzhao_ | https://www.irccloud.com/pastebin/uEx6kn3F/ | 08:44 |
| cgoncalves | http://paste.openstack.org/show/727387/ | 08:44 |
| bbbbzhao_ | Seem lvs not support ipv4 to ipv6... | 08:44 |
| bbbbzhao_ | I can show the rs only in amp ns. | 08:45 |
| bbbbzhao_ | https://www.irccloud.com/pastebin/PsUCtxSb/ | 08:46 |
| cgoncalves | right, we have to netcat within the amphora-proxy namespace | 08:46 |
| bbbbzhao_ | And when I request the VS, the syslog will full of no destination. | 08:46 |
| bbbbzhao_ | Yeah, I haven't add nc in it yet. Just testing the udp listener support tcp/http checker.. doing test the config generate is correct..;-) | 08:47 |
| bzhao__ | cgoncalves: I think there is a bug that if we specify the ipv6 RS(member), it doesn't check whether the persistence value match the ipv6 version.. | 08:50 |
| bzhao__ | cgoncalves: (⊙o⊙) many cases not cover...just the normal ones.. =。= | 08:51 |
| *** salmankhan has joined #openstack-lbaas | 08:56 | |
| *** ktibi has joined #openstack-lbaas | 08:57 | |
| cgoncalves | bzhao__, should I see a process listening on a udp socket? because I don't see any on netstat | 08:59 |
| bzhao__ | cgoncalves: No, we can see a process listening, as the lvs will send out the packets directly based on kernel. But we can see a connection in the ns by conntrack -L | 09:03 |
| bzhao__ | sorry can't see a process listening a udp port. | 09:03 |
| bbbbzhao_ | https://www.irccloud.com/pastebin/HDO327Pm/ | 09:04 |
| bbbbzhao_ | This is the version in my ubuntu env | 09:04 |
| cgoncalves | rm_work, johnsom: ^ !!! :D centos wins this time by shipping a newer keepalived version than ubuntu :P | 09:06 |
| bzhao__ | haha | 09:06 |
| *** yamamoto has quit IRC | 09:15 | |
| cgoncalves | /proc/net/nf_conntrack doesn't show the udp one | 09:15 |
| bzhao__ | cgoncalves: I will install a conntrack tool for show... I'm not very familiar with the kernel configuration. Please hold on. | 09:21 |
| cgoncalves | bzhao__, sure, no problem. thank you! | 09:22 |
| bbbbzhao_ | https://www.irccloud.com/pastebin/icx9VPOL/ | 09:26 |
| bbbbzhao_ | Before L21, no new connection, as I didnt request the VIP. After L21, I request with iperf, we can see the socket connect in L27.. | 09:27 |
| cgoncalves | I can not get that even after requesting udp on the VIP :/ | 09:43 |
| bzhao__ | =。= | 09:43 |
| openstackgerrit | zhouchangxun proposed openstack/octavia master: Change the driver to a singleton https://review.openstack.org/589100 | 09:49 |
| *** yamamoto has joined #openstack-lbaas | 09:50 | |
| bzhao__ | cgoncalves: Could you check the kernel configuration? net.ipv4.ip_forward and net.ipv4.vs.conntrack enabled, ip_vs kernel module had been loaded? | 10:17 |
| *** yamamoto has quit IRC | 10:19 | |
| *** luksky has quit IRC | 10:28 | |
| *** kobis1 has joined #openstack-lbaas | 10:57 | |
| *** kobis1 has left #openstack-lbaas | 10:58 | |
| *** kobis1 has joined #openstack-lbaas | 10:59 | |
| *** luksky has joined #openstack-lbaas | 11:02 | |
| *** nmagnezi_ has quit IRC | 11:18 | |
| cgoncalves | bzhao__, both set to 0, ip_vs kernel module enabled | 11:46 |
| *** amuller has joined #openstack-lbaas | 11:47 | |
| bzhao__ | cgoncalves: you need to set both them to 1 in amp namespace, also check the iptables is configured. | 11:47 |
| bbbbzhao_ | -A POSTROUTING -o eth1 -p udp -j MASQUERADE | 11:48 |
| bbbbzhao_ | like this | 11:48 |
| cgoncalves | oh, wait. that is inside the namespace. they are set to 1, yes | 11:49 |
| cgoncalves | I see that net.ipv6.conf.all.forwarding should have also been set to 1 but it is not | 11:50 |
| bzhao__ | cgoncalves: yeah, if you use ipv6, you need to set to 1 | 11:51 |
| cgoncalves | it should have been set when starting amphora-netns systemd | 11:51 |
| cgoncalves | ExecStart=-/sbin/ip netns exec {{ amphora_nsname }} sysctl -w net.ipv6.conf.all.forwarding=1 | 11:51 |
| bzhao__ | cgoncalves: So the logic check the vip's ip version, if it is ipv4, it just config net.ipv4.ip_forward=1 | 11:52 |
| bzhao__ | cgoncalves: I do that in plug.py... | 11:52 |
| bzhao__ | cgoncalves: :( | 11:53 |
| cgoncalves | wait, why are we modprobing and sysctl'ing both in plug.py and systemd service? | 11:54 |
| bzhao__ | hold on, checking the systemd script | 11:57 |
| *** yamamoto has joined #openstack-lbaas | 12:11 | |
| bzhao__ | Oh, as the plug.py is for the first VIPPost request and making sure that the kernel setting is good(which is a alternative for pre-setting the kernel), I saw the latest code, johnsom move it when upload_listener_config in util.install_netns_systemd_service(), and just write the script, not exec start/restart. So I think the script one may make sure they are correct when amp instance reboot, and the plug.py is for | 12:11 |
| bzhao__ | running time. Just a guess. ;-). Maybe there are some enhance approaches for it. | 12:11 |
| *** longkb has quit IRC | 12:23 | |
| bzhao__ | Leaving for home.. | 12:29 |
| *** nmagnezi_ has joined #openstack-lbaas | 12:30 | |
| *** velizarx has quit IRC | 12:43 | |
| *** velizarx has joined #openstack-lbaas | 12:48 | |
| *** yamamoto has quit IRC | 12:50 | |
| *** nmagnezi_ has quit IRC | 12:51 | |
| *** fnaval has joined #openstack-lbaas | 13:00 | |
| *** erjacobs has joined #openstack-lbaas | 13:32 | |
| johnsom | Yeah, this is a bit funky at the moment. It was disjoint for haproxy. Basically it is run in plug and written out to systemd for reboots | 13:43 |
| *** velizarx has quit IRC | 13:59 | |
| *** velizarx has joined #openstack-lbaas | 14:01 | |
| *** celebdor has joined #openstack-lbaas | 14:17 | |
| erjacobs | Hi, does anyone have a minute to help me create a loadbalancer in devstack? Every time i try and make one the provision_status changes to error | 14:19 |
| *** nmagnezi_ has joined #openstack-lbaas | 14:19 | |
| *** nmagnezi_ has left #openstack-lbaas | 14:19 | |
| *** nmagnezi_ has joined #openstack-lbaas | 14:19 | |
| *** kobis1 has quit IRC | 14:25 | |
| *** colby_ has quit IRC | 14:26 | |
| *** nmagnezi_ has quit IRC | 14:30 | |
| *** velizarx has quit IRC | 14:40 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: [UDP] Support HTTP GET and TCP check in udp healthmonitor https://review.openstack.org/589180 | 14:46 |
| *** yamamoto has joined #openstack-lbaas | 14:47 | |
| johnsom | erjacobs Check the devstack@o-cw log with journalctl "journalctl --u devstack@o-cw | less" | 14:47 |
| johnsom | cgoncalves Trying to catch up, it's the keepalived in CentOs that is seg faulting? | 14:52 |
| erjacobs | thanks johnsom | 14:57 |
| johnsom | Nova problem? | 14:57 |
| bbbbzhao_ | johnsom: He was running ipv4 VS, but ipv6 RS, then hit the seg fault. | 15:04 |
| *** luksky has quit IRC | 15:05 | |
| bbbbzhao_ | checking the lvs with ipvsadm -Ln, it doesn't support this case, only can see the configured ipv4 VS, no ipv6 RS. | 15:07 |
| *** celebdor has quit IRC | 15:07 | |
| *** celebdor has joined #openstack-lbaas | 15:09 | |
| *** pcaruana has quit IRC | 15:12 | |
| erjacobs | johnsom the --u flag is ambiguous, my options are '--user' '--utc' '--update-catalog' '--user-unit' '--unit' '--until' | 15:14 |
| johnsom | erjacobs Opps, I meant one dash. Sorry for the typo | 15:14 |
| erjacobs | johnsom http://paste.openstack.org/show/727420/ , any thoughts? | 15:16 |
| johnsom | erjacobs The key line there is: Nova failed to build the instance due to: 503 Service Unavailable: Service Unavailable: The server is temporarily unable to service your: request due to maintenance downtime or capacity: problems. Please try again later. | 15:19 |
| johnsom | Your nova is not functioning | 15:19 |
| erjacobs | ok | 15:20 |
| erjacobs | thanks for the help! | 15:21 |
| johnsom | Sure, NP | 15:21 |
| *** yamamoto has quit IRC | 15:27 | |
| xgerman_ | R | 15:42 |
| xgerman_ | \R | 15:42 |
| xgerman_ | learning all about user modes… | 15:45 |
| johnsom | Yeah, I need to figure out the private message issue | 15:51 |
| johnsom | Let me know if you come up with the right settings | 15:51 |
| johnsom | I am trying "/mode <username> +R" | 15:55 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Allows failover if port is not deallocated by nova https://review.openstack.org/585864 | 15:58 |
| *** erjacobs has quit IRC | 16:00 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Allows failover if port is not deallocated by nova https://review.openstack.org/585864 | 16:02 |
| *** nmagnezi_ has joined #openstack-lbaas | 16:08 | |
| *** ramishra has quit IRC | 16:10 | |
| bbbbzhao_ | johnsom: Hi, Michael, today I just finish coding the HTTP/TCP healthmonitor support in udp, not test the order change and the followup Uts addition. I will continue to do the rest tomorrow. ;-). I have to go, it's nearly 1:00 am, here. My wife will kick me. ;-(. | 16:42 |
| johnsom | Ok, thanks! | 16:45 |
| *** ktibi has quit IRC | 16:53 | |
| bbbbzhao_ | johnsom: Thank you very much. | 16:53 |
| *** luksky has joined #openstack-lbaas | 16:59 | |
| cgoncalves | johnsom, keepalived seg faulting, yeah. as bbbbzhao_ already wrote, it happened when virtual server ip is v4 and real servers v6. after changing RS to v4, keepalived runs yet I don't see the udp socket open | 17:09 |
| *** yamamoto has joined #openstack-lbaas | 17:24 | |
| bbbbzhao_ | cgoncalves: ;-). Last time, I just download and install a lower version keepalived and test, it works. Before I install that, I found the exist keepalived in centos didn't route any traffic, even though other configs are correct, then I stop the keepalive and just using ipvsadm to config lvs, it works. Then I decided to install another version of keepalived. Just my experience on centos testing. | 17:24 |
| *** salmankhan has quit IRC | 17:28 | |
| cgoncalves | bbbbzhao_, ok, thanks for that info. I need to spend some time to set up a keepalived+lvs on a full blown centos image | 17:33 |
| jiteka | Hello, I've got a question regarding amphora VM creation | 17:52 |
| jiteka | If octavia-health-manager process is not able to contact the amphora VMs on udp 5555 | 17:52 |
| jiteka | Is it triggering tasker_flow to revert the LB creation or only octavia-worker is taking the decision to revert everything when amphora are not reachable for too long after build ? | 17:52 |
| johnsom | jiteka It is the octavia-worker that will revert if it cannot make contact with a newly build amphora instance | 17:54 |
| jiteka | ok then I understood correctly this part :) | 17:55 |
| jiteka | thanks johnsom | 17:55 |
| johnsom | health manager will not start tracking the amphora until it has gone Active | 17:55 |
| jiteka | I found a way to avoid octavia worker to revert everything when failing as right now I need time to debug it and ssh on the amphora itself | 17:57 |
| jiteka | Beside the both VMs, associated neutron port and lb-uuid security group, which other resource should I look for when doing the cleaning manually myself ? | 17:57 |
| *** yamamoto has quit IRC | 17:57 | |
| jiteka | https://docs.openstack.org/octavia/latest/configuration/configref.html#task_flow.disable_revert | 17:58 |
| johnsom | Right, that is the right key to disable reverts. | 17:58 |
| johnsom | I think that is the list, but I may be forgetting something | 17:58 |
| jiteka | hmm server group too | 18:09 |
| jiteka | :) | 18:09 |
| *** amuller has quit IRC | 18:25 | |
| *** ctracey has quit IRC | 18:41 | |
| *** ctracey has joined #openstack-lbaas | 18:42 | |
| *** numans has quit IRC | 18:54 | |
| *** yamamoto has joined #openstack-lbaas | 19:17 | |
| *** abaindur has joined #openstack-lbaas | 19:29 | |
| *** yamamoto has quit IRC | 19:31 | |
| *** yamamoto has joined #openstack-lbaas | 19:36 | |
| *** yamamoto has quit IRC | 19:47 | |
| *** kobis1 has joined #openstack-lbaas | 19:59 | |
| *** kobis1 has quit IRC | 20:15 | |
| *** rtjure has joined #openstack-lbaas | 20:31 | |
| johnsom | cgoncalves You wouldn't happen to have that amp still around would you? Curious what the amphora-netns systemd log said. | 20:31 |
| *** celebdor has quit IRC | 21:07 | |
| *** harlowja has joined #openstack-lbaas | 21:21 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: DNM: Leave VIP NIC plugging for keepalived https://review.openstack.org/589292 | 21:32 |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: DNM: Leave VIP NIC plugging for keepalived https://review.openstack.org/589292 | 21:33 |
| *** yamamoto has joined #openstack-lbaas | 21:41 | |
| *** nmagnezi_ has quit IRC | 21:49 | |
| *** yamamoto has quit IRC | 21:51 | |
| *** rcernin has joined #openstack-lbaas | 22:15 | |
| *** fnaval has quit IRC | 22:22 | |
| *** luksky has quit IRC | 22:25 | |
| *** yamamoto has joined #openstack-lbaas | 22:43 | |
| *** yamamoto has quit IRC | 22:47 | |
| openstackgerrit | Merged openstack/octavia master: Cleanup Octavia create VIP ports on LB delete https://review.openstack.org/581168 | 23:07 |
| openstackgerrit | Merged openstack/octavia master: Clarify that the driver support library is interim https://review.openstack.org/588320 | 23:07 |
| *** hvhaugwitz has quit IRC | 23:23 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Followup patch for UDP support https://review.openstack.org/587690 | 23:23 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Followup patch for UDP support https://review.openstack.org/587690 | 23:30 |
| *** hvhaugwitz has joined #openstack-lbaas | 23:35 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!