| opendevreview | Andre Aranha proposed openstack/octavia master: Change FIPS jobs to centos-9-stream https://review.opendev.org/c/openstack/octavia/+/831840 | 11:03 |
|---|---|---|
| spatel | johnsom hey | 17:51 |
| johnsom | spatel Hi | 17:51 |
| spatel | what could be wrong here - https://paste.opendev.org/show/bP7JdWsLvWE66gXdTSlu/ | 17:51 |
| spatel | I am not seeing ip address demo-net=10.0.0.202 in amphora image | 17:52 |
| johnsom | spatel For security reasons we isolate tenant traffic in a network namespace. Use "sudo ip netns exec amphora-haproxy ip a" | 17:52 |
| spatel | oh!! | 17:53 |
| spatel | i can see it now | 17:53 |
| spatel | trying to troubleshoot issue.. i have 2 www server behind LB but not able to curl from VIP address | 17:55 |
| spatel | i am able to curl from amphora image | 17:55 |
| johnsom | What is the output of "openstack loadbalancer status show" ? | 17:55 |
| spatel | https://paste.opendev.org/show/bIsSQzgPbbcgKaRPEzie/ | 17:57 |
| spatel | OFFLINE not sure why | 17:57 |
| johnsom | Yeah, that is odd. Can you check that the admin state is up for all of those objects? "openstack loadbalancer show web-lb-1", etc. | 17:58 |
| spatel | https://paste.opendev.org/show/b38m46BhNRj25WXPkIfL/ | 17:59 |
| johnsom | It kind of looks like maybe the pool or listener is disabled | 18:00 |
| spatel | hmm | 18:00 |
| johnsom | check "openstack loadbalancer listener show listen-port-80" and openstack loadbalancer pool show pool-80" as well | 18:01 |
| spatel | https://paste.opendev.org/show/bDsiPKO7dens0kPKJe4T/ | 18:01 |
| johnsom | Hmm listener looks good | 18:02 |
| spatel | everything looks ok for me.. then not sure why its offline | 18:03 |
| spatel | i can curl www machine from amphora | 18:03 |
| spatel | I have used PING for monitoring is that ok? | 18:03 |
| johnsom | It's not great, since this is HTTP, try switching it to an HTTP health monitor | 18:03 |
| johnsom | Or TCP if they aren't web servers | 18:04 |
| spatel | Let me switch | 18:04 |
| spatel | I can't do it without removing pool | 18:05 |
| johnsom | PING has a bunch of gotchas with neutron security groups, etc. that an cause it to get odd results. | 18:05 |
| johnsom | Yes, you can, just remove the health monitor and add a new one | 18:05 |
| spatel | hmm ok | 18:05 |
| spatel | i have changed it to HTTP but still same problem | 18:07 |
| spatel | operating status: Offline | 18:07 |
| johnsom | can you paste an updated "openstack loadbalancer status show"? | 18:07 |
| johnsom | Oh, I wonder if your lb-mgmt-net isn't working right. Maybe the heartbeats aren't coming into the controllers. Though the curl to the VIP should still work even if lb-mgmt-net is not getting the heartbeats. | 18:08 |
| spatel | https://paste.opendev.org/show/bB2LuweXEmeobeVwab9e/ | 18:09 |
| spatel | I can ping lb-mgmt-net from worker and manager | 18:09 |
| johnsom | if "openstack loadbalancer stats show web-lb-1" doesn't show any metrics (all zeros) it's likely a problem with the lb-mgmt-net | 18:10 |
| johnsom | There should be values in "total connections" for the curl calls to the vip | 18:10 |
| johnsom | If that is zero, check your octavia.conf file, [health_manager] controller_ip_port_list setting. Make sure the health manager IP and port for your controller(s) is correct there. | 18:13 |
| johnsom | Also make sure your health manager process is running | 18:14 |
| spatel | hmm | 18:15 |
| spatel | oh let me check health_manager | 18:15 |
| spatel | give me sec i check all those setting | 18:15 |
| johnsom | Sure, NP | 18:15 |
| spatel | https://paste.opendev.org/show/bdM8jhqA24LcpYOrj2cg/ | 18:16 |
| spatel | 10.30.1.230 is my host IP address | 18:16 |
| spatel | is this correct? | 18:17 |
| johnsom | For the o-hm0 interface? | 18:17 |
| spatel | its 10.1.0.10 | 18:17 |
| spatel | i should use that IP means... | 18:17 |
| spatel | let me check it and restart service... that could be my problem | 18:18 |
| johnsom | Ok, so that is probably the problem. The amps are sending to 10.30.1.230 over the lb-mgmt-net via o-hm0 interfaces on the controllers. | 18:18 |
| johnsom | When you change the octavia.conf config, you will need to either failover the loadbalancer or use "openstack loadbalancer amphora configure [--wait] <amphora-id>" to push the updated endpoint to the amp | 18:18 |
| spatel | ok, do i need to change on worker config also or not? | 18:19 |
| johnsom | Yes | 18:19 |
| johnsom | All except the API will need to be updated | 18:19 |
| spatel | cool | 18:19 |
| spatel | johnsom he amphora ID is nova list ? | 18:27 |
| johnsom | no | 18:27 |
| spatel | where should i find amphora ID? | 18:27 |
| johnsom | openstack loadbalancer amphora list | 18:27 |
| spatel | :) | 18:28 |
| spatel | johnsom its still showing OFFLINE | 18:35 |
| spatel | thinking i can redeploy LB | 18:35 |
| johnsom | Try loadbalancer failover | 18:35 |
| spatel | one odd thing i can't see 5555 port listen on anywhere on host machine netstat -natp | grep 5555 | 18:36 |
| spatel | oh its UDP | 18:38 |
| johnsom | The health manager should listen on it on the o-hm0 ip. You did fix the binding address too right? | 18:39 |
| spatel | now i can see 5555 bind to 10.1.0.10 (earlier i was looking at TCP but now i found in UDP) | 18:39 |
| spatel | udp 0 0 10.1.0.10:5555 0.0.0.0:* | 18:40 |
| johnsom | +1 | 18:40 |
| spatel | redeploying whole LB :) its on the way | 18:40 |
| spatel | is this normal ? - https://paste.opendev.org/show/baH8eVseNQZGoQ6tYZSd/ | 18:46 |
| spatel | assuming amphora isn't ready yet so its just keep trying | 18:46 |
| johnsom | Yeah, that is normal. It is waiting for nova to full boot the vm | 18:48 |
| spatel | pool member show error | 18:48 |
| spatel | https://paste.opendev.org/show/bt6l2Ffj4E1VUOkTNH3z/ | 18:49 |
| johnsom | Well, you fixed the heartbeat! | 18:50 |
| spatel | ? | 18:50 |
| johnsom | So the web server isn’t responding as the health monitor expects | 18:50 |
| spatel | i can curl so port 80 is alive | 18:50 |
| spatel | from amphora i can curl web server | 18:51 |
| spatel | let me change to PING and see | 18:51 |
| johnsom | If you curl / on a web server with verbose on, it doesn’t send a 302 does it? | 18:51 |
| spatel | sudo ip netns exec amphora-haproxy curl http://10.0.0.100/ | 18:53 |
| spatel | it return default apache page | 18:53 |
| spatel | i switch back to PING but still error | 18:54 |
| spatel | lets debug and see what is going on | 18:55 |
| spatel | in my other datacenter i used PING for health and it works without issue | 18:55 |
| johnsom | Ok, yeah, that is odd. | 18:55 |
| johnsom | Check the haproxy log file too, just to see what it says | 18:55 |
| spatel | in which place amphora keep lb config? | 18:57 |
| spatel | i think /var/lib/octavia/ | 18:58 |
| johnsom | Yes | 18:58 |
| johnsom | Unlikely to be a config problem | 18:58 |
| spatel | https://paste.opendev.org/show/bsxcPsMngHs0en2xOqAP/ | 19:00 |
| johnsom | That says the ping is not getting a response | 19:01 |
| spatel | hmm but i can ping from amphora :) | 19:04 |
| spatel | let me debug this... | 19:05 |
| spatel | or i can use http and see | 19:05 |
| spatel | here is the ping script | 19:05 |
| spatel | assuming it will run inside namespace | 19:06 |
| johnsom | Yes, the health monitor checks run from inside the namespace | 19:07 |
| johnsom | Worst case, you can run tcpdump from inside the namespace and see the packets leaving and the response | 19:09 |
| spatel | johnsom i am doing all basic OS level debug,, i will let you know :) | 19:22 |
| johnsom | +1 | 19:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!