| spatel | johnsom hey! | 16:34 |
|---|---|---|
| spatel | around? | 16:34 |
| johnsom | spatel Hi | 16:34 |
| johnsom | Did you see I found the patch you needed? | 16:34 |
| spatel | No | 16:34 |
| spatel | send me again | 16:34 |
| johnsom | spatel https://bugs.launchpad.net/octavia/+bug/2024188 | 16:35 |
| johnsom | https://review.opendev.org/c/openstack/octavia-dashboard/+/889300 | 16:35 |
| johnsom | It's related to a change in the Chrome browsers | 16:35 |
| spatel | reading.. | 16:35 |
| spatel | oh boy! got it so its browser issue | 16:36 |
| spatel | I have one more issue for you and not sure how to troubleshoot related Octavia | 16:37 |
| spatel | I have created k8s cluster and spun up hello-world app. Which I expose to octavia | 16:38 |
| johnsom | Ok | 16:38 |
| spatel | But now my loadbalancer member going in Error state | 16:39 |
| spatel | https://paste.opendev.org/show/bP16FrMlTvuT1eT5fYz1/ | 16:39 |
| spatel | That means octavia not able to reach hello-world app | 16:39 |
| johnsom | The health check is failing | 16:39 |
| johnsom | So, here are a few things to check: | 16:40 |
| spatel | For testing I have created foo1 vm on same network and did curl to member with port 30594 and that works.. | 16:40 |
| spatel | Feels like my octavia LB has some security-group issue which not able to reach to member | 16:40 |
| johnsom | 1. When you created the member, did you specify a subnet that can reach the hello-world instances? If you didn't specify a subnet, can the VIP network reach those instances? | 16:41 |
| johnsom | Nope, it's not a security group issue | 16:41 |
| johnsom | 2. Is your hello-world app on port 80 and responding with HTTP and a status code of 200? Check your health monitor configuration, what is it checking for? | 16:42 |
| johnsom | It could be a security group issue on your hello-world instances, but it's not in Octavia | 16:42 |
| johnsom | Can you paste bin your health monitor config? "openstack loadbalancer healthmonitor show " | 16:44 |
| spatel | K8s deployment created LB so it did all right based on what I am looking here | 16:45 |
| johnsom | Also one of the member config | 16:46 |
| spatel | But I have. created foo1 VM using same network and from that VM i did curl to members and it works that means there is not security-group on hello-world app | 16:46 |
| johnsom | My guess is you forgot to specify a subnet when creating the members in Octavia | 16:47 |
| spatel | how do i check health monitor config? | 16:47 |
| spatel | Let me post all the LB command output hold on | 16:48 |
| johnsom | openstack loadbalancer healthmonitor show <health monitor uuid> | 16:48 |
| johnsom | openstack loadbalancer member show <pool ID> <member ID> | 16:48 |
| spatel | https://paste.opendev.org/show/blUbaV2NfzktlLbzR7E8/ | 16:49 |
| johnsom | Hmm, ok, so that is a TCP health check, it will just do a SYN handshake on the port and not check HTTP health. That is ok. | 16:50 |
| spatel | https://paste.opendev.org/show/bb9VeSzg0lMyUZ0Db4Gq/ | 16:50 |
| spatel | I did ssh into amphora and try to curl from amphora to members.. | 16:51 |
| johnsom | From inside the network namespace? | 16:51 |
| johnsom | Ok, so this member will connect from subnet dd0b518a-114d-4b34-b465-1c35a5ad8017 to 10.0.0.45 on port 30594 | 16:52 |
| johnsom | Can that subnet reach that address/port is the question. | 16:52 |
| spatel | I can't ping member IPs from amphora VM - root@amphora-ee4400fe-3c90-45d0-8b1b-8dbfb574d1ce:~# ip netns exec amphora-haproxy ping 10.0.0.95 | 16:53 |
| johnsom | From inside the amp, to test with curl, you have to run "sudo ip netns exec amphora-haproxy curl http://10.0.0.45:30594" | 16:53 |
| spatel | Ping and curl both not working :( | 16:54 |
| johnsom | Yeah, that is a sign that the members aren't reachable from that subnet. Either by routing, security group, or subnet scope | 16:54 |
| johnsom | So, Octavia is correct, they are not healthy | 16:54 |
| spatel | but security-group is mostly for INGRESS not for EGRESS correct? | 16:54 |
| spatel | If I create LB without k8s and just deploy apache web server then everything works | 16:55 |
| johnsom | The security groups in Octavia are fine, there is not issue there. It would be on the hello-world instances | 16:55 |
| spatel | But from other VM i can access hello-world app so why do you think its issue on hello-world app? | 16:56 |
| spatel | I can curl curl http://10.0.0.45:30594 from my foo1 test vm sitting on same subnet/network of k8s. | 16:56 |
| johnsom | Because I know the SGs in Octavia are correct. | 16:56 |
| johnsom | From the dd0b518a-114d-4b34-b465-1c35a5ad8017 subnet? | 16:57 |
| spatel | Yes from subnet - dd0b518a-114d-4b34-b465-1c35a5ad8017 | 16:57 |
| spatel | I created test VM on subnet dd0b518a-114d-4b34-b465-1c35a5ad8017 and ping members 10.0.0.45 and 10.0.0.95 and both pingable and I can curl both of them | 16:58 |
| johnsom | Then I would start tracking down what is broken in the k8s networking as something is odd | 16:58 |
| spatel | felt like something is wrong with octavia networking or tenant network.. | 16:58 |
| johnsom | Probably would start with tcpdump on the hello-world side | 16:58 |
| spatel | Feels like something is wrong either my networking or VxLAN tunnel | 16:58 |
| spatel | Thanks for checking. Let me dig deeper and see | 16:59 |
| johnsom | Ok, good luck | 16:59 |
| spatel | I found one bug similar to sec group - https://bugs.launchpad.net/cdk-addons/+bug/1884995 | 16:59 |
| spatel | I am on Xena release | 17:00 |
| johnsom | We haven't had a bug in Octavia security groups in like four-five years. It hasn't changed at all. | 17:01 |
| spatel | okie!! | 17:02 |
| johnsom | Ah, so that bug says the SG on the k8s nodeport was wrong | 17:02 |
| johnsom | You could also try this, on your test VM, setup apache and add it as a member, I but it is online since it's a VM and not in K8s | 17:04 |
| spatel | johnsom sometime my nova giving me error for amphora image - exception.ImageUnacceptable: Image 5a06e450-7aed-418b-a7c5-59372cd080e1 is unacceptable: Image has no associated data | 17:47 |
| spatel | very odd that its happening on only few compute nodes | 17:49 |
| spatel | Look like my glance has some issue.. let me check | 17:51 |
| johnsom | Yeah, seems like a glance issue | 18:03 |
| opendevreview | Brian Haley proposed openstack/octavia master: Don't fail if a provider driver cannot be loaded in Octavia API https://review.opendev.org/c/openstack/octavia/+/780215 | 18:32 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!