| mnasiadka | hello | 13:06 |
|---|---|---|
| mnasiadka | gthiemonge: https://review.opendev.org/c/openstack/octavia-dashboard/+/775561 - are you planning on working on this, or maybe there's a way I could help? | 13:06 |
| gthiemonge | mnasiadka: hi, we discussed this change at the last ptg, the main concern is that the ovn-provider doesn't support most of the default options in the LB wizard, and we don't want to add provider-specific code to the dashboard (like if provider == ovn-provider then available protocols = [tcp, udp]) | 13:51 |
| mnasiadka | Ok, but today there's no way to use the ovn-octavia-provider via Horizon dashboard | 13:53 |
| mnasiadka | Surely there must be a way out of this :) | 13:53 |
| gthiemonge | and IIRC if the user selects an unsupported option (like http protocol), the error is not propagated to the UI, it fails, without any explanations | 13:53 |
| gthiemonge | I think if we have a popup that says "unsupported protocol http" that could work | 13:54 |
| gthiemonge | but right now, this commit would bring bad user experience | 13:55 |
| mnasiadka | ok, so UI error propagation would need to be solved first | 14:18 |
| gthiemonge | https://opendev.org/openstack/octavia-dashboard/src/branch/master/octavia_dashboard/static/app/core/openstack-service-api/lbaasv2.service.js#L151-L156 | 14:19 |
| gthiemonge | I think we should be able to get the error in the catch function and display it on the screen | 14:20 |
| gthiemonge | mnasiadka: unfortunatly the API validates only one option at a time, so it could be really painful for the end user to find the correct options to set for the ovn-provider | 15:13 |
| spatel | johnsom Hey! | 17:00 |
| spatel | I am seeing very strange issue with one of my k8s application when I expose to octavia LB using UDP port - https://paste.opendev.org/show/bwl8E2tHizLVwuUXcRJH/ | 17:01 |
| spatel | Pool is showing OFFLINE | 17:01 |
| spatel | does Octavia support UDP ? | 17:01 |
| johnsom | It has since Rocky | 17:01 |
| spatel | I am running Xena | 17:02 |
| johnsom | I think that is a known and fixed bug related to the fact there are no health monitors on the pool. I think it is just rolling that up as OFFLINE instead of active. | 17:03 |
| spatel | ? | 17:03 |
| spatel | you are saying its BUG? | 17:03 |
| johnsom | Yes. I think it is just that status field has the wrong word in it. The LB works right? | 17:04 |
| spatel | NO, its not working | 17:04 |
| johnsom | Hmm, ok, what protocol is going over UDP? | 17:05 |
| johnsom | Can we add a health monitor to the pool so we can see the status of the members? | 17:05 |
| johnsom | Hmm, it implies RTP.... | 17:07 |
| johnsom | Can you do a member show on one of those? | 17:07 |
| spatel | UDP | 17:10 |
| spatel | This is all managed by k8s so not sure how to tell k8s to edit health monitor | 17:10 |
| johnsom | Ok, let's start with a member show | 17:11 |
| spatel | let me get you output hold on | 17:11 |
| spatel | is this correct command? openstack loadbalancer member show 5ac0af83-0797-4411-95dd-e20e1caa45b9 | 17:12 |
| johnsom | No, you need to include the pool UUID as well | 17:12 |
| spatel | https://paste.opendev.org/show/bG5rZ5pMwWszyJOAP4uS/ | 17:13 |
| johnsom | openstack loadbalancer member show bf2bb75f-edcb-4073-9a8f-b53b8b6d0cab 5ac0af83-0797-4411-95dd-e20e1caa45b9 | 17:13 |
| spatel | Found it | 17:13 |
| johnsom | Ok, this looks fine. now a "openstack subnet show 975fe679-61ba-4f4c-abb9-56416917fbb4" | 17:14 |
| spatel | https://paste.opendev.org/show/bOZFCbOlkYgFTwcCwJ1v/ | 17:14 |
| spatel | I have other apps exposed with TCP and they are working fine | 17:15 |
| spatel | one odd thing, I can't ssh to lbaas-mgmt IP even I am on same subnet | 17:17 |
| spatel | otherwise I can get on VM and try to debug | 17:17 |
| johnsom | That looks fine as well. Give me one minute to look at the xena code | 17:18 |
| spatel | ok | 17:18 |
| johnsom | Hmm, that implies you are having neutron issues. | 17:18 |
| spatel | let me debug that | 17:19 |
| spatel | johnsom I am inside amphora | 17:31 |
| spatel | How do I check haproxy logs to see what is going on | 17:31 |
| johnsom | Ok, sudo and run "ipvsadm" | 17:32 |
| johnsom | HAProxy is not used for UDP, it is all done in the kernel itself | 17:32 |
| spatel | I can see /var/log/haproxy.log | 17:32 |
| spatel | https://paste.opendev.org/show/bMzcHWvx6gnbW80X9C7p/ | 17:33 |
| spatel | You are saying its not using haproxy but ipvsadm? | 17:33 |
| johnsom | It's not using HAProxy, it's using the linux kernel for UDP | 17:34 |
| spatel | I see | 17:34 |
| johnsom | Give me one minute to create a UDP LB | 17:34 |
| spatel | cool | 17:35 |
| johnsom | We may need to run that ipvsadm command in the network namespace. | 17:35 |
| johnsom | Ugh, my development environment is not in a good state as I'm working on SR-IOV. Give me ~10 minutes to boot an Octavia VM | 17:40 |
| spatel | no worry or you can tell me and I will run command | 17:45 |
| spatel | https://paste.opendev.org/show/b8cTWxsmfW4pHhMjWzjN/ | 17:45 |
| johnsom | While we wait, try "ip netns exec amphora-haproxy ipvsadm" (Trying to remember the syntax off my head) | 17:45 |
| johnsom | Ok, so there is no configured UDP LB in that amphora. Could the connectivity issue cause the LB to go into ERROR? | 17:46 |
| johnsom | Have you tried a failover? | 17:46 |
| johnsom | Ok, the VM is ready, give me one minute to create a UDP LB | 17:55 |
| johnsom | Yeah, ok, inside the netns you should see: | 18:00 |
| johnsom | https://www.irccloud.com/pastebin/ilWJfdBQ/ | 18:01 |
| johnsom | You can check if the configuration is even there, it's in /var/lib/octavia/lvs | 18:02 |
| johnsom | Also, just to make sure, the amphora image you are using is from xena right? | 18:02 |
| spatel | give me a min.. in other call | 18:16 |
| spatel | back | 18:34 |
| spatel | I am not seeing that info in my output | 18:35 |
| spatel | /var/lib/octavia/lvs: No such file or directory | 18:35 |
| spatel | Yes its Xena image | 18:35 |
| johnsom | So I am guessing your network connectivity issues caused a provisioning failure. The LB should be in ERROR. I would try an "openstack loadbalancer failover" command and see if that fixes it. | 18:37 |
| spatel | what if I found yoga image or some latest version | 18:37 |
| johnsom | No, Xena should be fine. We gate test all of this | 18:37 |
| spatel | ok my network issue was my problem. when I ssh from LXC container it works | 18:38 |
| spatel | what is openstack loadbalancer failover command for? | 18:39 |
| spatel | I did deploy this LB 5 time after destroying... | 18:39 |
| spatel | all the time same issue | 18:39 |
| johnsom | It rebuilds the amphora instances to resolve ERROR provisioning status when something failed like neutron or nova | 18:39 |
| spatel | I am running octavia in single instance (without HA) | 18:39 |
| spatel | done - openstack loadbalancer failover 95e35a5b-109b-4bca-b70a-1d1f0ed9e071 | 18:40 |
| johnsom | That is ok, but it will cause an traffic outage in single topology | 18:40 |
| spatel | This is LAB environment so trying to save resources :) | 18:41 |
| johnsom | Ok, log into the new instance once the LB goes active, check the /var/lib/octavia/lvs directory | 18:41 |
| johnsom | Also, when you are there, can you "cat /opt/amphora-agent.gitref ? | 18:41 |
| spatel | johnsom no luck still pool is offline | 18:42 |
| spatel | let me try hold on | 18:42 |
| spatel | I got this output - 8ccc0d01764c234ce9d3dc0191b5a0952111e2ef | 18:43 |
| johnsom | That is a bobcat image | 18:44 |
| spatel | oh wait.. really? | 18:44 |
| johnsom | https://review.opendev.org/c/openstack/octavia/+/876041 | 18:45 |
| johnsom | It should still work, but it's not a tested combination | 18:45 |
| spatel | Oh i have this image - test-only-amphora-x64-haproxy-ubuntu-focal.qcow2 | 18:45 |
| spatel | openstack-ansible download and imported.. I thought its Xena | 18:46 |
| johnsom | Yeah, those are built regularly and are for gate testing only | 18:46 |
| spatel | even its bobcat it should work right? | 18:46 |
| johnsom | It should.... | 18:49 |
| spatel | otherwise I can download - https://minio.services.osism.tech/openstack-octavia-amphora-image/octavia-amphora-haproxy-xena.qcow2 | 18:49 |
| spatel | That is what I thought that bobcat should work because it is latest code.. | 18:50 |
| johnsom | Well, something odd is going on. We can try to debug by looking in the amphora-agent log, but that is a pretty big jump in versions. | 18:52 |
| johnsom | I need to step away for a bit. Let me know if you find something in the agent log. I would expect it to work, but we don't test that combination, so maybe there is something odd in a four version jump | 18:55 |
| spatel | hmm! | 19:00 |
| spatel | I will poke around also try Xena image and see what works | 19:01 |
| spatel | Thanks for stick around with me :) | 19:01 |
| spatel | Have a good weekend | 19:02 |
| spatel | johnsom just to close the loop here, after download Xena haproxy amphora I can see ipvsadm showing some good stuff and LB pool is ONLINE | 20:24 |
| spatel | https://paste.opendev.org/show/bmz3koPtM4IyfXInOBMw/ | 20:24 |
| johnsom | Yeah, that looks right. | 20:24 |
| spatel | So its clear that image should be compatible with release | 20:24 |
| johnsom | I wonder what changed in the bobcat image that is making that incompatible. | 20:24 |
| spatel | Or may be octavia xena doesn't like bobcat image | 20:25 |
| johnsom | Yeah, but it *should*. We should be maintaining compatible APIs. | 20:25 |
| spatel | Then time to open BUG :) | 20:26 |
| johnsom | Yeah, go for it. It can't hurt to report it | 20:26 |
| johnsom | https://bugs.launchpad.net/octavia | 20:26 |
| spatel | Sure I will open one | 20:26 |
| spatel | done - https://bugs.launchpad.net/octavia/+bug/2037759 | 20:32 |
| johnsom | Thanks! | 20:32 |
| spatel | are there any logs of ipvsadm? | 20:35 |
| johnsom | ipvsadm is just a configuration tool, it does not handle any traffic, etc. The traffic is all handled in the kernel. So, there is very little log information for UDP at the moment. You can get statistics and such from the ipvsadm tool or via the /proc/ filesystem. Octavia log offloading does not support UDP or SCTP at this time. | 20:38 |
| johnsom | Any errors from UDP traffic would go into the kernel log file. | 20:38 |
| spatel | got it | 20:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!