| opendevreview | Merged openstack/octavia stable/xena: Fix amphora-agent elements for RHEL9 https://review.opendev.org/c/openstack/octavia/+/830205 | 00:43 |
|---|---|---|
| *** ysandeep|out is now known as ysandeep | 06:28 | |
| *** ysandeep is now known as ysandeep|afk | 10:02 | |
| *** ysandeep|afk is now known as ysandeep | 12:00 | |
| *** ysandeep is now known as ysandeep|away | 12:58 | |
| viks__ | I have tried `soft-anti-affinity` with `ACTIVE_STANDBY` and i have 2 compute nodes.. but it is not scheduling the instances to different compute nodes? any suggestions what could be going wrong | 13:27 |
| *** ysandeep|away is now known as ysandeep | 13:42 | |
| gthiemonge | viks__: hey, it should create amphorae on different compute nodes, but if it cannot because a compute node is down or because all the compute nodes have already one amp from the same LB, it may create the amphora on the same compute node | 14:40 |
| gthiemonge | with "normal" anti-affinity, the creating of the VMs would be denied by nova | 14:40 |
| viks__ | gthiemonge: i tried creating both anti-affinit and soft-anti-affinity server groups in my cluster and tried to create 2 instances together in my 2 compute node setup. In both cases, it got created in different nodes as expected. But in case of octavia, with anti-affinity, it creates in two nodes, where master instance is stuck in BOOTING and BACKUP instance IN ACTIVE state.. Not sure what is going wrong | 14:44 |
| viks__ | https://www.irccloud.com/pastebin/7M59wXQf/ | 14:46 |
| gthiemonge | viks__: mmh here BOOTING is not good, it means that a VM is spawned but is not reachable yet | 14:52 |
| *** ysandeep is now known as ysandeep|out | 14:52 | |
| johnsom | viks__ It seems like a nova problem with your compute host. While it is “booting” do a openstack show server on it and see if nova posted an error. | 14:53 |
| gthiemonge | viks__: there should be a nova vm with a name that includes the amphora id (3b7f9338-a050-448d-a000-38d1136b52ba), maybe you can fetch the console logs for this VM | 14:53 |
| johnsom | Yeah, looking at the console would be good too | 14:53 |
| viks__ | ok.. as posted above instance in active state, amphora status is in booting state and when opening the console, getting `Something went wrong, connection is closed` | 15:07 |
| viks__ | i'm using linuxbridge in my setup.. i hope thats not the issue | 15:08 |
| johnsom | Not at all | 15:09 |
| johnsom | Can you open the console of the other instance? | 15:11 |
| johnsom | My guess here is one of two compute hosts has a problem. Manually booting a vm on the bad host will also fail. | 15:14 |
| viks__ | yes... other instance i can open the console... as i mentioned, i have tested anti-affinity and soft-anti-affinity by creating a server group, with the same set of hosts, it it worked as expected | 15:18 |
| viks__ | i have also tested disabling one of the nodes and creating instance, and the instance creation works fine | 15:20 |
| johnsom | Yeah, so that other host has a nova problem. I would start with the server show output, then start digging in the nova logs, then check for qemu logs on the host | 15:31 |
| viks__ | johnsom: ok.. the server show does not show any error as it's in active state in , only network it is not getting 2 ip's for one of the instances | 15:37 |
| viks__ | also it gets 2 ip's `lb-mgmt-net=172.16.2.137, 172.16.0.253` and what is the use of the second ip? | 15:47 |
| viks__ | in linuxbridge logs, i see 2 tap interfaces gets created for the one which is having 2 ip's , for the other only only 1 tap inteface getting created.. what process creates these 2 interfaces for amphora instance? | 15:58 |
| johnsom | viks__ that is not normal and octavia has no need for two IPs on the lb-mgmt-net. Check your octavia.conf and make sure the amphora boot network config is set correctly | 16:17 |
| johnsom | That doesn’t explain why one instance would work however. Does the working standalone also get two ips? | 16:20 |
| viks__ | yes.. standalone also gets 2 ips and it works fine in that case | 16:22 |
| viks__ | soft-anti-affinity is also works fine and it gets 2 ips... but it does create instances in the same compute host | 16:23 |
| johnsom | Yeah, anti-affinity in nova has a number of caveats. | 16:27 |
| viks__ | ok | 16:28 |
| viks__ | not sure how to make ACTIVE and STANDBY instances to go to separate compute hosts | 16:31 |
| johnsom | For example I filed this bug: https://bugs.launchpad.net/nova/+bug/1863190 | 16:33 |
| johnsom | I think your path forward is to debug why nova is unable to boot the instance on the other compute host. I.e. why the console can' t even connect. That is digging through nova/qemu logs. There is not special about the Octavia vms, they are simple vms created via the standard nova apis. | 16:35 |
| viks__ | johnsom: ok... let me dig further.. the puzzling part for me is , why it works when i create instances with anti-affinity server group and not with octavia.. anyways i'll try to investigate | 16:40 |
| opendevreview | Michael Johnson proposed openstack/octavia stable/wallaby: Add fips jobs https://review.opendev.org/c/openstack/octavia/+/827900 | 19:39 |
| opendevreview | Spencer Harmon proposed openstack/octavia master: Add rpc notification for load balancers. https://review.opendev.org/c/openstack/octavia/+/831051 | 21:10 |
| johnsom | spencerharmon Hi and thanks for the patch | 21:21 |
| spencerharmon | Hi! Very happy to contribute :) | 21:21 |
| johnsom | It looks like you posted it fine. So you have gerrit skills now. grin | 21:22 |
| spencerharmon | Hahaha. I know! Exciting milestone. | 21:22 |
| johnsom | There are usually people around this channel, so if you have questions don't be shy. | 21:22 |
| johnsom | There is another notifications patch that someone has been working on recently: https://review.opendev.org/c/openstack/octavia/+/784628 | 21:23 |
| johnsom | You might want to compare approaches | 21:23 |
| spencerharmon | Yeah, I reviewed this patch. I actually started off testing it, but what I found is that it was nicer to have specific tasks for notifications so I can import it in a provider driver. | 21:24 |
| johnsom | I have also felt that the notifications made sense to be in tasks | 21:24 |
| spencerharmon | I hope I soled some of the other concerns from the comments as well. NOTIFIER singleton, for example. | 21:25 |
| spencerharmon | *solved | 21:26 |
| johnsom | Cool. Yeah, it's been a while since I reviewed the other patch. My time is split between Octavia and Designate these days. | 21:27 |
| johnsom | Anyway, please feel free to reach out if you have any questions. | 21:28 |
| spencerharmon | Certainly will! Thanks for the help and kind words so far. Looking forward to feedback and the rest of the review process! | 21:29 |
| opendevreview | Spencer Harmon proposed openstack/octavia master: Add rpc notification for load balancers. https://review.opendev.org/c/openstack/octavia/+/831051 | 21:38 |
| spencerharmon | Sorry; realized I had the reference/actual backwards on my unit test assertions. | 21:39 |
| johnsom | No worries, that catches a lot of people | 21:46 |
| spencerharmon | I have ~50% success getting it right hahaha! | 21:47 |
| johnsom | Yep | 21:47 |
| opendevreview | Michael Johnson proposed openstack/octavia master: Fix prometheus-proxy systemd service https://review.opendev.org/c/openstack/octavia/+/831062 | 22:48 |
| opendevreview | Spencer Harmon proposed openstack/octavia master: Add rpc notification for load balancers. https://review.opendev.org/c/openstack/octavia/+/831051 | 23:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!