| openstackgerrit | Merged openstack/octavia stable/train: Ignore DELETED amphorae when performing certificate rotation https://review.opendev.org/c/openstack/octavia/+/754057 | 00:04 |
|---|---|---|
| openstackgerrit | Merged openstack/octavia-tempest-plugin master: Fix Go lint errors https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/758616 | 00:04 |
| *** spatel has joined #openstack-lbaas | 00:52 | |
| *** xgerman has quit IRC | 00:56 | |
| *** guilhermesp has quit IRC | 00:56 | |
| *** xgerman has joined #openstack-lbaas | 00:57 | |
| *** nicolasbock has quit IRC | 00:57 | |
| *** guilhermesp has joined #openstack-lbaas | 00:58 | |
| *** nicolasbock has joined #openstack-lbaas | 00:58 | |
| *** openstackgerrit has quit IRC | 01:38 | |
| *** spatel has quit IRC | 01:54 | |
| *** zzzeek has quit IRC | 02:14 | |
| *** zzzeek has joined #openstack-lbaas | 02:15 | |
| *** sapd1 has joined #openstack-lbaas | 03:31 | |
| *** sapd1 has quit IRC | 03:38 | |
| *** lemko has quit IRC | 04:26 | |
| *** lemko has joined #openstack-lbaas | 04:26 | |
| *** vishalmanchanda has joined #openstack-lbaas | 05:38 | |
| *** gcheresh has joined #openstack-lbaas | 05:46 | |
| *** gcheresh has quit IRC | 06:29 | |
| *** spatel has joined #openstack-lbaas | 06:33 | |
| *** spatel has quit IRC | 06:37 | |
| *** rcernin has quit IRC | 07:04 | |
| *** xgerman has quit IRC | 07:04 | |
| *** rcernin has joined #openstack-lbaas | 07:17 | |
| *** rcernin has quit IRC | 07:18 | |
| *** sapd1 has joined #openstack-lbaas | 07:35 | |
| *** ccamposr has joined #openstack-lbaas | 07:46 | |
| *** ccamposr__ has quit IRC | 07:49 | |
| *** luksky has joined #openstack-lbaas | 07:55 | |
| *** ccamposr__ has joined #openstack-lbaas | 08:02 | |
| *** ccamposr has quit IRC | 08:05 | |
| lxkong | hi, i have an issue with UDP listener. I successfully created a load balancer with UDP listener, pool, and a member that provides UDP service. I can access the UDP server inside the amphora by `ip netns exec amphora-haproxy nc -vuz $member_ip $udp_port`, but can't access via the VIP, any chance you know why? | 08:13 |
| lxkong | if creating health monitor, it can also get the right response. Just access VIP failed. | 08:14 |
| lxkong | The keepalivedlvs service looks fine, https://dpaste.com/8AJSPBWKT | 08:15 |
| *** tkajinam has quit IRC | 08:31 | |
| *** tkajinam has joined #openstack-lbaas | 08:32 | |
| *** sapd1 has quit IRC | 08:50 | |
| *** sapd1 has joined #openstack-lbaas | 08:53 | |
| *** ramishra has quit IRC | 09:19 | |
| gthiemonge | lxkong: can you check the network traffic (with tcpdump) on the member port in the amphora? I had similar issues in the past because masquerading was not correctly enabled, packets were forwarded by the amphora but the UDP service was not able to reply because the source ip address was not correct | 09:39 |
| *** ramishra has joined #openstack-lbaas | 09:40 | |
| lxkong | gthiemonge: thanks for reply, but what do you mean by 'source ip address was not correct'? | 09:50 |
| *** gcheresh has joined #openstack-lbaas | 09:50 | |
| gthiemonge | lxkong: masquerading is enabled in the amphora for UDP LBs, so it means that the amphora rewrites the source ip of the packets that are sent to the servers | 10:05 |
| *** sapd1 has quit IRC | 10:05 | |
| *** spatel has joined #openstack-lbaas | 10:05 | |
| gthiemonge | lxkong: when it wasn't working for me: tcpdump showed that the source ip was the client source ip (the client that sent a packet to the LB) | 10:05 |
| *** spatel has quit IRC | 10:11 | |
| *** gcheresh has quit IRC | 10:53 | |
| *** spatel has joined #openstack-lbaas | 11:54 | |
| *** spatel has quit IRC | 11:58 | |
| *** zzzeek has quit IRC | 12:36 | |
| *** zzzeek has joined #openstack-lbaas | 12:37 | |
| *** cgoncalves has quit IRC | 12:45 | |
| *** cgoncalves has joined #openstack-lbaas | 12:46 | |
| *** cgoncalves has quit IRC | 12:47 | |
| *** spatel has joined #openstack-lbaas | 13:00 | |
| *** spatel has quit IRC | 13:05 | |
| *** cgoncalves has joined #openstack-lbaas | 13:40 | |
| *** tkajinam has quit IRC | 14:07 | |
| *** kevinz has quit IRC | 14:35 | |
| *** damien_r has joined #openstack-lbaas | 14:44 | |
| *** TrevorV has joined #openstack-lbaas | 14:48 | |
| *** vishalmanchanda has quit IRC | 17:58 | |
| *** ccamposr has joined #openstack-lbaas | 18:36 | |
| *** ccamposr__ has quit IRC | 18:38 | |
| *** spatel has joined #openstack-lbaas | 18:44 | |
| spatel | johnsom: Hi | 18:44 |
| spatel | I have create LB and this is what amphora looks like - http://paste.openstack.org/show/800750/ | 18:45 |
| spatel | 10.68.x.x is front end VIP | 18:45 |
| spatel | 10.66.x.x my www server located | 18:46 |
| johnsom | Yes, that looks correct | 18:46 |
| spatel | why do i have two VIP ip for front end? | 18:46 |
| spatel | look like i am going to waste lots of public IP in that case | 18:46 |
| johnsom | One is the VIP itself that floats, one is the base port that is actually active on that instance | 18:47 |
| spatel | Even in SINGLE amphora deployment? | 18:47 |
| spatel | I am not looking for Active-Standby | 18:47 |
| johnsom | Yeah, that is how it is implemented including single. We need to have a VIP port that reserves the IP in neutron permanently. Unfortunately we can't rely on unplugging it and moving it in the case of a failure as nova will not let go of the port should the host go down. | 18:48 |
| johnsom | So, if you are concerned about using up public IPs with two per amp, the best option we have is to put the VIP on a private subnet and use floating IPs. | 18:49 |
| spatel | My cloud doesn't support floating IP, we are using VLAN base provider | 18:50 |
| spatel | anyway just wanted to confirm that what is going on. | 18:51 |
| johnsom | Yeah, we have considered writing an alternate single IP network driver, but it comes with the caveat that if the host goes down, we can't move the port and IP, nova won't let it or storage go until the host comes back up. | 18:51 |
| spatel | One more issue, my LB is up but i can't ping VIP, does it use security group or something? | 18:51 |
| johnsom | Yes, the SGs Octavia uses only open the ports needed. no ICMP | 18:52 |
| spatel | I can't telnet on port also.. very strange. | 18:53 |
| spatel | from amphora VM i can ping my gateway IP | 18:53 |
| spatel | from Amphora i can ping outside world but no traffic coming in. | 18:54 |
| spatel | let me debug it.. | 18:54 |
| johnsom | Ok | 18:54 |
| spatel | can i modify security group and allow ICMP? | 18:55 |
| spatel | i can see octavia_sec_grp | 18:56 |
| spatel | let me try | 18:56 |
| johnsom | Not currently. They are unique security groups per LB and they are generated in the code. | 18:56 |
| spatel | :( | 18:56 |
| johnsom | We probably should add a configuration option to turn on ping. Feel free to open an RFE story for that. | 18:56 |
| johnsom | https://storyboard.openstack.org/#!/dashboard/stories | 18:56 |
| spatel | ping is very important :) without that hard to troubleshoot | 18:57 |
| spatel | I think operator should have control to modify rules | 18:57 |
| johnsom | Well, security people feel otherwise most of the time. As you said, telnet to a TCP port should work. | 18:57 |
| spatel | no telnet also not working on port 80 | 18:58 |
| spatel | It seems something is wrong with security-group. | 18:58 |
| johnsom | Yeah, so then something is likely wrong outside the LB like a missing route, or HTTP proxy in the way, or other security group issue | 18:58 |
| spatel | Let me debug and find out with tcpdump etc.. | 18:59 |
| spatel | oh wait... | 19:02 |
| spatel | in amphora-haproxy route table has no default route | 19:03 |
| -spatel- Destination Gateway Genmask Flags Metric Ref Use Iface | 19:03 | |
| -spatel- 10.66.0.0 0.0.0.0 255.255.248.0 U 0 0 0 eth2 | 19:03 | |
| -spatel- 10.68.0.0 0.0.0.0 255.255.248.0 U 0 0 0 eth1 | 19:03 | |
| johnsom | Ah, so the subnet in neutron is not setup correctly. | 19:03 |
| spatel | let me verify hold on | 19:03 |
| spatel | They looks good and both has default route.. | 19:04 |
| spatel | even i spin up VM with that subnet and it works | 19:04 |
| johnsom | Can you paste the openstack subnet show for the 68 subnet? | 19:05 |
| spatel | both 10.66 and 10.68 has default route in neutron and lets say DHCP handover default route then how does amphora handle that? | 19:05 |
| johnsom | Neutron gives us all of the route information directly via the subnet configuration. | 19:06 |
| spatel | if both subnet trying to set default route then it will create issue | 19:06 |
| spatel | Let me ask question differently. | 19:07 |
| johnsom | Right, there are policy based routing tables in use inside the amphora | 19:07 |
| spatel | hmm | 19:08 |
| johnsom | It's a complicated, but very tested/stable system. | 19:08 |
| johnsom | Can you paste the subnet show? | 19:08 |
| spatel | yes let me give you every single info hold on | 19:08 |
| johnsom | I'm on vacation, so do not have a reference amphora to give you the example output for the routing tables. | 19:09 |
| spatel | johnsom: here you go http://paste.openstack.org/show/800751/ | 19:13 |
| spatel | johnsom: no worry, you should enjoy your vacation (you shouldn't be here.. haha) | 19:14 |
| johnsom | spatel So, on the VIP subnet, 68, it shows "gateway_ip | None " and there are no host routes, so there is no gateway defined for that subnet | 19:14 |
| spatel | oh wait.. you are right.. | 19:15 |
| johnsom | You should set the default gateway for that subnet in neutron "gateway_ip" | 19:15 |
| spatel | good catch.. may be i am always pinging those VM from directly connected host and not other subnet.. (damn it) | 19:15 |
| spatel | let me try to do that and verify | 19:16 |
| johnsom | You will need to do a failover of the LB after changing neutron for it to pick it up or wait until the DHCP interval is up | 19:16 |
| spatel | i can destroy and re-create this is all in my lab | 19:17 |
| spatel | johnsom: added gateway now re-building new lb | 19:22 |
| spatel | johnsom: you are awesome!! its working now | 19:25 |
| spatel | thanks | 19:25 |
| spatel | Last question: How does it remove default route of my web server side subnet? | 19:26 |
| johnsom | Well, they go into separate policy based routing tables. | 19:27 |
| johnsom | Play around with it in your lab and enter the netns and look at the various routing tables. | 19:28 |
| johnsom | I think there is still some open issue with member subnets that has not yet been resolved. There is an open story on it. But in general it will work. | 19:29 |
| johnsom | If the members don't have gateways defined, it will route out the VIP subnet default gateway, like a one-armed load balancer. | 19:29 |
| spatel | johnsom: sure i will look into and poke around. | 19:30 |
| spatel | Thank you! | 19:30 |
| johnsom | No problem. | 19:30 |
| spatel | I doubt you are on vacation.. lol | 19:30 |
| johnsom | Ha, well, in theory I am | 19:30 |
| spatel | :) | 19:33 |
| *** xgerman has joined #openstack-lbaas | 19:33 | |
| spatel | Let me destroy amphora and see if octavia create one more me. | 19:33 |
| spatel | hmm i destroyed amphora and it did start creating new vm but that vm got stuck in paused stat (12 instance-0000004b paused) | 19:35 |
| *** luksky has quit IRC | 19:46 | |
| *** luksky has joined #openstack-lbaas | 19:46 | |
| spatel | This is the error i am getting - http://paste.openstack.org/show/800752/ | 19:52 |
| johnsom | Something is wrong with nova, check the nova logs | 20:01 |
| spatel | default default] [instance: 53472774-1008-4b04-8780-3770e2f58171] Failed to allocate network(s): nova.exception.VirtualInterfaceCreateExcept | 20:07 |
| spatel | very odd.. | 20:07 |
| spatel | let me see.. | 20:08 |
| *** jamesdenton has quit IRC | 20:51 | |
| *** jamesdenton has joined #openstack-lbaas | 20:51 | |
| *** gcheresh has joined #openstack-lbaas | 20:58 | |
| *** openstackgerrit has joined #openstack-lbaas | 21:01 | |
| openstackgerrit | Merged openstack/octavia stable/train: Fix backend certificate file paths https://review.opendev.org/c/openstack/octavia/+/754342 | 21:01 |
| spatel | johnsom: on horizon GUI why my flavor drop down menu showing no flavor? | 21:18 |
| johnsom | What does “openstack loadbalancer flavor show” give for the flavor you setup? | 21:21 |
| *** TrevorV has quit IRC | 21:36 | |
| spatel | No matching flavor | 21:45 |
| spatel | johnsom: https://ibb.co/LpjrQnX | 21:46 |
| spatel | i did create flavor using octavia account so it should see them | 21:47 |
| johnsom | Did you create the flavor using my guide? https://docs.openstack.org/octavia/latest/admin/flavors.html | 21:48 |
| spatel | no, i did create normal way. let me read this.. | 21:49 |
| spatel | johnsom: can i add huge page, because all my compute using hugepage | 21:49 |
| johnsom | You can add that to the compute flavor and then either use it as the default or add an octavia flavor that points to the hugepage compute flavor | 21:51 |
| spatel | johnsom: i have deployed cloud using openstack-ansible and it created default octavia flavor that one also not visible. so that is interesting | 21:55 |
| spatel | may be that flavor has option os-flavor-access:is_public | False | 21:56 |
| johnsom | Hmm, I didn’t think ansible creates an octavia fflavor by default | 21:56 |
| johnsom | Octavia flavors don’t have that | 21:57 |
| spatel | it did, because i just deploy using osa and now i can see m1.amphora flavor in list | 21:57 |
| johnsom | No, that is a compute flavor for nova | 21:57 |
| johnsom | It is not an Octavia flavor | 21:57 |
| johnsom | Read the guide link I sent | 21:58 |
| johnsom | Most of the OpenStack services have flavors, they just mean different things | 21:58 |
| spatel | johnsom: you are right.. now i know what you trying to say.. | 21:59 |
| spatel | openstack loadbalancer flavor list <-- this is empty list | 21:59 |
| spatel | I thought openstack flavor list will be visible in that GUI | 21:59 |
| johnsom | Nope, different flavors | 22:00 |
| *** tkajinam has joined #openstack-lbaas | 22:00 | |
| spatel | That is very clear now :) i got confused when its saying flavor | 22:01 |
| spatel | lets saying i reboot my compute node and while its rebooting amphora think VM is dead and start rebuilding one on other compute node in that case how it will handle | 22:13 |
| spatel | Trying to think about disaster senarios | 22:13 |
| *** gcheresh has quit IRC | 22:14 | |
| johnsom | Yeah, so if the host reboots and nova brings up the amphora inside the health check timeout (60 seconds by default), nothing happens (assuming standalone mode here). | 22:14 |
| johnsom | It picks up where it left off and keeps going. The caveat is TLS offload, which the encrypted ram disk will have been cleared. | 22:15 |
| johnsom | If it doesn’t make the time or it does have TLS, a failover will start and the old instance is marked for deletion and it’s ports are disabled | 22:16 |
| spatel | that was my answer | 22:17 |
| spatel | it will delete old instance and keep new one | 22:17 |
| spatel | johnsom: sounds good. | 22:17 |
| johnsom | Yeah, and if nova fails to delete it, we have a zombie killer as well | 22:18 |
| spatel | nice | 22:18 |
| johnsom | It just keeps deleting it until nova stops failing | 22:18 |
| spatel | Let me try to reproduce more disaster scenario to understand how does it handle :) | 22:19 |
| johnsom | Ok | 22:19 |
| johnsom | I am about to go rack leaves, so I will be offline for a while | 22:20 |
| spatel | johnsom: thank you for answering all my question :) | 22:20 |
| spatel | Have a great weekend and stay safe! | 22:20 |
| johnsom | Rake, helps if I can type | 22:20 |
| johnsom | You too | 22:21 |
| spatel | :) | 22:21 |
| *** spatel has quit IRC | 22:22 | |
| *** jamesdenton has quit IRC | 22:40 | |
| *** jamesdenton has joined #openstack-lbaas | 22:40 | |
| *** luksky has quit IRC | 23:20 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!