| *** TMM has quit IRC | 00:20 | |
| *** TMM has joined #openstack-lbaas | 00:22 | |
| *** rm_work has quit IRC | 00:37 | |
| *** logan- has quit IRC | 00:38 | |
| *** rm_work has joined #openstack-lbaas | 00:38 | |
| *** rm_work has quit IRC | 00:38 | |
| *** logan- has joined #openstack-lbaas | 00:40 | |
| *** yamamoto_ has joined #openstack-lbaas | 00:45 | |
| *** yamamoto has quit IRC | 00:46 | |
| *** sapd__x has quit IRC | 00:58 | |
| *** mnaser has quit IRC | 01:13 | |
| openstackgerrit | Merged openstack/octavia master: Fix PROXYV2 pools https://review.opendev.org/c/openstack/octavia/+/756934 | 01:13 |
|---|---|---|
| *** fyx has quit IRC | 01:14 | |
| *** wolsen has quit IRC | 01:14 | |
| *** mnaser has joined #openstack-lbaas | 01:16 | |
| *** rm_work has joined #openstack-lbaas | 01:18 | |
| *** rm_work has quit IRC | 01:18 | |
| *** rm_work has joined #openstack-lbaas | 01:29 | |
| *** rm_work has quit IRC | 01:29 | |
| *** fyx has joined #openstack-lbaas | 01:33 | |
| *** xgerman has joined #openstack-lbaas | 01:37 | |
| *** wolsen has joined #openstack-lbaas | 01:46 | |
| *** rm_work has joined #openstack-lbaas | 02:01 | |
| *** wolsen has quit IRC | 02:04 | |
| *** yamamoto_ has quit IRC | 02:15 | |
| *** yamamoto has joined #openstack-lbaas | 02:16 | |
| *** sshnaidm|ruck is now known as sshnaidm|off | 02:25 | |
| *** lxkong has quit IRC | 02:38 | |
| *** lxkong has joined #openstack-lbaas | 02:39 | |
| openstackgerrit | Merged openstack/octavia master: Install HAProxy 2.x in CentOS amphora images https://review.opendev.org/c/openstack/octavia/+/744102 | 02:41 |
| *** wolsen has joined #openstack-lbaas | 02:48 | |
| *** sapd1_x has quit IRC | 02:49 | |
| *** zzzeek has quit IRC | 03:01 | |
| *** zzzeek has joined #openstack-lbaas | 03:02 | |
| *** sapd__x has joined #openstack-lbaas | 03:21 | |
| *** rcernin has quit IRC | 03:27 | |
| *** nicolasbock_ has joined #openstack-lbaas | 03:30 | |
| *** TMM has quit IRC | 03:38 | |
| *** nicolasbock has quit IRC | 03:38 | |
| *** tobias-urdin has quit IRC | 03:38 | |
| *** beagles has quit IRC | 03:38 | |
| *** nicolasbock_ is now known as nicolasbock | 03:38 | |
| *** rcernin has joined #openstack-lbaas | 03:41 | |
| *** psachin has joined #openstack-lbaas | 03:42 | |
| *** TMM has joined #openstack-lbaas | 03:43 | |
| *** beagles has joined #openstack-lbaas | 03:43 | |
| *** TMM has quit IRC | 03:43 | |
| *** TMM has joined #openstack-lbaas | 03:43 | |
| *** xgerman has quit IRC | 04:08 | |
| *** vishalmanchanda has joined #openstack-lbaas | 05:03 | |
| openstackgerrit | Merged openstack/octavia master: Support deletion of all tags https://review.opendev.org/c/openstack/octavia/+/755828 | 05:19 |
| *** sapd__x has quit IRC | 05:42 | |
| *** sapd__x has joined #openstack-lbaas | 05:47 | |
| *** gcheresh has joined #openstack-lbaas | 06:57 | |
| *** openstackgerrit has quit IRC | 07:23 | |
| *** gcheresh has quit IRC | 07:59 | |
| *** openstackgerrit has joined #openstack-lbaas | 08:33 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia master: Add haproxy-source backend element in DIB https://review.opendev.org/c/openstack/octavia/+/698086 | 08:33 |
| cgoncalves | rm_work, https://review.opendev.org/c/openstack/python-octaviaclient/+/772848 | 08:45 |
| *** rpittau|afk is now known as rpittau | 08:52 | |
| *** zzzeek has quit IRC | 09:01 | |
| *** rcernin has quit IRC | 09:02 | |
| *** yamamoto has quit IRC | 09:02 | |
| *** zzzeek has joined #openstack-lbaas | 09:03 | |
| *** gcheresh has joined #openstack-lbaas | 09:07 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-dashboard master: Add ciphers options for listeners and pools https://review.opendev.org/c/openstack/octavia-dashboard/+/718550 | 09:21 |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-dashboard master: Add TLS enabled support for pools https://review.opendev.org/c/openstack/octavia-dashboard/+/763352 | 09:21 |
| *** gcheresh has quit IRC | 09:21 | |
| *** rcernin has joined #openstack-lbaas | 09:21 | |
| *** yamamoto has joined #openstack-lbaas | 09:38 | |
| *** psachin has quit IRC | 09:46 | |
| rm_work | cgoncalves: +W, but I hate these LC tests (and LC in general) | 09:47 |
| rm_work | It's like a constant game of chase-the-version, just install in a damn virtualenv | 09:48 |
| rm_work | Or a container | 09:48 |
| *** yamamoto has quit IRC | 09:48 | |
| rm_work | Who in their right mind does like "sudo pip install" in 2021 | 09:49 |
| rm_work | If I knew someone not using virtualenvs, first I'd teach them how to use virtualenvs, then if they still didn't do it, STRAIGHT TO PYTHON JAIL | 09:50 |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-dashboard master: Allows creation of resources without forcing the creation of children https://review.opendev.org/c/openstack/octavia-dashboard/+/763353 | 10:02 |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-dashboard master: Add TLS versions support for listeners and pools https://review.opendev.org/c/openstack/octavia-dashboard/+/763569 | 10:02 |
| *** lxkong has quit IRC | 10:15 | |
| *** lxkong has joined #openstack-lbaas | 10:16 | |
| *** yamamoto has joined #openstack-lbaas | 10:19 | |
| *** rcernin has quit IRC | 10:23 | |
| *** zzzeek has quit IRC | 10:30 | |
| *** zzzeek has joined #openstack-lbaas | 10:32 | |
| *** spatel has joined #openstack-lbaas | 10:36 | |
| *** spatel has quit IRC | 10:40 | |
| openstackgerrit | Carlos Gonçalves proposed openstack/octavia master: Install centos-release-nfv-extras https://review.opendev.org/c/openstack/octavia/+/763215 | 10:49 |
| *** zzzeek has quit IRC | 10:54 | |
| *** zzzeek has joined #openstack-lbaas | 10:55 | |
| *** xarlos has quit IRC | 11:00 | |
| *** gcheresh has joined #openstack-lbaas | 11:01 | |
| openstackgerrit | Carlos Gonçalves proposed openstack/octavia-tempest-plugin master: Fix copy output in two-node jobs https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/752936 | 11:03 |
| *** sshnaidm|off has quit IRC | 11:09 | |
| *** sshnaidm|off has joined #openstack-lbaas | 11:10 | |
| *** rcernin has joined #openstack-lbaas | 11:33 | |
| *** gcheresh has quit IRC | 11:49 | |
| *** sapd__x has quit IRC | 11:55 | |
| *** servagem has quit IRC | 12:27 | |
| *** servagem has joined #openstack-lbaas | 12:28 | |
| *** rcernin has quit IRC | 12:32 | |
| *** spatel has joined #openstack-lbaas | 12:37 | |
| *** ccamposr__ has joined #openstack-lbaas | 12:40 | |
| *** spatel has quit IRC | 12:41 | |
| *** ccamposr has quit IRC | 12:43 | |
| openstackgerrit | Merged openstack/python-octaviaclient master: Fix lower-constraints job https://review.opendev.org/c/openstack/python-octaviaclient/+/772848 | 12:44 |
| openstackgerrit | Carlos Gonçalves proposed openstack/python-octaviaclient master: Add ALPN support for pools https://review.opendev.org/c/openstack/python-octaviaclient/+/752096 | 12:47 |
| *** yamamoto has quit IRC | 12:53 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia master: Add SCTP support in Amphora https://review.opendev.org/c/openstack/octavia/+/753247 | 12:59 |
| *** rm_work has quit IRC | 13:00 | |
| *** jamesdenton has quit IRC | 13:00 | |
| *** mnaser has quit IRC | 13:00 | |
| *** jrosser has quit IRC | 13:00 | |
| *** rm_work has joined #openstack-lbaas | 13:01 | |
| *** jamesdenton has joined #openstack-lbaas | 13:01 | |
| *** mnaser has joined #openstack-lbaas | 13:01 | |
| *** jrosser has joined #openstack-lbaas | 13:01 | |
| *** wolsen has quit IRC | 13:02 | |
| *** fyx has quit IRC | 13:03 | |
| *** f0o has quit IRC | 13:03 | |
| *** andy_ has quit IRC | 13:03 | |
| *** zigo has quit IRC | 13:03 | |
| *** sorrison has quit IRC | 13:03 | |
| *** wolsen has joined #openstack-lbaas | 13:03 | |
| *** fyx has joined #openstack-lbaas | 13:03 | |
| *** f0o has joined #openstack-lbaas | 13:03 | |
| *** andy_ has joined #openstack-lbaas | 13:03 | |
| *** zigo has joined #openstack-lbaas | 13:03 | |
| *** sorrison has joined #openstack-lbaas | 13:03 | |
| *** wolsen has quit IRC | 13:04 | |
| *** yamamoto has joined #openstack-lbaas | 13:21 | |
| *** sapd__x has joined #openstack-lbaas | 13:25 | |
| *** yamamoto has quit IRC | 13:26 | |
| *** zzzeek has quit IRC | 13:29 | |
| *** zzzeek has joined #openstack-lbaas | 13:31 | |
| *** yamamoto has joined #openstack-lbaas | 13:37 | |
| *** spatel has joined #openstack-lbaas | 13:51 | |
| *** zzzeek has quit IRC | 13:52 | |
| *** spatel has quit IRC | 13:53 | |
| *** zzzeek has joined #openstack-lbaas | 13:56 | |
| *** zzzeek has quit IRC | 14:02 | |
| *** zzzeek has joined #openstack-lbaas | 14:03 | |
| *** wolsen has joined #openstack-lbaas | 14:14 | |
| *** gcheresh has joined #openstack-lbaas | 14:24 | |
| mchlumsky | Hi there folks! We just upgraded our Octavia deployment to Ussuri from Stein. It was a mostly flawless experience. Thanks for all your hard work, much appreciated! | 14:26 |
| mchlumsky | Our one snag in the upgrade came when we did the failovers and I have a feeling it's our own fault. We have a project that uses a vip subnet with a host route defined (destination: 10.0.0.0/8, nexthop: 10.x.y.z). Our lb-mgmt-net is a 10.a.b.0/23 and our control plane sits on a 10.c.d.0/23 so that host route on the vip subnet interferes with the | 14:26 |
| mchlumsky | communication between the amphora | 14:26 |
| mchlumsky | s and the control plane. Not sure if I'm asking for help or a shoulder to cry on. | 14:26 |
| *** sshnaidm|off has quit IRC | 14:27 | |
| *** sshnaidm|off has joined #openstack-lbaas | 14:27 | |
| gthiemonge | mchlumsky: the management interface and the vip/members interfaces are in distinct namespaces in the amphora, so it should not be a problem | 14:35 |
| mchlumsky | yeah, I thought that as well but I think there might be a race condition. When I look on the amphora (centos 8 based) there is only the default namespace and I can see the host route in ip route | 14:38 |
| *** yamamoto has quit IRC | 14:57 | |
| *** gcheresh has quit IRC | 15:20 | |
| *** rpittau is now known as rpittau|afk | 15:27 | |
| *** sapd__x has quit IRC | 15:34 | |
| johnsom | mchlumsky: The host routes are honored in the network namespace haproxy-amphora. As gthiemonge mentioned, they are separated from the lb-mgmt-net. What OS are the amphora images? | 15:44 |
| mchlumsky | CentOS 8 | 15:45 |
| johnsom | Hmm, there was a patch recently messing with the route files on centos 8. I wonder if that broke them. | 15:46 |
| johnsom | I did the Ubuntu side, but someone else did the CentOS. | 15:47 |
| johnsom | If there is only the default namespace in the amp, something else major is wrong though. I.e. ip netns doesn’t show the other namespace | 15:48 |
| mchlumsky | I'll keep digging. It's not that hard to reproduce | 15:53 |
| haleyb | johnsom: it wasn't this one you were thinking of was it? https://review.opendev.org/c/openstack/octavia/+/700513/ | 15:55 |
| johnsom | Yeah, that was the one. But if the netns is missing, it didn’t get that far | 15:56 |
| *** sshnaidm|off has quit IRC | 16:09 | |
| *** sshnaidm|off has joined #openstack-lbaas | 16:14 | |
| *** jaibanez has joined #openstack-lbaas | 16:41 | |
| *** jaibanez has quit IRC | 16:43 | |
| mchlumsky | the /etc/netns directory is missing completely | 17:18 |
| johnsom | I don't think that is used | 17:18 |
| johnsom | If you are in the amphroa, run "sudo ip netns" | 17:19 |
| mchlumsky | there are no namespaces | 17:20 |
| mchlumsky | the weird thing is that the other amphora did get net namespaces. | 17:20 |
| johnsom | Hmm, so something went wrong super early. Can you check the /var/log/messages/syslog for the amphora-agent and see if there are errors there? | 17:20 |
| johnsom | Also check the amphora-agent log, but typically the errors will go in the main syslog file | 17:21 |
| *** xgerman has joined #openstack-lbaas | 17:24 | |
| mchlumsky | no syslog file in centos 8. I guess journalctl? | 17:28 |
| *** vishalmanchanda has quit IRC | 17:32 | |
| johnsom | It's "messages" I think | 17:34 |
| mchlumsky | http://paste.openstack.org/show/802143/ The vip network comes up after amphora-agent starts. | 19:08 |
| johnsom | Yeah, that looks like the amphora has not been configured yet, just the initial boot | 19:10 |
| johnsom | No errors there | 19:10 |
| mchlumsky | ah so the NICs don't all have to be up before the amphora agent starts? | 19:13 |
| johnsom | Nope | 19:13 |
| mchlumsky | I don't really see any errors anywhere. I do notice that there is this message: `kernel: virtio_net virtio4 ens7: renamed from eth0` and it's normally followed later in the logs by a similar message saying it renames ens7 to eth1 but this second message doesn't happen in the problematic amphorae | 19:41 |
| johnsom | The interface names don't matter in amphora. We don't rely on them as they are fluid across distros | 19:48 |
| johnsom | Could this amphora be a spare? i.e. not in use by a load balancer? | 19:48 |
| johnsom | what is it's status in "openstack loadbalancer amphora list"? | 19:48 |
| mchlumsky | ALLOCATED | 19:52 |
| mchlumsky | both MASTER and BACKUP roles | 19:52 |
| johnsom | So it is assigned to a load balancer. So, if it's hasn't been configured, there must be some messages in the worker logs. | 19:53 |
| mchlumsky | octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out | 19:56 |
| johnsom | So it wasn't reachable over the lb-mgmt-net from that controller. I would check the neutron networking, the management port on that controller, etc. | 19:56 |
| mchlumsky | Keep in mind it has that route set that sends traffic out over a network that isn;t the lb-mgmt-net | 19:57 |
| johnsom | Yeah, but the log earier shows the controller never sent out any configuration to the amp, so that all isn't even there yet | 19:58 |
| mchlumsky | Does the controller need to reach the amphora for it to configure the net namespace? Until the net namespace is configured, the bad route is in the default namespace. Chicken and egg problem? | 20:00 |
| johnsom | All that is in that amp now is the lb-mgmt-net interface configured by nova and the routes associated with that. The amphora-agent, when it gets a config, will create the netns, configured the vip interface, add the routes, etc. But you aren't getting that far yet. | 20:00 |
| mchlumsky | ok, this makes more sense now. | 20:01 |
| *** rcernin has joined #openstack-lbaas | 20:09 | |
| mchlumsky | The 10/8 route that interferes with the lb-mgmt-net is getting set by networkmanager via dhcp when the second interface (the one from vip_subnet_id) gets added. It happens early enough that it can keep the contoller from pushing config to the amphora so it can never move that route into the amphora namespace. | 20:10 |
| *** rcernin has quit IRC | 20:44 | |
| *** rcernin has joined #openstack-lbaas | 21:10 | |
| *** rcernin has quit IRC | 21:15 | |
| *** rcernin has joined #openstack-lbaas | 21:27 | |
| *** yamamoto has joined #openstack-lbaas | 22:11 | |
| *** servagem has quit IRC | 22:14 | |
| rm_work | Interesting | 22:18 |
| *** yamamoto has quit IRC | 22:21 | |
| *** yamamoto has joined #openstack-lbaas | 22:25 | |
| *** stand has quit IRC | 22:26 | |
| *** armax has quit IRC | 22:43 | |
| *** rcernin has quit IRC | 22:47 | |
| *** ccamposr has joined #openstack-lbaas | 22:47 | |
| rm_work | johnsom: so how do we accelerate https://review.opendev.org/c/openstack/taskflow/+/772953 ? :D | 22:47 |
| rm_work | who else do we ping? | 22:47 |
| *** ccamposr__ has quit IRC | 22:49 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Bump the minimum coverage to 92% https://review.opendev.org/c/openstack/octavia/+/760382 | 23:02 |
| rm_work | ^^ just a rebase | 23:02 |
| *** rcernin has joined #openstack-lbaas | 23:04 | |
| johnsom | rm_work Any of the oslo cores. I will try to remember to bring it up at the meeting Monday as well | 23:10 |
| *** rcernin has quit IRC | 23:19 | |
| *** xgerman has quit IRC | 23:34 | |
| *** zzzeek has quit IRC | 23:48 | |
| *** zzzeek has joined #openstack-lbaas | 23:50 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!