| opendevreview | Susanta gautam proposed openstack/octavia master: Add simple notifications for load balancers https://review.opendev.org/c/openstack/octavia/+/784628 | 05:09 |
|---|---|---|
| opendevreview | Vishal Manchanda proposed openstack/octavia-dashboard master: Updating python testing classifier as per Yoga testing runtime https://review.opendev.org/c/openstack/octavia-dashboard/+/826986 | 05:12 |
| opendevreview | Susanta gautam proposed openstack/octavia master: Add simple notifications for load balancers https://review.opendev.org/c/openstack/octavia/+/784628 | 08:06 |
| zigo | Hi there! Your thoughts about the bug I just filled ? | 09:22 |
| zigo | https://storyboard.openstack.org/#!/story/2009815 | 09:22 |
| gthiemonge | zigo: Hi, thanks for reporting it | 10:00 |
| gthiemonge | zigo: AFAIK there's a kind of lock for the LBs in Octavia (PENDING_* means the LB is busy), but Octavia doesn't it when rotating the certs | 10:04 |
| gthiemonge | zigo: one question about your story: did you run a failover from the CLI that was denied because it happens when octavia tried to rotate the certs? | 10:05 |
| zigo | gthiemonge: The cli reported an HTTP 409: the load balancer is ummutable. | 10:17 |
| gthiemonge | zigo: if the CLI replied with 409, it means that you didn't trigger the failover. But if the amphora was not reachable, it is possible that the health-manager service triggered it | 10:21 |
| opendevreview | Tom Weininger proposed openstack/octavia-tempest-plugin master: Add tests for creating a fully populated LB https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/824999 | 10:55 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Fix member DRAIN state https://review.opendev.org/c/openstack/octavia/+/826897 | 11:52 |
| opendevreview | Tom Weininger proposed openstack/octavia-dashboard master: Display Draining state correctly https://review.opendev.org/c/openstack/octavia-dashboard/+/826905 | 12:24 |
| opendevreview | Tom Weininger proposed openstack/octavia-dashboard master: Display Draining state correctly https://review.opendev.org/c/openstack/octavia-dashboard/+/826905 | 12:29 |
| gthiemonge | FYI stable/train, stable/ussuri, centos-8-stream & octavia-tempest-plugin jobs are failing because of a new pip issue | 14:34 |
| opendevreview | Tom Weininger proposed openstack/octavia-dashboard master: Display Draining state correctly https://review.opendev.org/c/openstack/octavia-dashboard/+/826905 | 15:42 |
| *** tkajinam is now known as Guest1307 | 18:43 | |
| opendevreview | Gregory Thiemonge proposed openstack/octavia master: Optimize DB object to provider dict conversions https://review.opendev.org/c/openstack/octavia/+/827169 | 19:17 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia master: Remove incorrect info message https://review.opendev.org/c/openstack/octavia/+/827170 | 19:26 |
| opendevreview | Merged openstack/octavia-lib master: Add "PROMETHEUS" protocol. https://review.opendev.org/c/openstack/octavia-lib/+/812257 | 20:30 |
| michchap | o/ hello! I'm seeing the issue described here https://bugs.launchpad.net/octavia/+bug/1841016 after upgrading from rocky to stein. If I create a new secgroup as the octavia user and set that in amp_secgroup_list then I can make new loadbalancers, but the old ones won't failover because the ports can't be connected I think due to the same issue. Does anyone know what I'm missing? | 21:43 |
| johnsom | michchap What error do you get in the worker log when you attempt a failover? | 22:09 |
| michchap | PlugNetworkException: Error plugging amphora (compute_id: b608c4b2-3048-41bb-937f-517beca43752) into port 4b8f152b-7356-4b19-9da8-5c3624f335fd. | 22:10 |
| johnsom | There are not other related error messages there? | 22:10 |
| michchap | a little further up I see ERROR octavia.network.drivers.neutron.allowed_address_pairs BadRequest: Port 4b8f152b-7356-4b19-9da8-5c3624f335fd not usable for instance b608c4b2-3048-41bb-937f-517beca43752. (HTTP 400) (Request-ID: req-cf7d8a5e-68e3-4de7-9388-2a0319f56ff7) | 22:10 |
| michchap | was there a change between rocky and stein regarding how the users/roles needed to be set up? | 22:11 |
| johnsom | Hmm, that is a new one. | 22:11 |
| johnsom | No | 22:11 |
| michchap | swapping the configured secgroup is probably not the right thing to do then | 22:12 |
| johnsom | What port is 4b8f152b-7356-4b19-9da8-5c3624f335fd | 22:12 |
| michchap | 4b8f152b-7356-4b19-9da8-5c3624f335fd | octavia-lb-vrrp-ba8e835f-23fd-4ee8-a13d-3bcd4a50f7b7 | fa:16:3e:25:bc:08 | ip_address='192.168.0.37', subnet_id='5f4e34b4-f5aa-460f-a19a-c128148ea631' | DOWN | | 22:13 |
| johnsom | Hmm, do you have dns integration enabled in neutron? | 22:13 |
| michchap | what I don't understand is why when the octavia user has the admin role, it's failing to use the configured security group in that launchpad bug | 22:13 |
| johnsom | Does that port have a dns_name? | 22:13 |
| johnsom | That launchpad bug was marked invalid, who knows what was going on there | 22:14 |
| michchap | that's the error I'm seeing after the rocky-stein upgrade | 22:14 |
| michchap | using a pretty vanilla kolla-ansible deployment | 22:15 |
| johnsom | This is the key error: Port 4b8f152b-7356-4b19-9da8-5c3624f335fd not usable for instance b608c4b2-3048-41bb-937f-517beca43752. | 22:15 |
| michchap | I don't see any DNS names on the port | 22:15 |
| johnsom | I'm trying to think of why neutron/nova would reject a port | 22:15 |
| michchap | my guess was the service user lacking permission | 22:15 |
| michchap | the same as it seems to lack permission to use the security group | 22:16 |
| michchap | but the admin role is on octavia@Default in service@Default | 22:17 |
| johnsom | Nova - PortNotUsable - If a requested port is not owned by the same tenant that the instance is created under. | 22:18 |
| johnsom | Yeah, so did your upgrade code somehow change the service account? | 22:18 |
| michchap | I'll triple check I guess, but I think it's remained octavia/service/Default | 22:19 |
| johnsom | Check the project ID on the port and on the instance. With Octavia, those should always match as both are created with the octavia service account | 22:19 |
| michchap | looks like that port is created as the admin user | 22:20 |
| johnsom | The account that Octavia will use to create those resources is configured in the [service_auth] section of the octavia.conf. | 22:21 |
| michchap | Yeah, I suspect kolla has changed the default. The octavia user is probably the correct way going forward but if it breaks failover I guess we'll have to recreate all the LBs after the upgrade | 22:23 |
| michchap | ah, yep the project name is set to admin in kolla rocky -_- | 22:24 |
| johnsom | Yeah, I don't know kolla, so can't help much there. But changing the octavia service account with existing LBs is going to be a problem. | 22:24 |
| michchap | if there's no migration path when doing so, we'll just have to deal with it | 22:25 |
| michchap | thanks for the help! | 22:25 |
| johnsom | Sure, NP | 22:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!