| opendevreview | Merged openstack/octavia master: Use centos amphora image in the FIPS jobs https://review.opendev.org/c/openstack/octavia/+/827843 | 01:42 |
|---|---|---|
| mnasiadka | Good morning - I'm having an issue with octavia and ovn-octavia-provider - that OVN NB connection timeouts are affecting the octavia api service itself (e.g. when it's trying to do it's job - the whole WSGI API service is hanging/frozen) - shouldn't those operations be done in the background? | 06:16 |
| gthiemonge | mnasiadka: Hi, does it timeout on startup? the Octavia API service loads 3rd party providers at init time, so if a driver is broken (ex: connection failures), it impacts the API service | 06:24 |
| mnasiadka | gthiemonge: no, not on startup, let me paste some logs into paste service | 06:24 |
| gthiemonge | mnasiadka: BTW the ovn-octavia-provider is maintained by the neutron team. The Octavia team provides the framework that enables these drivers | 06:25 |
| mnasiadka | Yes, I understand - one thing is to speak with them around those timeouts - second is to get Octavia to not be frozen on those timeouts - as I understand ;-) | 06:26 |
| mnasiadka | basically adding members to an OVN load balancer ends with a timeout | 06:27 |
| gthiemonge | yeah that sounds like an ovn provider issue | 06:27 |
| mnasiadka | ok then, let me go complain there :) | 06:31 |
| gthiemonge | mnasiadka: I'll keep on eye on it | 06:47 |
| mnasiadka | gthiemonge: thanks | 06:53 |
| opendevreview | Gaudenz Steinlin proposed openstack/octavia master: Reconfigure amphora network interfaces seamlessly https://review.opendev.org/c/openstack/octavia/+/812368 | 08:00 |
| opendevreview | Gaudenz Steinlin proposed openstack/octavia master: Fix plugging member subnets on existing networks https://review.opendev.org/c/openstack/octavia/+/665402 | 08:00 |
| opendevreview | Gaudenz Steinlin proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/c/openstack/octavia/+/660239 | 08:00 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/wallaby: Drop Bionic support https://review.opendev.org/c/openstack/octavia/+/830592 | 11:07 |
| opendevreview | Gaudenz Steinlin proposed openstack/octavia master: Reconfigure amphora network interfaces seamlessly https://review.opendev.org/c/openstack/octavia/+/812368 | 12:23 |
| opendevreview | Gaudenz Steinlin proposed openstack/octavia master: Fix plugging member subnets on existing networks https://review.opendev.org/c/openstack/octavia/+/665402 | 12:23 |
| opendevreview | Gaudenz Steinlin proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/c/openstack/octavia/+/660239 | 12:23 |
| viks__ | hi, octavia documentation says to create : `openstack flavor create --id 200 --vcpus 1 --ram 1024 --disk 2 "amphora" --private` . Is this sufficient for a production setup? or what is the suggested flavor for a production system? Can someone please guide me on this? | 13:36 |
| gthiemonge | viks__: this is close to what we recommend to our users, I don't know if some users are using more than 1 vcpu, we haven't done anything in the amphora to optimize multiple vcpus (cpu-pinning, etc...) but it's on the roadmap, 1GB ram is enough for 1 vcpu, 2GB for the disk may be not enough if you don't offload the logs nad you have many connections | 13:43 |
| viks__ | gthiemonge: Thanks for the quick reply. What is the recommended disk size or how to offload logs? log rotation? or any other way? | 13:45 |
| gthiemonge | viks__: we have logrotate in the amp but it is triggered once a day, so you can still fill the disk with logs in 24h (a change was proposed at https://review.opendev.org/c/openstack/octavia/+/810996) | 14:00 |
| gthiemonge | viks__: log offloading support (you need a syslog server): https://docs.openstack.org/octavia/latest/admin/log-offloading.html | 14:01 |
| viks__ | gthiemonge: ok thanks... I was also trying compute node failure scenario where the amphora instance is present. It worked fine with loadbalancer topology : SINGLE eventhough there was some delay due to new instance creation in other available compute node. However in ACTIVE/STANDBY topology, compute host failure did not work for me somehow. I also used anti-affinity settings stiill no luck. Also tried SINGLE topology with | 14:10 |
| viks__ | `spare_amphora_pool_size` which also did not work. I'm using train release... any suggestions on this? | 14:10 |
| gthiemonge | viks__: hmm it should work in ACTIVE_STANDBY, you can log in to both amphorae (with ssh) and check the logs (keepalived logs), you should be able to see which amp is the MASTER, and when triggering a failure on the compute node that runs this amphora, the BACKUP amphora will detect it and it shuld become the MASTER amphora | 14:13 |
| gthiemonge | I would not recommand using spare amphora, this feature was removed in the Xena release | 14:14 |
| gthiemonge | viks__: and after BACKUP becomes MASTER, octavia should re-spawn a new amphora to replace the missing one | 14:14 |
| viks__ | gthiemonge: if my setup has 2 compute nodes, it spawns both instances in the same compute node... so i tried to use anti-affinity settings... but then one of the instance stuck in BOOTING when i create http load balancer and curl to the backend instance via floating ip does not work.. not sure what i'm doing wrong? should i use ACTIVE?STANDBY or SINGLE topology? | 14:21 |
| gthiemonge | when using ACTIVE_STANDBY, you need 3 compute nodes if you have anti-affinity (it would work with soft-anti-affinity), because if one node is down, nova needs a compute node that doesn't already have an amphora from the same lb | 14:24 |
| gthiemonge | "one of the instance stuck in BOOTING" -> this is an issue, does it go into ERROR after a timeout? | 14:25 |
| viks__ | gthiemonge: yes.. it goes in to error state | 14:25 |
| gthiemonge | viks__: it may mean that nova failed to start a VM, or that there's an issue with the network connectivity | 14:25 |
| gthiemonge | viks__: so check the octavia worker logs, you will find ERROR-level messages | 14:26 |
| viks__ | gthiemonge: ok... will investigate more on it.. Also how do i delete loadbalancer stuck in ` Pending Create` state? | 14:27 |
| gthiemonge | viks__: you cannot delete them, they should go into ERROR after a timeout, then you can remove them. If they don't (after a long wait) and you're sure that octavia is not doing anything with this LB (check activity in the logs), you need to update the DB manually to set them to ERROR | 14:29 |
| viks__ | gthiemonge: ok.. thanks | 14:31 |
| viks__ | what is the recommended topology for production setup: SINGLE or ACTIVE_STANDBY? | 14:32 |
| gthiemonge | I recommend active-standby if you want to recover really quickly from failures | 14:35 |
| viks__ | gthiemonge: ok.. thanks | 14:36 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/wallaby: Drop Bionic support https://review.opendev.org/c/openstack/octavia/+/830592 | 14:37 |
| opendevreview | Gaudenz Steinlin proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/c/openstack/octavia/+/660239 | 15:09 |
| *** gmann is now known as gmann_afk | 15:59 | |
| gthiemonge | #startmeeting Octavia | 16:00 |
| opendevmeet | Meeting started Wed Feb 23 16:00:31 2022 UTC and is due to finish in 60 minutes. The chair is gthiemonge. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'octavia' | 16:00 |
| gthiemonge | Hi! | 16:00 |
| tweining | hi | 16:01 |
| johnsom | o/ | 16:01 |
| gthiemonge | #topic Announcements | 16:02 |
| gthiemonge | ** Yoga-3 milestone | 16:02 |
| gthiemonge | This is the feature freeze and final release for client libraries | 16:03 |
| gthiemonge | AFAIK we are good for the features | 16:03 |
| gthiemonge | and johnsom proposed a release highlights commit for Octavia: | 16:03 |
| johnsom | I have proposed our feature highlights: | 16:03 |
| johnsom | #link https://review.opendev.org/c/openstack/releases/+/830518 | 16:03 |
| gthiemonge | #link https://review.opendev.org/c/openstack/releases/+/830518 | 16:03 |
| opendevreview | Merged openstack/octavia-dashboard master: Display Draining state correctly https://review.opendev.org/c/openstack/octavia-dashboard/+/826905 | 16:03 |
| gthiemonge | ;-) | 16:03 |
| johnsom | lol, yeah, that.... | 16:03 |
| gthiemonge | johnsom: thanks! | 16:03 |
| gthiemonge | for python-octaviaclient, I still see one bugfix: | 16:04 |
| gthiemonge | #link https://review.opendev.org/c/openstack/python-octaviaclient/+/808030 | 16:04 |
| johnsom | If you have any changes, please comment today. I plan to remove the WIP tomorrow | 16:04 |
| gthiemonge | ok thanks | 16:04 |
| gthiemonge | now we can focus on bugfixes for the Yoga release | 16:06 |
| gthiemonge | reminder: the link to the priority review list is in the topic of this channel: | 16:07 |
| gthiemonge | #link https://etherpad.openstack.org/p/octavia-priority-reviews | 16:07 |
| gthiemonge | any other announcements folks? | 16:07 |
| gthiemonge | johnsom: ? | 16:07 |
| johnsom | Not that I can think of at the moment. | 16:07 |
| gthiemonge | #topic Brief progress reports / bugs needing review | 16:08 |
| gthiemonge | I have focused on reviews, I also created a lot of backports because we merged a lot of commits on master these last days | 16:08 |
| tweining | do we know for certain that the ipv6 issue is fixed? | 16:09 |
| gthiemonge | well if you see some random gate failures with ipv6 tests, ping me | 16:10 |
| johnsom | I did capture all of the system state on one of the IPv6 errors in my test patch. But I didn't dig into it as Greg said he found the issue. | 16:11 |
| gthiemonge | the ipv6 issues may still impact octavia-tempest-plugin because the backports are still in review | 16:11 |
| johnsom | If you are bored, the logs are probably still there. | 16:11 |
| gthiemonge | who is bored here? | 16:11 |
| johnsom | Yeah, I don't have the time to spare for that. | 16:11 |
| tweining | I hope once it's backported https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/824999 will finally succeed | 16:13 |
| gthiemonge | yeah I hope so | 16:14 |
| gthiemonge | one last thing, the periodic job octavia-amphora-image-build is broken on wallaby, the job is still using ubuntu bionic instead of focal | 16:15 |
| gthiemonge | I backported the fix: | 16:15 |
| gthiemonge | #link https://review.opendev.org/c/openstack/octavia/+/830592 | 16:15 |
| johnsom | Opps | 16:15 |
| gthiemonge | installing pyroute2 0.6.5 triggered the issue in wallaby | 16:16 |
| *** gmann_afk is now known as gmann | 16:17 | |
| gthiemonge | (so this is a recent breakage) | 16:17 |
| gthiemonge | #topic Open Discussion | 16:17 |
| gthiemonge | any other topics? | 16:17 |
| johnsom | Fun trivia note, including rc's and dot releases across all of the versions, we have put out 83 releases of Octavia over the history of the project. | 16:19 |
| tweining | I tried to profile octavia-tempest/octavia-api/octavia-cw but didn't really find a bottleneck. | 16:20 |
| gthiemonge | wow that's impressive | 16:20 |
| johnsom | We will have to watch for 100 and have a hangout or something | 16:21 |
| gthiemonge | tweining: thanks for trying ;-) | 16:21 |
| tweining | https://review.opendev.org/c/openstack/octavia/+/830441 is your interested in the code | 16:21 |
| tweining | *if you're | 16:22 |
| johnsom | Yeah, thanks for having a look. I will look at the patch. | 16:22 |
| tweining | the api and worker are quite fast actually | 16:22 |
| johnsom | Yeah, I would expect they aren't bad. That is why I don't understand no-op API tests taking a minute or more to run. | 16:24 |
| gthiemonge | Ok I think that's all for today! | 16:29 |
| gthiemonge | thank you Folks | 16:29 |
| gthiemonge | #endmeeting | 16:29 |
| opendevmeet | Meeting ended Wed Feb 23 16:29:46 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:29 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/octavia/2022/octavia.2022-02-23-16.00.html | 16:29 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/octavia/2022/octavia.2022-02-23-16.00.txt | 16:29 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/octavia/2022/octavia.2022-02-23-16.00.log.html | 16:29 |
| tweining | good night! ;) | 16:30 |
| opendevreview | Tom Weininger proposed openstack/octavia-dashboard stable/xena: Display Draining state correctly https://review.opendev.org/c/openstack/octavia-dashboard/+/830659 | 16:39 |
| opendevreview | Tom Weininger proposed openstack/octavia-dashboard stable/wallaby: Display Draining state correctly https://review.opendev.org/c/openstack/octavia-dashboard/+/830660 | 16:40 |
| opendevreview | Tom Weininger proposed openstack/octavia-dashboard stable/victoria: Display Draining state correctly https://review.opendev.org/c/openstack/octavia-dashboard/+/830661 | 16:40 |
| opendevreview | Tom Weininger proposed openstack/octavia-dashboard stable/ussuri: Display Draining state correctly https://review.opendev.org/c/openstack/octavia-dashboard/+/830662 | 16:40 |
| opendevreview | Tom Weininger proposed openstack/octavia-dashboard stable/train: Display Draining state correctly https://review.opendev.org/c/openstack/octavia-dashboard/+/830663 | 16:41 |
| opendevreview | Michael Johnson proposed openstack/octavia master: Increase CentOS 8 Stream FIPS periodic timeout https://review.opendev.org/c/openstack/octavia/+/830680 | 19:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!