| *** terdei has quit IRC | 00:02 | |
| *** terdei has joined #openstack-lbaas | 00:03 | |
| *** TrevorV has joined #openstack-lbaas | 00:33 | |
| *** devfaz has quit IRC | 00:35 | |
| *** devfaz has joined #openstack-lbaas | 00:35 | |
| *** rcernin_ has joined #openstack-lbaas | 00:45 | |
| *** rcernin has quit IRC | 00:47 | |
| *** njohnston has quit IRC | 01:21 | |
| openstackgerrit | Merged openstack/octavia master: trivial fix https://review.opendev.org/732375 | 02:06 |
|---|---|---|
| *** ramishra has quit IRC | 02:46 | |
| *** ramishra has joined #openstack-lbaas | 02:46 | |
| openstackgerrit | Merged openstack/octavia master: Add some details on enable_anti_affinity option https://review.opendev.org/742670 | 03:12 |
| *** rcernin_ has quit IRC | 03:19 | |
| *** rcernin_ has joined #openstack-lbaas | 03:34 | |
| *** psachin has joined #openstack-lbaas | 03:35 | |
| *** TrevorV has quit IRC | 03:43 | |
| *** vishalmanchanda has joined #openstack-lbaas | 04:55 | |
| *** armax has quit IRC | 05:24 | |
| *** gcheresh has joined #openstack-lbaas | 05:26 | |
| *** kevinz has joined #openstack-lbaas | 05:34 | |
| openstackgerrit | Vishal Manchanda proposed openstack/octavia-dashboard master: [goal] Migrate testing to ubuntu focal https://review.opendev.org/744710 | 05:52 |
| *** JustAFlerkin has quit IRC | 06:43 | |
| *** rcernin_ has quit IRC | 06:51 | |
| *** rcernin_ has joined #openstack-lbaas | 06:54 | |
| *** ataraday has joined #openstack-lbaas | 07:25 | |
| *** ataraday has quit IRC | 07:27 | |
| *** rcernin_ has quit IRC | 07:35 | |
| *** ccamposr__ has joined #openstack-lbaas | 07:38 | |
| *** ccamposr has quit IRC | 07:40 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Add experimental amphorav2 jobs https://review.opendev.org/737993 | 07:42 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Add option to set default ssl ciphers in haproxy https://review.opendev.org/685337 | 07:58 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Add retry for getting amphora VM https://review.opendev.org/726084 | 07:58 |
| *** ataraday has joined #openstack-lbaas | 08:06 | |
| *** rcernin_ has joined #openstack-lbaas | 08:39 | |
| *** rcernin_ has quit IRC | 08:46 | |
| openstackgerrit | Merged openstack/python-octaviaclient stable/ussuri: Cap pylint for stable branches https://review.opendev.org/748024 | 08:48 |
| openstackgerrit | Merged openstack/octavia master: Switch to live from noop drivers https://review.opendev.org/748163 | 09:16 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Fix amphorav2 bytes error https://review.opendev.org/738407 | 09:22 |
| *** zzzeek has quit IRC | 09:32 | |
| *** zzzeek has joined #openstack-lbaas | 09:33 | |
| ataraday | gthiemonge, Hi, could you review https://review.opendev.org/#/c/738292/ ? | 09:40 |
| gthiemonge | ataraday: Hi, sure | 09:41 |
| ataraday | gthiemonge, Thanks! | 09:42 |
| *** zzzeek has quit IRC | 09:48 | |
| *** zzzeek has joined #openstack-lbaas | 09:49 | |
| openstackgerrit | Merged openstack/octavia stable/stein: Remove scenario bionic job from check https://review.opendev.org/738797 | 10:06 |
| openstackgerrit | Merged openstack/octavia master: Add a message for the amphora image size https://review.opendev.org/746771 | 10:06 |
| *** zzzeek has quit IRC | 10:08 | |
| *** zzzeek has joined #openstack-lbaas | 10:10 | |
| *** sapd1_x has joined #openstack-lbaas | 10:21 | |
| *** rcernin_ has joined #openstack-lbaas | 10:42 | |
| *** rcernin_ has quit IRC | 10:47 | |
| *** armax has joined #openstack-lbaas | 10:55 | |
| *** zzzeek has quit IRC | 10:57 | |
| *** zzzeek has joined #openstack-lbaas | 11:00 | |
| *** gcheresh has quit IRC | 11:00 | |
| gthiemonge | ataraday: not really familiar with taskflow but how are arguments passed to execute() functions? what defines the order of the args? (cf https://review.opendev.org/#/c/738292/13/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py) | 11:25 |
| ataraday | gthiemonge, They described in flows https://review.opendev.org/#/c/738292/13/octavia/controller/worker/v2/flows/amphora_flows.py@707 and are taken from store that is passed in worker https://review.opendev.org/#/c/738292/13/octavia/controller/worker/v2/controller_worker.py@357 | 11:29 |
| gthiemonge | thanks | 11:33 |
| gthiemonge | ataraday: and for the order? why did johnsom swap 'amphorae' and 'amphora_index' in v1/tasks/amphora_driver_tasks.py? | 11:34 |
| ataraday | I think it takes parameters by mapping names, so order here more for consistency | 11:35 |
| *** gcheresh has joined #openstack-lbaas | 11:54 | |
| gthiemonge | ataraday: Ok thanks! | 11:56 |
| *** njohnston has joined #openstack-lbaas | 11:59 | |
| *** TMM has quit IRC | 12:57 | |
| *** TMM has joined #openstack-lbaas | 12:57 | |
| johnsom | Yeah, consistency reasons. In other places I had that order | 13:18 |
| *** TrevorV has joined #openstack-lbaas | 13:46 | |
| *** rcernin_ has joined #openstack-lbaas | 14:44 | |
| *** rcernin_ has quit IRC | 14:49 | |
| *** ataraday_ has joined #openstack-lbaas | 15:52 | |
| johnsom | #startmeeting Octavia | 16:00 |
| openstack | Meeting started Wed Sep 2 16:00:02 2020 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 16:00 | |
| openstack | The meeting name has been set to 'octavia' | 16:00 |
| johnsom | Hi everyone (that isn't on vacation this week)! | 16:00 |
| gthiemonge | o/ | 16:00 |
| aannuusshhkkaa | hello! | 16:00 |
| ataraday_ | hi | 16:00 |
| johnsom | If you are on vacation, well, why are you reading this? grin | 16:00 |
| johnsom | #topic Announcements | 16:00 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 16:00 | |
| johnsom | Final release of python-octaviaclient is next week! | 16:01 |
| johnsom | #link https://releases.openstack.org/victoria/schedule.html | 16:01 |
| johnsom | Also, feature freeze for all of the other repositories is next week as well. | 16:01 |
| johnsom | We had a gate outage this week with the release of a new setuptools. A change was added to devstack to resolve that issue | 16:02 |
| johnsom | Any other announcements this week? | 16:03 |
| johnsom | We have been doing pretty good burning down the priority review list. Thank you! | 16:03 |
| johnsom | #topic Brief progress reports / bugs needing review | 16:04 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 16:04 | |
| johnsom | I have mostly been focusing on the priority review list, doing reviews, rebases, ordering patches with dependencies, etc. | 16:04 |
| aannuusshhkkaa | awaiting reviews on a couple of our patches.. https://review.opendev.org/#/c/746822/4 https://review.opendev.org/#/c/737111/27 https://review.opendev.org/#/c/742294/11 | 16:05 |
| ataraday_ | Do some reviews and update amphorav2 changes | 16:05 |
| johnsom | ataraday_ Thank you for your help there. | 16:05 |
| johnsom | aannuusshhkkaa Those are on the top of my list. | 16:06 |
| aannuusshhkkaa | yay! looking forward to it.. | 16:06 |
| gthiemonge | I did some reviews, and I'm working on SCTP support... and downstream stuff | 16:06 |
| ataraday_ | Will we make amphorav2 changes into Victory? They seems are down in review priority list? | 16:07 |
| johnsom | ataraday_ Yeah, I think we can do that. If they aren't feature patches, we still have a bit of time to get through those. I am also fine if we want to move those up into the other lists. | 16:08 |
| johnsom | The requirements patch is probably a "feature" level patch of priority | 16:08 |
| johnsom | The upgrade check probably as well. | 16:08 |
| johnsom | I will move those up after the meeting. I don't think there is a reason to have them separate | 16:09 |
| johnsom | #topic Priority bug reviews for Victoria | 16:09 |
| *** openstack changes topic to "Priority bug reviews for Victoria (Meeting topic: Octavia)" | 16:09 | |
| ataraday_ | johnsom, OK, thanks! | 16:09 |
| johnsom | #link https://etherpad.opendev.org/p/octavia-priority-reviews | 16:09 |
| johnsom | We are getting into the critical time to get features in. There is currently 29 feature patches on the list. | 16:10 |
| johnsom | I think we should start talking about patches that may not make it. | 16:10 |
| johnsom | If you see any patches on that list that you think are not ready or don't need to make Victoria, please add a comment to the patch line. I will also make some proposals of patches to defer later today. I will make a break in the list. | 16:12 |
| johnsom | If I put something on that list that is important to you, please feel free to move them back up. | 16:12 |
| gthiemonge | ack | 16:12 |
| johnsom | We just need to start making a cut line and prioritize | 16:12 |
| johnsom | Any other questions/comment/ideas on the priority list? | 16:13 |
| gthiemonge | nop | 16:14 |
| johnsom | Ok. | 16:14 |
| johnsom | #topic Open Discussion | 16:14 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 16:14 | |
| johnsom | Other topics this week? | 16:14 |
| johnsom | Ok, short and sweet. Thanks everyone for your reviews, we are making good progress! | 16:16 |
| johnsom | #endmeeting | 16:16 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 16:16 | |
| openstack | Meeting ended Wed Sep 2 16:16:32 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:16 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-09-02-16.00.html | 16:16 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-09-02-16.00.txt | 16:16 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-09-02-16.00.log.html | 16:16 |
| gthiemonge | thanks johnsom | 16:16 |
| ataraday_ | thanks! | 16:17 |
| *** ataraday_ has quit IRC | 16:23 | |
| openstackgerrit | Vishal Manchanda proposed openstack/octavia-dashboard master: [goal] Migrate testing to ubuntu focal https://review.opendev.org/744710 | 16:36 |
| *** psachin has quit IRC | 17:09 | |
| *** ccamposr__ has quit IRC | 17:24 | |
| *** rouk has joined #openstack-lbaas | 18:10 | |
| rouk | @johnsom we (mloza too) are having issues with health manager apparently reviving deleted amphora rows | 18:12 |
| rouk | rows with ips that overlap existing amphora management ips come back from the dead eventually, is there some better place we could purge these? | 18:13 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add proxy v2 protocol support https://review.opendev.org/747801 | 18:13 |
| johnsom | rouk Housekeeping process should purge those | 18:14 |
| rouk | it does not. | 18:14 |
| rouk | should we just be dumping the rows, instead of setting deleted? | 18:14 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/controller/housekeeping/house_keeping.py#L118 | 18:14 |
| rouk | yeah, but these arent old, really. | 18:15 |
| rouk | theres plenty of times where an amphora will come up on the same ip as an old one, and then the old record gets revived by health checks erroring. | 18:15 |
| johnsom | Oh, I see, neutron is recycling IPs very quickly? | 18:15 |
| rouk | hundreds of LBs, it happens. | 18:15 |
| rouk | neutron doesnt have a LRU allocation or something to try and spread them out, so reuse happens very quickly when people are recycling things | 18:16 |
| johnsom | Yeah, hmm. My initial thought would be to tune the purge interval to be very short and have it check often. | 18:16 |
| johnsom | Well, it does assign then in sequence, but I could see a large deployment with high rate of change being an issue. | 18:17 |
| johnsom | Just a sec, let me recommend some config tuning | 18:17 |
| rouk | can we configure that easily? this particular burst we have is due to the previous issue of mloza setting deleted on all amphoras at once, and doing 100 LB rebuilds in an hour. | 18:17 |
| rouk | but, he originally was doing the delete cause this was happening in somewhat isolated cases before | 18:17 |
| rouk | due to random ip reuse, it was just like, 4 instead of 100 | 18:18 |
| johnsom | In the [house_keeping] section of the octavia.conf, set the amphora_expiry_age to some small amount of seconds, like 600 for 10 minutes, or even lower if you don't care about historical records | 18:18 |
| rouk | and for the already revived records on re-used ips that keep coming back? do i just drop the rows manually? | 18:19 |
| rouk | or if i set them deleted with a short interval, they should get cleaned before revived? | 18:19 |
| johnsom | It should then check and delete them every 30 seconds based on the cleanup_interval default setting | 18:20 |
| rouk | dont really care about historical amphora records. they do stay deleted for >30s if we set them deleted, so they should be down long enough for the cleanup to hit | 18:20 |
| rouk | we will try that | 18:21 |
| johnsom | Yeah, setting that low should purge them pretty fast, if 30 seconds is too long, drop the cleanup_interval too | 18:21 |
| rouk | it should be fine, health manager usually revives them within the next day or so | 18:21 |
| johnsom | Also note, if you have more than one housekeeping running, they will all take a turn at cleaning up | 18:21 |
| rouk | amphora_expiry_age 30s should basically be clean up everything every run, right? | 18:22 |
| johnsom | That is an interesting scenario however. If you want you could open a story for us to think about how to work better with re-issued IPs. | 18:22 |
| rouk | oh, you should never move from deleted -> error | 18:23 |
| johnsom | Yeah, or 29 for sure | 18:23 |
| rouk | imo | 18:23 |
| rouk | thats the fix | 18:23 |
| rouk | what cases do we need to move from deleted -> error? a deletion error would go pending_delete -> error | 18:23 |
| johnsom | Yeah, I agree actually. | 18:23 |
| rouk | also, for mass rebuilds where this stuff happens quickly, a amphora delete command would be cool | 18:23 |
| johnsom | Once it's DELETED, it should be in that state until the record is completely purged | 18:24 |
| rouk | maybe with an admin flag | 18:24 |
| *** ccamposr has joined #openstack-lbaas | 18:24 | |
| rouk | since the reason why we were doing db edits, is cause theres no amphora delete command. | 18:24 |
| rouk | but theres some ugly cases where it would help | 18:24 |
| johnsom | Ahead of you on that one: https://review.opendev.org/718293 | 18:24 |
| rouk | oi, nice. | 18:24 |
| johnsom | And https://review.opendev.org/718311 | 18:24 |
| rouk | perdy | 18:25 |
| johnsom | Those should make Victoria | 18:25 |
| johnsom | Just need to get the reviews | 18:25 |
| rouk | so yeah, i think banning deleted -> error on health check would be good, alternatively, dont check health on deleted records | 18:26 |
| rouk | idk if health manager have it cached, or if they check db for what the look for every time? | 18:26 |
| johnsom | It checks the DB | 18:26 |
| rouk | then yeah, health checks on deleted are frivilous | 18:26 |
| johnsom | Yeah, so if you could capture your situation and thoughts in a story we can work on fixing that. | 18:27 |
| johnsom | We don't cache at the controllers as we support multiple processes. We let the DB cache the content and coordinate via the DB locking. | 18:28 |
| rouk | "Bad bot, go away! Request aborted." i hate ubuntu one so much. | 18:29 |
| rouk | needs to go away | 18:29 |
| johnsom | lol, that was part of the reasoning to moving us to storyboard, but I guess that work has stalled | 18:29 |
| rouk | yeah, just trying to log in | 18:29 |
| johnsom | If you can't get logged in, send me an e-mail with the details an I will open it for you. | 18:30 |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/train: Fix API sort key for complex columns https://review.opendev.org/749410 | 18:38 |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/stein: Fix API sort key for complex columns https://review.opendev.org/749411 | 18:40 |
| *** vishalmanchanda has quit IRC | 18:43 | |
| *** zzzeek has quit IRC | 19:17 | |
| *** zzzeek has joined #openstack-lbaas | 19:18 | |
| *** servagem has quit IRC | 19:21 | |
| *** servagem has joined #openstack-lbaas | 19:22 | |
| *** zzzeek has quit IRC | 19:25 | |
| *** zzzeek has joined #openstack-lbaas | 19:27 | |
| *** zzzeek has quit IRC | 19:53 | |
| *** zzzeek has joined #openstack-lbaas | 19:56 | |
| *** zzzeek has quit IRC | 20:50 | |
| *** zzzeek has joined #openstack-lbaas | 20:51 | |
| *** sapd1_x has quit IRC | 21:04 | |
| *** rcernin_ has joined #openstack-lbaas | 21:06 | |
| *** rcernin_ has quit IRC | 21:12 | |
| *** sapd1_x has joined #openstack-lbaas | 21:17 | |
| *** rcernin_ has joined #openstack-lbaas | 22:15 | |
| *** TrevorV has quit IRC | 22:18 | |
| *** rcernin_ has quit IRC | 23:04 | |
| *** rcernin has joined #openstack-lbaas | 23:04 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Adjust scenario tests for NotImplemented skip https://review.opendev.org/714004 | 23:24 |
| *** irclogbot_3 has quit IRC | 23:29 | |
| *** irclogbot_0 has joined #openstack-lbaas | 23:33 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!