| *** yamamoto has quit IRC | 00:03 | |
| openstackgerrit | Anushka Singh proposed openstack/octavia master: Change amphora statistics to use deltas https://review.opendev.org/740815 | 00:21 |
|---|---|---|
| openstackgerrit | Anushka Singh proposed openstack/octavia master: Change amphora statistics to use deltas https://review.opendev.org/740815 | 00:56 |
| *** armax has joined #openstack-lbaas | 00:57 | |
| *** yamamoto has joined #openstack-lbaas | 01:26 | |
| *** yamamoto has quit IRC | 02:00 | |
| *** spatel has joined #openstack-lbaas | 02:10 | |
| *** sapd__x has joined #openstack-lbaas | 02:20 | |
| *** spatel has quit IRC | 02:49 | |
| *** rcernin has quit IRC | 02:56 | |
| *** rcernin has joined #openstack-lbaas | 02:57 | |
| *** ramishra has quit IRC | 03:13 | |
| *** ramishra has joined #openstack-lbaas | 03:14 | |
| *** spatel has joined #openstack-lbaas | 03:22 | |
| *** spatel has quit IRC | 03:22 | |
| *** spatel has joined #openstack-lbaas | 03:23 | |
| *** spatel has quit IRC | 03:23 | |
| *** spatel has joined #openstack-lbaas | 03:30 | |
| *** spatel has quit IRC | 03:30 | |
| *** armax has quit IRC | 03:38 | |
| *** psachin has joined #openstack-lbaas | 03:39 | |
| *** yamamoto has joined #openstack-lbaas | 03:59 | |
| *** yamamoto has quit IRC | 03:59 | |
| *** yamamoto has joined #openstack-lbaas | 04:00 | |
| *** sapd__x has quit IRC | 04:12 | |
| *** vishalmanchanda has joined #openstack-lbaas | 04:50 | |
| *** sapd__x has joined #openstack-lbaas | 04:58 | |
| *** gcheresh has joined #openstack-lbaas | 05:55 | |
| *** osmanlicilegi has quit IRC | 06:02 | |
| *** osmanlicilegi has joined #openstack-lbaas | 06:06 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Migrate grenade job to native Zuul v3 https://review.opendev.org/726741 | 06:18 |
| *** ltomasbo has joined #openstack-lbaas | 06:23 | |
| *** gcheresh has quit IRC | 06:42 | |
| *** takamatsu has quit IRC | 06:59 | |
| *** rcernin has quit IRC | 07:07 | |
| *** born2bake has joined #openstack-lbaas | 07:13 | |
| *** happyhemant has joined #openstack-lbaas | 07:17 | |
| *** rcernin has joined #openstack-lbaas | 07:17 | |
| *** gcheresh has joined #openstack-lbaas | 07:32 | |
| *** also_stingrayza is now known as stingrayza | 07:54 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 07:56 | |
| *** dmsimard has quit IRC | 08:25 | |
| *** dmsimard has joined #openstack-lbaas | 08:26 | |
| *** rcernin has quit IRC | 08:27 | |
| *** rcernin has joined #openstack-lbaas | 08:48 | |
| *** yamamoto has quit IRC | 08:55 | |
| *** yamamoto has joined #openstack-lbaas | 09:00 | |
| *** yamamoto has quit IRC | 09:02 | |
| *** yamamoto has joined #openstack-lbaas | 09:08 | |
| *** rcernin has quit IRC | 09:17 | |
| openstackgerrit | Merged openstack/octavia-tempest-plugin master: Define and use octavia nodesets https://review.opendev.org/738246 | 09:18 |
| *** ataraday_ has joined #openstack-lbaas | 09:39 | |
| *** happyhemant has quit IRC | 10:07 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Add non-voting octavia-v2-dsvm-scenario-centos-8 https://review.opendev.org/741147 | 10:31 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add experimental CentOS 8 Stream amphora job https://review.opendev.org/698450 | 11:00 |
| *** yamamoto has quit IRC | 11:01 | |
| *** wuchunyang has joined #openstack-lbaas | 11:22 | |
| *** sapd__x has quit IRC | 11:22 | |
| *** rcernin has joined #openstack-lbaas | 11:31 | |
| *** yamamoto has joined #openstack-lbaas | 11:40 | |
| *** wuchunyang has quit IRC | 11:44 | |
| *** rcernin has quit IRC | 11:47 | |
| openstackgerrit | Jay Liu proposed openstack/octavia master: Alternative Distributor for L3 Active-Active, N+1 Amphora Setup https://review.opendev.org/723864 | 12:03 |
| *** yamamoto has quit IRC | 12:06 | |
| *** yamamoto has joined #openstack-lbaas | 12:23 | |
| *** tkajinam has quit IRC | 12:24 | |
| *** yamamoto has quit IRC | 12:42 | |
| *** ataraday_ has quit IRC | 12:55 | |
| *** wuchunyang has joined #openstack-lbaas | 13:02 | |
| *** wuchunyang has quit IRC | 13:06 | |
| *** yamamoto has joined #openstack-lbaas | 13:24 | |
| *** wuchunyang has joined #openstack-lbaas | 13:41 | |
| *** yamamoto has quit IRC | 13:47 | |
| rm_work | cgoncalves: you remember a patch for pool <--> listener type matrix? | 13:52 |
| rm_work | ah it did land, found it | 13:55 |
| *** ramishra has quit IRC | 13:59 | |
| dmsimard | What's the right approach when lb-mgmt-net is full and we need to add a subnet ? Is it problematic due to keepalived/vrrp ? | 13:59 |
| dmsimard | I've found https://bugs.launchpad.net/octavia/+bug/1693795 but it hasn't moved in a while | 14:00 |
| openstack | Launchpad bug 1693795 in octavia "lb-mgmt-net do not support pool of subnets" [Wishlist,Triaged] - Assigned to Santhosh Fernandes (santhosh-fernandes) | 14:00 |
| cgoncalves | dmsimard, octavia uses storyboard.openstack.org | 14:01 |
| johnsom | Octavia doesn't use launchpad | 14:01 |
| dmsimard | I go where google leads me :D | 14:01 |
| * dmsimard searches storyboard | 14:02 | |
| johnsom | Bad google | 14:02 |
| johnsom | Also keepalived/vrrp doesn't run over the lb-mgmt-net. It's only over the VIP network | 14:03 |
| johnsom | I don't know that we have had the case where the lb-mgmt-net was full. It's a private network, so can be large | 14:04 |
| johnsom | As long as they are routable, you should be able to add networks however. | 14:05 |
| dmsimard | yeah, it looks like the network team routed just a /24 :/ | 14:07 |
| dmsimard | I'll look into it, thanks for your help | 14:07 |
| *** also_stingrayza has joined #openstack-lbaas | 14:17 | |
| *** ramishra has joined #openstack-lbaas | 14:19 | |
| *** mchlumsky has joined #openstack-lbaas | 14:20 | |
| *** stingrayza has quit IRC | 14:21 | |
| *** ramishra has quit IRC | 14:28 | |
| *** ramishra has joined #openstack-lbaas | 14:48 | |
| *** ZhuXiaoYu has joined #openstack-lbaas | 15:01 | |
| *** TrevorV has joined #openstack-lbaas | 15:14 | |
| *** ramishra has quit IRC | 15:46 | |
| *** yamamoto has joined #openstack-lbaas | 15:46 | |
| *** ataraday_ has joined #openstack-lbaas | 15:52 | |
| johnsom | #startmeeting Octavia | 16:00 |
| openstack | Meeting started Wed Jul 15 16:00:12 2020 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 16:00 | |
| openstack | The meeting name has been set to 'octavia' | 16:00 |
| cgoncalves | hi | 16:00 |
| johnsom | Hi everyone | 16:00 |
| ataraday_ | hi | 16:00 |
| gthiemonge | hi | 16:00 |
| aannuusshhkkaa | hi | 16:00 |
| *** shtepanie has joined #openstack-lbaas | 16:00 | |
| ZhuXiaoYu | hi | 16:00 |
| rm_work | o/ | 16:01 |
| johnsom | #topic Announcements | 16:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 16:01 | |
| johnsom | OpenStack 10 years celebration July 16th 1500 UTC | 16:01 |
| johnsom | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015839.html | 16:01 |
| johnsom | I wanted to make sure everyone knew the foundation is having a virtual celebration of ten years of OpenStack | 16:02 |
| johnsom | That is the only announcement I have this week. Any others? | 16:02 |
| johnsom | #topic Brief progress reports / bugs needing review | 16:02 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 16:03 | |
| johnsom | I have been deep into porting the failover flow updates to the amphora v2 driver. It's slow going, but I think I have the tasks all ported now. I have started on the amphora failover flow. | 16:03 |
| ataraday_ | I would like to point out that there are 8 changes for review about amphorav2 required to be ready/merged listed in https://etherpad.openstack.org/p/octavia-priority-reviews | 16:03 |
| ataraday_ | And I will have a vacation next week | 16:04 |
| ataraday_ | will we make till end of July? | 16:04 |
| johnsom | Nice, I will take a look at those. | 16:05 |
| *** armax has joined #openstack-lbaas | 16:05 | |
| johnsom | I assume you are talking about making Amphora v2 default decision for MS2 milestone? | 16:05 |
| cgoncalves | I've completed the failover flows backports to Ussuri and Train, now passing Python 2 CI jobs too. work on the jobboard on/off continued (thank you for the reviews thus far, ataraday_!) | 16:05 |
| johnsom | I have added a topic later on the agenda to talk about that. I think we have some problems in the Amphora v2 driver we need to work on. | 16:06 |
| ataraday_ | johnsom, yeah, ok lets discuss later | 16:06 |
| cgoncalves | I also posted small but important patches, and would appreciate reviews on those. I added them to the review priority list as well as cleaned it up by removing already merged patches | 16:07 |
| cgoncalves | #link https://etherpad.opendev.org/p/octavia-priority-reviews | 16:07 |
| johnsom | cgoncalves Thanks for working on the priority list! | 16:08 |
| aannuusshhkkaa | We are working on changing amphora stats to report deltas instead of absolutes. We have written code for the amphora, still have to write code for the other side. | 16:08 |
| johnsom | Nice | 16:09 |
| johnsom | #topic Queens and Rocky End Of Life | 16:09 |
| *** openstack changes topic to "Queens and Rocky End Of Life (Meeting topic: Octavia)" | 16:09 | |
| johnsom | Someone added this topic to the agenda. Please run with it | 16:09 |
| cgoncalves | o/ | 16:10 |
| cgoncalves | Adam proposed some time ago a patch to EOL Queens. | 16:11 |
| cgoncalves | Adam Workflow-1'd because, per my understanding, we could still want to make a final release. I think we are past that time since it has entered extended maintenance. | 16:11 |
| cgoncalves | if that is accurate, would we be good to officially EOL queens? | 16:12 |
| johnsom | We can still merge the patches on Queens if we want even though it is in extended maintenance. It just will not have anymore releases | 16:12 |
| cgoncalves | there are a few open stable/queens patches. however, queens and rocky CI pipelines are broken due to devstack/wsgi/... issues | 16:13 |
| johnsom | I am good with moving forward with the EOL. I don't think we have any reasons to not move forward. | 16:13 |
| johnsom | Ugh, yeah, those are ... troublesome | 16:13 |
| cgoncalves | there was a patch in devstack project to fix issues but I don't think that has made much progress since | 16:14 |
| cgoncalves | so, Rocky is mostly also in the same boat: broken CI, in extended maintenance mode (no point releases, no CI commitments). should we EOL Rocky too? | 16:15 |
| johnsom | I think we need to finish the EOL of Queens first, but I am also very open to EOL Rocky. | 16:16 |
| rm_work | link it and i will un-minus-workflow | 16:17 |
| johnsom | #link https://review.opendev.org/#/c/719099/ | 16:17 |
| cgoncalves | 2 sec. finding your patch | 16:17 |
| cgoncalves | #link https://review.opendev.org/#/c/719099/ | 16:17 |
| rm_work | So, no more release, go ahead with queens EOL? | 16:18 |
| rm_work | sounds good to me | 16:18 |
| *** maciejjozefczyk has quit IRC | 16:18 | |
| johnsom | Should we just abandon the open patches? | 16:18 |
| johnsom | I guess if devstack and the gates are broken they aren't going to be able to merge any time soon | 16:19 |
| cgoncalves | +1 to abandon | 16:19 |
| johnsom | Ok, maybe we wait for the EOL tag to land, then abandon these. | 16:19 |
| johnsom | Does someone want to propose EOL of Rocky? Are we ready for that? | 16:21 |
| johnsom | I think most of the distributions are using Queens as LTS and not Rocky, so maybe that is ok | 16:22 |
| cgoncalves | I can propose if you would like. I think coming from the PTL or stable liaison would be best | 16:22 |
| johnsom | Yeah, I agree. I expect Rocky will have some commentary... grin | 16:23 |
| *** TrevorV has quit IRC | 16:23 | |
| *** yamamoto has quit IRC | 16:24 | |
| *** ramishra has joined #openstack-lbaas | 16:24 | |
| cgoncalves | the fact devstack Rocky is broken for most if not everyone for weeks is already half-way to have it accepted | 16:25 |
| johnsom | Not a lot of comments here. I guess go ahead and propose it and see if anyone comments on the proposal. | 16:25 |
| cgoncalves | OK. I'll do it asap | 16:25 |
| johnsom | Yeah, I had the same comment about Queens. Both of the recent CVEs in other projects didn't bother to backport to queens. | 16:25 |
| johnsom | Ok, thank you cgoncalves | 16:26 |
| johnsom | #topic Active/Active specification | 16:26 |
| *** openstack changes topic to "Active/Active specification (Meeting topic: Octavia)" | 16:26 | |
| johnsom | #link https://review.opendev.org/#/c/723864 | 16:26 |
| johnsom | I want to highlight we have an open specification that has had a recent update. | 16:27 |
| johnsom | Also ZhuXiaoYu has joined the meeting today if you have any questions/comments on the specification | 16:27 |
| johnsom | I have not yet had a chance to review the update. How are the proposals for changes to neutron going? | 16:28 |
| ZhuXiaoYu | yes, please speak to me if you have any advise with this | 16:28 |
| *** TrevorV has joined #openstack-lbaas | 16:28 | |
| ZhuXiaoYu | I remove the same subnet limit | 16:29 |
| ZhuXiaoYu | and continue using existing update router api | 16:30 |
| ZhuXiaoYu | and about the FIP, sorry but it is necessary | 16:30 |
| *** wuchunyang has quit IRC | 16:30 | |
| johnsom | #link https://review.opendev.org/#/c/729532 | 16:31 |
| johnsom | That is the ECMP in neutron patch | 16:31 |
| johnsom | Hmm, yeah, if FIPs are necessary, that might be an issue. We would at a minimum need to document somehow. | 16:31 |
| ZhuXiaoYu | yes, our neutron team is planning to use HMARK in iptables instead of normal routes | 16:32 |
| ZhuXiaoYu | this could support not to redistribute flows when remove a amphora | 16:33 |
| johnsom | I have put down a task of re-reviewing the comments/specification. I hope others in the Octavia community can also comment/review. | 16:34 |
| johnsom | ZhuXiaoYu Are there any questions you have for the Octavia team we can answer now? Or just the items in the comments of the specification? | 16:35 |
| ZhuXiaoYu | In fact, we're more concerned about how to get it passed. | 16:37 |
| johnsom | Ok, so what you need is reviews. Thanks! | 16:38 |
| johnsom | #topic State of Amphora v2 driver | 16:38 |
| *** openstack changes topic to "State of Amphora v2 driver (Meeting topic: Octavia)" | 16:38 | |
| ataraday_ | I would like to mention that first 3 changes from the list - are actually important fixes for amphorav2 issues and should be backported. | 16:38 |
| johnsom | So this is my topic. As I have been working on the failover port, I have found some issues I think we need to work on. | 16:39 |
| johnsom | I have created an etherpad to track: | 16:39 |
| johnsom | #link https://etherpad.opendev.org/p/octavia-amphora-v2 | 16:39 |
| johnsom | Most are around the fact that when a flow fails and is re-dispatched, it's not bounded in any way. | 16:40 |
| johnsom | So, for example, I have a code issue (expected during the work I'm doing) that causes a port data model to not serialize correctly. | 16:40 |
| johnsom | This caused the controller to go into an endless loop trying to re-dispatch the flow to another worker. | 16:41 |
| johnsom | The problem is that the revert isn't triggering, so each attempt creates a new port in neutron | 16:41 |
| johnsom | In a matter of seconds I have 255 bogus ports and no more IP addresses left of the subnet. | 16:42 |
| johnsom | So, this is a big concern for me with the v2 driver. I think we need to look into how to make sure it doesn't run away and consume all of the cloud resources. | 16:43 |
| *** gcheresh has quit IRC | 16:44 | |
| ataraday_ | Could you put steps of repro for this on etherpad? | 16:44 |
| ataraday_ | or a story | 16:45 |
| johnsom | Yes, I can. Granted they will be "artificial" steps, but I think some cloud outage scenarios will also trigger this behavior. | 16:45 |
| johnsom | I will do this after the meeting. | 16:46 |
| ataraday_ | johnsom, thanks! | 16:46 |
| johnsom | ataraday_ Thank you for the comment about the progress update notification. I also agree I need to look at that. | 16:47 |
| johnsom | Let's work on these issues. I hope we can have some initial comments/ideas in a week or so. That way we can make an good decision on the "v2 default" at MS2 | 16:48 |
| johnsom | If you find any other concerns, please feel free to update the etherpad. | 16:48 |
| johnsom | Any other discussion on the V2 amphora driver? | 16:49 |
| johnsom | Ok. More work to do, but I don't think it's a show stopper yet. | 16:51 |
| johnsom | #topic Open Discussion | 16:51 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 16:51 | |
| johnsom | Any other topics this week? | 16:51 |
| cgoncalves | just a FYI that all our scenario CI jobs run on nested-virt nodepool instances with KVM rather than TCG | 16:52 |
| johnsom | Nice! | 16:52 |
| johnsom | That should help a lot. | 16:52 |
| johnsom | I just hope we don't find many broken hypervisors. | 16:53 |
| cgoncalves | this change improves CI job build times significantly. please keep an eye open for random failures, it could be issues with nested virtualization and do let us know | 16:53 |
| johnsom | Also a quick mention, there are a lot of open patches on the tempest plugin. It would be nice to get some reviews on those. | 16:54 |
| johnsom | #link https://review.opendev.org/#/q/project:openstack/octavia-tempest-plugin+status:open | 16:54 |
| cgoncalves | also, I've observed the CentOS 8 job taking as low as ~45 minutes to run so great improvement considering before it was taking 2h30 to TIMOUT | 16:54 |
| johnsom | Really? That is great news! Maybe we can re-add a centos job | 16:55 |
| cgoncalves | sure! here: | 16:55 |
| cgoncalves | #link https://review.opendev.org/#/c/741147/ | 16:55 |
| cgoncalves | job history: | 16:55 |
| cgoncalves | #link https://zuul.opendev.org/t/openstack/builds?job_name=octavia-v2-dsvm-scenario-centos-8 | 16:55 |
| johnsom | Nice, it even passes | 16:56 |
| cgoncalves | 41 mins 39 secs !! | 16:56 |
| rm_work | noice | 16:56 |
| cgoncalves | lol to "it even passes" | 16:56 |
| johnsom | Well, it's been a while since we had a working job. I was worried there would be some bit-rot | 16:56 |
| *** ltomasbo has quit IRC | 16:57 | |
| *** ataraday_ has quit IRC | 16:58 | |
| johnsom | Any other topics this week? | 16:58 |
| *** psachin has quit IRC | 16:58 | |
| johnsom | Ok, thank you everyone! Have a good week. | 16:59 |
| cgoncalves | thank you | 16:59 |
| johnsom | #endmeeting | 16:59 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 16:59 | |
| openstack | Meeting ended Wed Jul 15 16:59:28 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-07-15-16.00.html | 16:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-07-15-16.00.txt | 16:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-07-15-16.00.log.html | 16:59 |
| *** ZhuXiaoYu has quit IRC | 17:03 | |
| *** haleyb has quit IRC | 17:45 | |
| *** gcheresh has joined #openstack-lbaas | 17:55 | |
| *** mloza has quit IRC | 18:04 | |
| openstackgerrit | Anushka Singh proposed openstack/octavia master: Change amphora statistics to use deltas https://review.opendev.org/740815 | 18:11 |
| *** vishalmanchanda has quit IRC | 18:20 | |
| *** yamamoto has joined #openstack-lbaas | 18:21 | |
| *** gcheresh has quit IRC | 18:35 | |
| openstackgerrit | Stephanie Djajadi proposed openstack/octavia master: Change amphora statistics to use deltas https://review.opendev.org/740815 | 18:49 |
| *** yamamoto has quit IRC | 19:01 | |
| *** KeithMnemonic has joined #openstack-lbaas | 19:12 | |
| *** wuchunyang has joined #openstack-lbaas | 19:20 | |
| *** haleyb has joined #openstack-lbaas | 19:29 | |
| *** wuchunyang has quit IRC | 19:45 | |
| *** wuchunyang has joined #openstack-lbaas | 19:51 | |
| *** TMM has quit IRC | 19:58 | |
| *** TMM has joined #openstack-lbaas | 19:58 | |
| *** gcheresh has joined #openstack-lbaas | 20:08 | |
| *** wuchunyang has quit IRC | 20:37 | |
| *** wuchunyang has joined #openstack-lbaas | 20:44 | |
| *** gcheresh has quit IRC | 20:49 | |
| *** yamamoto has joined #openstack-lbaas | 21:00 | |
| *** TrevorV has quit IRC | 21:18 | |
| *** yamamoto has quit IRC | 21:35 | |
| *** yamamoto has joined #openstack-lbaas | 21:36 | |
| johnsom | Flow 'octavia-failover-amphora-flow' (0ffa8c69-840b-497e-9e84-bd5d69eef89c) transitioned into state 'SUCCESS' from state 'RUNNING' | 22:02 |
| johnsom | Some progress | 22:02 |
| *** logan- has quit IRC | 22:17 | |
| *** rcernin has joined #openstack-lbaas | 22:19 | |
| *** logan- has joined #openstack-lbaas | 22:19 | |
| *** rcernin has quit IRC | 22:20 | |
| *** rcernin has joined #openstack-lbaas | 22:21 | |
| *** spatel has joined #openstack-lbaas | 22:29 | |
| *** spatel has quit IRC | 22:34 | |
| *** armax has quit IRC | 22:34 | |
| *** armax has joined #openstack-lbaas | 22:36 | |
| *** tkajinam has joined #openstack-lbaas | 22:55 | |
| *** tkajinam has quit IRC | 22:55 | |
| *** tkajinam has joined #openstack-lbaas | 22:58 | |
| *** yamamoto has quit IRC | 22:59 | |
| *** born2bake has quit IRC | 23:28 | |
| *** yamamoto has joined #openstack-lbaas | 23:30 | |
| *** yamamoto has quit IRC | 23:32 | |
| *** yamamoto has joined #openstack-lbaas | 23:32 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!