Wednesday, 2020-09-09

openstackgerritMerged openstack/octavia master: Deprecate the Spares Pool feature for removal in X  https://review.opendev.org/74168600:44
rm_workjohnsom: seeing some weirdness possibly again in failover, looking into it, just a heads up in case you were also aware of any issues01:21
rm_workran into `octavia.network.base.PortNotFound: port not found` during a failover of an LB that had a user's bring-your-own-port01:22
rm_workstill trying to track down what caused the failover to begin with, it happened while i was out, and everything is gone now, so sifting through logs01:23
johnsomNo, I don’t know of any open issues with failover01:23
rm_workkk01:23
rm_workfailover failed, ended up in ERROR with 4 amp records <_<01:23
rm_workmanual failover fixed it somehow01:23
johnsomThat seems like the old code behavior01:24
johnsomIs like the worker updated and the hm not?01:25
rm_workdon't think that's possible given how many iterations it's been, but yeah i will look01:25
rm_workif we are still running OOOOLD HM code somehow, I will boggle01:26
rm_workhmm looks like cert refresh failed due to a timeout O_o01:47
rm_workand then... it tried to replace the amp maybe?01:49
rm_workor... some other failover was attempted actually much later01:49
rm_workand that's when the port not found error happened -- trying to figure out which port01:49
rm_work`vrrp_port = self.get_port(amp.vrrp_port_id)`01:51
rm_workso, the vrrp port didn't exist... weird, i thought that was a solved issue01:51
rm_work1) we might want to retry on a connection timeout on a cert rotation?01:57
openstackgerritMerged openstack/octavia master: Refactoring amphora stats driver interface  https://review.opendev.org/73711102:10
rm_work`1 retries with interval 5 seconds have failed for retry-octavia-delete-amphora-flow-07a4407d-35e7-49e5-a18f-183c965a9527. Giving up.` then it does a "revert-delete-port" but obviously that's bogus, you can't revert that :D02:11
rm_workthen the port doesn't exist for the failover02:11
rm_workwhich... i thought you made it create all new ports, but I remember there was one you had to reuse -- hmm02:12
johnsomIt should recreate everything, but if we don’t own the port I think I don’t mess with it02:13
rm_workyeah this is the vrrp port anyway02:14
rm_worktrying to figure out why a failover 30m later fails because it can't find the vrrp port that was deleted earlier02:15
johnsomThat so seems like old code. The base ports are all rebuilt now02:15
johnsomBecause nova is dumb and will not release them02:16
rm_workyeah.... doesn't look like old code involved though, which is why i'm confused.02:19
rm_workhmmm ok02:25
rm_workhttps://github.com/openstack/octavia/blob/master/octavia/controller/worker/v1/flows/amphora_flows.py#L650-L65102:29
rm_workhttps://github.com/openstack/octavia/blob/master/octavia/controller/worker/v1/flows/amphora_flows.py#L320-L32302:29
*** rcernin has quit IRC02:29
rm_workhttps://github.com/openstack/octavia/blob/master/octavia/network/drivers/neutron/allowed_address_pairs.py#L68802:30
rm_workjohnsom: ^^ and there is where no matter what happens, during a failover it will try to get the vrrp port, fail, and revert02:31
rm_workif the vrrp port is gone02:31
rm_workif i'm reading this right02:31
johnsomHmmm02:31
johnsomI will have to look tomorrow, having dinner now02:32
johnsomSeems like it should already have rebuilt that port by then, but that is off my head02:32
*** armax has quit IRC03:10
*** armax has joined #openstack-lbaas03:10
*** zzzeek has quit IRC03:18
*** zzzeek has joined #openstack-lbaas03:19
*** sapd1_x has joined #openstack-lbaas03:27
*** rcernin has joined #openstack-lbaas03:29
*** rcernin has quit IRC03:29
*** rcernin has joined #openstack-lbaas03:29
*** zzzeek has quit IRC03:55
*** zzzeek has joined #openstack-lbaas03:58
*** sapd1_x has quit IRC04:03
*** sapd1_x has joined #openstack-lbaas04:04
*** njohnston has quit IRC04:22
*** sapd1_x has quit IRC04:35
*** zzzeek has quit IRC04:45
*** zzzeek has joined #openstack-lbaas04:48
*** zzzeek has quit IRC05:17
*** zzzeek has joined #openstack-lbaas05:18
*** gcheresh_ has joined #openstack-lbaas05:26
*** sapd1 has quit IRC05:48
*** sapd1 has joined #openstack-lbaas05:48
*** vishalmanchanda has joined #openstack-lbaas06:14
*** ataraday_ has joined #openstack-lbaas06:34
openstackgerritAnn Taraday proposed openstack/octavia stable/ussuri: Add missing reload method in amphora noop driver  https://review.opendev.org/73906406:42
openstackgerritAnn Taraday proposed openstack/octavia stable/ussuri: Fix missing params in amphora base and noop driver  https://review.opendev.org/73906506:42
openstackgerritAnn Taraday proposed openstack/octavia stable/ussuri: Fix missing dependency in amphora create flow  https://review.opendev.org/74240106:43
*** ramishra has quit IRC06:51
*** ramishra has joined #openstack-lbaas07:24
*** sapd1_x has joined #openstack-lbaas07:52
*** rcernin has quit IRC08:20
gthiemongeataraday_: Hi, could you review https://review.opendev.org/#/c/714004/ please?08:21
*** ianychoi__ has quit IRC08:40
kklimondaHi, I've deployed octavia on ussuri with ovn ml2 driver with amphora driver - it seems to be working fine, however I constantly see the following in ovn-controller logs:09:08
kklimonda```pinctrl|INFO|Claiming virtual lport a5a3f456-622a-46fa-94e7-9805b153e640 for this chassis with the virtual parent b8f77b5e-6886-4506-ae55-7f76b374a07c```09:08
kklimondait seems to be repeating every 5 seconds for every vip port09:14
*** sapd1_x has quit IRC09:15
*** lxkong has joined #openstack-lbaas09:28
ataraday_gthiemonge, Hi! Done09:29
*** lxkong has quit IRC09:34
openstackgerritAnn Taraday proposed openstack/octavia stable/train: Fix memory consumption issues with default connection_limit  https://review.opendev.org/74766209:36
*** lxkong has joined #openstack-lbaas09:39
ataraday_gthiemonge, Question about https://review.opendev.org/#/c/738381/ - do we still need some changes for amphora backend?09:46
kklimondabtw, could octavia/ovn combo be used to implement active-active LB, where ovn takes a role of a "director" and balances traffic to available amphorae?09:55
gthiemongeataraday_: yes, amphora support will be included in a follow-up commit (in the next cycle), but some other providers might start implementing sctp now10:21
openstackgerritMerged openstack/octavia stable/ussuri: Add missing reload method in amphora noop driver  https://review.opendev.org/73906410:43
*** rcernin has joined #openstack-lbaas10:52
openstackgerritMerged openstack/octavia stable/ussuri: Fix missing params in amphora base and noop driver  https://review.opendev.org/73906511:07
openstackgerritMerged openstack/octavia stable/ussuri: Fix missing dependency in amphora create flow  https://review.opendev.org/74240111:07
*** sapd1_x has joined #openstack-lbaas11:17
ataraday_gthiemonge, OK, thanks!11:17
*** rcernin has quit IRC11:28
openstackgerritMerged openstack/octavia-tempest-plugin master: Adjust scenario tests for NotImplemented skip  https://review.opendev.org/71400411:39
*** rcernin has joined #openstack-lbaas11:45
*** sapd1_x has quit IRC11:56
openstackgerritAnn Taraday proposed openstack/octavia stable/ussuri: Update amphora v2 for the failover refactor  https://review.opendev.org/75063212:06
*** jamesdenton has quit IRC12:08
*** njohnston has joined #openstack-lbaas12:08
*** jamesdenton has joined #openstack-lbaas12:09
openstackgerritAnn Taraday proposed openstack/octavia master: Bump taskflow version  https://review.opendev.org/75063812:23
*** rcernin has quit IRC12:41
ataraday_I will miss today meeting, I want to raise for consideration this change https://review.opendev.org/#/c/750638/ - if we want to bump taskflow version in V.13:02
openstackgerritGregory Thiemonge proposed openstack/octavia stable/ussuri: Fix memory consumption issues with default connection_limit  https://review.opendev.org/74765113:05
*** ataraday_ has quit IRC13:07
*** rcernin has joined #openstack-lbaas13:14
*** TrevorV has joined #openstack-lbaas13:24
*** sapd1_x has joined #openstack-lbaas13:27
*** rcernin has quit IRC13:41
johnsomkklimonda OVN errors should be discussed in the #openstack-neutron channel. We can't help you much with errors in OVN.13:53
kklimondafair enough13:56
johnsomkklimonda Also on the active-active topic. Maybe. Right now it doesn't support all of the load balancing capabilities that would be needed by a distributor.13:57
kklimondajohnsom: is there something I could read about what's lacking on ovn side?14:03
johnsomkklimonda https://docs.openstack.org/octavia/latest/user/feature-classification/index.html14:03
rm_workjohnsom: yeah i am pretty sure this is an issue14:19
rm_workwhat is the best way to step through this with you?14:19
rm_workHangout?14:19
johnsomrm_work Sure, I'm in a meeting for another 30 or so14:20
rm_workkk np me too14:20
*** armax has joined #openstack-lbaas14:21
johnsomrm_work I am free now, so whenever you are ready14:35
rm_workkk one minute14:35
openstackgerritMichael Johnson proposed openstack/octavia master: Handle network deltas when delete member  https://review.opendev.org/73400214:40
rm_workjohnsom: alright14:46
rm_workjohnsom: https://meet.google.com/btb-hvbw-epu14:47
*** ramishra has quit IRC15:30
*** zzzeek has quit IRC15:31
*** zzzeek has joined #openstack-lbaas15:32
*** zzzeek has quit IRC15:37
*** zzzeek has joined #openstack-lbaas15:38
*** zzzeek has quit IRC15:42
openstackgerritAdam Harwell proposed openstack/octavia master: Failover stop threshold  https://review.opendev.org/65681115:44
*** zzzeek has joined #openstack-lbaas15:45
*** ramishra has joined #openstack-lbaas15:46
*** zzzeek has quit IRC15:50
*** zzzeek has joined #openstack-lbaas15:51
*** zzzeek has quit IRC15:56
*** zzzeek has joined #openstack-lbaas15:59
johnsom#startmeeting Octavia16:00
openstackMeeting started Wed Sep  9 16:00:49 2020 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Octavia)"16:00
openstackThe meeting name has been set to 'octavia'16:00
gthiemongeo/16:00
johnsomHi everyone16:01
johnsomHappy feature freeze week.16:01
rm_worko/16:01
johnsom#topic Announcements16:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:01
johnsomFinal release of python-octaviaclient is this week!16:01
johnsomFeature freeze for octavia is this week!16:02
rm_worksemi-distracted, got a second meeting overlapping right now >_<16:02
johnsom#link https://releases.openstack.org/victoria/schedule.html16:02
johnsomKendall did a nice write up of the intern work on Octavia:16:03
johnsom#link https://superuser.openstack.org/articles/contributor-qa-a-college-students-experience-contributing-to-the-openstack-ussuri-release/16:03
johnsomIf you haven't seen it16:03
johnsomThe summit schedule just went live as well:16:03
johnsom#link https://www.openstack.org/summit/2020/summit-schedule16:03
johnsomI have not looked to see if we are on it. lol16:04
*** zzzeek has quit IRC16:04
johnsomAny other announcements this week?16:04
*** zzzeek has joined #openstack-lbaas16:05
johnsom#topic Brief progress reports / bugs needing review16:05
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:05
rm_work"The amphorae of application delivery & security with Octavia" by A10 :P16:05
rm_worklol16:05
johnsomMy focus has pretty much been purely on reviews and rebasing patches.16:05
gthiemongereviews and internal stuff here16:06
gthiemongeBTW ataraday asked if we could review https://review.opendev.org/#/c/750638/16:06
johnsomgthiemonge Ah, thanks for highlighting that. It's not on the priority review list and I would have missed it.16:07
johnsomI have added it16:08
gthiemongethanks johnsom16:08
johnsomOk, on to the main event:16:08
johnsom#topic Priority bug reviews for Victoria16:08
*** openstack changes topic to "Priority bug reviews for Victoria (Meeting topic: Octavia)"16:08
johnsom#link https://etherpad.opendev.org/p/octavia-priority-reviews16:08
johnsomWe need to land anything from the top two (three, but lib is done) sections by tomorrow-ish.16:09
johnsomIf there is anything there that you are ok not making it into Victoria, please move it down.16:09
johnsomOtherwise, please help prioritize getting those reviewed and merged.16:10
johnsomI think we are doing pretty good really. We have got a lot of work done over the last week or two. Thank you!16:10
*** sapd1_x has quit IRC16:10
johnsomAny other discussion/questions/comments on the priority list?16:11
johnsom#topic Victoria cycle highlights16:12
*** openstack changes topic to "Victoria cycle highlights (Meeting topic: Octavia)"16:12
johnsomIt's that time again where we submit highlights for the release.16:13
johnsomI have put up a proposed patch:16:13
johnsom#link https://review.opendev.org/75052816:13
johnsomPlease review that and give me comments by Friday when I will remove the WIP flag and submit them.16:13
johnsomJust a reminder, the highlights are up to five items to highlight for the release for press reasons. It's not a replacement for the release notes which are much more detailed.16:15
johnsom#topic Open Discussion16:15
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:15
johnsomAny other topics today?16:15
johnsomgthiemonge just pointed out to me that the grenade job is timing out. That is probably due to the new scenario tests. I will fix that here after the meeting.16:16
johnsomAfter this week we are in feature freeze. No new features should be merged.16:18
johnsomWe can continue to work on bug fixes, docs, and tests however.16:18
johnsomIf there are no more topics I will give you all some time back to do reviews!16:19
gthiemongethanks johnsom16:19
johnsomo/16:20
johnsom#endmeeting16:20
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews"16:20
openstackMeeting ended Wed Sep  9 16:20:21 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:20
openstackMinutes:        http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-09-09-16.00.html16:20
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-09-09-16.00.txt16:20
openstackLog:            http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-09-09-16.00.log.html16:20
*** ramishra has quit IRC16:43
*** dosaboy has quit IRC16:51
openstackgerritMichael Johnson proposed openstack/octavia master: Update grenade tempest waiter check interval  https://review.opendev.org/75073616:52
johnsomlet's give that a try16:52
johnsomAh, that isn't going to be enough. I need to narrow the job list down again. Before we ran one smoke check, now it's running a bunch.16:55
openstackgerritMichael Johnson proposed openstack/octavia master: Update grenade job to run one smoke test  https://review.opendev.org/75073616:57
*** spatel has joined #openstack-lbaas17:02
spatelI have question for octavia, I am testing in lab and we are running VLAN base provider so no more NATing (My cisco router is gateway for all vms). when i create LB in that case amphora attaching it as floating IP (instead of directly adding external IP to amphora VM)17:05
spatelHow does my external IP get attached to direct on amphora and not do NAT.17:05
johnsomOctavia doesn't use floating IPs17:05
spateljohnsom: good to see you again :)17:06
johnsomo/17:06
spatelI just spun up lb1 and ssh into amphora and found only single IP address on VM which is lbaas-mgmt17:07
johnsomSo, when you create your load balancer, you specify any neutron network as your VIP network. They can be a provider network.17:07
johnsomYeah, all tenant traffic is isolated in a network namespace.17:07
johnsom"sudo ip netns exec amphora-haproxy ip a"17:07
spatelon controller node?17:07
johnsomThe tenant traffic interfaces are all in that network namespace inside the amphora17:08
johnsomDid you find the interface you expected?17:10
spateljohnsom: ah! i can see it now17:11
johnsomNote that you will not see the VIP address with the "ip a" command, it will be a base port IP17:11
spatelearlier i was using "ip a"  :)17:11
spateljohnsom: you nailed it17:11
johnsomYeah, that will only show the control plane interface on the lb-mgmt-net17:11
johnsomWe isolate the tenant traffic for security17:12
spateljohnsom: look at this output - http://paste.openstack.org/show/797663/17:13
johnsomYep, that looks pretty good to me17:14
spatelso i have only single interface ens3 and all traffic flowing through ?  or it has two distinct interface ens3 and eth1 ?17:14
johnsomCurrently, for tenant traffic you have one interface. When you add a member server, that has a different subnet than the VIP, it will add another interface.17:15
spatelhmm17:15
johnsomIf the pool member is on the same network as the VIP, you will only have one interface.17:16
spateli have pool member on same subnet17:16
spatelwhat are these two address ?   192.168.1.14 and 192.168.1.23 ?17:17
johnsomYeah, that creates a "one-armed" load balancer.17:17
spateldon't you think it should have single IP (if everything located on single subnet?)17:18
johnsom14 is the base port address. The amphora synchronization traffic and VRRP use that interface. The 23 is the VIP interface. The 14 address is permanent to this amphora. The 23 address will move between amphora in failure cases.17:18
spatelI have noticed my lb1 created two amphora, look like its Active-standby17:19
johnsomYes. that topology allows failover between the amphora in around a second. Less if you do manual tuning.17:20
johnsomIf you were to nova delete this amphora you are on, your traffic through the load balancer will continue on the standby.17:20
johnsomAnd then Octavia will rebuild the missing amphora17:21
spatelLets say if i configure for SINGLE amphora (not HA) in that case if my compute node die with some reason, does octavia create new amphora on other compute and heal my services itself?17:21
johnsomYes, it does. But it will take longer to resume traffic flow. Typically 30 seconds to a  minute.17:21
openstackgerritMichael Johnson proposed openstack/octavia master: Add a release note about HAProxy 2.0  https://review.opendev.org/75053117:27
spatelhmm can we adjust that time to lower?17:27
johnsomFor SINGLE instances most of the time is waiting for a new VM to boot in nova.17:27
spatelit would be great if octvia integrate with senlin so it will auto-scale ELB (like AWS)17:27
johnsomFor ACTIVE/STANDBY you can configure how aggressive it is with the octavia.conf.17:27
johnsomAuto scale is on the list of to-do features. It won't need senlin, it will handle it on it's own.17:28
johnsomWork started on this recently with the new metrics work.17:28
johnsomWhat level of connections and bandwidth are you expecting to need?17:29
openstackgerritMichael Johnson proposed openstack/octavia master: Add a release note about HAProxy 2.0  https://review.opendev.org/75053117:30
johnsomUgh, the focal jobs switched. now the lower constraints job is failing with a missing package dependency17:42
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the lower-const job by bumping cffi to 1.14.0  https://review.opendev.org/75076717:55
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the lower-constraints job for focal  https://review.opendev.org/75076718:19
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the lower-constraints job for focal  https://review.opendev.org/75076718:25
*** spatel has quit IRC18:28
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the lower-constraints job for focal  https://review.opendev.org/75076718:31
johnsomblah, what a chain of required changes. Sorry for the noise18:31
*** zzzeek has quit IRC18:33
*** zzzeek has joined #openstack-lbaas18:34
*** zzzeek has quit IRC18:41
*** zzzeek has joined #openstack-lbaas18:42
*** zzzeek has quit IRC18:50
*** zzzeek has joined #openstack-lbaas18:50
*** spatel has joined #openstack-lbaas18:51
spateljohnsom: thanks! sorry i was away for sometime so didn't see your last messages18:51
johnsomNo worries18:51
spatel"Auto scale is on the list of to-do features" do you know how they are going to do active-active?18:52
spatelthey need one more LB in front of octavia to scale horizontally18:53
johnsomWell, auto scale also include more RAM and vCPUs as well.18:53
johnsomThere are three active/active proposals. Just no developers to work on them at the moment.18:53
spateljohnsom: how does AWS doing that, i believe they are using CNAME style RR method.18:54
*** zzzeek has quit IRC18:55
johnsomYeah, that will break session persistence. AWS has three different "load balancers" that handle it differently18:55
*** zzzeek has joined #openstack-lbaas18:56
spatelhmm18:56
*** zzzeek has quit IRC19:03
*** zzzeek has joined #openstack-lbaas19:05
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the lower-constraints job for focal  https://review.opendev.org/75076719:17
johnsomOh boy py3.8 opened a can of worms. I guess I shouldn't have lost track of that test patch I was running.19:18
johnsomasync is now a reserved word, lol19:18
*** zzzeek has quit IRC19:21
*** zzzeek has joined #openstack-lbaas19:23
johnsompyroute2 released today with the fix to our regression19:29
johnsomWell, the regression in pyroute219:29
rm_worki thought that happened in like 3.619:32
johnsomWell we had versions in lower-constraints of sqlalchemy and pyroute2 that used it....19:32
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the lower-constraints job for focal  https://review.opendev.org/75076719:40
johnsomI guess it was in oslo.db, not sqlalchemy19:40
*** vishalmanchanda has quit IRC19:45
*** openstackgerrit has quit IRC20:17
*** openstackgerrit has joined #openstack-lbaas20:28
openstackgerritMichael Johnson proposed openstack/octavia master: Fix the lower-constraints job for focal  https://review.opendev.org/75076720:28
*** spatel has quit IRC20:34
rm_worki meant i thought async became reserved in python in like 3.620:39
*** spatel has joined #openstack-lbaas20:46
*** spatel has quit IRC20:52
*** TMM has quit IRC21:04
*** TMM has joined #openstack-lbaas21:04
openstackgerritMichael Johnson proposed openstack/octavia master: Update grenade job to run one smoke test  https://review.opendev.org/75073621:22
*** gcheresh_ has quit IRC21:31
rm_workk this looks good, I am assuming the commands work because i don't know how to actually verify other than yeah the syntax seems valid :D22:12
johnsomlol22:20
johnsomWhen you get a chance, could you take a quick pass at these gate fix patches: https://review.opendev.org/#/c/750767/ and https://review.opendev.org/#/c/750736/22:24
johnsomThe lower-constraint one was mostly done but zuul got restarted, so it started over.22:24
*** TrevorV has quit IRC22:42
*** rcernin has joined #openstack-lbaas22:54
*** rcernin has quit IRC22:56
*** rcernin has joined #openstack-lbaas22:56
*** zzzeek has quit IRC23:17
*** zzzeek has joined #openstack-lbaas23:19
*** dosaboy has joined #openstack-lbaas23:28
*** zzzeek has quit IRC23:38
*** zzzeek has joined #openstack-lbaas23:40
*** zzzeek has quit IRC23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!