Wednesday, 2018-07-04

rm_workxgerman_: ah you mean a rebase?00:00
rm_workmy own patch should have added enough skip_tests?00:00
*** yamamoto has quit IRC00:04
*** annp has quit IRC00:20
*** annp has joined #openstack-lbaas00:21
*** harlowja has joined #openstack-lbaas00:26
*** harlowja has quit IRC00:27
*** longkb has joined #openstack-lbaas00:43
*** yamamoto has joined #openstack-lbaas01:00
*** yamamoto has quit IRC01:06
sapd_rm_work: How do you tun your loadbalancer? What tool did you use?01:21
rm_work?01:22
*** hongbin has joined #openstack-lbaas01:27
*** yamamoto has joined #openstack-lbaas02:02
*** yamamoto has quit IRC02:07
*** sapd_ has quit IRC02:45
*** yamamoto has joined #openstack-lbaas03:04
*** yamamoto has quit IRC03:09
*** sapd has joined #openstack-lbaas03:44
*** ramishra has joined #openstack-lbaas03:49
bzhao__johnsom:  ping,  hi michael, "net.ipv4.vs.conntrack" configuration(/proc/sys/net/ipv4/vs) not exist before a keepalived start which hold a lvs,  and the amp namespace happen in plug vip during create lb, so sysctl --system into amp namespace happened there, at that moment, amp ns can not load the kernel configuraion.  If we do not reload the system configuraion in udp logic, that seems  "net.ipv4.vs.conntrack"  still03:52
bzhao__is 0... :(.. But for "net.ipv4.ip_forward" is OK.03:52
*** hongbin has quit IRC03:55
johnsombzhao__ Hi03:55
johnsombzhao__ Ok, it is likely a kernel module that exposes that path, so try this, before you start the keepalived, run an "lsmod" then start keepalived and do another "lsmod" and compare.  Then you should see some kernel modules loaded. If so, we can modprobe those modules (or insmod depending on the module) to load it early and be able to set the setting03:57
johnsombzhao__ It is 9pm before a holiday here, so I'm not at my computer any longer.03:58
*** yamamoto has joined #openstack-lbaas04:05
bzhao__johnsom:  OK, thanks very much. I think today it could be done. Also happy holiday and good night. :)04:10
johnsombzhao__: ok, cool. I worked on UDP today. I left one comment about the stats.  Good news is the performance looks pretty good in my tests.04:12
johnsomWill continue to focus on the on Thursday04:12
johnsomI got my other priority task done today04:13
bzhao__johnsom:  HA. :). Thanks. Never mind. But I lack so many tests, I will add them then.04:13
*** yamamoto has quit IRC04:13
bzhao__johnsom:  Sorry , Uts or some tests04:14
bzhao__johnsom:  Take a good rest. :)04:14
*** yamamoto has joined #openstack-lbaas04:18
*** kobis has joined #openstack-lbaas05:25
*** kobis has quit IRC05:42
*** ispp has joined #openstack-lbaas06:56
*** kobis has joined #openstack-lbaas07:09
*** rcernin_ has quit IRC07:13
*** AlexStaf has quit IRC07:16
*** ispp has quit IRC07:20
*** ianychoi has quit IRC07:20
*** tesseract has joined #openstack-lbaas07:28
*** kiennt26 has joined #openstack-lbaas07:38
*** ktibi has joined #openstack-lbaas07:50
*** yboaron has joined #openstack-lbaas07:56
*** ispp has joined #openstack-lbaas08:01
*** ktibi has quit IRC08:37
*** ktibi has joined #openstack-lbaas08:40
*** sapd has quit IRC08:56
*** sapd has joined #openstack-lbaas08:56
openstackgerritMerged openstack/octavia master: Enable oslo_config mutable configurations  https://review.openstack.org/57939108:57
*** apuimedo has joined #openstack-lbaas09:16
*** kobis has quit IRC09:58
*** yboaron_ has joined #openstack-lbaas10:01
*** yboaron has quit IRC10:04
*** kiennt26 has quit IRC10:07
*** kobis has joined #openstack-lbaas10:27
*** crazik has joined #openstack-lbaas10:47
crazikhello10:47
crazikWhat should happen if I shutdown MASTER amphora (ACTIVE_STANDBY layout)?10:49
crazikswitch to STANDBY (role->MASTER) and recreate second with STANDBY role?10:50
crazikLooks like at my setup something is wrong, both amphoras were replaced by new ones.10:51
*** yboaron has joined #openstack-lbaas11:03
*** yboaron_ has quit IRC11:05
*** annp has quit IRC11:33
*** peereb has joined #openstack-lbaas11:52
*** longkb has quit IRC12:14
*** ispp has quit IRC12:32
*** yamamoto has quit IRC12:33
*** ispp has joined #openstack-lbaas12:34
*** yamamoto has joined #openstack-lbaas12:54
*** yamamoto has quit IRC12:54
*** yamamoto has joined #openstack-lbaas12:55
*** ispp has quit IRC14:04
*** xcc has joined #openstack-lbaas14:05
*** ispp has joined #openstack-lbaas14:05
*** ramishra has quit IRC14:18
*** ramishra has joined #openstack-lbaas14:22
johnsomcrazik: yes, the failed amp should be replaced. Note, the roles in the database are settings, not the current state of which amp is master14:47
*** ramishra has quit IRC14:51
*** ramishra has joined #openstack-lbaas14:53
*** yboaron has quit IRC15:04
*** ramishra has quit IRC15:20
*** kobis has quit IRC15:21
*** kobis has joined #openstack-lbaas15:54
*** sapd1 has joined #openstack-lbaas16:11
*** peereb has quit IRC16:21
*** ispp has quit IRC16:21
*** tesseract has quit IRC16:30
*** ktibi_ has joined #openstack-lbaas16:42
*** ktibi has quit IRC16:45
*** kobis has quit IRC17:18
cgoncalvesyay! neutron-lbaas stable/queens fixed. kudos to johnsom!17:33
johnsomOh good!  Now we have a bunch of rechecks...17:34
cgoncalvesall done17:36
johnsomThanks!17:36
cgoncalvesprobably worth tagging stable/queens once all cherry-picks get merged17:37
cgoncalvesI've been seeing quite a few DNS issues at CI recently. for example: http://logs.openstack.org/07/579707/1/check/neutron-lbaasv2-dsvm-py3x-api-namespace/09f9d06/job-output.txt.gz#_2018-07-04_17_33_46_99033517:44
cgoncalvesI can resolve http://mirror.ca-ymq-1.vexxhost.openstack.org/wheel locally17:45
cgoncalvesmnaser, ^ do you happen to know?17:46
mnasercgoncalves: interesting, let me see.17:59
cgoncalvesmnaser, more here (apt command): http://logs.openstack.org/14/578014/1/check/octavia-v1-dsvm-scenario-multinode/20c191c/job-output.txt.gz#_2018-07-04_17_46_18_29272217:59
mnasercgoncalves: i've noticed this issue before but i dont actually have control over mirror vm17:59
mnaseroh thats interesting17:59
mnasera dns failure17:59
mnaserlet me talk to infra17:59
cgoncalvesthanks18:00
*** sapd1 has quit IRC18:06
crazikjohnsom: thanks!18:25
*** kobis has joined #openstack-lbaas18:39
*** kobis has quit IRC19:07
*** kobis has joined #openstack-lbaas19:08
*** kobis has quit IRC19:09
*** kobis has joined #openstack-lbaas19:10
*** kobis has quit IRC19:19
nmagnezio/19:56
johnsom#startmeeting Octavia20:00
openstackMeeting started Wed Jul  4 20:00:04 2018 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.20:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.20:00
*** openstack changes topic to " (Meeting topic: Octavia)"20:00
openstackThe meeting name has been set to 'octavia'20:00
johnsomHi folks!20:00
nmagneziO/20:00
cgoncalveshi20:00
johnsomI'm guessing this will be a quick one as the US is on holiday today.20:00
johnsomBut since we are an international team I figured we should still have our meeting as scheduled.20:00
johnsom#topic Announcements20:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"20:01
cgoncalvesoh, right. what are you doing here today?! :)20:01
nmagnezijohnsom, don't miss out the fireworks! :)20:01
johnsomcgoncalves It's called dedication to the PTL role....  Grin20:01
johnsomMy normal reminder, we have a priority bug list for the Rocky release:20:01
johnsom#link https://etherpad.openstack.org/p/octavia-priority-reviews20:02
nmagnezijohnsom, four more years!.. mm.. cycles :)20:02
johnsomThank you to everyone that has been helping there!20:02
* johnsom sighs20:02
johnsomAlso a heads up, python-octaviaclient needs to have it's final Rocky release before July 26th per the release calendar20:02
johnsom#link https://releases.openstack.org/rocky/schedule.html20:03
johnsomSo any additions we want to get into the "official" Rocky version of the client needs to be in by the 26th.20:03
nmagneziaye. We have some client patches to get in.\20:03
johnsomI think we have two right now, backup members and the UDP extensions (which I have been testing).20:03
johnsomIf there are any others not on that list, let me know so I can keep track of them and make sure we get them in20:04
cgoncalvesFWIW I quickly tested the backup member patch and looked good to me20:05
johnsomThe last item I have is about the recent queens breakage in neutron-lbaas. However I understand this is now resolved.20:05
johnsomCool, thanks!20:05
cgoncalvesit is. thank YOU!20:05
nmagnezijohnsom, great job!20:05
johnsomFYI, the trouble with stable/queens started with this patch: https://review.openstack.org/57652620:06
nmagneziI reviewed bunch of queens backports today so we can have those in soon20:06
cgoncalvesjobs are arbitrarily failing due to DNS failures. rechecking should do the trick20:06
johnsomSomeone added neutron-lbaas to upper constraints and then later global requirements.20:06
johnsomIf you look in those files, none of the "plugins" horizon or neutron are in those files20:07
johnsomThis is because the tests for these plugins need to install the "parent" project, neutron in our case, from a filesystem path (zuul) or git (tox) and you can't set an upper-constraint limit on those install types.20:08
*** kobis has joined #openstack-lbaas20:09
johnsomAnyway, we got those changes reverted.  They should not have been approved as requirements changes on stable branches are against the stable policy, but that is a different issue.20:09
johnsomcgoncalves Thanks for the rechecks and looking into that DNS issue. I know Mr. Naser is working to fix it.20:09
*** kobis has quit IRC20:10
johnsomAny other announcements today?20:10
johnsom#topic Brief progress reports / bugs needing review20:10
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"20:10
*** kobis has joined #openstack-lbaas20:11
johnsomI have wrapped up the initial version of the neutron-lbaas to octavia LB migration tool. I have been working on the gate to test it, which was a pain with little Ansible gotchas.20:11
* nmagnezi still waits for the magical local.conf :-)20:11
johnsomLike the one where it merges stderr into the stdout content from the "neutron" command20:11
johnsomYeah, I need to do another spin to take another approach at getting two drivers enabled in neutron-lbaas.20:12
johnsomI will poke at that after the meeting.20:12
johnsomAnyway, the second test will migrate a non-octavia LB (even though there is no octavia driver for it) as a test.20:13
nmagneziWill that migration tool work for migration from n-lbaas (any provider) to Octavia as a keystone endoint?20:13
johnsomShould be good soon-ish.  I plan to make it a periodic gate20:13
johnsomYes, any neutron-lbaas provider to Octavia. Octavia should have the appropriate provider installed of course.20:14
johnsomIt sounds like VMware is making good progress on their provider driver, so likely to see that soon.  They already have a third party gate setup.20:14
johnsomThe tool will not migrate from one provider to a different provider on Octavia. That is not it's intent.20:15
johnsomI think that is possible, but a bunch more work that probably wouldn't make Rocky20:15
johnsomThen after that work I am focusing on helping with the UDP support.20:16
nmagneziI think we should think about at least haproxy-ns to amphora driver migration since both are reference implementations. But maybe that is something I will follow up with, after you migrate what you plan for Rocky20:16
johnsomI have been doing some testing. I was able to push 4.76 Gbits/sec through it on my little devstack setup, so performance looks decent. We just have some stuff to finish up there.20:17
johnsomYeah, technically the namespace driver ceased to be a reference driver in Liberty, but I'm not sure that was communicated well.20:17
johnsomAgreed though, that is a really good use case.20:18
johnsomFeel free to start work on it.... grin20:18
nmagneziSadly many folks still use it, and migration path might help them to move :-)20:18
nmagneziHaha, first we finish yours. I plan to test it20:19
johnsomI have "ideas" of how to do it if someone wants to work on it.20:19
johnsomCool, thanks!20:19
johnsomIt "should" work now, I'm really just poking at the gate tests20:19
johnsomAfter UDP is in good shape I need to work on an active/standby test gate. (internal request)20:20
johnsomAny other progress updates?20:21
johnsom#topic Open Discussion20:22
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"20:22
johnsomAny other topics today?20:22
cgoncalvesnot much from my side. essentially backporting patches and working in tripleo to have an octavia scenario with octavia tempest tests20:22
johnsomCool. I really appreciate the help with backports BTW.20:23
johnsomI know you mentioned cutting a new release, should we do that around MS3 time or would you like it sooner?20:23
*** fnaval has joined #openstack-lbaas20:24
cgoncalvesI *think* I don't have a preference. I just thought that it would be nice to release since we have backported a couple of good stuff20:24
nmagneziOn my side, I'm currently digging into Active Standby, looks like data plane takes longer than expected time to recover (but the backup amp does send GARPS, so.. still looking)20:25
johnsomYeah, agreed.  Ok, I will set a mental upper bound on MS3 timeline20:25
johnsomYeah, that is interesting. If we see the GARPs coming out of the backup that became master, but traffic isn't getting to us, then something is fishy in neutron land.  Well worth looking into.20:26
cgoncalvesnmagnezi, that reminds me of the MASTER/BACKUP role topic we discussed offline20:26
nmagneziAnd by longer.. It took it 8 minutes to recover on my devstack. So definitely something worth looking into20:26
johnsomYeah, that is crazy20:26
cgoncalvesbut maybe we can leave it to another time20:26
nmagnezicgoncalves, yeah, this is something I'm also looking into. Trying to find a way to determine the nodes state in a reliable way so we can report it to Octavia.20:27
johnsomcgoncalves We can chat about it if you want. The meeting is scheduled for an hour20:27
*** aojea_ has joined #openstack-lbaas20:27
johnsomnmagnezi Let me share a video demo I prepared for Vancouver but didn't get to present.20:27
nmagnezijohnsom, please do :)20:27
cgoncalvesjohnsom, I'd rather like having you reviewing https://review.openstack.org/#/c/568361/ xD20:27
johnsom#link https://drive.google.com/open?id=1wx1kkLjUxwNAOpd9KeTGSAV-INZS58fD20:28
johnsomThat is a video I recorded of a failover flow (it was part of a dashboard demo)20:29
cgoncalvesjohnsom, in a nutshell we think that showing amphora roles as MASTER or BACKUP confuses users. folks tend to think the MASTER one is the active, and the other the standby20:29
cgoncalvesand that on an amp failover, they would expect the BACKUP to become MASTER and the new spawned amp the BACKUP20:30
cgoncalvescool, thanks for the video!20:30
johnsomcgoncalves Yes, I understand.  This is why we called it "role" and not "status" as it's really about configuration settings applied to the amphora20:30
johnsomLuckily it's only an admin API that exposes that.20:31
nmagneziToday when I created an HA lb, without even doing anything I got it exactly the opposite from the listed roles20:31
nmagneziO_O20:31
johnsomYeah, and that is perfectly ok.20:32
johnsomThe amps are autonomous on their "active" status20:32
nmagneziBy that logic I agree with you. It's just counterintuitive20:33
cgoncalveswhy would admins care about configuration settings if they can't (in many cases) log-in to the amps? what is the use case for exposing the roles?20:33
nmagnezicgoncalves, for swapping amps (to update images) and I would even say to understand the impact of evacuating a specific compute node that has amps running on it, No?20:35
johnsomNone really. It was just an API that Adam wanted to be able to see the details of an amphora.  Amphora are really supposed to be "hidden" things that are an implementation detail.  I think that column just got swept up with the more interesting columns about the amp20:35
cgoncalvesnmagnezi, I don't follow. what you said sounds to me more of a reason for exposing the current status of the amps than their roles20:36
nmagnezicgoncalves, yup20:36
johnsomNo, the Role column has no operational value really. It is just there to track the priority value and preemption settings that get applied to each in the pair.20:36
nmagneziSorry, you got it right20:37
cgoncalvesjohnsom, ok. I don't see right now a use case where admins/operators would be interested in consuming that column. Adam may have and might be able to share20:38
cgoncalvesanyhow, I brought this up because I expect users to get confused about this20:39
johnsomHmm, looking in the code I'm not sure that even matters anymore....  We may have removed the differentiation between the settings on each (other than the priority which is tracked separate)20:39
johnsom*If* we can find a reliable way to get the actual "active" status of an amp (long discussion with Nir about that yesterday), then we certainly can add a status column for it.20:40
johnsomThe challenge is getting an accurate status from keepalived.20:40
johnsomThis also brings us back around to Mr. Ohara20:41
cgoncalvesright. I read the discussion you had to amuller back in end of May here too20:41
cgoncalvess/to/with/20:42
*** kobis has quit IRC20:42
johnsomgrin20:42
*** kobis has joined #openstack-lbaas20:42
johnsomYeah, it does have a bit of usefulness. I also don't want to break the autonomy of the amps in this failover. They should be free to switch whenever they detect an issue, which may be faster than the heartbeat interval.20:44
johnsomVRRP failover should be around a second or less depending on the configuration. Heartbeat is usually spaced higher than that.20:46
johnsomSo, please feel free to work on this.  We can also add a column "active status" or something and just put "unkown" in it....  grin20:46
cgoncalvesright. I think, though, that keepalived has a notification subsystem. we could leverage that and callback octavia20:47
cgoncalveslol20:47
nmagnezihaha20:47
johnsomYeah, I couldn't get it to reliably give me the actual state.20:47
*** kobis has quit IRC20:47
johnsomI would skip some, or give bogus "Failed" while in the master role, etc.20:47
johnsomMaybe it's been fixed and we can make it work.20:48
johnsomThe active IP thing might also solve this if we go clean that up.20:48
cgoncalvesinteresting. maybe I can play a bit with that some day20:48
johnsomYeah, right now I don't have cycles to go play with it again.  As long as it is *working* it's lower on the priority list.20:49
cgoncalvesknowing status of amps would improve maintenance ops greatly (e.g. failover first standby amp for image update than active amp)20:50
johnsomWell, technically, not really.20:50
johnsomSince if you happen to kill the master, it's going to failover inside the TCP timeout window, so it should be transparent other than a slight delay in that packet.20:51
johnsomThis is why it's a bit funny when the marketing folks ask how many milliseconds does it take for a HA failover. With TCP flows it's just a metric of jitter, not a window of downtime.20:53
cgoncalveshmmm the first failover would fail to the standby amp, ok, then we'd need to failover once more to also rotate the now active amp20:53
johnsomCorrect, that is what the LB failover flow does20:54
johnsomWhen you call the LB failover API20:54
cgoncalvesok, you're right in saying that technically it's possible20:54
*** ThomasWhite has quit IRC20:55
cgoncalvesI'd just like to avoid failing over active amps twice20:55
cgoncalveslow prio for now :)20:55
johnsomI'm not following, how would you failover one of the amps twice?20:56
johnsomIt should be, amp A and amp B, each get one failover20:57
cgoncalvesright. I'm saying it would involve failing over twice insie the TCP timeout window as you mentioned before20:58
johnsomOh, that would be impressive if you could cycle your first amp inside one packet timeout20:58
cgoncalvesamp A (MASTER), amp B (BACKUP) running image X. failover amp A to image X+1. amp B (now ACTIVE) is still on image X. we'd need to fail it over to update it to image X+1 too20:59
nmagnezijohnsom, cgoncalves, my connection went down few minutes ago, sorry.21:00
nmagnezijohnsom, cgoncalves, good night/day folks!21:00
johnsomCorrect, but amp failover (not vrrp failover) of A takes 10+ seconds on a good day, so B would be VRRP master and handling traffic long before you are ready to do amp B.21:00
johnsomAh, yeah, out of time....21:00
johnsomThanks folks!21:01
johnsom#endmeeting21:01
*** openstack changes topic to "Discussion of OpenStack Load Balancing (Octavia) | https://etherpad.openstack.org/p/octavia-priority-reviews"21:01
openstackMeeting ended Wed Jul  4 21:01:06 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-07-04-20.00.html21:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-07-04-20.00.txt21:01
openstackLog:            http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-07-04-20.00.log.html21:01
cgoncalvesgood discussion. thanks for helping clarify, and for the time today on a holiday, Michael!21:01
johnsomSo in a full LB failover, two packets would see a slightly higher latency, or increased jitter.21:02
johnsomSure, let's keep talking about this so we are all on the same page.21:02
cgoncalvesack :)21:03
johnsomAlso, there is a demo in this Tokyo summit presentation: https://youtu.be/8n7FGhtOiXk21:03
johnsomAh, here to be exact: https://youtu.be/8n7FGhtOiXk?t=23m40s21:03
cgoncalvesnice!21:04
*** ktibi_ has quit IRC21:57
*** threestrands has joined #openstack-lbaas22:05
*** threestrands has quit IRC22:05
*** threestrands has joined #openstack-lbaas22:05
*** threestrands has quit IRC22:06
*** rcernin has joined #openstack-lbaas22:27
*** aojea_ has quit IRC22:29
*** kobis has joined #openstack-lbaas23:10
*** kobis has quit IRC23:15

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!