Friday, 2018-10-05

*** Swami has quit IRC00:04
abaindurjohnsom: or rm_work: is there a way to scale up/down a loadbalancer?00:05
abaindurhttps://developer.openstack.org/api-ref/load-balancer/v2/#update-a-load-balancer00:05
rm_workNot yet00:05
abaindurI dont see a flavor ID parameter for update00:05
abaindurbut i only see it for create00:06
johnsomNot at this point.00:06
johnsomDo you need to?00:06
abaindurNo, we were just wondering, a nice to have00:06
abaindurso i guess right now only solution is to delete it, then create a new one?00:06
johnsomYeah, with flavors we will allow different sized amphroa and likely enable threading. Beyond that is the ActiveActive work00:06
abaindurmeaning we would also need to delete the members, pools, listeners... and recreate all those resources again from scartch00:07
johnsomCorrect. or change the flavor setting in the cfg and do a failover.00:07
abaindurthe flavor in the cfg is the default one, if flavor_id not specified in the create POST call, right?00:08
abaindurotherwise it creates using flavor in the request00:08
abaindurhttps://developer.openstack.org/api-ref/load-balancer/v2/#create-a-load-balancer00:08
johnsomTechnically you can do a nova resize, but it will reboot the instance, which will delete the TLS certs.00:08
johnsomNo, the flavor_id in the API is for Octavia flavors, not nova flavors. Octavia flavors aren't fully implemented yet.00:09
johnsomI am planning to finish that for Stein00:09
colin-was setting up some benchmark tests for my amusement and ran iperf on the mgmt NIC of my amphora, was surprised to get 2x the throughput i did to a similar backend when using the VIP through the same amphora. does that make sense and is netns expected to suffer to a large degree?00:09
abaindurnot sure I follow then. if its defined in the cfg file, what is purpose of the flavor_id in POST call?00:09
johnsomNo. I have got 14Gbps with iperf3 thorugh our amp00:10
colin-ok00:10
abainduramp_flavor_id is "Nova instance flavor id for the Amphora"... so what is the purpose of the flavor_id in current create call?00:10
johnsomabaindur the flavor in the config file is the nova flavor. The flavor in the Octavia API is for Octavia flavors (not nova)00:10
colin-and you expect the performance characteristics to be similar for traffic using the namespaced NIC as it would be for traffic on the native one?00:11
abaindurWhat is an Octavia flavor then :)00:11
abaindurah i see.. you said its not implemented yet, in stein00:12
johnsomOctavia flavors will implement this spec: https://docs.openstack.org/octavia/latest/contributor/specs/version1.0/flavors.html00:12
abaindurso currently, even as of rocky, the flavor_id in create is basically ignored00:12
johnsomcolin- Well, native will always be faster, but the way we use the namespace it doesn't have much performance penalty. We don't use bridges and veth pairs and all that muck that takes the perf down.00:13
johnsomabaindur Correct00:13
johnsomabaindur We lost a lot of developers off the project all of a sudden and it just didn't get finished per the original plan.00:14
johnsomWell, OpenStack lost a lot of devs, and our project was one that suffered.00:14
johnsomcolin- http://paste.openstack.org/show/731546/00:16
johnsom1 vCPU, 1GB RAM, Rocky codebase00:17
johnsomOf course a million variables in the clouds will change those numbers, so your mileage will vary.00:17
colin-yeah00:20
colin-what is your line speed?00:20
colin-i'm not approaching even 50% line speed on my tests run similarly so definitely some key differences00:20
johnsomThe bypass was 15.4 Gbits/sec00:21
johnsomcolin- Oh, my bad, it had this patch in it, so I guess it was a Stein amp image: https://review.openstack.org/#/c/598379/00:24
johnsomI did those changes internal before I upstreamed them.00:24
colin-understood, thanks for the additional details. between two nova vms in the same network i can get almost double what i can through an octavia vip so i must have something that is off00:25
colin-i'll take a look at those changes you linked specifically about tcp kernel splicing maybe they could hepl00:25
johnsomYeah, the splicing is helpful to bring the CPU load down. Also, make sure your haproxy package from your OS vendor isn't forcing it off on the command line for the systemd wrapper. I saw that happening somewhere too.00:26
*** rcernin has quit IRC00:27
johnsomOk, off to dinner. catch you all tomorrow00:28
*** rcernin has joined #openstack-lbaas00:29
colin-ttyl, enjoy00:30
*** dlundquist has joined #openstack-lbaas00:38
*** dlundquist has quit IRC00:43
*** dlundquist has joined #openstack-lbaas00:44
*** hongbin has joined #openstack-lbaas01:57
*** JudeCross has quit IRC01:57
*** ramishra has joined #openstack-lbaas02:09
*** JudeCross has joined #openstack-lbaas02:17
*** yamamoto has quit IRC02:23
*** yamamoto has joined #openstack-lbaas02:23
*** abaindur has quit IRC02:23
*** dosaboy has quit IRC02:35
*** annp has joined #openstack-lbaas03:01
*** JudeCross has quit IRC03:04
*** JudeCross has joined #openstack-lbaas03:05
*** sapd1 has quit IRC03:10
*** sapd1 has joined #openstack-lbaas03:10
openstackgerritMichael Johnson proposed openstack/octavia master: Fix IPv6 in Active/Standby topology  https://review.openstack.org/58929203:27
*** hongbin has quit IRC03:42
*** rcernin has quit IRC04:42
*** rcernin has joined #openstack-lbaas04:46
*** openstackgerrit has quit IRC04:52
*** openstackgerrit has joined #openstack-lbaas04:57
*** aojea has joined #openstack-lbaas05:34
*** celebdor has joined #openstack-lbaas05:39
*** aojea has quit IRC05:51
*** Emine has quit IRC05:56
*** phuoc has joined #openstack-lbaas06:33
*** phuoc_ has quit IRC06:36
*** ccamposr has joined #openstack-lbaas06:43
*** aojea has joined #openstack-lbaas06:43
*** JudeCross has quit IRC06:49
*** pcaruana has joined #openstack-lbaas06:57
*** JudeCross has joined #openstack-lbaas06:58
*** rcernin has quit IRC07:04
*** maciejjozefczyk has joined #openstack-lbaas07:05
*** rtjure has quit IRC07:09
*** rtjure has joined #openstack-lbaas07:12
*** abaindur has joined #openstack-lbaas07:14
*** dosaboy has joined #openstack-lbaas07:38
*** abaindur has quit IRC07:47
*** velizarx has joined #openstack-lbaas07:48
maciejjozefczykhello!08:03
maciejjozefczykMy amphorea which is build on Xenial is playing with me, sometimes VIP interface is renamed from ethX to ensX and then renamed back. After some seconds its missing in the system.08:04
maciejjozefczykAfter a big digging it said on the net to try Bionic08:04
maciejjozefczykSo I tried amphore'a with Bionic and while attaching VIP interface to amp on amphore-agent logs I see08:05
maciejjozefczyk[2018-10-05 07:57:46 +0000] [970] [DEBUG] POST /0.5/plug/vip/192.168.0.5108:05
maciejjozefczyk::ffff:10.10.10.54 - - [05/Oct/2018:07:57:46 +0000] "POST /0.5/plug/vip/192.168.0.51 HTTP/1.1" 500 123 "-" "Octavia HaProxy Rest Client/0.5 (https://wiki.openstack.org/wiki/Octavia)"08:05
maciejjozefczykHow to debug it?08:05
maciejjozefczykIn conf i have defiend the debug=True08:05
maciejjozefczykI need to do manual CURL with certificate validation to check that/08:06
maciejjozefczykok… got it, syslog is the key08:07
maciejjozefczyksorry for spamming ;)08:07
*** salmankhan has joined #openstack-lbaas08:08
*** velizarx has quit IRC08:11
*** velizarx has joined #openstack-lbaas08:12
*** JudeCross has quit IRC08:26
*** JudeCross has joined #openstack-lbaas08:43
*** aojea has quit IRC08:44
*** JudeCross has quit IRC08:47
*** aojea has joined #openstack-lbaas08:50
rm_workmaciejjozefczyk: i am not sure how much Bionic is actually tested/functional yet for amps08:53
rm_workmaciejjozefczyk: but it's interesting you're suffering from the net naming issue, as we specifically turn that off08:53
maciejjozefczykrm_work: my fault08:53
maciejjozefczykrm_work: I need a duck, basically :)08:54
rm_workdid you get it working with xenial?08:55
maciejjozefczykyep08:57
maciejjozefczykim just curious, because I'm testing Active-Standby with my network architecture08:58
maciejjozefczykand I see there is no ability to set "HA mode" while creating LB?08:58
maciejjozefczykthe only way is config parameter loadbalancer_topology?08:58
rm_workare you using my L3 network patch?08:59
rm_workyes08:59
rm_workwe would need flavor support08:59
rm_workto allow for configurable topology08:59
maciejjozefczykrm_work: unfortunately im not using the L3 network driver08:59
rm_workah08:59
maciejjozefczykdue I was not able to select other subnet for VIP and VRRP08:59
maciejjozefczykok, but anyway about this patch09:00
rm_worki'm still confused about that09:01
rm_workwhy would you need to select a subnet?09:01
rm_workto make a FLIP, you only select a network?09:01
rm_workI thought09:01
maciejjozefczykI have usecase when client has his own private tenant network, lets name it 'private'09:01
maciejjozefczykand under this 'private' network client has some backends09:01
maciejjozefczykI would like to have VRRP working inside this private network09:02
rm_workhmm09:02
maciejjozefczykand on VIP port I would like to create a FLIP, which is from other network (External network)09:02
rm_workyes, that is how it works now, I believe09:03
rm_workthough actually i can't remember how i decide which network to put the VRRP ports on09:03
maciejjozefczykbasically its similiar to usecase given here: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/networking_guide/sec-octavia09:03
rm_worksince it can't be the FLIP network09:03
rm_workah yeah probably my driver just selects the management network for the VRRP network09:05
rm_workbecause I also assumed "no tenant networks" in the design09:05
maciejjozefczykyep, thats right09:05
rm_workok, i'll think about that09:05
maciejjozefczykso basically vrrp port, VIP port and FIP are from the same subnet09:05
rm_worknot sure if there's an easy way to allow that09:05
rm_workerr, that can't be true i think09:06
rm_workbut i'll check09:06
maciejjozefczykplease check :)09:06
rm_workyou can always submit a patch after mine in the chain, and change what you need :P09:07
maciejjozefczykfor now I tested the allowed_access_pairs driver under my tenant network with floating IP that points on VIP port, and it worked perfecto!09:07
rm_workah, cool09:07
rm_worki wonder if it will work on failover though09:07
maciejjozefczykyea, sure, rm_work we're like touching the topic now, like a little research09:07
maciejjozefczykrm_work: im checking it09:07
rm_workif you don't have the same L2, it shouldn't09:07
maciejjozefczykrouter and FIP along with VRRP works in the same L209:08
maciejjozefczykso the only thing is that keepalived needs to move the alias on VRRP interface from one amphorea to another09:08
maciejjozefczykit must work ;)09:08
maciejjozefczykanyway, we're going in OVH do deploy octavia soon09:09
rm_workah yeah if you have guaranteed L2 adjacency, you're good09:09
maciejjozefczykwe have dedicated team for service similiar to Octavia and they will adopt Octavia to our use-case09:09
maciejjozefczykso for sure we gonna upstream some patches09:09
rm_workcool09:10
maciejjozefczykfor nowe we have like 2 or 3 cases that needs to be solved, but little ones09:10
maciejjozefczykfor now*09:10
*** salmankhan has quit IRC09:25
*** Emine has joined #openstack-lbaas09:32
*** salmankhan has joined #openstack-lbaas09:32
*** JudeCross has joined #openstack-lbaas09:38
*** JudeCross has quit IRC09:43
maciejjozefczykrm_work: HA works perfect!09:51
maciejjozefczyk<1 sec of downtime09:51
*** yamamoto has quit IRC10:07
*** yamamoto has joined #openstack-lbaas10:08
*** yamamoto has quit IRC10:09
*** yamamoto has joined #openstack-lbaas10:10
*** yamamoto has quit IRC10:13
*** yamamoto has joined #openstack-lbaas10:14
*** yamamoto has quit IRC10:16
*** yamamoto has joined #openstack-lbaas10:17
*** Emine has quit IRC10:17
*** yamamoto has quit IRC10:21
*** yamamoto has joined #openstack-lbaas10:27
*** yamamoto has quit IRC10:32
*** yamamoto has joined #openstack-lbaas10:33
*** yamamoto has quit IRC10:38
*** annp has quit IRC10:53
*** yamamoto has joined #openstack-lbaas11:08
*** rpittau_ has quit IRC11:17
*** yamamoto has quit IRC11:18
*** yamamoto has joined #openstack-lbaas11:19
*** yamamoto has quit IRC11:23
*** yamamoto has joined #openstack-lbaas11:24
*** salmankhan has quit IRC11:29
*** JudeCross has joined #openstack-lbaas11:39
*** JudeCross has quit IRC11:44
*** JudeCross has joined #openstack-lbaas12:23
*** JudeCross has quit IRC12:28
*** aojea has quit IRC12:32
*** salmankhan has joined #openstack-lbaas12:33
*** dosaboy has quit IRC12:35
*** dosaboy has joined #openstack-lbaas12:36
*** dosaboy has quit IRC12:36
*** aojea has joined #openstack-lbaas12:36
*** aojea has quit IRC12:41
*** dosaboy has joined #openstack-lbaas12:41
*** dosaboy has left #openstack-lbaas12:42
*** ccamposr has quit IRC12:52
*** dosaboy has joined #openstack-lbaas12:56
*** dims_ has quit IRC12:58
*** celebdor has quit IRC13:02
*** celebdor has joined #openstack-lbaas13:16
*** ramishra has quit IRC13:28
*** velizarx has quit IRC13:46
*** velizarx has joined #openstack-lbaas13:50
*** celebdor has quit IRC14:14
*** fnordahl has joined #openstack-lbaas14:16
*** JudeCross has joined #openstack-lbaas14:24
*** JudeCross has quit IRC14:28
*** numans has quit IRC14:47
*** sabomia has joined #openstack-lbaas14:54
johnsomFYI, the charms folks are starting work on an Octavia charm: https://review.openstack.org/60828315:03
johnsomBummer, so the Bionic kernel does not solve the KVM issues we are seeing....15:15
johnsomhttp://logs.openstack.org/39/600539/18/check/octavia-v2-dsvm-scenario/6c294a7/controller/logs/libvirt/qemu/instance-0000000a_log.txt.gz15:15
*** Emine has joined #openstack-lbaas15:23
*** pcaruana has quit IRC15:39
cgoncalvesshall we switch OS to centos? ;-)15:48
*** velizarx has quit IRC15:50
*** salmankhan has quit IRC16:04
*** ianychoi_ is now known as ianychoi16:19
*** sabomia has quit IRC16:27
*** celebdor has joined #openstack-lbaas16:34
*** pcaruana has joined #openstack-lbaas16:42
johnsomI am open to the possibility if it fixes our issue.....  How is that test patch going?16:45
*** dims has joined #openstack-lbaas16:46
openstackgerritLuis Tomas Bolivar proposed openstack/octavia master: Enabling SG customization on loadbalancer listerners  https://review.openstack.org/60256416:49
*** salmankhan has joined #openstack-lbaas16:49
*** pcaruana has quit IRC16:50
openstackgerritLuis Tomas Bolivar proposed openstack/octavia master: Enabling SG customization on loadbalancer listerners  https://review.openstack.org/60256416:54
ltomasbojohnsom, cgoncalves ^ I updated it with your suggestions16:55
johnsomCool, so you figured out a work around?  Sorry, I got pulled into an internal exercise so, haven't followed closely16:56
ltomasbojohnsom, me? I just added the option to have it disable by default, as well as a whitelisting option (cgoncalves' suggestion) to even if it is active, only enable it for certain projects16:58
johnsomhmmm, not sure I am a fan of the whitelisting16:58
johnsomI will have a look though.  Thanks for the patience on this, it's a tricky one17:00
*** JudeCross has joined #openstack-lbaas17:00
ltomasbojohnsom, great! thanks! and don't worry, I understand it is not the right thing to do17:01
ltomasbojohnsom, it is a pity that changing the SG on the VIP was not working, that would have been a better compromise17:02
johnsomAgreed17:02
johnsomOr even a way to "AND" SGs so that we could own one and the tenant could own the other....17:03
ltomasbothanks for the feedback (and for the time spent on the weekly meetings on this)17:03
*** yamamoto has quit IRC17:08
*** salmankhan has quit IRC17:09
*** yamamoto has joined #openstack-lbaas17:09
*** yamamoto has quit IRC17:14
xgerman_ I am spending my morning with LB in Error… OSP shutting down rabbit with each upgrade is really a bummer…17:24
johnsomThat is fixed with the HM perf patch17:24
xgerman_ok, well, they won’t backport to OSP 12 - so there’s that17:25
johnsomSigh17:25
*** celebdor has quit IRC17:38
*** salmankhan has joined #openstack-lbaas17:53
*** salmankhan has quit IRC17:55
*** logan- has left #openstack-lbaas17:55
*** salmankhan has joined #openstack-lbaas17:55
*** yamamoto has joined #openstack-lbaas17:59
*** logan- has joined #openstack-lbaas18:08
*** Swami has joined #openstack-lbaas18:36
*** sabomia` has joined #openstack-lbaas19:37
sabomia`hey guys, what could explain for an otherwise functional load balancer to remain in operational status offline?19:40
johnsomWhich object is marked operating status offline, the LB itself?19:45
sabomia`yes the LB itself and the listener19:45
sabomia`the pool is online19:45
sabomia`and the LB seems to function as expected19:45
sabomia`running OSA Queens 17.0.719:45
johnsomCheck that your health manager process is actually getting the health heartbeats.19:46
johnsomfor example, you could look in the octavia database a query the amphora_health table;19:46
johnsomhttps://www.irccloud.com/pastebin/3ytFpIiu/19:46
johnsomIf it is empty or very old dates, you may have a problem with the heartbeats getting back to the controller(s).19:47
*** Emine has quit IRC19:47
sabomia`http://paste.openstack.org/show/731600/19:50
sabomia`this eems unrelated to the current lb though19:50
sabomia`I had a networking issue on it before, so that may explain it. I got that sorted out and somewhat successfully deployed the first VM19:50
sabomia`*LB19:50
johnsomYeah, ok, so the health heartbeats are not making it back.  Things to check:19:52
johnsomhttps://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.controller_ip_port_list19:52
johnsomMake sure the controller_ip_port_list is accurate to your health manager processes.19:53
johnsomExample: controller_ip_port_list example: 10.0.0.1:5555, 10.0.1.2:555519:53
johnsomMake sure your amps can send UDP back on port 5555 to the controllers.19:54
sabomia`thanks johnsom, I'll look at that in a sec.20:09
*** celebdor has joined #openstack-lbaas20:33
openstackgerritMichael Johnson proposed openstack/octavia master: Fix IPv6 in Active/Standby topology  https://review.openstack.org/58929220:33
*** salmankhan has quit IRC20:49
*** rtjure has quit IRC21:07
johnsomsabomia` They can have IPs from a different subnet, but that is unusual. Typically the lb-mgmt-net will have the same addresses both on the controller-ip list and the amphora-agent interface22:04
colin-johnsom: any suggestions for where i can search for evidence of what's throttling my throughput wrt what we discussed yesterday?22:06
colin-resource utilization on all concerned hosts is healthy throughout and i don't see anything obvious in journal22:06
colin-cherry picking the changes you linked yesterday for testing22:07
johnsomcolin- I would start by eliminating the parts between point A and B.  So, co-locate your source instance, amp, and target on the same host. This removes the NICs, etc. from the picture.22:07
colin-ah, good idea. thanks22:07
colin-that will vastly simplify things22:07
johnsomChecking the splicing stuff I mentioned yesterday are good. Then I would work back, like adding OVS, NICs, etc.22:08
sabomia`johnsom: that may be my issue here? I couldn't get it to work though with the amphoras and management ips on the same bridge. Once a amphora VM would get deployed I would lose connectivity on my bridge. I followed xgerman_ install instructions using separate subnets from https://github.com/xgerman/rpc-octavia/blob/master/INSTALLATION.md22:11
sabomia`Following his instructions got my load balancers to work, I am just still not getting any health monitoring msgs22:12
*** aojea has joined #openstack-lbaas22:12
johnsomsabomia` Those instructions have one subnet for the lb-mgmt-net22:12
sabomia`they refer to 172.29.248.0/22 and br-lbaas in the provider network and 10.0.252.0/22 + lbaas-mgt in the octavia management network22:14
johnsomI think that is a mistake22:21
*** aojea has quit IRC22:45
openstackgerritMerged openstack/octavia stable/queens: Add note to lower constraints for Jinja and pyOpenSSL  https://review.openstack.org/60614222:47
*** celebdor has quit IRC22:48
*** Swami has quit IRC23:04
*** aojea has joined #openstack-lbaas23:17
openstackgerritMerged openstack/octavia master: HTTPS HMs need the same validation path as HTTP++  https://review.openstack.org/60809923:20
*** aojea has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!