Thursday, 2019-08-01

openstackgerritAdam Harwell proposed openstack/python-octaviaclient master: Add l7policy and l7rule to octavia quota  https://review.opendev.org/59156800:00
openstackgerritAdam Harwell proposed openstack/octavia master: Add support for the oslo_middleware http_proxy_to_wsgi  https://review.opendev.org/63973600:03
openstackgerritAdam Harwell proposed openstack/octavia master: Add support for oslo_middleware http_proxy_to_wsgi  https://review.opendev.org/63973600:11
rm_workfixed a couple of those patches00:15
rm_workand you could do another review pass00:15
rm_workfor example https://review.opendev.org/#/c/667316/ is good now I think00:15
rm_workand the ones I just proposed00:15
rm_workI think the authors were kinda hit-and-run so I fixed them up00:15
rm_workjohnsom: ^^00:16
rm_workvoted +1 on your release00:17
*** ivve has quit IRC00:24
johnsomOk, thanks00:31
*** abaindur has joined #openstack-lbaas00:32
*** abaindur has quit IRC00:45
*** abaindur has joined #openstack-lbaas00:46
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Add l7policy and l7rule to octavia quota  https://review.opendev.org/59156800:48
johnsomSo all of those iptables act/stdby gates are broken somehow:00:53
johnsomhttps://review.opendev.org/#/c/673909/00:53
johnsomspare pool too00:53
johnsomI need to sign off for at list a bit, so can't look into them00:55
*** ricolin has joined #openstack-lbaas01:04
*** happyhemant has quit IRC01:35
openstackgerritMerged openstack/octavia master: Fix flavor profile API handling of None updates  https://review.opendev.org/66234401:43
*** abaindur has quit IRC01:53
*** ramishra has joined #openstack-lbaas02:41
*** hongbin has joined #openstack-lbaas03:45
*** hongbin has quit IRC03:46
*** psachin has joined #openstack-lbaas03:55
*** ramishra has quit IRC04:17
openstackgerritMerged openstack/octavia master: Add support for oslo_middleware http_proxy_to_wsgi  https://review.opendev.org/63973604:21
*** ramishra has joined #openstack-lbaas04:30
*** goldyfruit has joined #openstack-lbaas04:59
*** goldyfruit has quit IRC05:04
*** gcheresh has joined #openstack-lbaas05:38
*** gcheresh_ has joined #openstack-lbaas05:43
*** gcheresh has quit IRC05:43
*** abaindur has joined #openstack-lbaas05:48
*** abaindur has quit IRC05:50
*** abaindur has joined #openstack-lbaas05:57
*** maciejjozefczyk has joined #openstack-lbaas05:59
openstackgerritMerged openstack/octavia master: Build amphora image using centos-minimal element  https://review.opendev.org/61230906:13
*** takamatsu has joined #openstack-lbaas06:13
*** henriqueof has joined #openstack-lbaas06:17
*** ricolin_ has joined #openstack-lbaas06:26
*** ricolin has quit IRC06:29
*** maciejjozefczyk_ has joined #openstack-lbaas07:05
*** maciejjozefczyk has quit IRC07:08
cgoncalvesjohnsom, the iptables act/stdby jobs failed all for different reasons (teardown, lb connect timeout, LB create error). rocky and queens jobs are okay. same for spare pool07:09
*** pcaruana has quit IRC07:12
*** rpittau|afk is now known as rpittau07:13
*** abaindur has quit IRC07:14
*** tesseract has joined #openstack-lbaas07:20
*** pcaruana has joined #openstack-lbaas07:42
*** lemko has joined #openstack-lbaas07:48
*** ivve has joined #openstack-lbaas07:51
*** goldyfruit has joined #openstack-lbaas08:04
*** goldyfruit has quit IRC08:09
*** happyhemant has joined #openstack-lbaas08:09
*** maciejjozefczyk_ is now known as maciejjozefczyk08:17
*** tkajinam has quit IRC08:27
*** ivve has quit IRC08:30
*** ivve has joined #openstack-lbaas08:40
*** psachin has quit IRC09:00
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: test centos-minimal++  https://review.opendev.org/66484009:29
*** yamamoto has joined #openstack-lbaas09:36
*** ivve has quit IRC09:37
*** goldyfruit has joined #openstack-lbaas09:55
*** sapd1_x has joined #openstack-lbaas09:59
*** goldyfruit has quit IRC10:00
*** rpittau is now known as rpittau|bbl10:04
*** yamamoto has quit IRC10:07
*** lemko has quit IRC10:11
openstackgerritMerged openstack/neutron-lbaas stable/queens: Allow to create loadbalancer with chinese character  https://review.opendev.org/66306710:14
*** ricolin__ has joined #openstack-lbaas10:28
*** yamamoto has joined #openstack-lbaas10:30
*** yamamoto has quit IRC10:30
*** ricolin_ has quit IRC10:31
*** yamamoto has joined #openstack-lbaas10:51
*** yamamoto has quit IRC10:55
*** rcernin has quit IRC10:58
*** ivve has joined #openstack-lbaas11:07
*** yamamoto has joined #openstack-lbaas11:10
*** goldyfruit has joined #openstack-lbaas11:14
*** goldyfruit has quit IRC11:21
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add missing keystoneauth1 to requirements.txt  https://review.opendev.org/67400911:31
*** ivve has quit IRC11:37
cgoncalveshmm, I'm seeing now all but -queens act-stby jobs in https://review.opendev.org/#/c/664840/ failing. all at tear down -- lb as children11:48
cgoncalvess/as/has/11:48
*** boden has joined #openstack-lbaas11:54
*** mloza has quit IRC11:55
cgoncalvesFYI, gthiemonge (thanks!) pointed me to https://logs.opendev.org/40/664840/9/check/octavia-v2-act-stdby-iptables-dsvm-scenario/113dea8/controller/logs/screen-o-cw.txt.gz?level=WARNING#_Aug_01_10_24_44_80085212:55
cgoncalveslistener delete on active-standby is definitely broken12:55
*** rpittau|bbl is now known as rpittau12:57
*** yamamoto has quit IRC13:21
*** AustinR has quit IRC13:25
*** AustinR has joined #openstack-lbaas13:41
*** yamamoto has joined #openstack-lbaas13:43
*** spatel has joined #openstack-lbaas14:06
*** lemko has joined #openstack-lbaas14:08
*** mkuf has joined #openstack-lbaas14:16
*** Vorrtex has joined #openstack-lbaas14:33
*** mkuf_ has joined #openstack-lbaas14:53
*** mkuf has quit IRC14:54
*** mkuf has joined #openstack-lbaas14:59
*** mkuf_ has quit IRC14:59
*** yamamoto has quit IRC15:02
*** yamamoto has joined #openstack-lbaas15:04
*** yamamoto has quit IRC15:04
*** yamamoto has joined #openstack-lbaas15:05
*** maciejjozefczyk has quit IRC15:07
straldihi all there is somewhere I could found a documentation to install octavia in an openstack rocky-release installed in a CentOS O. S.?15:08
*** yamamoto has quit IRC15:09
straldihere https://docs.openstack.org/octavia/pike/contributor/guides/dev-quick-start.html15:10
straldiI can find just or using dev-stack in ubuntu or just the things to do without real commands tested.15:10
*** Vorrtex has quit IRC15:12
*** yamamoto has joined #openstack-lbaas15:13
*** yamamoto has quit IRC15:13
*** yamamoto has joined #openstack-lbaas15:14
*** yamamoto has quit IRC15:18
*** gcheresh_ has quit IRC15:21
*** Vorrtex has joined #openstack-lbaas15:25
*** gthiemon1e has joined #openstack-lbaas16:00
*** gthiemon1e has quit IRC16:01
openstackgerritGregory Thiemonge proposed openstack/octavia master: Fix wrong package names or versions for centos-minimal images  https://review.opendev.org/67408716:36
*** rpittau is now known as rpittau|afk16:40
*** lemko has quit IRC16:47
*** sapd1_x has quit IRC16:49
openstackgerritGregory Thiemonge proposed openstack/octavia master: Fix wrong package names or versions for centos-minimal images  https://review.opendev.org/67408716:53
*** ivve has joined #openstack-lbaas16:55
cgoncalvesstraldi, hi there. there's a patch open to add an install guide for Ubuntu -- https://review.opendev.org/#/c/672842/16:55
cgoncalvesstraldi, it should be relatively simple to follow it on CentOS. the difference should be setting up the package repository pointing to RDO Rocky (if you want to install from RPMs) and doing 'yum install' instead of 'apt install'16:56
cgoncalvesstraldi, the RDO Rocky you could use is https://trunk.rdoproject.org/centos7-rocky/current-tripleo-rdo/16:58
cgoncalvesthe yum repo file to place in /etc/yum.repos.d: https://trunk.rdoproject.org/centos7-rocky/current-tripleo-rdo/delorean.repo16:59
cgoncalvesstraldi, it would be great if you later could share the steps you took to install it on CentOS :) the community wuold appreciate it much17:01
*** AustinR has quit IRC17:01
*** AustinR has joined #openstack-lbaas17:07
*** ramishra has quit IRC17:18
*** ricolin__ is now known as ricolin17:30
*** AustinR has quit IRC17:32
*** AustinR has joined #openstack-lbaas17:33
*** abaindur has joined #openstack-lbaas17:52
*** yamamoto has joined #openstack-lbaas17:57
*** abaindur has quit IRC17:59
*** AustinR has quit IRC18:02
*** AustinR has joined #openstack-lbaas18:04
*** yamamoto has quit IRC18:04
colin-is there anywhere i can read more about what sort of config this api accepts and in what format? https://docs.openstack.org/api-ref/load-balancer/v2/index.html?expanded=configure-amphora-detail#configure-amphora18:25
colin-if it helps, one operation that comes to mind is that let's say i want to overwrite the 'controller_ip_port_list' in the amphora-agent.conf18:26
*** Vorrtex has quit IRC18:27
colin-also, unrelated, any tips for easy ways to locally override the keystone catalog to send requests through a development instance of a given service? not finding much related to openstackclient about that18:35
colin-disregard my last dns should make that trivial18:42
*** Vorrtex has joined #openstack-lbaas18:43
*** AustinR has quit IRC18:52
*** abaindur has joined #openstack-lbaas18:52
*** abaindur has quit IRC18:52
*** abaindur has joined #openstack-lbaas18:53
johnsomcolin- So, the configure-amphora doesn't accept any parameters, it will update any config setting that is "mutable" https://opendev.org/openstack/octavia/src/branch/master/octavia/common/config.py#L24418:53
*** AustinR has joined #openstack-lbaas18:53
johnsomI guess it would be "nice" to comment these in this file: https://opendev.org/openstack/octavia/src/branch/master/octavia/amphorae/backends/agent/templates/amphora_agent_conf.template18:54
johnsomBonus points to you if you add the "  # mutable" to that file.18:54
johnsomcolin- You can also override the endpoints in the octavia.conf18:55
johnsomhttps://opendev.org/openstack/octavia/src/branch/master/etc/octavia.conf#L42318:55
colin-got it, was thinking more for getting traffic from a local openstack client to the local copy of the api service19:02
johnsomcolin- Oh, yeah, there are command line flags and/or environment variables for the OpenStack client19:21
*** tesseract has quit IRC19:23
cgoncalvesloadbalancer_topology is mutable? how would that work on change?19:26
johnsomThe "default" topology would be the new value for all newly create load balancers.19:28
cgoncalvesright. say there's a LB running on SINGLE. the operator changes default topology to active-standby in controller and then calls that amphora API19:31
colin-how does one propose changes on that site? it seems read only with no login option19:39
colin-funny considering the topic of the change to the doc is mutable related19:40
johnsomcolin- That is just the Octavia git repo in a web UI19:56
colin-so normal review site, ok19:57
johnsomcgoncalves You have good point there.19:58
*** ltomasbo has quit IRC20:02
*** dalvarez has quit IRC20:03
*** colby_ has joined #openstack-lbaas20:17
colby_Hey Everyone. I have a question. We upgraded to queens and are now able to use the pkcs12 certs. It looks like you dont need to set acl any more on a conatiner now. Seems like you can add the cert via command line but the dashboard does not see it. Do you still need to add acl for the dashboard to see the SSL cert?20:18
colby_sorry upgraded to rocky I meant20:19
johnsomcolby_ Can you check that you have this patch? https://review.opendev.org/#/c/641577/20:20
colby_Im checking now20:23
johnsomIt kind of looks like we are behind on doing bug fix releases of dashboard.20:24
colby_yea I dont have that. Im using RPM releases and Im on 2.0.0-120:24
johnsomYeah, it looks like there are some un-released bug fixes on both rocky and queens.20:25
johnsomOur stable release guy just logged off for the night. Let me see if I can get those releases rolling. Though it will probably be a while on the RPM getting updated.20:26
johnsomIt's a pretty small fix, you could try it by hand updating.20:27
colby_yea Im doing manual patch to test20:30
colby_hmm I still only see a cert from when I did this on the old version using the acl user add. I dont see my new pkcs12 cert I uploaded to barbican20:37
johnsomhmm, rm_work do you know if dashboard got updated for the pkcs12 bundles?20:39
*** Vorrtex has quit IRC20:40
*** spatel has quit IRC20:41
*** boden has quit IRC20:41
*** AustinR has quit IRC21:03
*** AustinR has joined #openstack-lbaas21:22
abaindurhello, im wondering how and if Octavia handles this issue. The fix was reverted, so original issue still exists:21:24
abaindurhttps://bugs.launchpad.net/neutron/+bug/160840021:24
openstackLaunchpad bug 1608400 in neutron "Neutron should not add ARP entry for allowed-address-pair-fixed-ip in DVR router" [High,Fix released] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan)21:24
abaindurthen reverted in https://bugs.launchpad.net/neutron/+bug/177399921:25
openstackLaunchpad bug 1773999 in neutron "Allowed Address Pairs doesn’t work after neutron-port update" [Undecided,Fix released]21:25
abaindurIt seems like when a VIP failsover, a GARP is sent. Which is fine for L2. But the static ARP entry in a neutron router is not updated21:25
abaindurIt was then reverted per second bug, because "it was possible to mitigate the issue of allowed-address pairs and DVR by neutron-port update."21:26
johnsomabaindur There are a lot of bugs open against DVR. We work around that one by agressively GARP-ing the VIP address if we are running in Active/Standby mode (thus multiple ports).21:26
abaindurDoes Octavia issue a neutron port update everytime a failover from active to standby occurs?21:26
abaindurGARP won't fix the *static* ARP entry inside a qrouter namespace21:27
johnsomThe fact that DVR ignores GARP updates is a known bug in DVR. It was being worked on in Stein, but I'm not sure if they merged or not.21:27
abaindurSo basically - Active/Standby Failover (or really, any VIP moving from one port to another where its an allowed addr pair) - is broken?21:29
johnsomNo, DVR is broken21:30
openstackgerritMerged openstack/octavia master: Add failover logging to show the amphora details.  https://review.opendev.org/66731621:30
johnsomThe tracking bug for DVR is here: https://bugs.launchpad.net/neutron/+bug/177445921:30
openstackLaunchpad bug 1774459 in neutron "Update permanent ARP entries for allowed_address_pair IPs in DVR Routers" [High,In progress] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan)21:30
johnsomThis impacts anyone doing any activity that updates ARP tables using GARP21:31
johnsomIf you are running DVR21:31
abaindurright, i didnt mean issue is w/ octavia, just that this is an example of where Octavia would break21:32
johnsomMy understanding is that it does work, except that floating IPs pointing to the VIP will not work.21:32
abaindurYea that issue you linked seems like what we were seeing21:32
abaindurrather seems exatly like the first bug, which was committed then reverted21:33
johnsomYeah, this issue has been open in one bug or another since DVR was introduced21:34
abaindurand with a non-DVR router i assume this is not an issue?21:40
johnsomCorrect, to my knowledge it is only the DVR driver that has a failure in this situation.21:43
rm_workHmm not sure about the dashboard :(21:43
abaindurjohnsom: Do you think another potential way Octavia could handle this is by triggering a port update whenever a failure occurs?21:49
abaindurThe bug that reverted the first attempt at fixing it seems to think so https://bugs.launchpad.net/neutron/+bug/177399921:49
openstackLaunchpad bug 1773999 in neutron "Allowed Address Pairs doesn’t work after neutron-port update" [Undecided,Fix released]21:49
johnsomabaindur We looked at that, it is so slow it defeats the purpose of using VRRP.21:49
abainduralso, we tried the same - updating the bound port of the LB VM by just admin shutting it on/off, then the static ARP entries were updated21:50
abaindurand fixed our issue. I'm thinking if Octavia did this each time?21:50
johnsomWe would not be able to failover in a second or less in that case. Really, we want to failover inside the TCP retry window. Round trips to the async neutron API for port actions, is slow....21:51
abaindurI guess you're right.... at that point in time our VIP was basically unreachable21:55
*** henriqueof has quit IRC22:07
*** henriqueof has joined #openstack-lbaas22:07
johnsomI encourage you to ask about the DVR issues in the neutron channel. Help raise awareness that it needs to get fixed. They are tired of hearing from me. grin22:07
openstackgerritMichael Johnson proposed openstack/octavia master: Add long-running provider agent support  https://review.opendev.org/67414022:24
johnsomThat appears to be functional, but I still need to do docs, tests, release notes22:24
*** fyx_ has joined #openstack-lbaas22:30
*** jrosser_ has joined #openstack-lbaas22:30
*** fyx has quit IRC22:37
*** jrosser has quit IRC22:37
*** fyx_ is now known as fyx22:37
*** jrosser_ is now known as jrosser22:37
openstackgerritMichael Johnson proposed openstack/octavia master: Add get method support to the driver-agent  https://review.opendev.org/66502922:48
openstackgerritMichael Johnson proposed openstack/octavia master: Move to using octavia-lib constants  https://review.opendev.org/67371222:48
openstackgerritMichael Johnson proposed openstack/octavia master: Add long-running provider agent support  https://review.opendev.org/67414022:48
*** tkajinam has joined #openstack-lbaas22:51
rm_workjohnsom / abaindur: that is basically what my code did for my L3 driver, and yes, it was SLOW. ~10s failovers. That said, it might be ok as a workaround for *in addition to* the ARPing, since 10s is much better than *never*. :D22:51
rm_workhttps://review.opendev.org/#/c/435612/154/octavia/amphorae/backends/health_daemon/health_daemon.py@6822:52
rm_workbroadcast the "help me" alert to every HM22:53
rm_workand one of them would handle it22:53
abaindurIs that fix in?22:53
johnsomIt's not a fix22:53
abaindur"workaround" but yea... this DVR issue seems to be a blocker for us in using any kind of lbaas22:53
johnsomIt's an alternate idea22:54
johnsomStandalone topology works. It also works fine if you put the VIPs on a non-DVR network22:54
johnsomabaindur I am assuming you have tested this right? I remember Swami telling me they had a workaround in DVR for this and were working on the *right* solution now.22:57
openstackgerritMichael Johnson proposed openstack/octavia master: Add get method support to the driver-agent  https://review.opendev.org/66502922:58
openstackgerritMichael Johnson proposed openstack/octavia master: Move to using octavia-lib constants  https://review.opendev.org/67371222:58
abaindurrm_work: is https://review.opendev.org/#/c/435612 to basically allow us to address amphora by a floating IP, rather than directly by fixed IP?22:58
abaindurso the VIP port is associated with a floating IP?22:59
abaindurerr not VIP port22:59
abaindurbut the amphora ports on VIP subnet have floating IPs?22:59
rm_workyes22:59
rm_worki was using that driver for about a year in production22:59
rm_workextensive testing on failover times in our environment showed an average of ~10s23:00
abaindurah yea, i think that would be useful as well. since we need 2 way connectivity between amphora and hosts where octavia runs23:00
rm_workbasically the VIP port *is* a floating-ip in that driver23:00
rm_workalso, that timing is really down to whatever the time for your neutron floating-ip associate takes, so it COULD be faster, our environment it was shitty23:02
rm_workthis driver also works for failover in cases where all of octavia is not in a single L2 (because the VRRP failovers only work if both hosts are in the same L2) which a ton of larger deployments don't guarantee because one L2 doesn't scale up very well23:03
rm_workI tried to drum up some support for getting it merged but there was not enough interest23:03
rm_workat this point it needs some major rebasing work to be able to be used23:03
*** ivve has quit IRC23:05
johnsomrm_work Are you looking at that dashboard pkcs12 thing or should I open a bug for it?23:09
openstackgerritMichael Johnson proposed openstack/octavia master: Add get method support to the driver-agent  https://review.opendev.org/66502923:09
*** ianychoi has quit IRC23:12
rm_worki was not23:12
rm_worki don't UI23:12
rm_workhow does frontend code, what is it23:13
rm_worki don't even23:13
johnsomI don't think it's UI, I think it's the barbican client code23:13
rm_workorly?23:13
johnsomIt is this, very non-UI file: https://github.com/openstack/octavia-dashboard/blob/master/octavia_dashboard/api/rest/barbican.py23:13
openstackgerritMichael Johnson proposed openstack/octavia master: Move to using octavia-lib constants  https://review.opendev.org/67371223:14
johnsomI am obviously having a rebase-big-chain-inept-afternoon23:14
*** rcernin has joined #openstack-lbaas23:21
openstackgerritMichael Johnson proposed openstack/octavia master: Add get method support to the driver-agent  https://review.opendev.org/66502923:53
*** trident has quit IRC23:54
*** trident has joined #openstack-lbaas23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!