Tuesday, 2018-09-11

« Monday, 2018-09-10 Index Wednesday, 2018-09-12 »
*** yamamoto has quit IRC00:09
*** Swami has quit IRC00:18
*** abaindur has quit IRC00:43
*** abaindur has joined #openstack-lbaas00:44
*** abaindur_ has joined #openstack-lbaas00:47
*** abaindur has quit IRC00:48
*** sanfern has joined #openstack-lbaas00:51
*** abaindur_ has quit IRC01:09
sapd1rm_work: I fixed unit test. So It passes all test now.01:40
*** yamamoto has joined #openstack-lbaas01:51
openstackgerritsapd proposed openstack/octavia master: Support REDIRECT_PREFIX action for L7Policy  https://review.openstack.org/60108601:52
openstackgerritYang JianFeng proposed openstack/octavia master: Add quota support to octavia's l7policy and l7rule  https://review.openstack.org/59062002:15
*** dmellado has quit IRC02:18
openstackgerritYang JianFeng proposed openstack/octavia master: [WIP] Refactor 'check_quota_met' and 'decrement_quota'  https://review.openstack.org/59666502:39
*** rcernin has quit IRC02:47
*** rcernin has joined #openstack-lbaas02:48
*** rcernin has quit IRC02:48
*** rcernin has joined #openstack-lbaas02:49
*** sanfern has quit IRC02:53
openstackgerritYang JianFeng proposed openstack/octavia master: [WIP] Refactor 'check_quota_met' and 'decrement_quota'  https://review.openstack.org/59666502:55
openstackgerritCarlos Goncalves proposed openstack/octavia master: Make health checks resilient to DB outages  https://review.openstack.org/60087603:01
cgoncalvesrm_work, FYI: http://paste.openstack.org/show/729841/03:05
cgoncalves29 JOINs03:05
*** phuoc has joined #openstack-lbaas03:06
*** phuoc_ has quit IRC03:08
*** phuoc_ has joined #openstack-lbaas03:08
*** phuoc has quit IRC03:11
*** reedipb has quit IRC04:06
*** sanfern has joined #openstack-lbaas04:15
*** ramishra has joined #openstack-lbaas04:22
*** fnaval has quit IRC04:34
*** threestrands has joined #openstack-lbaas04:34
*** threestrands has quit IRC04:34
*** threestrands has joined #openstack-lbaas04:34
johnsomJust a reminder, most of the core team is at the PTG this week and are making the most of the time by collaborating with other project teams. Answers to questions and reviews may be delayed.04:38
johnsomabaindur: 1 yes, 2 yes via the housekeeping process, 3 yes - following a normal renewal process that should be true, 4 there is a discussion of this in the maintenance guide in our docs. I think again, if you do a proper renewal I think you can continue to operate. However this is renewing the certs issued by the CAs, the CAs themselves may have other implications.04:44
sapd1johnsom: Could you discuss about redirect prefix which I implemented.04:59
*** pcaruana has joined #openstack-lbaas05:00
*** reedipb has joined #openstack-lbaas05:09
*** pcaruana has quit IRC05:09
*** yamamoto has quit IRC05:31
openstackgerritYang JianFeng proposed openstack/octavia master: Refactor 'check_quota_met' and 'decrement_quota'  https://review.openstack.org/59666505:37
openstackgerritYang JianFeng proposed openstack/octavia master: Refactor 'check_quota_met' and 'decrement_quota'  https://review.openstack.org/59666505:59
*** AlexeyAbashkin has joined #openstack-lbaas06:06
*** yamamoto has joined #openstack-lbaas06:09
*** AlexeyAbashkin has quit IRC06:10
*** yamamoto has quit IRC06:26
*** annp has joined #openstack-lbaas06:31
*** Emine has quit IRC06:32
openstackgerritYang JianFeng proposed openstack/octavia master: Add compute_flavor field for amphora api  https://review.openstack.org/58291406:38
openstackgerritYang JianFeng proposed openstack/python-octaviaclient master: Add l7policy and l7rule to octavia quota  https://review.openstack.org/59156806:39
*** velizarx has joined #openstack-lbaas06:44
openstackgerritYang JianFeng proposed openstack/octavia master: [WIP] Add listener and pool protocol validation.  https://review.openstack.org/59404006:47
*** tesseract has joined #openstack-lbaas06:51
openstackgerritYang JianFeng proposed openstack/octavia master: [WIP] Add listener and pool protocol validation.  https://review.openstack.org/59404007:00
*** rcernin has quit IRC07:08
*** pcaruana has joined #openstack-lbaas07:13
*** dmellado has joined #openstack-lbaas07:16
*** ccamposr has joined #openstack-lbaas07:18
openstackgerritYang JianFeng proposed openstack/octavia master: Add listener and pool protocol validation.  https://review.openstack.org/59404007:27
*** velizarx has quit IRC07:30
*** yamamoto has joined #openstack-lbaas07:40
*** velizarx has joined #openstack-lbaas07:42
*** hvhaugwitz has quit IRC07:45
*** hvhaugwitz has joined #openstack-lbaas07:45
*** velizarx has quit IRC07:48
*** velizarx has joined #openstack-lbaas07:54
*** luksky has joined #openstack-lbaas07:55
*** AlexeyAbashkin has joined #openstack-lbaas08:01
*** yamamoto has quit IRC08:05
*** velizarx has quit IRC08:16
openstackgerritYang JianFeng proposed openstack/octavia master: Add listener and pool protocol validation.  https://review.openstack.org/59404008:17
*** velizarx has joined #openstack-lbaas08:20
openstackgerritYang JianFeng proposed openstack/octavia master: Add listener and pool protocol validation.  https://review.openstack.org/59404008:24
*** threestrands has quit IRC08:30
*** Emine has joined #openstack-lbaas08:31
openstackgerritYang JianFeng proposed openstack/octavia master: Add listener and pool protocol validation.  https://review.openstack.org/59404008:41
*** fnaval has joined #openstack-lbaas09:07
*** fnaval has quit IRC09:11
*** huseyin has joined #openstack-lbaas09:16
*** huseyin has left #openstack-lbaas09:17
*** Huseyin_ has joined #openstack-lbaas09:19
openstackgerritYang JianFeng proposed openstack/octavia master: Add listener and pool protocol validation.  https://review.openstack.org/59404009:19
*** huseyin has joined #openstack-lbaas09:20
*** Huseyin_ has quit IRC09:20
*** crazik has joined #openstack-lbaas09:37
*** yamamoto has joined #openstack-lbaas09:37
crazikhello09:37
crazikI have a question: how to prepare to the planned network maintenance, to avoid amphora recreation? Can I simply disable some octavia services to avoid such events (in case of missed healthchecks)?09:39
crazikI had bad experience on DB connectivity issues09:39
*** tesseract has quit IRC09:46
*** tesseract has joined #openstack-lbaas09:50
*** tesseract has quit IRC09:50
*** tesseract has joined #openstack-lbaas09:52
*** fnaval has joined #openstack-lbaas10:07
*** tesseract has quit IRC10:11
*** fnaval has quit IRC10:11
*** yamamoto has quit IRC10:13
*** sanfern has quit IRC10:19
*** sanfern has joined #openstack-lbaas10:20
*** rpittau has joined #openstack-lbaas10:20
*** rpittau has quit IRC10:21
*** rpittau has joined #openstack-lbaas10:21
*** sanfern has quit IRC10:28
*** yamamoto has joined #openstack-lbaas11:06
*** velizarx has quit IRC11:13
*** yamamoto has quit IRC11:13
*** sanfern has joined #openstack-lbaas11:23
*** annp has quit IRC11:23
*** velizarx has joined #openstack-lbaas11:26
*** yamamoto has joined #openstack-lbaas11:29
*** yamamoto has quit IRC12:06
*** velizarx has quit IRC12:07
*** fnaval has joined #openstack-lbaas12:07
*** velizarx has joined #openstack-lbaas12:10
*** fnaval has quit IRC12:11
*** reedipb has quit IRC12:20
*** yamamoto has joined #openstack-lbaas12:33
*** velizarx has quit IRC12:53
*** velizarx has joined #openstack-lbaas12:55
*** jlaffaye_ has joined #openstack-lbaas13:00
jlaffaye_hello, what is the difference between the service_auth and keystone_authtoken sections of the octavia conf file ?13:01
johnsomcrazik: yes, just stop the health manager processes and resume after your maintenance13:04
crazikoh, thank you johnsom :)13:05
crazikanyone did octavia upgrade from 2.0.1 to 2.0.2?13:05
crazik;)13:05
johnsomjlaffaye_: service_auth is the keystone credential octavia uses when accessing other services, such as nova and neutron.  Keystone_authtoken is used to verify user tokens.13:06
*** fnaval has joined #openstack-lbaas13:07
johnsomcrazik: Carlos has a patch posted to address some DB issue scenarios.  Please review it for him.13:08
*** fnaval has quit IRC13:11
jlaffaye_johnsom: thanks, I guess I was confused because the octavia::keystone::authtoken puppet module takes a user/pass13:22
crazikjohnsom: any url for this patch?13:22
*** reedipb has joined #openstack-lbaas13:23
jlaffaye_so I have a lb stuck in PENDING_DELETE because I messed up with oslo messaging config, what can I do as an operator to fix this king of thing ?13:39
*** fnaval has joined #openstack-lbaas13:44
johnsomcrazik https://review.openstack.org/#/c/600876/14:02
johnsomjlaffaye_ PENDING_* states means a controller has ownership of the object. You can wait for that controller to timeout attempting retries, then it will go to ERROR and be deletable, etc.14:03
jlaffaye_well it appears to be idle after an exception in glanceclient and the state is still PENDING14:07
crazikhmm14:21
crazikyeah, the same thing with amqp probably is needed.14:23
*** yamamoto has quit IRC14:27
*** yamamoto has joined #openstack-lbaas14:28
*** yamamoto has quit IRC14:28
*** yamamoto has joined #openstack-lbaas14:28
*** yamamoto has quit IRC14:33
*** sapd1_ has joined #openstack-lbaas14:44
*** pcaruana has quit IRC14:47
jitekahello, I did some testing today on spare pool feature with our environment with octavia queens and I realise that,14:48
jitekanew LB are not using READY amphora but always booting new amphora VMs.14:48
jitekaSpare amphora seems to be only used in case of faillover14:48
jiteka1. Is it the expected behaviour ?14:48
jiteka2. Is it behaving the same when not using ACTIVE_STANDBY topology ?14:48
jiteka3. Is that behaviour is changing in Rocky release ?14:48
jitekaNote : actually in my last test, even failover was creating new amphora instead of using the spare ones14:48
jitekahttp://paste.openstack.org/show/729880/14:48
johnsomYes, if you have anti-affinity enabled, spares pool (READY) amps will be bypassed14:51
jitekajohnsom: ok it makes sense with what I read in the operator guide then14:53
johnsomYeah, we can't add existing instances to nova server groups.  We could probably get creative, but we also don't want to cross the line into becoming a nova scheduler...  sigh14:54
jitekajohnsom: was thinking about something creative like a reconciliator service that periodically audit the amphora fleet cluster by cluster and detect the one running on the same compute to trigger some live-migration14:56
jitekabut... it's far from a small config change14:56
jitekaand not sure it would scale really good on a large number of LB14:57
huseyinHello, does anyone have an idea about the error “ERROR octavia.controller.worker.controller_worker TemplateSyntaxError: expected token 'end of statement block', got '.'” in the worker logs?14:58
huseyinAccording to this bug report : https://bugzilla.redhat.com/show_bug.cgi?id=1551821 it is about Jinja2 version14:58
openstackbugzilla.redhat.com bug 1551821 in openstack-octavia "Octavia requires jinja2 2.10" [High,Closed: errata] - Assigned to cgoncalves14:58
huseyinBu after upgrading to Jinja2 2.10 the problem still exists14:59
huseyinpip show Jinja215:00
huseyinName: Jinja215:00
huseyinVersion: 2.1015:00
huseyinSummary: A small but fast and easy to use stand-alone template engine written in pure python.15:00
huseyinHome-page: http://jinja.pocoo.org/15:00
huseyinAuthor: Armin Ronacher15:00
huseyinAuthor-email: armin.ronacher@active-4.com15:00
huseyinLicense: BSD15:00
huseyinLocation: /usr/local/lib/python2.7/dist-packages15:00
huseyinRequires: MarkupSafe15:00
huseyinRequired-by: octavia, Flask15:00
huseyinAny other idea?15:03
*** luksky has quit IRC15:05
sapd1_huseyin: I think because your image is built by source from Master branch. So it will be conflict version with your octavia-worker. So I think you should install octavia-* rocky branch.15:06
johnsomDid you customize the haproxy templates or are you running the stock templates?15:07
huseyinsapd1_: Actually the amphora image is the one that I created while using ocata15:08
huseyinjohnsom: No I did not15:08
huseyinIs it required to create a new image for queens?15:08
sapd1_huseyin: Yep. Please build a new one.15:09
johnsomOk, the Ocata image version likely has the wrong jinja version in it if you are running a newer control plane. We would have to check the release notes15:09
huseyinOh OK. Thanks a lot. I will create a new one immediately15:09
*** josephrsandoval has joined #openstack-lbaas15:10
jitekathanks for the help johnsom :)15:10
jitekaafter chaning "enable_anti_affinity" to false, it use the spare amps15:11
*** yamamoto has joined #openstack-lbaas15:16
*** fnaval has quit IRC15:30
*** phuoc has joined #openstack-lbaas15:31
*** phuoc_ has quit IRC15:33
*** bcafarel has joined #openstack-lbaas15:37
*** fnaval has joined #openstack-lbaas15:37
*** Emine has quit IRC15:44
openstackgerritCarlos Goncalves proposed openstack/octavia master: Make health checks resilient to DB outages  https://review.openstack.org/60087615:46
huseyinthanks sapd1_ and johnsom15:50
huseyincreating a new amphora image solved the problem15:50
huseyinnew image uses Jinja 2.10 btw15:50
*** nmanos has joined #openstack-lbaas15:54
sapd1_huseyin: are you running octavia for production or test only?15:59
huseyinsapd1_: I am testing now and planning to use in production after the tests16:01
huseyinsapd1_: I started to test while using ocata but some packages like barbicanclient has problems16:02
huseyinsapd1_: so I upgraded to queens last week, and running tests now16:03
*** huseyin has left #openstack-lbaas16:03
*** huseyin has joined #openstack-lbaas16:03
sapd1_huseyin: have you benchmark your LB yet16:04
huseyinsapd1_: no I didn’t16:04
sapd1_I concern about performance.16:05
huseyinsapd1_: what do you mean by concern? Could you please explain a bit more?16:06
*** ramishra has quit IRC16:07
huseyinoctavia is the default implementation for the openstack as of pike release, isn’t it?16:07
huseyinfor lbaas service I mean16:07
johnsomYes it is16:07
sapd1_huseyin: Because currently octavia run only one haproxy process. So if you running on multi core. It does not increase performance . :D16:08
johnsomsapd1_ I think you will be surprised at the performance. I have seen the underlying cloud networking performance is more of a limiting factor that Octavia.16:08
huseyinwhat kind of performance problems did you find?16:08
johnsomI have been able to do 14.5Gbps with a 1vcpu amp16:09
sapd1_huseyin: maximum new http request. :D16:09
johnsomBut those numbers are super dependent on your cloud platform and application....  So, your milage will vary16:09
johnsomI have seen around 33,000 reqs per second when passing actual traffic.16:10
sapd1_johnsom: Yes. I try with SR-IOV so It's higher than OVS16:10
sapd1_what does req/s mean? How did you monitor it?16:10
johnsomYeah, there wasn't SR-IOV or OVS in that deployment16:11
johnsomUsing benchmarking tools, such as Tsung, weighttp, etc.16:11
huseyinThat sounds great16:11
johnsomFar surpasses what I see most clouds being able to handle.16:12
*** velizarx has quit IRC16:12
sapd1_johnsom: how many members in a pool? How many clients did you use in that test?16:12
johnsomThree member servers, 10,000 "users"16:13
johnsomBut again, please note, this is HIGHLY dependent on your cloud and application.16:14
sapd1_yep. I just use ab and apache2 as web server backend. And "hello world" text only for benchmark16:14
johnsomThis is why we don't quote performance, many clouds cannot achieve that due to the cloud configuration, etc.16:14
johnsomAh, yeah, that will not really stress Octavia.16:15
johnsomYou will hit limits in ab and apache2 before you stress Octavia16:15
huseyinjohnsom: is there any plan to backup amp with keepalived for the redundancy?16:15
sapd1_huseyin: It has already implemented.16:16
johnsomhuseyin Built in since Mitaka16:16
sapd1_johnsom: Let me try tsung and weighttp tomorrow. You only run 1 vcpu amp to get 33k req/s, aren't you?16:17
johnsomCorrect16:17
sapd1_unbelievable16:17
johnsomsapd_1 Grab my performance patch I posted too16:17
sapd1_johnsom: where I can find that patch? :D16:18
*** huseyin has quit IRC16:18
johnsomsapd1_ https://review.openstack.org/59837916:19
sapd1_johnsom: Next week I will run Octavia in production. :|16:19
johnsomNice16:20
johnsomWelcome to the club! grin16:20
sapd1_My boss want LB get higher performance I mean higher 6k req/s :D16:21
johnsomsapd1_ We do that easily today, out of the box16:21
sapd1_oh. 6k req/s I collect from haproxy socket. (rate - measurements)16:21
johnsomJust check your cloud networking to make sure it can handle that by bypassing Octavia.  That is the biggest issue I see typically, the underlying cloud can't do it16:22
*** hongbin has joined #openstack-lbaas16:22
sapd1_run ab process on three servers client. Each of process can get 2k req/s16:22
johnsomYeah, get a better benchmark tool, and faster web server.16:23
cgoncalvesjohnsom, 14.5Gbps? that's really good. what was the frames size, 1500 bytes? :)16:23
johnsomYes, in TCP mode. Octavia test only, so co-located on the same compute host.16:24
johnsomsapd1_ Since you are doing "hello world" consider an in-memory web service, such as using this handler: https://github.com/perusio/nginx-hello-world-module16:25
sapd1_9Gbps in TCP :D16:25
johnsomsapd1_ the splice settings will help that16:25
*** josephrsandoval has quit IRC16:26
johnsomAlso check that your haproxy enables splice, I saw some distro packages that disabled it on the command line.16:26
sapd1_johnsom:  I saw your patch. It will enable splice in haproxy.16:28
sapd1_johnsom: My backend server is not overload. Because If I stress test direct to backend server (apache2) result is 4k req/s :D16:30
johnsomYeah, that gave me a reasonable BW speed bump.16:30
johnsomsapd_1 Ok, so consider a better benchmark tool as well.  But you aren't going to get 6k/s with a web server only doing 4k.... grin16:31
sapd1_johnsom: I have 10 servers member for that test .16:32
johnsomOk16:32
sapd1_each of them can get 4k req/s :D16:32
sapd1_In the theory, I can get 40k req/s if haproxy can handle it. :D16:33
sapd1_johnsom: Do you think we should use a/a with DNS Round robin? We can create A record in designate for all AMP.16:35
johnsomNo, it adds a huge latency when using DNS16:35
johnsomAnd that breaks a bunch of features as well16:36
sapd1_johnsom: I think ELB/ALB from AWS are using DNS16:37
johnsomYes they are for ELB, ALB is similar to Octavia16:37
johnsomDoesn't mean it's the best solution....16:37
sapd1_I have checked proposal for A/A, which use a director component.16:38
johnsomYes, there are around three proposals for A/A16:39
sapd1_So I have to use vRouter, don't I?16:39
johnsomNo16:39
johnsomThe reference A/A spec has no external resource requirements.16:39
johnsomBut, really do you have a use case that requires a/a?  We find it hard to find16:40
sapd1_I have some customer, they require high performance for Loadbalancer :(16:41
sapd1_s/customer/customers/16:41
johnsomI would be interested to hear what their current rates are.16:42
sapd1_They said they need 20k requests/s.16:43
johnsomI think with Stein work we can reach that without A/a16:43
johnsoma/a target is 100k/s really16:43
sapd1_johnsom: It also depends on my network infrasture which you mentioned  before16:45
sapd1_johnsom: I confused with current connections and new connections in haproxy.16:46
johnsomsapd1_ are you interested in doing development on a/a?16:50
sapd1_I have a proposal for a/a using RNS for my company.16:50
johnsomOk. The reference A/A is stalled with no developers (I had to shift the priority on that down). It's at the stage where it needs some Ryu/Ken development work. The flows are all defined, it's just doing the library coding work.16:52
sapd1_johnsom: Can I join :D16:54
sapd1_johnsom: I don't understand Ryu/Ken mean.16:55
johnsomsapd1_ Of course.  If you have time we can sync up next week.16:55
johnsomsapd1_ Oh, it's an OpenFlow / SDN controller library for python.16:55
johnsomsapd1_ https://osrg.github.io/ryu/16:56
johnsomsapd1_ I thought you were going to work on cinder storage stuff.16:56
johnsomsapd1_ Ryu is becoming neutron ken16:57
sapd1_johnsom: Last month I have to work to release Load balancer for public cloud. So I can't follow that patch. ( Do you mention boot from volume? - I guess)16:58
*** ccamposr has quit IRC16:59
johnsomRight16:59
sapd1_johnsom:  I will re-read it (https://docs.openstack.org/octavia/latest/contributor/specs/version0.9/active-active-topology.html)17:02
*** hongbin_ has joined #openstack-lbaas17:06
*** luksky has joined #openstack-lbaas17:07
*** hongbin has quit IRC17:08
*** Swami has joined #openstack-lbaas17:08
*** AlexeyAbashkin has quit IRC17:09
*** sapd1_ has quit IRC17:13
*** huseyin has joined #openstack-lbaas17:15
*** nmanos has quit IRC17:15
*** hongbin_ has quit IRC17:22
*** hongbin has joined #openstack-lbaas17:27
*** pcaruana has joined #openstack-lbaas17:32
*** hongbin has quit IRC17:46
*** irenab has quit IRC18:02
*** irenab has joined #openstack-lbaas18:03
*** abaindur has joined #openstack-lbaas18:04
abaindurjohnsom: hey, you around?18:04
abaindurI needed some clarification about cert expiry behavior18:05
abaindurrm_work: or maybe you since i see several commits by you related to cert stuf?18:06
*** huseyin has left #openstack-lbaas18:08
abaindur1. If the self-generated amphora server certs expire, octavia takes care of this automatically, injecting it into the amphora using the internal API and req. no intervention on our part, correct?18:09
abaindur2. if the octavia controller's client cert expires, no failover or anything else needs to be done also, right? Besides obviously updating the cert in file on host and restarting services. But no failover or anything else18:09
johnsomabaindur: I answered your questions already.  See eavesdrop.openstack.org if you don’t have the scroll back18:10
abaindurah sorry, i logged off and didnt see em18:10
johnsomJust about to be in another meeting though18:10
johnsomThat site has the channel logs18:11
abaindurhmm i cant find them18:12
abaindurhttp://eavesdrop.openstack.org/irclogs/%23openstack-dns/%23openstack-dns.2018-09-10.log.html18:12
abaindurwhooops18:12
abaindurwrong channel18:12
abaindur:)18:12
rm_workyep i think johnsom got all the answers you needed hopefully18:14
rm_workbut essentially: yes18:14
abaindurok still a little confused about #4... "4 there is a discussion of this in the maintenance guide in our docs. I think again, if you do a proper renewal I think you can continue to operate. However this is renewing the certs issued by the CAs, the CAs themselves may have other implications."18:14
abaindurat least how i'm doing it now, to "renew" the server CA, i'm just re-running the create_certificates.sh script to generate a new server CA18:15
abaindurwouldn't that create a new server CA, thus making the amphora unable to validate, since amp certs were signed by old server CA?18:16
rm_workerr, yes, i believe so18:16
rm_workbut i think there is a way to renew a CA cert as well18:17
rm_workhmm18:17
rm_workit's an interesting question, i haven't been deeply involved enough with that CA stuff :P18:17
abaindur"If the amphora CA changed in a way which jeopardizes validation of the amphora certificate an operator can manually upload newly issued amphora certificates by switching off validation of the old amphora certificate. This requires a client certificate which can be validated by the client CA file on the amphora. Refer to Octavia HAProxy Amphora API for more details."18:17
abaindurhttps://docs.openstack.org/octavia/queens/admin/guides/operator-maintenance.html#best-practice18:17
abaindurunder the Rotating Amphora Certificates section18:17
abainduri cant find that API or how to do that in haproxy api: https://docs.openstack.org/octavia/queens/contributor/api/haproxy-amphora-api.html#upload-ssl-server-certificate-pem-file-for-controller-communication18:18
abaindurtheres somehow a way to disable cert validation for unencrypted communication, via the amphora API?18:18
openstackgerritCarlos Goncalves proposed openstack/octavia master: Make health checks resilient to DB outages  https://review.openstack.org/60087618:21
abaindursorry for all the questions - trying to figure out best way to automate cert issuance and renewal, with least amount of manual intervention (like failing over amphora) needed or downtime18:21
johnsomabaindur: did you get what you need. There is also discussion of this in the maintenance guide.19:18
johnsomabaindur: did you get what you need. There is also discussion of this in the maintenance guide.19:18
abaindursee above ^^19:18
abaindurYeah, I read the maintenance guide19:18
abaindurWe are concerned about what needs to be done when the CA's expire. Seems like this requires triggering a manual failover, thus manual intervention, of every amphora which would be disruptive19:19
abaindurand require some kind of planning and donwtime by end customers19:19
abaindurjohnsom: "switching off validation of the old amphora certificate" - which API is that? i dont see it documented in haproxy-amphora-api doc19:21
*** luksky11 has joined #openstack-lbaas19:22
abaindurit would be nice if Octavia monitors expiry of the server and client CA's themselves, could turn off validation like mentioned above, then generate new CSRs/certs and upload them to the amphora via API19:23
abaindurperhaps if the ca configs are empty, it even generates its own self signed CA's and monitors them for expiry19:24
*** luksky has quit IRC19:25
johnsomxgerman_: wrote that part, maybe he can clarify19:30
xgerman_abaindur: it says: "Octavia will also monitor those certificates and refresh them before they expire."19:53
xgerman_so you don’t have to worry about amphora certs19:53
xgerman_now, if you need to rotate the certs on the control plane that will require manual work or some automation tool.19:55
*** pcaruana has quit IRC19:55
abaindurxgerman_: so reissuing both the server CA (which issues amp certs), or the client CA requires failing over every amphora?19:56
abaindureither of the two CA's certs themselves19:57
xgerman_Yes, but depnding on your security requirements youc an set them to 100 years19:58
abaindurxgerman_: any reason why there isn't a way to upload the client_ca into the amph via the API? as we do with amp cert refresh19:59
abaindurin this case octavia should still be able to talk to the amp, and upload the new client CA19:59
abaindurIf that could be done, then we'd need manual intervention/failover of all amphora only when we reissue the server CA20:00
xgerman_how often are you rotating certs?20:01
abaindurcurrently never :) because we havent deployed yet20:02
xgerman_:-) We assume control plane is “safe” and so you can choose a very long expiration date...20:02
abaindurtrue but here the host has the server CA's priv key and cert and passhrase stored in plaintext on filesysstem20:03
abaindurso its located on same DCs or even same hosts as VMs and amphora20:05
xgerman_Yes, we never wanted to be a CA. But last time I asked the OpenStack people they said install dogtag20:05
xgerman_no, certs are on control hosts… if they are on the same hosts as VMs you are not building a big cloud. Most clouds we have seen had dedicated control hosts20:06
abaindurthis is just in test env, but its needed to be in same DC/rack as the hosts where VMs reside, no?20:07
abaindursimilar to how OVS and l3 agents in neutron are deployed20:07
abaindurat least, because the octavia-worker needs to be ablt to talk to the amphora VM directly over the LB mgmt network20:07
abaindurso the controller host's networking needs access to the amphora's fixed IP20:08
abainduroctavia-api service is completely elsewhere, but -worker, -housekeeping, and health-manager said they need access to LB mgmt network20:09
johnsomDon't forget that lb-mgmt-net is isolated and not accessible from the code/interfaces handling tenant traffic.20:10
*** hongbin has joined #openstack-lbaas20:12
johnsomI don't disagree that there could be more automation here. There is a long history with this code and "OpenStack CA-as-a-Service" that never materialized, etc.  So, yes,  given today's environment we probably should make some changes to automate some parts and allow shorter expiration on some of the control plane certs.  We are an open volunteer community, so patches are welcome!20:13
johnsomAdding stories describing your needs is a start as well.20:14
abaindurive been trying to dig thru the code to see if i can hack up something myself, at least for auto-updating and monitoring client_ca expiry, so if i come up with something i'll def send for a review :)20:16
xgerman_Yeah, as I said in practice most people run with really long expiration dates and at most three octavia-control instances — so we are talking about keeping track of four certs...20:18
xgerman_But  agree with johnsom if we can automate make it better — I am all for it20:19
*** Emine has joined #openstack-lbaas21:45
*** Emine has quit IRC21:49
*** luksky11 has quit IRC21:50
*** hongbin has quit IRC21:58
*** hongbin has joined #openstack-lbaas22:01
*** josephrsandoval has joined #openstack-lbaas22:12
colin-had a pretty easy time with the queens -> rocky upgrade, thanks for keeping that smooth :)22:21
xgerman_happy dance22:22
colin->UDP protocol support requires an update to the amphora image to support UDP protocol statistics reporting and UDP-CONNECT health monitoring22:22
colin-was this the update that happened to my existing amphora automatically after the control plane services were online?22:22
xgerman_No, this is something you will need to failover the amphora for22:23
colin-understood22:23
xgerman_but udp should apply to new amps IF you deployed the image22:24
colin-is that upgrade note expanded elsewhere in the project? just curious about the nature of the amp image changes22:37
*** rcernin has joined #openstack-lbaas22:43
*** fnaval has quit IRC23:02
*** josephrsandoval has quit IRC23:11
*** josephrsandoval has joined #openstack-lbaas23:12
*** josephrsandoval has quit IRC23:16
*** amuller has quit IRC23:42
*** Swami has quit IRC23:44
« Monday, 2018-09-10 Index Wednesday, 2018-09-12 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!