Friday, 2019-05-31

*** ianychoi has joined #openstack-lbaas00:00
*** mithilarun has quit IRC00:55
*** mithilarun has joined #openstack-lbaas00:56
*** mithilarun has quit IRC01:00
v1k0d3nhey folks. can someone tell me what may be causing this error in the octavia logs `Exception during message handling: Error: [('PEM routines', 'get_name', 'no start line'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')]`01:13
v1k0d3nobviously i'm assuming it's something with my pem file, but i tried following the instructions on how to set this up. that kinda got me to this point, and now i'm scratching my head a bit.01:13
*** yamamoto has joined #openstack-lbaas01:16
openstackgerritsunkai proposed openstack/neutron-lbaas master: touch a test.sh for myself  https://review.opendev.org/66237202:17
*** ricolin has joined #openstack-lbaas02:54
johnsomv1k0d3n: view your private key files, they should have a text header. Double check you followed the instructions accurately, they have been widely used/tested.  If they are binary files, you generated DER format instead of PEM as the instructions list.03:11
v1k0d3nthat was the first thing i checked actually03:13
v1k0d3njohnsom: it's definitely text. the guide is how i created them.03:13
johnsomMaybe the controller process doesn’t have permission to read it?03:14
v1k0d3nhmmm...now that could be.03:15
v1k0d3nlet me check.03:15
v1k0d3noctavia owns is03:17
v1k0d3ni would imagine it needs to be 600 still, right?03:18
johnsomYes03:18
*** sapd1_x has joined #openstack-lbaas03:59
*** sapd1_x has quit IRC04:07
*** pcaruana has joined #openstack-lbaas05:03
*** ricolin has quit IRC05:05
*** rcernin has quit IRC05:07
*** ramishra_ has quit IRC05:26
*** ramishra has joined #openstack-lbaas05:42
*** yamamoto has quit IRC06:01
*** rcernin has joined #openstack-lbaas06:10
*** sapd1_x has joined #openstack-lbaas06:11
*** mkuf_ has joined #openstack-lbaas06:11
*** mkuf has quit IRC06:12
*** mkuf has joined #openstack-lbaas06:17
*** mkuf_ has quit IRC06:21
*** luksky has joined #openstack-lbaas06:30
*** yamamoto has joined #openstack-lbaas06:36
*** yamamoto has quit IRC06:41
*** yamamoto has joined #openstack-lbaas06:41
*** ramishra has quit IRC06:47
*** ramishra has joined #openstack-lbaas06:49
*** yamamoto has quit IRC06:51
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add UDP test scenario  https://review.opendev.org/65651506:59
*** yamamoto has joined #openstack-lbaas06:59
*** ccamposr has quit IRC07:01
*** ivve has quit IRC07:01
*** ccamposr has joined #openstack-lbaas07:08
*** tesseract has joined #openstack-lbaas07:12
*** yamamoto has quit IRC07:18
*** yamamoto has joined #openstack-lbaas07:24
*** luksky has quit IRC07:45
*** rcernin has quit IRC07:52
*** sapd1_x has quit IRC07:58
*** happyhemant has joined #openstack-lbaas08:05
*** luksky has joined #openstack-lbaas08:29
*** ramishra has quit IRC08:46
*** ramishra has joined #openstack-lbaas08:46
*** sapd1_x has joined #openstack-lbaas09:19
*** yboaron_ has quit IRC09:36
*** yboaron_ has joined #openstack-lbaas09:37
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add UDP test scenario  https://review.opendev.org/65651509:42
*** gthiemon1e is now known as gthiemonge09:46
*** yamamoto has quit IRC09:58
*** yboaron_ has quit IRC10:07
openstackgerritGregory Thiemonge proposed openstack/octavia stable/queens: DNM Testing CI  https://review.opendev.org/66242810:27
*** yamamoto has joined #openstack-lbaas10:32
*** yboaron_ has joined #openstack-lbaas10:38
*** yamamoto has quit IRC10:40
*** yboaron_ has quit IRC10:40
*** yboaron_ has joined #openstack-lbaas10:41
*** yboaron_ has quit IRC10:46
*** yboaron_ has joined #openstack-lbaas10:51
*** yboaron_ has quit IRC10:53
*** yboaron_ has joined #openstack-lbaas10:54
*** yboaron_ has quit IRC10:56
*** yboaron_ has joined #openstack-lbaas10:56
*** yboaron_ has quit IRC10:58
*** yboaron_ has joined #openstack-lbaas10:59
*** yboaron_ has quit IRC11:04
*** yboaron_ has joined #openstack-lbaas11:07
*** yboaron_ has quit IRC11:09
*** yboaron_ has joined #openstack-lbaas11:10
*** yboaron_ has quit IRC11:12
*** yboaron_ has joined #openstack-lbaas11:15
*** yboaron_ has quit IRC11:16
*** yboaron_ has joined #openstack-lbaas11:17
*** sapd1_x has quit IRC11:18
*** yboaron_ has quit IRC11:22
*** yboaron_ has joined #openstack-lbaas11:38
*** yboaron_ has quit IRC11:42
*** yboaron_ has joined #openstack-lbaas11:44
*** yboaron_ has quit IRC11:47
*** logan- has quit IRC12:07
*** logan- has joined #openstack-lbaas12:07
*** boden has joined #openstack-lbaas12:11
*** openstack has joined #openstack-lbaas12:30
*** ChanServ sets mode: +o openstack12:30
*** yamamoto has joined #openstack-lbaas12:32
*** yamamoto has quit IRC12:36
*** ramishra has quit IRC12:59
*** ramishra has joined #openstack-lbaas12:59
*** yamamoto has joined #openstack-lbaas13:07
*** goldyfruit has joined #openstack-lbaas13:23
bodenHi, I have a question regarding oslo.config and how to handle duplicate opts... Currently both Octavia and Neutron register the 'host' opt (https://opendev.org/openstack/octavia/src/branch/master/octavia/common/config.py#L39) so there's a collision when trying to use neutron + octavia13:29
bodenERROR octavia DuplicateOptError: duplicate option: host13:29
bodenWhat's the best way to handle this scenario?13:29
*** yamamoto has quit IRC13:50
*** yamamoto has joined #openstack-lbaas13:51
*** yamamoto has quit IRC13:51
bodenI created a bug https://storyboard.openstack.org/#!/story/200580813:51
*** yamamoto has joined #openstack-lbaas13:51
*** yamamoto has quit IRC13:56
*** yamamoto has joined #openstack-lbaas14:01
*** yamamoto has quit IRC14:05
johnsomboden: They are separate processes, how are you getting them mixed?14:12
bodenjohnsom I think by (non-neutron) code that uses both neutron and octavia14:13
bodenisn't it realistic to think some consumer could use the APIs from both neutron and octavia? if so, this issue becomes apparent14:13
johnsomboden: nothing should ever import octavia code.14:14
bodenjohnsom I think the reality is some people are still transisitoning to octavia-lib http://codesearch.openstack.org/?q=from%20octavia%5C.&i=nope&files=&repos=14:15
bodenI will check and see if its possible to remove the octavia imports, but I'm not sure we're there yet14:16
johnsomboden: We have always documented that you should never import Octavia. It has never been in G-R.14:16
bodenjohnsom sure that's fine, you can reject the bug as you see fit14:17
johnsomboden We never had a transition from importing Octavia to using octavia-lib.  That doesn’t mean people didn’t do it, but doing so would be bugs against those projects.14:19
bodenjohnsom ack14:19
*** luksky has quit IRC14:24
*** openstackstatus has joined #openstack-lbaas14:34
*** ChanServ sets mode: +v openstackstatus14:34
*** goldyfruit_ has joined #openstack-lbaas14:40
*** goldyfruit has quit IRC14:42
*** Vorrtex has joined #openstack-lbaas14:47
*** spatel has joined #openstack-lbaas14:59
spatelHello folks!!!15:00
-openstackstatus- NOTICE: Gerrit is now entering its maintenance window. Expect Gerrit outages in the near future. We will notify when it is back up and running.15:07
*** ChanServ changes topic to "Gerrit is now entering its maintenance window. Expect Gerrit outages in the near future. We will notify when it is back up and running."15:07
johnsomspatel o/ Welcome to the Octavia club15:12
johnsomgrin15:12
spatel:)15:18
*** luksky has joined #openstack-lbaas15:18
v1k0d3nhey johnsom, super sorry to bother, but that pem error i mentioned yesterday...it doesn't exactly look like there's anything wrong with the certs/keys that are being used in octavia. but could it potentially be an issue with barbican?15:27
v1k0d3ni'm not 100% sure the barbican installation is completed from a conf standpoint yet.15:27
v1k0d3nbut i think that _might_ be down the line a bit. i get those errors (above) after a few of these...15:28
johnsomv1k0d3n Could be, which log is it in?  What is the context around the error?15:28
v1k0d3nhttps://www.irccloud.com/pastebin/CMcMtZNP/15:28
v1k0d3nwhich i would imagine is around the ssh key?15:29
johnsomv1k0d3n Those are normal, it means nova hasn't finished booting the instance yet.15:29
v1k0d3nit does succeed after that. ok....that makes sense.15:29
johnsomIt will loop with that warning, waiting for the instance to respond15:29
v1k0d3nso then i see the outputs (text) of the pem files.15:29
v1k0d3nwhich i think is expected in the logs.15:29
v1k0d3nthose all say success as well.15:29
johnsomv1k0d3n They should be encrypted15:30
johnsomMaybe you are running an old version?15:30
v1k0d3nthey look encrypted actually i believe.15:30
v1k0d3none sec, and let me validate that though.15:30
v1k0d3nyeah definitely appear to be encrypted.15:31
v1k0d3nthen in goes through a tree of logs (all seem ok)15:31
v1k0d3nhttps://www.irccloud.com/pastebin/YN2kSFyM/15:32
johnsomOk, that is an error15:32
johnsomThe error is going to be right before that tree15:32
v1k0d3nok, good to know.15:32
v1k0d3nhere's the full flow w/end error....15:32
v1k0d3nhttps://www.irccloud.com/pastebin/6FWYlYz2/15:32
* redrobot pokes head in after hearing "barbican"15:33
v1k0d3nand that's that last line that i can't seem to search very well on. the googles doesn't really like searching messages like that.15:33
johnsomYeah, that tree doesn't give us a lot to go on, we need the ERROR line before that tree starts15:33
v1k0d3nhmmm15:33
johnsomredrobot o/15:33
v1k0d3ni guess another service log perhaps?15:34
redrobotjohnsom, 👋15:34
v1k0d3no/ :)15:34
johnsomredrobot This looks like an Octavia config issue, so I think you are safe.15:34
v1k0d3nmost likely....the config is...whoa.15:34
v1k0d3ni know you helped me with some config stuff before johnsom. i think _most_ things are worked through.15:35
johnsomYeah, this looks like the TLS between the controller and the amphora instance has an issue. One of the private keys is not setup correctly.15:35
v1k0d3nbut trying to find bread crumbs to the next thing is tough. the logs that describe network, key, etc. those are very nice...and helped a lot.15:35
v1k0d3nok. i saw a couple of guides on setting this up.15:36
johnsomIf you grep for "ERROR" you should get the exception line.15:36
v1k0d3ni'm running stein, and if i'm not mistaken, that's where i got the docs from on how to create the keys.15:36
v1k0d3nok let me check real quick.15:36
johnsomThis is the guide: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html15:36
v1k0d3nis this the one you're referring to?15:37
v1k0d3n`2019-05-31 15:15:38.755 19 ERROR oslo_messaging.rpc.server [-] Exception during message handling: Error: [('PEM routines', 'get_name', 'no start line'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')]`15:37
johnsomYes, that is probably it.15:37
v1k0d3nok, that's the guide i followed.15:37
johnsomOk, let's debug this if you have time.15:38
v1k0d3ni ended up validating the password/key when i _first_ saw this error.15:38
v1k0d3njust to make sure that amphora could also read/use it.15:38
v1k0d3ni definitely have time.15:38
v1k0d3ni appreciate the help!15:38
johnsomOk, from the guide, the first file I want to look at is:  /etc/octavia/certs/client.cert-and-key.pem15:39
johnsomYou don't need to paste these15:39
johnsomI will explain what to look for15:39
v1k0d3nok one sec, it's in a kolla container...so let me check both the source and destination in the container.15:40
johnsomIf you view that file, you should see two sections, one after the other.15:40
johnsomOne section should have a header "-----BEGIN CERTIFICATE-----" and the other section should have a header of "-----BEGIN PRIVATE KEY-----". These same files need to be loaded on all of your controller instances if you are running more than one controller instance.15:42
v1k0d3nhmmmm15:43
v1k0d3nthat may be the problem. octavia wants cert and key in the same file for that client cert?15:43
johnsomYes, step 1 in this section creates that file: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html#configuring-octavia15:44
johnsomIt's actually a limitation of a python library we are using.15:44
v1k0d3nok, that makes a lot of sense as to the why; for sure.15:44
v1k0d3nlet me try to concat this, and see what happens.15:46
v1k0d3nnow i realize what you were saying last night.15:46
johnsomWe try to do strong security with Octavia, so the setup is a bit complicated. This is why I wrote that detailed guide with all of the required steps.15:46
v1k0d3nanother breadcrumb...this helps. should know something soon.15:46
v1k0d3ni thought i saw a script that sort of does everything in an automated way for testing too, right?15:48
johnsomNot really, there is a script that creates some files, but it's not complete for a deployment15:48
v1k0d3nthis is for testing only right now. and then i was going to produce better certs after testing through it and after understanding it better.15:48
johnsomIt is for our testing. You could be successful using it, but the guide is setup to not use that script. The setup the script creates is not secure. It uses the same CA for both sides of the channel, etc.15:49
johnsomReally, at some point, we should get rid of that script.15:50
v1k0d3nlimiting docs to only something that works might reduce confusion, and how much you guys get pinged by noobs like me :)15:51
johnsomIs there a doc pointing to that script?15:51
*** goldyfruit__ has joined #openstack-lbaas15:52
johnsomAh, the quick start does.... Ok, I will fix that right now.15:52
v1k0d3nwhen i googled octavia installation, the net was super wide...ranging from useful (this guide) to old (just because how OS docs work) to that script.15:52
v1k0d3nyes, that...exactly.15:52
v1k0d3ndocs are hard to keep up with; i get it.15:52
v1k0d3ni did a lot of work in the openstack-helm docs a long time ago...and they're still rough (and we redid them probably 6 different times/ways).15:53
v1k0d3njust the nature of the beast.15:53
johnsomYes, we evolve pretty quickly, so sometimes things get lost.  We try to keep https://docs.openstack.org/octavia/latest/ up to date15:53
v1k0d3nand of course, everyone has an opinion on docs :P15:53
v1k0d3ngood to know. well, that's one reason why i'm using stein...hoping that i'm not that far off from latest. because i've been down that train with other services...15:54
v1k0d3nwhen things change quickly; it's just so hard to go back and update docs.15:54
v1k0d3nstill to this day why OSH doesn't really have a true release yet.15:54
johnsomYeah, you are in good shape with Stein15:55
*** goldyfruit_ has quit IRC15:55
v1k0d3na coworker of mine is trying with Rocky...any issues there that you're aware of?15:55
johnsomNo, we try to maintain back to Queens, but newer is obviously going to be "better". grin15:56
johnsomAnother thing to note, so far we have been able to make it so you can run newer Octavia on older clouds.15:57
johnsomWe don't gate test this, but it should work.  We use stable interfaces when talking to the other OpenStack services.15:57
*** ChanServ changes topic to "Discussions for OpenStack Octavia | Train PTG etherpad: https://etherpad.openstack.org/p/octavia-train-ptg"15:59
*** mithilarun has joined #openstack-lbaas16:01
*** boden has quit IRC16:09
openstackgerritMichael Johnson proposed openstack/octavia master: Clarify that the certificate guide should be used  https://review.opendev.org/66251216:13
johnsomOk, that should help folks find the certificate configuration guide.16:13
johnsomAt some point someone can create a new script, but I can't take that time right now.16:14
*** goldyfruit__ is now known as goldyfruit16:22
*** boden has joined #openstack-lbaas16:23
*** mithilarun has quit IRC16:26
*** ramishra has quit IRC16:29
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Adds "unset" action to the l7policy command  https://review.opendev.org/66251816:43
-openstackstatus- NOTICE: Gerrit is back up and running again. Thank you for your patience and sorry for the delay in this notification (we thought the statusbot was still busy updating channel topics).16:47
v1k0d3nso johnsom...success!16:51
v1k0d3ni actually just used the wrong client side pem. it was that one little miss  at the end :)16:51
*** mithilarun has joined #openstack-lbaas16:51
johnsomYay! Glad we got you up and running.16:52
v1k0d3ncan't thank you enough. i might run into other things, but so far, it looks like it's working. now it's just putting it all to work.16:53
v1k0d3nthank you for being there for us users. i think sometimes it's hard for us to remember that the users are the ones who promote the product. :)16:54
v1k0d3nyou guys have obviously not forgotten this.16:54
johnsomWe feel strongly about building a community.16:54
v1k0d3ni totally agree.16:55
v1k0d3na lot say that, but don't really do as well. you guys are backing that up totally.16:55
v1k0d3n`2019-05-31 16:49:16.322 19 DEBUG octavia.controller.worker.controller_worker [-] Flow 'octavia-create-loadbalancer-flow' (956982b6-212e-4275-9759-a0ded9e7045b) transitioned into state 'SUCCESS' from state 'RUNNING' _flow_receiver /var/lib/kolla/venv/local/lib/python2.7/site-packages/taskflow/listeners/logging.py:145`16:55
v1k0d3nseems like good so far.16:55
johnsomYep! That is the line you want to see16:56
johnsomAdd an HTTP listener, curl the VIP, you should get a 503 back since there are no  members defined yet.16:56
johnsomThat is the smoke test. If that works, you have the basics all working16:57
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Adds "unset" action to the l7policy command  https://review.opendev.org/66251817:05
v1k0d3ngood info johnsom; i'll try that now.17:06
*** goldyfruit has quit IRC17:06
johnsomThe next thing to check is ten seconds or so later, do a stats show and make sure the traffic shows up. That proves that the amphora can communicate back to the controller.17:07
*** goldyfruit has joined #openstack-lbaas17:42
*** sapd1_x has joined #openstack-lbaas17:46
*** sapd1_x has quit IRC17:51
v1k0d3njohnsom...an assumption here, but octavia will work with both tenant _and_ flat networks; is that right?17:53
v1k0d3ni am going to have a setup that's using only flat networks leveraging calico as the sdn controller.17:53
johnsomYes, if neutron supports it, we do17:53
*** goldyfruit has quit IRC17:56
v1k0d3n👍🏼17:57
*** rouk has joined #openstack-lbaas18:02
roukanyone know if i can get active connections to a pool member in octavia, when preparing for swapping out a member?18:03
johnsomrouk We do not currently have member level statistics, but you can set a member weight to 0 which will "drain" the connections from the member.18:04
*** tesseract has quit IRC18:04
johnsomI.e. no new connections will get scheduled to that member, but current connections can finish.18:04
roukis the member statistics a thing on a roadmap somewhere?18:12
rouktrying to do a 100 node rolling upgrade in heat is kinda scary if you have to wait [some random number] minutes for members to drain18:13
johnsomI don't think so, but certainly could be. You can open an RFE story here: https://storyboard.openstack.org/#!/dashboard/stories18:13
roukunless im going about the rolling upgrade the wrong way and i dont need member stats?18:13
johnsomYeah, you can't get the connection info from the member servers themselves?18:13
roukthats implementation specific, a10/f5 let you see the ongoing connections to members, so thats what everything was automated against18:14
johnsomYeah, I get the use case. We should add it to the API.18:15
johnsomrouk Let me know if you need help creating the RFE story for us18:16
roukjust a story with a couple tasks?18:17
johnsomA story with even just one task that captures the use case and desired API information.18:18
johnsomSimple is fine as long as it captures your need18:18
roukjohnsom: 3355218:23
johnsomrouk Looks good, thank you.18:24
v1k0d3njohnsom: so i tried curling the vip and got a connection refused...so i may have some addition issues to chase down.18:35
johnsomHmm, yeah. I am assuming the VIP network is reachable from where you curled.18:35
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Adds "unset" action to the l7rule command  https://review.opendev.org/66254018:38
*** yamamoto has joined #openstack-lbaas18:41
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Adds "unset" action to the l7rule command  https://review.opendev.org/66254018:44
*** yamamoto has quit IRC18:46
*** openstackgerrit has quit IRC19:01
*** ccamposr__ has joined #openstack-lbaas19:02
*** ccamposr has quit IRC19:06
*** happyhemant has quit IRC19:27
*** mithilarun has quit IRC19:28
v1k0d3nyeah it is johnsom19:37
v1k0d3ni can provide some details in a minute (calls).19:37
*** openstackgerrit has joined #openstack-lbaas20:03
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Adds "unset" action to the l7rule command  https://review.opendev.org/66254020:03
openstackgerritMichael Johnson proposed openstack/octavia master: Fix l7rule API handling of None updates  https://review.opendev.org/66256920:11
*** mithilarun has joined #openstack-lbaas20:25
*** Vorrtex has quit IRC20:28
*** goldyfruit has joined #openstack-lbaas21:00
*** boden has quit IRC21:19
*** mithilarun has quit IRC21:28
*** mithilarun has joined #openstack-lbaas21:29
*** spatel has quit IRC21:33
*** mithilarun has quit IRC21:33
*** mithilarun has joined #openstack-lbaas21:45
*** pcaruana has quit IRC21:54
*** spatel has joined #openstack-lbaas22:17
*** spatel has quit IRC22:21
*** yamamoto has joined #openstack-lbaas22:44
*** yamamoto has quit IRC22:48
*** mithilarun has quit IRC22:51
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Adds "unset" action to the quota command  https://review.opendev.org/66258822:52
johnsomWahoo, that should wrap up the unset patches! (Aside from tags which still need some work).22:53
openstackgerritMichael Johnson proposed openstack/octavia master: Fix flavor profile API handling of None updates  https://review.opendev.org/66234422:55
*** goldyfruit has quit IRC22:59
*** ccamposr__ has quit IRC23:01
*** mithilarun has joined #openstack-lbaas23:07
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Allow multiple VIPs per LB  https://review.opendev.org/66023923:09
*** goldyfruit has joined #openstack-lbaas23:09
*** goldyfruit has quit IRC23:35
*** goldyfruit has joined #openstack-lbaas23:38
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Allow multiple VIPs per LB  https://review.opendev.org/66023923:53
*** mithilarun has quit IRC23:54
*** mithilarun has joined #openstack-lbaas23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!