Tuesday, 2020-01-14

*** vishalmanchanda has joined #openstack-lbaas00:06
*** ianychoi_ has joined #openstack-lbaas00:09
*** ianychoi has quit IRC00:11
*** tkajinam has quit IRC00:56
*** tkajinam has joined #openstack-lbaas00:59
*** nicolasbock has quit IRC01:18
rm_workFYI I'm quiet this week because I'm heads down on some internal stuff for a little bit, but please ping me if there's reviews you need done, I will make time for that01:28
*** tkajinam has quit IRC02:36
*** tkajinam has joined #openstack-lbaas03:02
johnsomOk, thanks!03:33
*** psachin has joined #openstack-lbaas03:36
*** hongbin has joined #openstack-lbaas03:50
*** TrevorV has joined #openstack-lbaas03:53
*** TrevorV has quit IRC03:54
*** psachin has quit IRC04:09
*** sapd1_x has joined #openstack-lbaas04:25
openstackgerritVishal Manchanda proposed openstack/octavia-dashboard master: Remove six usage  https://review.opendev.org/70233604:31
openstackgerritMerged openstack/octavia-dashboard master: Drop Django 1.11 support  https://review.opendev.org/70093204:35
*** hongbin has quit IRC05:05
*** AlexStaf has joined #openstack-lbaas05:05
*** AlexStaf has quit IRC05:10
*** hongbin has joined #openstack-lbaas05:14
*** hongbin has quit IRC05:19
*** sapd1_x has quit IRC05:53
*** gcheresh has joined #openstack-lbaas05:53
*** gcheresh has quit IRC06:05
*** pcaruana has quit IRC06:17
*** gcheresh has joined #openstack-lbaas06:32
openstackgerritVishal Manchanda proposed openstack/octavia-dashboard master: Remove six usage  https://review.opendev.org/70233606:40
openstackgerritMerged openstack/octavia stable/rocky: Cap hacking version to <2  https://review.opendev.org/70204007:07
openstackgerritMerged openstack/octavia stable/stein: Cap hacking version to <2  https://review.opendev.org/70203907:10
*** tesseract has joined #openstack-lbaas07:41
*** rcernin has quit IRC07:52
*** maciejjozefczyk has joined #openstack-lbaas08:06
openstackgerritMerged openstack/octavia-lib master: Fix flake8 tox.ini directive  https://review.opendev.org/69329808:09
*** gcheresh has quit IRC08:15
*** gcheresh has joined #openstack-lbaas08:20
*** tkajinam has quit IRC08:24
*** dmellado has quit IRC08:55
*** dmellado has joined #openstack-lbaas08:56
*** pcaruana has joined #openstack-lbaas08:56
*** rpittau|afk is now known as rpittau09:16
*** ramishra has quit IRC09:31
*** gcheresh has quit IRC09:32
*** ramishra has joined #openstack-lbaas09:33
*** gcheresh has joined #openstack-lbaas09:37
*** ramishra has quit IRC10:08
*** ramishra has joined #openstack-lbaas10:16
*** gcheresh_ has joined #openstack-lbaas10:17
*** gcheresh has quit IRC10:17
*** pcaruana has quit IRC10:23
*** ajay33_ has joined #openstack-lbaas10:35
*** ccamposr has quit IRC10:56
*** ramishra has quit IRC10:58
*** ccamposr has joined #openstack-lbaas10:58
*** pcaruana has joined #openstack-lbaas11:02
*** ramishra has joined #openstack-lbaas11:05
*** ramishra has quit IRC11:10
*** rpittau is now known as rpittau|bbl11:11
*** ramishra has joined #openstack-lbaas11:11
*** ramishra has quit IRC11:16
openstackgerritAnn Taraday proposed openstack/octavia master: Add logging filter for AmpConnectionRetry exception  https://review.opendev.org/70055311:17
*** ramishra has joined #openstack-lbaas11:22
*** ramishra has quit IRC11:28
*** gcheresh_ has quit IRC11:49
*** gcheresh_ has joined #openstack-lbaas11:54
*** ramishra has joined #openstack-lbaas12:22
*** ramishra has quit IRC12:58
*** gcheresh_ has quit IRC13:03
*** gcheresh_ has joined #openstack-lbaas13:07
*** rpittau|bbl is now known as rpittau13:26
*** tkajinam has joined #openstack-lbaas13:56
*** AlexStaf has joined #openstack-lbaas14:19
*** TrevorV has joined #openstack-lbaas14:34
*** gcheresh_ has quit IRC14:44
*** gcheresh has joined #openstack-lbaas14:45
*** vesper has quit IRC14:53
*** vesper11 has joined #openstack-lbaas14:53
*** ramishra has joined #openstack-lbaas15:02
*** tkajinam has quit IRC15:10
*** maciejjozefczyk has quit IRC15:12
*** fyx has quit IRC15:13
*** dougwig has quit IRC15:13
*** fyx has joined #openstack-lbaas15:17
*** dougwig has joined #openstack-lbaas15:17
*** maciejjozefczyk has joined #openstack-lbaas15:19
*** gcheresh has quit IRC15:33
*** gcheresh has joined #openstack-lbaas15:40
*** fyx has quit IRC15:48
*** dougwig has quit IRC15:48
*** fyx has joined #openstack-lbaas15:51
*** dougwig has joined #openstack-lbaas15:52
*** maciejjozefczyk has quit IRC16:02
openstackgerritMerged openstack/octavia stable/queens: Cap hacking version to <2  https://review.opendev.org/70204116:06
*** ramishra_ has joined #openstack-lbaas16:07
*** ramishra has quit IRC16:09
*** gcheresh has quit IRC16:13
*** dosaboy has quit IRC16:28
*** AlexStaf has quit IRC16:44
*** maciejjozefczyk has joined #openstack-lbaas16:55
*** rpittau is now known as rpittau|afk17:01
*** ccamposr__ has joined #openstack-lbaas17:14
*** ccamposr has quit IRC17:14
*** ajay33_ has quit IRC17:24
*** dosaboy has joined #openstack-lbaas17:59
*** maciejjozefczyk has quit IRC18:02
*** maciejjozefczyk has joined #openstack-lbaas18:17
*** maciejjozefczyk has quit IRC18:56
*** maciejjozefczyk has joined #openstack-lbaas18:59
*** pcaruana has quit IRC19:16
*** maciejjozefczyk has quit IRC19:26
*** armax has joined #openstack-lbaas19:30
*** gmann is now known as gmann_afk19:36
openstackgerritBrian Haley proposed openstack/octavia master: Refactor 'check_quota_met' and 'decrement_quota'  https://review.opendev.org/59666519:48
*** maciejjozefczyk has joined #openstack-lbaas20:13
*** born2bake has joined #openstack-lbaas20:18
born2bakeHi guys20:18
born2bakeso installed octavia; good thing that at least now my amphora instances are not in spawning state and are active; however, still lb is in pending create state. octavia-worker.log - https://pastebin.com/ieq2YHju20:18
born2bake20.0.0.202 - lb ; 20.0.0.214 and 20.0.0.190 amphora instances20:18
born2bakesec groups are added20:19
johnsomborn2bake How are you running your nova? Is it inside virtualbox?20:19
johnsomThe controller will keep retrying to connect (thus the pending_create state) until nova finishes booting the instance or it hits the timeout in your configuration file. At which time it will move to "ERROR" state.20:21
*** AlexStaf has joined #openstack-lbaas20:23
*** maciejjozefczyk has quit IRC20:25
*** AlexStaf has quit IRC20:27
*** maciejjozefczyk has joined #openstack-lbaas20:29
*** AlexStaf has joined #openstack-lbaas20:29
*** ccamposr__ has quit IRC20:40
born2bakejohnsom no, I have cluster of 4 machines. nova is backed by ceph20:41
johnsomborn2bake Hmm, ok, so it's not likely that nova is taking too long to boot the instance (this is very common). So, let's look at the controller port on the lb-mgmt-net20:42
johnsomWhere you are running the octavia worker process, it needs a route to or a port on the lb-mgmt-net that is used to communicate with the amphora. How do you have that port/route setup?20:43
born2bakeI am using kolla-ansible, everything is deployed in kolla containers. I was following this guide: https://ssup2.github.io/record/OpenStack_Stein_%EC%84%A4%EC%B9%98_Kolla-Ansible_Ubuntu_18.04_ODROID-H2_Cluster/ ; However, I am using Train version of openstack20:44
born2bakeBefore that, I had issues that it was taking too long to boot the instance but now its working fine. instances are active20:45
born2bakecan you specify how do I find a port? Network setup is: I have provider network 10.0.0.0/16 and I can my instances with floating ip. I have octavia-network 20.0.0.0/24. octavia vxlan is attached to the router(with provider network).20:46
johnsomAh, yeah, ok. We here of a bunch of people using Kolla Ansible having problems because it doesn't setup the lb-mgmt-net for you. Most (maybe all) of the other deployment tools do.20:48
johnsomLet me look at the kolla docs to see if I can find the section on the lb-mgmt-net20:49
*** tesseract has quit IRC20:50
*** maciejjozefczyk has quit IRC20:50
born2bakewith kolla, you deploy octavia containers, then you add flavor,network,sec group, you add them in conf file and redeploy octavia20:50
johnsomYeah, let's see, in that document they are calling the lb-mgmt-net "octavia-net"20:51
johnsomYeah, the issue with kolla ansible is it doesn't setup the lb-mgmt-net for you, the docs say you have to do it by hand. The other deployment tools do that automatically when they setup the containers20:51
born2bakewhich other deployment tools? and what I can do if I am using kolla-ansible :) Also, thanks a lot for help cause I could not get any advice for long time :)20:52
johnsomAnsible, tripleo, charms, puppet, ....  pretty much all of them.  I don't know kolla much, so I have to dig.  There are so many deployment tools now the octavia team can't keep up with them really.20:55
johnsomGive me a minute, we can work through getting your deployment going.20:55
born2bakeif you need, I can send my octavia.conf files20:56
johnsomNo, not at the moment. This is configuration outside of the octavia.conf20:57
johnsomOk, so this document creates the "octavia-net" as a vxlan network in neutron. Then it creates a route from the cloud's external network to this "octavia-net". Do we agree on that part?20:59
born2bakeroute is added just for debug purposes as I understood. so you can ssh into amphora instances and check logs from your local network.  In kolla config you need to specify 2 interfaces:21:00
born2bakenetwork_interface: "enp2s0f0" - internal, api, etc21:01
born2bakeneutron_external_interface: "enp2s0f1" - ip interface without ip address, which will be used by provider network so you can assign floating ip's to your instances and access them.21:01
born2bakeneutron_tenant_network_types: "vxlan,vlan,flat"21:01
born2bakeso for my external interface, I am using physnet1 for 10.0.0.0/16..in that doc they use 192.168.0. etc21:01
*** TrevorV has quit IRC21:01
johnsomHmm, ok.  Well, the controller container(s) that run the (worker, health manager, and housekeeping) processes must have a way to access the "octavia-net"\21:02
johnsomThe document you linked implies they are creating a route in the container(s) to allow that access21:02
born2bakeahh so this process shall be done inside the containers?21:03
born2bakeoctavia containers*21:03
johnsomI think that is how the document is having you do it.21:03
born2bakeroute command within container is not allowed21:05
johnsomEither way, those processes need to have a way to reach IPs on the "octavia-net". The IPs on the "octavia-net" must also be able to send a packet back to the controller IPs configured in https://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.controller_ip_port_list21:05
johnsomWhat are the interfaces and IPs inside the worker container?21:06
johnsomDid they plug them directly?21:06
openstackgerritAdam Harwell proposed openstack/octavia master: Conf option to use VIP ip as source ip for backend  https://review.opendev.org/70253521:06
born2bakeinterfaces and ip's are the same what I have on my openstack controller21:08
born2bakejust for reference I was also following this guide: https://shreddedbacon.com/post/openstack-kolla/21:11
born2bakesimilar21:11
johnsomIs there a port there with an IP on the 20.0.0.x subnet?21:11
born2bakenope21:11
born2bakehttp://prntscr.com/qnqjvj21:11
jrosserimho the first example is made more complicated by using a vxlan network for octavia21:15
jrosserbecasue that implies the use of a router to de-encapsulate on the controller21:15
johnsomWhy is that?21:15
born2bakesecond one is used with vlans; i dont have vlans21:15
jrosserthe vlan example doesnt need that, and is therefore simpler and the octaiva mgmt network remains un-routed21:16
jrosserwhich is a ++ from my pov21:16
johnsomYeah, it just depends on how you get access to that neutron network. If you pop a port off via openvswitch it doesn't matter21:16
johnsomTypically the routed designs are easier for folks to setup, but really there are thousands of ways to do this.21:17
johnsomok, so that is a lot of interfaces and networks if it's inside the container.21:18
jrosserdon't you need a vlan anyway between the controllers because only one of them will be the active router21:18
johnsomAlso, inside your container, it may require you use the new "ip route" syntax instead of using the older "route" command21:18
born2bakemy setup is simple...2 network interfaces; 1 is used for openstack api, services etc. 2 - up without ip address. In openstack I created external with 10.0.0.0/16 which is my LAN, internal - 192.168.x which I assign to instances.21:18
*** AlexStaf has quit IRC21:19
born2bakeip route add -net 20.0.0.0/24 gw 10.0.53.19521:19
born2bakeError: any valid prefix is expected rather than "-net".21:19
*** ccamposr has joined #openstack-lbaas21:20
johnsomip route add 20.0.0.0/24 via 10.0.53.19521:21
born2bakeip route add 20.0.0.0/24 via 10.0.53.195 is not permitted and I cannot use sudo inside container21:21
johnsomHa, well, that might make things a bit hard....21:21
*** ccamposr has quit IRC21:23
johnsomIf you can't execute root level commands inside the container I'm not sure how you can fix/configure this21:23
johnsomSorry I don't have experience with kolla and how those are setup21:24
openstackgerritAdam Harwell proposed openstack/octavia master: Conf option to use VIP ip as source ip for backend  https://review.opendev.org/70253521:24
johnsomMaybe asking for help in the kolla channel would get a response. Basically you need your Octavia controller containers to have a way to access the "octavia-net".21:24
jrosserin the vxlan example the route is added on (Controller) which i take to mean the host, not the container21:25
johnsomWould the container have access to that?21:26
jrosserwhich for that style of containerisation presume the host is doing the NAT/port forwarding/dockerism necessary there21:26
johnsomIf I setup those containers I wouldn't expose all of the host networks into the container21:26
*** ccamposr has joined #openstack-lbaas21:26
johnsomHmm, yeah, could be.21:27
jrosserbut i !kolla so the advice to go to their channel and ask is good21:27
johnsomYeah, same, never used it21:27
jrossermy osa deployment looks very very different to that21:28
born2bakethey are doing the same :)21:28
born2bakeask octavia lol21:28
johnsomSigh21:28
johnsomYeah, osa, tripleo, devstack, all of them are different than this.21:29
jrosserborn2bake: this all depends what you care about. this ocatvia mgmt network is one of the very rare times in openstack that the control plane and the VM's get "a bit tangled up"21:29
jrosserand so your approach might be quite different in a trusted environment vs a public cloud, for example21:30
jrosserit's a shame you're getting batted back here as it's really the job of deployment tooling projects to provide you with some pointers/templates of best practice21:31
*** gcheresh has joined #openstack-lbaas21:32
johnsomYeah, there are so many deployment tools now the octavia team doesn't always have visibility to how they are setup. Heck, I have even lost track of how OSA is setup.21:34
jrosserit makes a vlan provider network21:34
jrosserand then a pile of iptables on the controllers on top of that21:35
johnsomlol. Yeah, I think last I saw it had a bunch of odd bridges going on, but I think someone was simplifying that21:36
jrosserbut it's up to the deployer to make sure the vlan id and address range is suitable, and to do "whatever is needed" to make br-lbaas be in all the right places21:36
johnsomRight, to us it's just a neutron network...21:37
born2bakebefore you joined a channel they mentioned:21:38
born2bake[21:23:37] <noxoid> born2bake, kolla uses net:host for docker containers. it uses the routes/ips of the parent node21:38
born2bake[21:24:13] <noxoid> i have octavia configured as a vlan on the host and tell neutron about it. the amphora are attached to that vlan neutron network21:38
born2bake[21:24:35] <noxoid> provider:network_type = vlan, provider:physical_network = physnet1, provider:segmentation_id = 100221:38
born2bake[21:24:54] <osmanlicilegi> born2bake: it's not a kolla issue. I would recommend you to setup an all-in-one environment to learn the basics of octavia. trying to understand it on production may confuse you.21:38
johnsomlol, well, it *is* a kolla issue.....21:39
johnsomOk, so that answers the question about how the container connects to the networks.21:39
johnsomIt's cloning in the host networks.21:39
jrosserborn2bake: what they kind of miss out is that the controllers will need actual IP on that vlan21:40
johnsomSo, on the "parent node" or host, whatever it is called. Do  you have the route to 20.0.0.x?   "ip route" should list the routes21:40
johnsomYeah, I didn't see that in the blog post21:41
johnsomOr routes, you can route in/out of the lb-mgmt-net just fine21:41
born2bakehm, not on a controller node. actually let me try to add that route to the controller21:41
jrosserunless the addition of the static route via the public ip of a neutron router means you don't need that lbaas mgmt net ip on the host itself.....21:42
*** gcheresh has quit IRC21:45
born2bakeokay its different now i guess21:46
born2bakehttps://pastebin.com/eHzLt30K21:46
born2bakei will try to restart octavia containers21:47
johnsomOk, so that log implied it could now connect, but the certificate configuration in the octavia.conf is wrong21:48
born2bakecerts in kolla shall be generated using -b 4.0.1 octavia version21:49
born2bakehttp://prntscr.com/qnqzun21:50
born2bakeit doesnt support up-to-date version as I understood21:50
johnsomWell, that screen shot is bad advice. Plus, this certificate configuration has been required since the first release of Octavia.21:51
johnsomHere is a guide to set it up correctly: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html21:51
born2bakehttps://pastebin.com/EPwdidVT21:52
johnsomTheir instructions might work, but it is not optimal.21:52
johnsomFirst, I would try booting another load balancer and confirming that the connectivity is working. If we get an SSL error, we know it's simply this certificate configuration that is wrong21:53
born2bakeyeah, cert conf is wrong21:54
johnsomOk. That guide document should help you sort it out. I went into detail in it. Granted this security is a bit complicated, but the guide is step by step21:55
born2bakejust to confirm...kolla still requires: ca_01.pem  cakey.pem  client.pem ; if those will not be inside config folder, deployment wont be possible. If I follow up-to-date guide generating certificates, can I just rename them? and if yes, what would be the right way to rename ?21:55
johnsomI can't speak to the kolla side and if it needs those or not.21:56
johnsomThis section of the guide https://docs.openstack.org/octavia/latest/admin/guides/certificates.html#configuring-octavia might help you map their filenames to how the guide creates the certificates21:56
born2bakeI was confused just by the names and they dont have documentation for octavia therefore, I am not sure what ca_01.pem  cakey.pem  client.pem are - comparing to server_ca.cert.pem server_ca.key.pem client_ca.cert.pem client.cert-and-key.pem and etc21:58
johnsomYeah, the guide is for a "dual CA" deployment, which is best for production security.21:58
johnsomIt looks like kolla was trying a single CA config21:59
born2bakeokay, i will try. Thank you very much for you help22:04
johnsomSure, good luck!  Maybe you can open some bugs for kolla to make this easier22:04
*** rcernin has joined #openstack-lbaas22:08
*** ccamposr has quit IRC22:10
born2bakehttp://prntscr.com/qnr9ja <322:15
johnsomNice!22:15
born2bakeHowever, I didnt use dual CA. Needs to raise that to kolla22:16
johnsomNow, the one thing to look at, is do the members become "operating_status" "ONLINE". If not, it may be that there is a route from the amphora to the controllers missing22:16
born2bakehow do I check that?22:17
born2bakehttp://prntscr.com/qnraq7 ?22:18
johnsomWhen you create the load balancer, and add a pool and members, with a health monitor, then in the UI it will have that status column22:18
johnsomThat is listener, click the pool tab22:18
born2bakeI have not added any instances to the pool22:19
born2bakelet me try22:19
born2bakeif I add instances to the pool, amphora images are spawning for ages22:27
johnsomspawning? What do you mean by that? We don't boot anything when adding a member to a pool22:27
born2bakewhen I create load balancer, it automatically creates 2 amphora instances which are spawning.22:28
johnsomYes, in active/standby topology configuration it creates two. It should take less than a minute from the create command for the LB to be ACTIVE22:29
johnsomTypically about 30 seconds22:29
born2bakehttp://prntscr.com/qnrg41 I guess its ok22:32
johnsomThat looks ok, if you click into the pool (which has no name), you should see the members and that they are only22:33
johnsomonline22:33
born2bakeyup, they are online. its awesome!22:34
born2bakethanks a lot johnsom22:34
johnsom+1 ok. Good stuff.  Enjoy!22:34
born2bakeHowever, I noticed if I assosiate floating ip to lb, I cant ping it.22:46
born2bakeahh got it...sorry. i can curl it :)22:47
*** gmann_afk is now known as gmann22:53
*** tkajinam has joined #openstack-lbaas22:55
*** armax has quit IRC23:30
*** born2bake has quit IRC23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!